Repository navigation
[Bug]: Context window exceeded error not triggering retry logic during episodic summarization (original error message missing in ExternalServiceAPIError) #1247
Description
Describe the bug
When the LLM returns a context length exceeded error, MemMachine’s retry logic fails to activate. The root cause is that ExternalServiceAPIError is raised without including the original error message from the LLM. As a result, _is_exceed_context_window_error() cannot detect keywords like “context length” or “token limit”, skipping the split-and-retry mechanism.
This leads to silent failure with “Summarization failed due to unexpected error”, risking data loss.
2026-03-18 19:43:14,833 [ERROR] memmachine_server.common.language_model.openai_chat_completions_language_model - [call uuid: e3758a9d-e2ef-481c-b14c-f381949b03e1] Giving up generating response after failed attempt 1 due to non-retryable BadRequestError
Traceback (most recent call last):
File "/path/to/mm/packages/server/src/memmachine_server/common/language_model/openai_chat_completions_language_model.py", line 219, in _generate_response
response = await self._client.chat.completions.create(**args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/mm/.venv/lib/python3.12/site-packages/openai/resources/chat/completions/completions.py", line 2676, in create
return await self._post(
^^^^^^^^^^^^^^^^^
File "/path/to/mm/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1884, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/mm/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1669, in request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "You passed 202753 input tokens and requested 0 output tokens. However, the model's context length is only 202752 tokens, resulting in a maximum input length of 202752 tokens. Please reduce the length of the input prompt. (parameter=input_tokens, value=202753)", 'type': 'BadRequestError', 'param': 'input_tokens', 'code': 400}}
2026-03-18 19:43:14,836 [ERROR] memmachine_server.episodic_memory.short_term_memory.short_term_memory - Summarization failed due to unexpected error
Traceback (most recent call last):
File "/path/to/mm/packages/server/src/memmachine_server/common/language_model/openai_chat_completions_language_model.py", line 219, in _generate_response
response = await self._client.chat.completions.create(**args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/mm/.venv/lib/python3.12/site-packages/openai/resources/chat/completions/completions.py", line 2676, in create
return await self._post(
^^^^^^^^^^^^^^^^^
File "/path/to/mm/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1884, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/mm/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1669, in request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "You passed 202753 input tokens and requested 0 output tokens. However, the model's context length is only 202752 tokens, resulting in a maximum input length of 202752 tokens. Please reduce the length of the input prompt. (parameter=input_tokens, value=202753)", 'type': 'BadRequestError', 'param': 'input_tokens', 'code': 400}}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/path/to/mm/packages/server/src/memmachine_server/episodic_memory/short_term_memory/short_term_memory.py", line 606, in _create_summary
result = await self._model.generate_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/mm/packages/server/src/memmachine_server/common/language_model/openai_chat_completions_language_model.py", line 160, in generate_response
output, function_calls_arguments, _, _ = await self._generate_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/mm/packages/server/src/memmachine_server/common/language_model/openai_chat_completions_language_model.py", line 259, in _generate_response
raise ExternalServiceAPIError(error_message) from e
memmachine_server.common.data_types.ExternalServiceAPIError: [call uuid: e3758a9d-e2ef-481c-b14c-f381949b03e1] Giving up generating response after failed attempt 1 due to non-retryable BadRequestError
Note: We are not using OpenAI’s cloud LLM. We are using an on-premise LLM that exposes an OpenAI-compatible API endpoint (vLLM). The error structure and behavior are identical to OpenAI’s, but the service is self-hosted.
Steps to reproduce
- Start MemMachine server.
- Use REST client to rapidly inject BEAM benchmark conversations (20x ~100K token chats, 35x ~500K token chats, 35x ~1M token chats) via add(...) with both episodic & semantic memory enabled in a loop.
- When LLM context window is exceeded (e.g., 202753 tokens vs. 202752 limit), observe:
- LLM returns BadRequestError with message: “You passed 202753 input tokens... context length is 202752”
- MemMachine logs ExternalServiceAPIError without original message
- No “Batch failed. Splitting X episodes into halves.” log
- Final error: “Summarization failed due to unexpected error”
Expected behavior
- On context window exceedance, _is_exceed_context_window_error() should detect “context length” in the error message.
- Split-and-retry logic should trigger: log “Batch failed. Splitting X episodes into halves.” and retry with halved input.
- Original LLM error message must be preserved in ExternalServiceAPIError for accurate detection.
Environment
- OS: Ubuntu 22.04.5 LTS
- MemMachine Version: v0.3.3.dev1+g75e871eb6
- Development language version: Python 3.12
- LLM Backend: On-premise LLM with OpenAI API compatibility (vLLM)
Additional context
I fixed this myself and it’s working well in our environment. If you think it’s safe, feel free to merge it directly — it’s just one line.
(Not sure if this bug also exists when using the official OpenAI cloud service — but since we’re on-prem, this fix is critical for us.)
Root Cause:
- In
openai_chat_completions_language_model.py, ExternalServiceAPIError is raised with a generic message:
original_error_msg = str(e)
error_message = (
f"[call uuid: {generate_response_call_uuid}] "
"Giving up generating response "
f"after failed attempt {attempt} "
f"due to non-retryable {type(e).__name__}" # before
)- The original LLM error (e.g., “You passed 202753 input tokens... context length is 202752”) is omitted.
→_is_exceed_context_window_error()receives no keywords → returns False → retry logic skipped.
Fix:
- Include original error message in ExternalServiceAPIError:
original_error_msg = str(e)
error_message = (
f"[call uuid: {generate_response_call_uuid}] "
"Giving up generating response "
f"after failed attempt {attempt} "
f"due to non-retryable {type(e).__name__}: {original_error_msg}" # after
)Verification:
- After applying the fix, context window errors from the LLM now correctly trigger split-and-retry logic. No summary data loss observed.
Activity
[-][Bug]: Context window exceeded error not triggering retry logic (original error message missing in ExternalServiceAPIError)[/-][+][Bug]: Context window exceeded error not triggering retry logic during episodic summarization (original error message missing in ExternalServiceAPIError)[/+]sscargal commentedon Mar 23, 2026
Thanks for reporting this issue @junttang. We'll take a look and update as soon as possible. It could be similar to #1252 and #1251.
junttang commentedon Mar 23, 2026
Thanks for merging. Looking forward to contributing more.
fix(server): preserve original LLM error in ExternalServiceAPIError (#…