Summary
When a message exceeds the LLM's context window, the background ingestion task logs the error but retries immediately in a tight loop — causing high CPU and unbounded database cost on Aurora Serverless v2.
Impact
On our prod environment, 177 uningested messages that are too large for gpt-4o-mini cause a tight retry loop that drives Aurora ACU to 12-16 (the configured max) and writer CPU to 100%.
Root Cause
In semantic_memory.py:772-787, the background ingestion loop:
while not self._is_shutting_down:
dirty_sets = await self._semantic_storage.get_history_set_ids(...)
if len(dirty_sets) == 0:
await asyncio.sleep(self._background_ingestion_interval_sec)
continue
try:
await ingestion_service.process_set_ids(dirty_sets)
except Exception:
logger.exception("background task crashed, restarting")When process_set_ids fails (e.g., OpenAI returns context_length_exceeded), the exception is caught, logged, and the loop restarts immediately with no backoff. The same messages are found again, the same error occurs, and the cycle repeats as fast as the database can serve the polling query.
The OpenAI error from prod logs:
openai.BadRequestError: Error code: 400 - {'error': {'message': 'Your input exceeds the context window
of this model. Please adjust your input and try again.', 'type': 'invalid_request_error',
'param': 'input', 'code': 'context_length_exceeded'}}
Suggested Fixes
-
Add exponential backoff on failure — sleep after exceptions, increasing the delay on consecutive failures (e.g., 2s → 4s → 8s → ... capped at 60s).
-
Skip permanently failing messages — after N retries for the same set_id, mark its messages as ingested (or add a failed state) so they don't block the loop forever.
-
Handle context_length_exceeded specifically — truncate or chunk oversized inputs before sending to the LLM, or skip them with a warning rather than raising an exception.
-
Always sleep between iterations — even when dirty_sets is not empty, sleep briefly between processing rounds to prevent CPU saturation on repeated failures.
Related
Summary
When a message exceeds the LLM's context window, the background ingestion task logs the error but retries immediately in a tight loop — causing high CPU and unbounded database cost on Aurora Serverless v2.
Impact
On our prod environment, 177 uningested messages that are too large for
gpt-4o-minicause a tight retry loop that drives Aurora ACU to 12-16 (the configured max) and writer CPU to 100%.Root Cause
In
semantic_memory.py:772-787, the background ingestion loop:When
process_set_idsfails (e.g., OpenAI returnscontext_length_exceeded), the exception is caught, logged, and the loop restarts immediately with no backoff. The same messages are found again, the same error occurs, and the cycle repeats as fast as the database can serve the polling query.The OpenAI error from prod logs:
Suggested Fixes
Add exponential backoff on failure — sleep after exceptions, increasing the delay on consecutive failures (e.g., 2s → 4s → 8s → ... capped at 60s).
Skip permanently failing messages — after N retries for the same set_id, mark its messages as ingested (or add a
failedstate) so they don't block the loop forever.Handle
context_length_exceededspecifically — truncate or chunk oversized inputs before sending to the LLM, or skip them with a warning rather than raising an exception.Always sleep between iterations — even when
dirty_setsis not empty, sleep briefly between processing rounds to prevent CPU saturation on repeated failures.Related