Semantic ingestion retries forever on context_length_exceeded errors #1252

Summary

When a message exceeds the LLM's context window, the background ingestion task logs the error but retries immediately in a tight loop — causing high CPU and unbounded database cost on Aurora Serverless v2.

Impact

On our prod environment, 177 uningested messages that are too large for gpt-4o-mini cause a tight retry loop that drives Aurora ACU to 12-16 (the configured max) and writer CPU to 100%.

Root Cause

In semantic_memory.py:772-787, the background ingestion loop:

while not self._is_shutting_down:
    dirty_sets = await self._semantic_storage.get_history_set_ids(...)
    if len(dirty_sets) == 0:
        await asyncio.sleep(self._background_ingestion_interval_sec)
        continue
    try:
        await ingestion_service.process_set_ids(dirty_sets)
    except Exception:
        logger.exception("background task crashed, restarting")

When process_set_ids fails (e.g., OpenAI returns context_length_exceeded), the exception is caught, logged, and the loop restarts immediately with no backoff. The same messages are found again, the same error occurs, and the cycle repeats as fast as the database can serve the polling query.

The OpenAI error from prod logs:

openai.BadRequestError: Error code: 400 - {'error': {'message': 'Your input exceeds the context window 
of this model. Please adjust your input and try again.', 'type': 'invalid_request_error', 
'param': 'input', 'code': 'context_length_exceeded'}}

Suggested Fixes

1. Add exponential backoff on failure — sleep after exceptions, increasing the delay on consecutive failures (e.g., 2s → 4s → 8s → ... capped at 60s).

2. Skip permanently failing messages — after N retries for the same set_id, mark its messages as ingested (or add a failed state) so they don't block the loop forever.

3. Handle context_length_exceeded specifically — truncate or chunk oversized inputs before sending to the LLM, or skip them with a warning rather than raising an exception.

4. Always sleep between iterations — even when dirty_sets is not empty, sleep briefly between processing rounds to prevent CPU saturation on repeated failures.

Related

• Background ingestion polling causes unbounded database load growth #1251 — the polling query itself is also O(n²), compounding this issue