[Bug]: Improve Asynchronous Summary Processing in ShortTermMemory.add_episodes #909

Activity

1. sscargal

   added the

   Bug

   issue type on Jan 7, 2026
2. sscargal commented on Jan 7, 2026

   

   sscargal

   on Jan 7, 2026

   ContributorAuthor

   More actions

   This issue reports a performance bottleneck in the ShortTermMemory.add_episodes method, specifically a serialization lock that limits throughput during asynchronous summary creation triggered by eviction. When high write concurrency occurs (e.g., during load testing via Locust), callers block if an eviction triggers and a previous summary is still running.

   Code Review

   • Relevant Function: ShortTermMemory.add_episodes

     async def add_episodes(self, episodes: list[Episode]) -> bool:
         async with self._lock:
             if self._closed:
                 raise RuntimeError(f"Memory is closed {self._session_key}")
             self._memory.extend(episodes)
             self._current_episode_count += len(episodes)
             self._current_message_len += sum(len(e.content) for e in episodes)
             full = self._is_full()
             if full:
                 await self._do_evict()
             return full

   • Eviction: Inside _do_evict, after preparing for a summary, if a previous summary is still running, it waits:

     if self._summary_task is not None:
         await self._summary_task
     self._summary_task = asyncio.create_task(self._create_summary(result))

   • Locking: Both episode addition and eviction are protected by an asyncio.Lock, preventing concurrent modifications; however, async summary creation still serializes callers until the summary task is freed.

   Cause of Bottleneck

   • The add_episodes flow waits for the previous summary task to complete—in the critical section protected by the lock.
   • This means multiple callers trying to add episodes will be serialized during evictions, and if the summary task is slow, they will block until it finishes.
   • The lock is held for the entire duration of waiting for the previous async summary, preventing concurrency and causing throughput degradation under load.

   Suggested Workaround/Solution

   A. Reduce Critical Section

   • Only hold the lock for truly critical operations: The lock is meant to protect memory state changes (episode, count, length). Waiting for the summary task's completion does not need to be inside the critical section.

   Proposed changes:

   1. Release the lock before asynchronously waiting for the summary_task:

      # in _do_evict
      # Instead of:
      if self._summary_task is not None:
          await self._summary_task
      # Do this:
      prev_task = self._summary_task
      self._summary_task = asyncio.create_task(self._create_summary(result))
      # Release lock, then await prev_task outside lock!
      # Example (pseudocode):
      async with self._lock:
          # ... prepare for summary ...
          prev_task = self._summary_task
          self._summary_task = asyncio.create_task(self._create_summary(result))
      if prev_task is not None:
          await prev_task

   2. Minimize Lock Holding

      • Update only the necessary state inside async with self._lock.
      • Spawn summary tasks outside of the lock if safe.
      • If memory consistency relies on summary completion, queue episode additions or use a separate awaitable for summary completion, but do not block all writers with the lock.

   B. Advanced: Use Task Queue/FIFO Pool

   • Consider a background summary worker that triggers summaries in a separate asyncio Task, or use an asyncio condition or event to signal when summary is available, allowing episode additions to proceed and just mark that the memory is "evicting/evicted".

   C. Short-term: Document for Users

   • Warn clients and testers: This bottleneck can be encountered any time rapid eviction (due to high episode volume) overlaps summary generation.

   D. Long-term Improvement

   • Summary generation should be completely decoupled from episode addition; perhaps implement an eviction queue, and let new episodes proceed even if summary is pending, so long as memory capacity allows.

   ---

   References

   • ShortTermMemory implementation (source)

   ---

   Summary:
   The bottleneck is caused by holding the serialization lock while waiting for async summary completion in add_episodes. You should release the lock before awaiting summary completion. Only protect the minimal memory state with the lock, and let summary processing happen outside the critical section. For maximal throughput, consider backgrounding summary eviction or creating an async queue for eviction summaries.
3. sscargal

   assigned

   sscargal

   and

   o-love

   on Jan 7, 2026
4. jealous

   added 5 commits that reference this issue on Jan 14, 2026

   [GH-909] Improve async summary performance.

   ...

   3c33102

   [GH-909] Improve async summary performance.

   ...

   08a2d80

   [GH-909] Improve async summary performance.

   ...

   17b2660

   [GH-909] Improve async summary performance.

   ...

   255f6e1

   [GH-909] Improve async summary performance.

   ...

   9e57b54
5. sscargal

   added a commit that references this issue on Jan 27, 2026

   [GH-909] Improve async summary performance. (#949)

   ...

   aa74179
6. sscargal

   closed this as completedin #949on Jan 27, 2026
7. SarahScargall

   added a commit that references this issue on Jan 29, 2026

   [MemMachineGH-909] Improve async summary performance. (MemMachine#949)

   ...

   67a5a28