LLM response caching for tests #484

Activity

1. elroy-bot commented on Sep 3, 2025

   

   elroy-bot

   on Sep 3, 2025

   OwnerAuthor

   More actions

   @claude implement

   👀React with 👀1Reacted by claude
2. claude commented on Sep 3, 2025

   

   claudebot

   on Sep 3, 2025 – with Claude

   Last edited by claude

   More actions

   Claude finished @elroy-bot's task —— View job • claude/issue-484-20250903-2251 • Create PR ➔

   ---

   LLM Response Caching Implementation ✅

   I have successfully implemented LLM response caching for tests as requested.

   Todo List

   • Read repository setup instructions (CLAUDE.md)
   • Explore current LLM client implementation
   • Understand ElroyContext structure
   • Convert LLM client to class-based architecture
   • Create cached LLM client for tests
   • Create test fixtures directory structure
   • Integrate cached client with ElroyContext for tests
   • Update test configuration to use cached client
   • Create test for cached client functionality
   • Create documentation for cache functionality
   • Run linting/testing if available
   • Commit and push changes

   Implementation Summary

   Key Components Created:

   1. CachedLlmClient (elroy/llm/cached_client.py):

      • Extends the existing LlmClient class
      • Caches responses for all LLM methods: query_llm, get_embedding, generate_chat_completion_message, etc.
      • Uses MD5 hashing of parameters to create consistent cache keys
      • Stores cache files as JSON in tests/fixtures/llm_cache/

   2. MockStreamParser (elroy/llm/mock_stream_parser.py):

      • Replays cached streaming responses for generate_chat_completion_message
      • Maintains compatibility with existing code expecting StreamParser

   3. Test Integration:

      • Added cached_ctx fixture to conftest.py for easy test usage
      • Updated george_ctx fixture to use cached responses
      • Created comprehensive tests in tests/llm/test_cached_client.py

   4. ElroyContext Enhancement:

      • Added set_custom_llm_client() method to allow overriding the default LLM client
      • Maintains backward compatibility

   Features

   • Test-only caching: Only affects test environment, production code unchanged
   • Offline testing: Tests can run without API access once responses are cached
   • Deterministic results: Same inputs always produce same outputs
   • CI/CD friendly: Cache files can be committed to version control
   • Automatic cache directory creation: Creates tests/fixtures/llm_cache/ as needed
   • Parameter-based caching: Different requests create separate cache entries

   Usage

   Tests can now use the cached_ctx fixture instead of ctx to get automatic LLM response caching:

   def test_something(cached_ctx):
       response = cached_ctx.llm.query_llm("What is 2+2?", "You are helpful")
       # First run makes API call and caches response
       # Subsequent runs use cached response

   All changes committed and pushed to the branch.

   ---