Repository navigation
[Bug]: Episodic memory search's respond time increases drastically when number of concurrency increases #837
Description
Activity
Thanks @szou-mv. I did similar Locust testing on MemMachine v0.1.x some time ago and found similar scalability problems. Here is what I saw:
Conclusion: Response times exceed 1 second when the number of users increases from 50 to 100. On my SUT, 50 active users is the maximum.
IMHO, the critical metric is Requests Per Second (RPS), rather than Number of Users. My RPS results were low. My results were better on x86 than on ARM (Mac).
My SUT:
- OS: Ubuntu 25.04 x86_64
- Kernel: Linux 6.14.0-29-generic
- CPU: AMD Ryzen 7 7735HS (x4) @ 3.19 GHz
- Memory: 8 GiB
Suggested Next Steps: Follow the tips outlined in Scaling a FastAPI Application: Handling Multiple Requests at Once to improve FastAPI/FastMCP performance to handle more clients. Enabling more workers should help and scale to the number of available CPUs. By default, FastAPI/FastMCP is single-threaded.
More testing data:
During pure read load testing (/api/v2/memories/search),
When number of concurrency increases, the throughput(RPS) drops.Concurrency Throughput (RPS) Note 1 2.08 Baseline 5 2.65 Best Performance 10 2.02 Performance starts to degrade 50 1.08 Critical Drop (~50% of baseline) Observation: At 50 concurrency, the RPS (1.08) is significantly lower than single-thread execution (2.08). This implies that processing 50 requests sequentially (one by one) would actually be faster than handling them in parallel. This indicates severe resource contention or locking issues within the server.
Test Methodology: For each concurrency setting, 100+ requests were executed to ensure statistical significance and rule out variance or cold-start anomalies.
Expected Behavior: Throughput should increase or plateau as concurrency rises. It should strictly NOT drop below the single-thread baseline.
RPS vs concurrency (1~10)
RPS vs concurrency (10~50)
I experimented with increasing the number of uvicorn workers for the
memmachine-appFastAPI server.Here is the Locust run with the default single-process (existing behavior).
Here is the Locust run with 7 workers on an 8 vCPU server
Here are the Locust report details
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- POST /api/v2/memories [add assistant msg] 6612 111(1.68%) | 11964 3 123983 6700 | 8.49 0.14 POST /api/v2/memories [add user msg] 6807 99(1.45%) | 11748 0 128082 6700 | 8.74 0.13 POST /api/v2/memories/search 1230 24(1.95%) | 25387 4 140393 13000 | 1.58 0.03 POST /api/v2/projects [create] 500 0(0.00%) | 1568 5 15907 700 | 0.64 0.00 --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- Aggregated 15149 234(1.54%) | 12614 0 140393 6800 | 19.44 0.30 Response time percentiles (approximated) Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ POST /api/v2/memories [add assistant msg] 6700 13000 17000 20000 31000 41000 50000 57000 116000 124000 124000 6612 POST /api/v2/memories [add user msg] 6700 13000 17000 20000 29000 39000 50000 56000 116000 128000 128000 6807 POST /api/v2/memories/search 13000 26000 40000 46000 64000 82000 95000 118000 139000 140000 140000 1230 POST /api/v2/projects [create] 700 1500 2300 2700 4000 5400 7900 15000 16000 16000 16000 500 --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ Aggregated 6800 13000 17000 20000 32000 44000 57000 77000 121000 139000 140000 15149 Error report # occurrences Error ------------------|--------------------------------------------------------------------------------------------------------------------------------------------- 38 POST /api/v2/memories [add assistant msg]: HTTPError('500 Server Error: Internal Server Error for url: /api/v2/memories [add assistant msg]') 50 POST /api/v2/memories [add user msg]: RemoteDisconnected('Remote end closed connection without response') 49 POST /api/v2/memories [add user msg]: HTTPError('500 Server Error: Internal Server Error for url: /api/v2/memories [add user msg]') 73 POST /api/v2/memories [add assistant msg]: RemoteDisconnected('Remote end closed connection without response') 21 POST /api/v2/memories/search: RemoteDisconnected('Remote end closed connection without response') 3 POST /api/v2/memories/search: HTTPError('500 Server Error: Internal Server Error for url: /api/v2/memories/search') ------------------|---------------------------------------------------------------------------------------------------------------------------------------------Contrary to expectations, adding more workers hurts performance. We don't see any significant increase in Requests per Second (RPS), but response times have increased significantly, indicating we're likely now dealing with a Database/Back-end problem.
Next Actions:
- Profile
memmachine-appto understand where we are spending the majority of the time - There is serialization within the core code.
- Profile the Postgres and Neo4J databases to see where the bottleneck may be. DBs are not tuned out of the box.
locustfile.pyHere is the
locustfile.pythat uses the client REST APIsimport random import uuid import datetime import os from locust import HttpUser, task, between, SequentialTaskSet, LoadTestShape from faker import Faker fake = Faker() # Configuration ORG_ID = "benchmark_org" PROJECT_ID = "benchmark_project" # Default to localhost:8080 if not specified DEFAULT_HOST = os.getenv("MEMMACHINE_URL", "http://localhost:8080") class ChatSessionBehavior(SequentialTaskSet): """ Base behavior for chat sessions. """ session_id = None num_messages = 0 def on_start(self): self.session_id = str(uuid.uuid4()) # Ensure project exists (idempotent check usually done at user start, but safe to assume here) def simulate_message_pair(self): """Send a user message and then an assistant message.""" # 1. User Message user_content = fake.sentence(nb_words=10) user_msg = { "content": user_content, "role": "user", "timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(), "metadata": {"session_id": self.session_id} } self.client.post( "/api/v2/memories", json={ "org_id": ORG_ID, "project_id": PROJECT_ID, "messages": [user_msg] }, name="/api/v2/memories [add user msg]" ) # 2. Assistant Message asst_content = fake.sentence(nb_words=15) asst_msg = { "content": asst_content, "role": "assistant", "timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(), "metadata": {"session_id": self.session_id} } self.client.post( "/api/v2/memories", json={ "org_id": ORG_ID, "project_id": PROJECT_ID, "messages": [asst_msg] }, name="/api/v2/memories [add assistant msg]" ) def simulate_search(self): """Perform a search related to the session.""" query = fake.sentence(nb_words=5).replace(".", "?") self.client.post( "/api/v2/memories/search", json={ "org_id": ORG_ID, "project_id": PROJECT_ID, "query": query, "top_k": 5, "filter": f"metadata.session_id='{self.session_id}'" }, name="/api/v2/memories/search" ) class ShortChatSession(ChatSessionBehavior): @task def run_session(self): self.on_start() # 3-5 message pairs num_pairs = random.randint(3, 5) for _ in range(num_pairs): self.simulate_message_pair() # 1 search at the end self.simulate_search() self.interrupt() class LongChatSession(ChatSessionBehavior): @task def run_session(self): self.on_start() # 15-20 message pairs num_pairs = random.randint(15, 20) for i in range(num_pairs): self.simulate_message_pair() # Search every ~5 messages if (i + 1) % 5 == 0: self.simulate_search() self.interrupt() class MemMachineUser(HttpUser): wait_time = between(1, 3) host = DEFAULT_HOST tasks = { ShortChatSession: 3, LongChatSession: 1 } def on_start(self): """Initialize user and ensure project exists.""" # Try to create project, ignore 409 (already exists) with self.client.post( "/api/v2/projects", json={ "org_id": ORG_ID, "project_id": PROJECT_ID, "description": "Benchmark Project" }, name="/api/v2/projects [create]", catch_response=True ) as response: if response.status_code == 409: response.success() elif response.status_code >= 400: response.failure(f"Failed to create project: {response.text}") class RampUpShape(LoadTestShape): """ A step load shape that ramps up users from 0 to 500. Stages: 1. Warm up: 10 users (1 min) 2. Low load: 50 users (2 mins) 3. Medium load: 100 users (2 mins) 4. High load: 250 users (3 mins) 5. Peak load: 500 users (5 mins) """ stages = [ {"duration": 60, "users": 10, "spawn_rate": 2}, {"duration": 180, "users": 50, "spawn_rate": 5}, {"duration": 300, "users": 100, "spawn_rate": 5}, {"duration": 480, "users": 250, "spawn_rate": 10}, {"duration": 780, "users": 500, "spawn_rate": 10}, ] def tick(self): run_time = self.get_run_time() for stage in self.stages: if run_time < stage["duration"]: tick_data = (stage["users"], stage["spawn_rate"]) return tick_data return None
app.pychangesChanges to app.py to use the new workers and switch from
uvicorn.Server(...).serve()(Original) touvicorn.run("module:app", ...). The original.serve()doesn't support workers, whereasuvicorn.rundoes.$ git diff src/memmachine/server/app.py diff --git a/src/memmachine/server/app.py b/src/memmachine/server/app.py index ee73a21..3a903ae 100644 --- a/src/memmachine/server/app.py +++ b/src/memmachine/server/app.py @@ -14,6 +14,7 @@ It includes: import argparse import asyncio import logging +import os import sys import time from collections.abc import Callable @@ -104,20 +105,6 @@ async def access_log_middleware(request: Request, call_next: Callable) -> Respon return response -async def start() -> None: - """Run the FastAPI application using uvicorn server.""" - config = load_configuration() - await uvicorn.Server( - uvicorn.Config( - app, - host=config.server.host, - port=config.server.port, - access_log=True, - log_level=str(config.logging.level).lower(), - ), - ).serve() - - def main() -> None: """Execute the CLI entry point for the application.""" # Load environment variables from .env file @@ -127,6 +114,10 @@ def main() -> None: else: load_dotenv() + # Configure basic logging to ensure we see startup messages + logging.basicConfig(level=logging.INFO) + print("DEBUG: memmachine-server entrypoint called", file=sys.stderr) + # Parse command line arguments parser = argparse.ArgumentParser(description="MemMachine server") parser.add_argument( @@ -147,7 +138,30 @@ def main() -> None: asyncio.run(run_mcp_server()) else: # HTTP mode for REST API - asyncio.run(start()) + config = load_configuration() + + # Determine number of workers + workers_env = os.getenv("MEMMACHINE_WORKERS") + if workers_env: + workers = int(workers_env) + else: + # Default to max(1, cpu_count - 1) + cpu_count = os.cpu_count() or 1 + workers = max(1, cpu_count - 1) + print(f"DEBUG: Calculated workers based on cpu_count={cpu_count}: {workers}", file=sys.stderr) + + logger.info(f"Starting server with {workers} workers") + print(f"DEBUG: Starting server with {workers} workers (stderr)", file=sys.stderr) + + # Use uvicorn.run() to correctly handle multiprocessing + uvicorn.run( + "memmachine.server.app:app", + host=config.server.host, + port=config.server.port, + workers=workers, + access_log=True, + log_level=str(config.logging.level).lower(), + ) except KeyboardInterrupt: logger.warning("Application cancelled by user.") sys.exit(130) # Standard exit code for Ctrl+C- Profile
Hi @sscargal, thanks very much for running the experiments and sharing the results on scaling with additional workers. Based on your suggestion, would it be helpful to involve @malatewang and @edwinyyyu in the profiling effort of
memmachine-appand databases? CC @YTLiangMVlocustfile.py
Download: locustfile.py
I have updated the testing and profiling tools to support more targeted benchmarks and deeper performance analysis.
Refactored to support configurable workloads via the LOCUST_WORKLOAD environment variable.
Workload Env Variable Behavior Mixed LOCUST_WORKLOAD=mixed (Default) Simulates standard chat behavior: User/Assistant messages + periodic searches. Write-Only LOCUST_WORKLOAD=write-only Simulates high-volume ingestion: Only sends User/Assistant messages. No search. Read-Only LOCUST_WORKLOAD=read-only Simulates heavy retrieval: Only search queries. Assumes existing data. How to run (CLI):
# Run mixed workload (default) locust -f locustfile.py # Run write-only workload locust -f locustfile.py --workload write-only
How to use (Browser UI):
- Start locust:
locust -f locustfile.py - Open http://localhost:8089
- You will see a Workload drop-down field in the startup form. Select mixed, write-only, or read-only.
- You will see a Memory Types drop-down field. Select all, episodic, or semantic.
profile_memmachine.sh
Download: profile_memmachine.sh
Enhanced to support speedscope output format, which allows for interactive, zooming flamecharts (essential for finding serialization issues).
IMPORTANT
The profiling script uses py-spy, which requires the SYS_PTRACE capability. I have added cap_add: [SYS_PTRACE] to
docker-compose.yml. You MUST restart your container for this to take effect: docker compose up -dArguments:
-f speedscope: Output JSON for speedscope.app.-f flamegraph: Output standard SVG (default).-d <seconds>: Duration to profile.-i <iterations>: Number of profiles to capture.
How to use
Run the script from the host
# Capture 30s profile in speedscope format (BEST for deep dive) ./profile_memmachine.sh -f speedscope -d 30 -i 1 # Capture standard flamegraph ./profile_memmachine.sh -f flamegraph
Reacted by Jing- Start locust:
Episodic Memory + Write Test
- MemMachine_Workers: 1
- Memory_Type: Episodic
- I/O Pattern: Write
Here are the raw results:
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- POST /api/v2/memories [add assistant msg] 686 0(0.00%) | 1820 204 4878 1700 | 1.36 0.00 POST /api/v2/memories [add user msg] 735 0(0.00%) | 8187 186 146657 1800 | 1.46 0.00 POST /api/v2/projects [create] 500 0(0.00%) | 221 8 857 220 | 0.99 0.00 --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- Aggregated 1921 0(0.00%) | 3840 8 146657 1100 | 3.81 0.00 Response time percentiles (approximated) Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ POST /api/v2/memories [add assistant msg] 1700 2400 2700 2800 3100 3400 3900 4200 4900 4900 4900 686 POST /api/v2/memories [add user msg] 1800 2600 2800 3000 3800 82000 86000 143000 147000 147000 147000 735 POST /api/v2/projects [create] 220 250 260 290 380 460 630 790 860 860 860 500 --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ Aggregated 1100 2100 2500 2700 3100 3800 81000 85000 146000 147000 147000 1921Here is one of the profile flamegraphs
The profiling SVG shows that
SCRAM Authenticationis a bottleneck. Under load, the default Postgres database connection pool (size 5) is insufficient. This forces the application to create and discard temporary "overflow" connections rapidly. Every new connection setup requires an expensive SCRAM authentication handshake, which saturates the CPU and stalls the server.Suggested next step - Increase Pool Size: Add configuration for
pool_sizeandmax_overflowto the database config.I increased the Postgres DB Connection Pool size
pool_sizeandmax_overflow settingsfrom 5 to 20 by default.pool_sizeis the number of steady DB connections your app keeps openmax_overflowis how many extra, temporary connections it may create above that when demand spikes. Tuning them is about matching your expected concurrent DB work to PostgreSQL’s capacity (max_connections, vCPU, memory) without oversubscribing the database.
This had a significant benefit for users <=100 as shown in the screenshot below. We now achieve a higher and more consistent RPS
I will add code to make this user configurable, but keep the defaults to allow users to tune the database
Testing the fix for #908 in PR #944
vCPUs: 8
Memory: 16GiB
OS: Ubuntu 25.04
MemMachine Workers: 5Test 1 - Episodic + Write-only
The results show some improvement in RPS, with peaks up to ~80 RPS.
Average RPS increased from 23 to 37.5 ~= 62% improvement.
Performance appears stable up to 250 users; above that, we observe erratic RPS and a significant increase in latency.This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. If this issue is still relevant, please comment to keep it open, or add the
keep-openlabel if it should remain open indefinitely (e.g., a roadmap item awaiting a champion). Thank you for your contributions.
Describe the bug
Following is the respond time chart of load test of POST /memories/search with types = ['episodic'].
When number of user is less than 10, the response time doesn't significantly raise.
However, when number of user reached about 12, the response time raised drastically.
When the number of user reached 30, it took about half a minute to return.
And then as a result, we can only handle 2~3 search requests per second.
(Pure semantic memory search doesn't have this issue)
Steps to reproduce
{"org_id": "org-bedbug", "project_id": "default", "query": "how often should I provide examples?", "top_k": 14, "types": ["episodic"]}loadtest_20251219_120117_test_info.log
Expected behavior
If we can't handle big concurrency, maybe we should add throttling mechanisms.
Environment
Memmachine Version: 65578fc
Env:
MacOS(M2chip 16GB ram)
Embedding model and memmachine are deployed on the same host.
embedding model: embeddinggemma:latest (ollama)
No significant memory/GPU pressure found during load testing.
Additional context
No response