Is your feature request related to a problem?
MemMachine may experience bursty traffic, and the current single-process FastAPI service may not handle incoming user requests effectively, leading to slow response times or dropped requests. There is currently no way to scale FastAPI worker processes for handling these bursts, which limits the system's ability to efficiently utilize available CPU resources and provide reliable service under fluctuating load.
Describe the solution you'd like
Introduce support in @app.py to manage a pool of FastAPI workers. Unfortunately, when deployed with Uvicorn, FastAPI uses a static worker pool configured at startup via the --workers flag, without built-in support for dynamic scaling between lower and upper bounds. This is "good enough" for now. We will investigate the feasibility of introducing a dynamic pool in the future.
Describe alternatives you've considered
N/A
Additional context
The ability to scale FastAPI workers effectively is critical as user adoption grows, and efficient CPU utilization is a recurring need for modern Python API services.
This should help #837, though it is expected to be a multi-phase approach/solution.
Is your feature request related to a problem?
MemMachine may experience bursty traffic, and the current single-process FastAPI service may not handle incoming user requests effectively, leading to slow response times or dropped requests. There is currently no way to scale FastAPI worker processes for handling these bursts, which limits the system's ability to efficiently utilize available CPU resources and provide reliable service under fluctuating load.
Describe the solution you'd like
Introduce support in @app.py to manage a pool of FastAPI workers. Unfortunately, when deployed with Uvicorn, FastAPI uses a static worker pool configured at startup via the
--workersflag, without built-in support for dynamic scaling between lower and upper bounds. This is "good enough" for now. We will investigate the feasibility of introducing a dynamic pool in the future.Describe alternatives you've considered
N/A
Additional context
The ability to scale FastAPI workers effectively is critical as user adoption grows, and efficient CPU utilization is a recurring need for modern Python API services.
This should help #837, though it is expected to be a multi-phase approach/solution.