fix(metrics): make Prometheus metrics correct with multiple workers, and wire the trackers up (port of #1523 to main) - #1678

…and wire the trackers up (MemMachine#1523)

* fix(metrics): aggregate Prometheus metrics across uvicorn workers

With MEMMACHINE_WORKERS > 1 uvicorn forks a worker per process and each
keeps its own registry, so /metrics returned whichever worker happened to
answer the scrape - about 1/N of the traffic, a different 1/N each time.
Consecutive scrapes then look like counter resets, and every rate,
histogram quantile and calls-per-request figure derived from them is
wrong in an unbounded direction.

Build a fresh registry per scrape and attach MultiProcessCollector when
PROMETHEUS_MULTIPROC_DIR is set, leaving the single-worker path on the
default registry so nothing changes for the default deployment.

Measured on an 8-tenant capacity run: counter resets over one ladder went
from 70 (4 workers) and 87 (8 workers) to 0, and five consecutive scrapes
of a loaded server returned identical values where they previously
diverged.

Requires PROMETHEUS_MULTIPROC_DIR to be set and a writable directory
mounted at it; the chart change that does so is separate.

* fix(metrics): hand every OperationTracker a metrics factory

OperationTracker accepts metrics_factory=None and then silently discards
every timing it takes - no error, no warning, no series. A component that
is fully instrumented but never given a factory is therefore
indistinguishable from one that was never instrumented at all, and the
only way to notice is to go looking for a metric that should exist.

The Neo4j store, the episode store and the session store each shipped
instrumented and unwired, which is why database latency appeared to be
unmeasurable: every call was timed and thrown away. Wire the factory
through the resource manager, the database manager and the event
backend's params so those trackers emit.

The tests assert at the call sites rather than on the components. An
earlier version tested that each store honours a factory it is given,
which passes whether or not anything passes one - it still passed with
the fix reverted. These fail when the wiring is removed.

* fix(metrics): choose a multiprocess directory when workers > 1

PROMETHEUS_MULTIPROC_DIR had to be set by whoever deployed the server. Set
MEMMACHINE_WORKERS=4 without it and nothing complains: each worker keeps its
own registry, a scrape is answered by whichever worker the load balancer
picked, and the numbers that come back look plausible. There is no error to
notice, so the only way to find out is to compare a counter against a request
count and see it come up short by a factor of the worker count.

The worker count is the thing that decides whether aggregation is needed, so
read it here and default the directory when it is above 1. An explicit setting
still wins, which is how two servers on one host keep their metrics apart. If
the chosen directory cannot be created the variable is removed again -
prometheus_client raises in every worker if it cannot open the directory it is
pointed at, and taking the server down over metrics would be the wrong trade.

Worker-count parsing moves into _worker_count() so start_http() and the
directory setup cannot disagree about it.

Verified: uvicorn spawns workers via multiprocessing.get_context("spawn"), so
children re-import in a fresh interpreter and inherit the variable set here.
13 new tests cover the default, the explicit override, nested creation, stale
file clearing, and both failure paths; 1228 server/common tests pass; ruff
check, ruff format --check and ty are clean (17 ty diagnostics, all
pre-existing and none in these files).

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP

---------

Co-authored-by: Claude Opus 5 <[email protected]>
(cherry picked from commit ace7752)
Signed-off-by: Haiyan Wang <[email protected]>