Cluster epoch: isolate GarnetEpoch and replace the busy-spin barrier with a jittered self-poll - #2203

Problem

Adding a primary to a live cluster can wedge the donor: MIGRATE ... SLOTSRANGE pins CPU (~3598% observed) and makes no progress for an hour. The donor stops answering PING and logs connection failures while recipients still own zero slots. Root cause (see investigation): the config-epoch barrier that a bump waits on is an uncapped cooperative busy-spin (await Task.Yield() over a full session scan) with no backoff, so a bump that cannot immediately converge burns whole cores indefinitely.

This PR isolates the epoch barrier and replaces the busy-spin with a jittered, capped self-poll that idles instead of pinning cores, and wires production to it.

What this PR does

1. Isolate the epoch into its own instance. Extracts epoch state out of ClusterProvider into a dedicated GarnetEpoch<TEpochObserver> (libs/cluster/Server/Epochs/). Quiescence is determined through a new IEpochObserver abstraction, with ServerEpochObserverSource wrapping the live StoreWrapper scan. This isolates and clarifies the current API and makes the barrier testable without a live server. TEpochObserver is constrained to struct so the JIT specializes the generic and the quiescence check devirtualizes/inlines.

2. Replace the busy-spin with a self-poll barrier (enabled in production). BumpAndWaitForEpochTransitionAsync increments the epoch, then:

• Fast path: a short adaptive SpinWait (40 iterations) covers the common case where sessions drain immediately.
• Slow path: a jittered, capped exponential-backoff self-poll (Task.Delay) of the same IEpochObserver quiescence predicate.

There is no waker. A releasing session does nothing — the session hot path is untouched (zero locking/allocation on release). Each parked bump re-scans on its own jittered schedule, so concurrent bumps need no coordination (correctness rests on quiescence monotonicity: once all sessions cross the target epoch, every later re-scan still observes quiescence), and a genuinely stuck bump idles at the capped poll rate (maxParkDelay, default 50 ms) instead of burning a core. The bounded slice is also the liveness backstop: a session that quiesces is always seen by the next re-scan. Per-bump jitter decorrelates concurrent waiters (no thundering herd).

Production (ClusterProvider.BumpAndWaitForEpochTransitionAsync) now calls this path directly. The legacy busy-spin is retained as BumpAndSpinWaitForEpochTransitionAsync only as an A/B baseline for the benchmark below; it shares the exact same IEpochObserver predicate, so the two differ only in how they wait, not in what they wait for.

3. Add a reusable backoff primitive. ExponentialBackoff (libs/common/) — a lock-free, allocation-free struct implementing capped exponential backoff with equal-jitter in the upper portion of each slice (Random.Shared). The epoch self-poll derives each park slice from it.

4. Prove the problem and the fix (BDN + custom diagnoser).

• CpuDiagnoser — a custom BenchmarkDotNet IDiagnoser reporting the per-op CPU the wall-clock mean cannot see, split into KernelMode CPU / UserMode CPU / Total CPU columns (kernel = PrivilegedProcessorTime, i.e. syscalls/scheduler). CPU is normalized over the actual workload iterations only.
• EpochBumpContention — an A/B benchmark that runs the identical contention under both strategies (WaitMode = Spin | Poll), measuring the cost the bumping thread (the waiter) pays to complete one config-epoch bump and wait for every session to cross the new epoch, while AcquiringThreads background sessions continuously acquire/release their epoch.

5. Component tests. GarnetEpochBarrierTests (7 tests) covering the fast path, idle/ahead sessions, poll completion on blocking-session release, re-acquire at/above target, cancellation, concurrent bumps each converging, and a release-race soak.

Benchmark evidence

Environment: BenchmarkDotNet v0.15.8, net10.0, Windows 11, AMD Ryzen 7 PRO 7840U (16 logical / 8 physical cores), IterationCount=12, WarmupCount=3. The CpuDiagnoser columns are whole-process CPU per operation, summed across all threads, so a value larger than the wall-clock mean means more than one core was busy.

Waiter penalty and CPU burn (EpochBumpContention)

What it measures: the cost paid by the bumping thread (the waiter) to complete one config-epoch bump and wait for every session to cross the new epoch, while AcquiringThreads background sessions continuously acquire and release their epoch (randomized short holds). The WaitMode axis runs the identical contention under both strategies. Waiter penalty is the mean wall-clock time of one bump-and-wait; the CPU columns are whole-process CPU consumed per bump.

Reading it: the legacy Spin barrier burns ~90–121 ms of CPU per bump — several cores for a ~15–20 ms wait, and the user-mode burn grows with thread count (90 → 106 → 121 ms) — and allocates 140–165 B per bump (Task.Yield continuation boxing). This is the production ~3598%-CPU saturation in miniature. The Poll barrier cuts bump-wait CPU to ~0.2–0.4 ms (≈250–600× less) and allocates nothing, at the cost of a modest, bounded ~9 ms of extra wall-clock per bump (the jittered park quantum replacing frantic spinning).

Why this trade-off is right: the config-epoch barrier fires rarely and sits off the data hot path. Trading ~9 ms of wall-clock per bump to eliminate a full-core CPU burn is exactly what keeps the donor answering PING and lets the migration scan make progress. CPU is the axis that matters on a live server — it is the difference between serving traffic during migration and starving.

Notes / follow-ups

• The self-poll design intentionally has no waker, signal, or queue: with nothing to release waiters, a FIFO/wake mechanism has no role, and quiescence monotonicity makes concurrent bumps safe without shared coordination. This is a deliberate simplification over the earlier wake-based prototype.
• End-to-end migrate/failover test reproducing the CPU symptom is a natural follow-up.