Issues and PRs

This PR consolidates the following issues that were filed, as well as additional stabilization work:

Issue #2174 — With the storage tier and AOF enabled, --recover never completes after an unclean shutdown when hash objects push the log heap over the LogSizeTracker budget. The server never logs "Ready to accept connections" and sits at low CPU. NeedToWaitForClose signalled the size tracker and returned RETRY_NOW, but the tracker's background resizer is not running during recovery, and IssueShiftAddress only advances HeadAddress once the page count reaches MaxAllocatedPageCount — so with only the heap over budget nothing moved and TryAllocateRetryNow spun forever. The same load with SET instead of HSET recovered in about 2 seconds.

PR #2175 — Fixes #2174 in the allocator: NeedToWaitForClose no longer waits on a size tracker whose resizer is not running, and the same guard is applied to NeedToShiftAddress so the MaxAllocatedPageCount cap is still enforced when eviction cannot be deferred. All three page-turn guards test IsRunning before querying the budget, because IsBeyondSizeLimitAndCanEvict reads HeadAddress/TailAddress, which Recovery has not yet set up. Adds UnstartedSizeTrackerAllocationTests (page cap holds with an attached-but-unstarted tracker, at 5/9/16 pages) and RespAdminCommandsTests.SeAofRecoverObjectsOverHeapBudgetTest (end-to-end AOF recovery over the heap budget).

PR #2177 — Refs #1436. ScanCursor always built its iterator with SinglePageBuffering, so a whole-log lookup scan (DBSIZE, INFO KEYSPACE, KEYS, slot deletion) read an on-disk page, processed it, and only then issued the next read; disk IO and the per-record liveness checks never overlapped. Full-log iterations (count == long.MaxValue) now use DoublePageBuffering so the next page is read while the current one is processed. Bounded-count callers such as SCAN keep SinglePageBuffering, since they create a new iterator per call and would waste the read-ahead. Measured on DBSIZE with --storage-tier, 64 MB memory, 4 MB pages, log almost entirely on disk: 60K × 128 KB (7.8 GB log) ~9.8 s → ~6.8 s; 4M × 64 B (316 MB log) ~2.0 s → ~1.8 s.

This does not make DBSIZE O(1), so #1436 is referenced rather than closed: with the storage tier enabled it remains a full scan whose cost is proportional to log size.

Fixes #2174

Original work by Federico Laurianti (@Laurianti) (#2175) and Shivam Sharma (@ShivamSharma43) (#2177); authorship is preserved in the commit history.

Additional fixes

Work beyond what the two PRs proposed. The ScanCursor hardening below started as follow-up work in #2177's own branch that had not been pushed; the rest was found while consolidating, or in review.

Full-log ScanCursor read-ahead hardening. Enabling DoublePageBuffering made frameSize > 1 reachable on the ScanCursor path for the first time, which exposed four latent defects in ScanIteratorBase:

• Frame reuse race. A read-ahead issued into nextFrame could still be in flight when the scan reached a page that mapped back to that frame without having consumed the prefetched one (BeginAddress advancing past the prefetched page is one way in). Reusing the buffer left two device reads writing the same memory and orphaned the completion event the new load replaced. A Debug.Assert documented the invariant but did not enforce it; it is replaced by WaitForPriorFrameLoad, which drains until the prior claim's deferred read has been issued, then waits off-epoch so the drain list can make progress. With frameSize == 1, nextFrame is always currentFrame, which is awaited before every return, so this is a no-op.
• Cancelled token carried into the replacement read. A failed load cancels its frame's token without replacing it, and only WaitForFrameLoad's failure path installs a fresh one — which a frame re-claimed without first being awaited as currentFrame never reaches. The replacement read then ran under an already-cancelled token, so the next wait on that frame threw immediately: it skipped a page that was genuinely being read, and signalled the frame reusable while that read was still writing into its buffer — the very hazard the previous item exists to prevent. WaitForPriorFrameLoad now renews a cancelled token where the frame is prepared for reuse. (Found in review by GitHub Copilot.)
• Per-page CountdownEvent allocation. AsyncReadPageFromDeviceToFrame allocated a fresh CountdownEvent for every page read via an out parameter. It now takes the event by ref and reuses the frame's existing one through PrepareFrameLoadCompletionEvent, so at most one lazily-created kernel wait handle exists per frame for the iterator's lifetime. Reuse is safe because a frame is only re-read once its previous load has completed.
• Failure bookkeeping. WaitForFrameLoad cleared loadedPages but left nextLoadedPages at the failed claim — a state the BufferAndLoad CAS loop could neither satisfy nor re-claim, so a later pass over a page ending at or below that claim would spin. Both vectors are now reset together.

Expired-key-deletion scan. Routed through the unbounded ScanCursor overload so it benefits from the same read-ahead.

AOF replay is now memory-budget-managed. StoreWrapper.ReplayAOF starts the size trackers before replaying. Replay is ordinary store traffic, but the resizer was previously started only after recovery completed, so replay ran unmanaged. With the allocator guard alone, replay no longer hangs but holds roughly five times the object heap — measured ~600 KB versus ~120 KB post-replay in the regression test — because eviction falls back to the page cap. Checkpoint recovery still runs without the resizer, since it owns the log and resets log addresses directly; ReplayAOF is the earliest safe point. Starting is idempotent, so the existing call in StoreWrapper.Start() is unaffected, and the cluster replication and replica disk-based sync replay paths are covered too.

The IsRunning guards are mostly inert during replay, since the resizer is running by then; they protect checkpoint recovery, which precedes replay, the post-CacheSizeTracker.Stop() shutdown window, and the dispatch window described below. Each fix was verified in isolation against SeAofRecoverObjectsOverHeapBudgetTest: the original hang reproduces only when both are reverted.

Size-tracker resizer dispatch window. LogSizeTracker.Start publishes RunState.Running and then queues the resizer body with Task.Run, so IsRunning reported true while the thread pool had not yet dispatched anything that could act on Signal(). The allocation path would defer eviction to that resizer and stall for as long as dispatch took — unbounded under thread-pool starvation, and not limited to recovery. IsRunning now also requires the body to have started; until then the allocator falls back to the same synchronous page-cap eviction it uses when there is no size tracker at all. Measured by injecting a 15 s dispatch delay:

RunState.Running is deliberately still published synchronously in Start rather than moved into the task body: Stop hands off by CAS-ing Running → StopRequested, so if the state were still NotStarted at that point the CAS would fail and Stop would return without signalling or waiting — leaving the body to start against an allocator whose epoch and buffer pool had already been torn down. That variant was tested and Stop(wait: true) returned in 24 ms instead of waiting, confirming the leak.

This one is verified by the measurement above rather than by an automated test. Holding the dispatch window open deterministically needs a test seam in LogSizeTracker, and the gated test built on one passed in isolation but failed intermittently in full-suite runs, flipping on the mere presence of a diagnostic write. Rather than land an intermittently failing test or an unexplained production seam, the seam and test were removed and the coverage gap is called out here.

Lost resize signals. CompletionEvent.Set retires the current semaphore generation and installs a fresh, unsignaled one, so a signal is observed only by a waiter holding a copy of the generation that was current when it was raised. Copying the struct before the work that may race with Set is the contract — it is why CompletionEvent is a struct — and every flushEvent consumer follows it: BlockAllocate captures into operationState.flushEvent before TryAllocateRetryNow, TryAllocateRetryNow/WaitToRetryNow keep a localFlushEvent and refresh it each pass, and TsavoriteLog captures before each wait. LogSizeTracker.ResizerTask was the one caller that did not: it awaited the field directly, re-read after ResizeIfNeeded, so a signal raised while resizing retired the generation it was about to wait on and it slept out the full ResizeTaskDelaySeconds having missed it. It now captures before resizing and waits on the capture. Resize failures are contained in their own catch so they cannot skip the wait, since the previous ordering relied on the wait preceding the resize to rate-limit a persistently failing resize.

CompletionEventTests pins the contract down: a signal is lost without a capture, preserved with one (same-thread and cross-thread), and a consumed capture returns immediately forever rather than missing later signals — Set releases int.MaxValue permits on the generation it retires, which is why the capture must be refreshed rather than reused.

LogSizeTracker.ToString formatted the IsBeyondSizeLimitAndCanEvict method group instead of calling it, so the diagnostic printed a delegate type name (<>f__AnonymousDelegate0...) where the eviction state should have been.

Shutdown window. The same page-turn guards also close a latent hang after CacheSizeTracker.Stop(): any allocation that went over budget once the resizer had stopped would have hit the identical livelock.

Testing

• New: ScanCursorReadAheadTests (bounded-count scans still use SinglePageBuffering; frame reclaim waits for an in-flight read-ahead), read-ahead failure coverage in ScanIteratorEpochFailureTests including ReclaimAfterFailedReadAheadRenewsTheFrameToken, CompletionEventTests (signalling contract), LogSizeTrackerAllocationTests (allocation completes when the heap is over budget and the resizer is not running — never started, and started-then-stopped), UnstartedSizeTrackerAllocationTests, and SeAofRecoverObjectsOverHeapBudgetTest, which now also asserts the heap replay actually held.
• Every fix above except the dispatch window was verified by reverting it and confirming the corresponding test fails: the allocator guards hang, the cancelled-token fix fails its assertion, and SeAofRecoverObjectsOverHeapBudgetTest reports ~600 KB of replay heap instead of ~120 KB.
• Full suites on net10.0 Debug/Windows, all passing: Garnet.test (1290), Garnet.test.scripting (638), Garnet.test.extensions (536), Tsavorite.test.hlog (606), Tsavorite.test (346), Tsavorite.test.recovery (249), Tsavorite.test.recordops (222).
• dotnet format --verify-no-changes clean on both Garnet.slnx and Tsavorite.slnx.

Note on the AOF recovery test's timeout

SeAofRecoverObjectsOverHeapBudgetTest bounds the recovering Start() so the livelock it guards fails as a named assertion rather than hanging the run. Recovery measures ~110 ms and, with the dispatch-window fix above, no longer depends on the thread pool scheduling the resizer, so the 60 s bound is three orders of magnitude of headroom rather than a timing assumption.