Skip to content

Fix CompletionEvent disposal and reset bugs, and coalesce LogSizeTracker signals - #2198

Merged
Badrish Chandramouli (badrishc) merged 4 commits into
mainfrom
tedhar-completion-event
Oct 2, 2026
Merged

Badrish Chandramouli (badrishc) merged 4 commits into
mainfrom
tedhar-completion-event

Conversation

@TedHartMS

@TedHartMS Ted Hart (TedHartMS) commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

CompletionEvent signals by allocating a SemaphoreSlim, swapping it in, releasing the retired
one, and abandoning it. That design is kept — it delegates sync/async waits, cancellation and
timeouts to the BCL — but the bugs around its edges are fixed, and one caller signalled far more
often than it needed to.

Bugs fixed

  • Client throws mid-command. Garnet.client had a divergent copy whose Set() disposed the
    retired semaphore, so a flush landing between NetworkWriter.TryAllocate handing back the capture
    and GarnetClient awaiting it threw ObjectDisposedException out of a user call. Duplicate
    deleted; the shared type is used via InternalsVisibleTo.
  • Shutdown hang. Dispose() disposed the semaphore without releasing it, and
    SemaphoreSlim.Dispose() does not pulse, so threads parked in the infinite waits
    (ShiftBeginAddress, TsavoriteLog.AllocateBlock, the ALLOCATE_FAILED retry) were never woken.
    It now releases the live generation and installs a permanently signaled tombstone.
  • Shutdown throw / NRE. A capture taken before disposal and waited on after threw
    ObjectDisposedException from inside an RMW with its epoch suspended; a wait on the live field
    threw NullReferenceException because Dispose() nulled it.
  • FLUSHDB/FLUSHALL can strand a session thread. AllocatorBase.ResetCore() called
    flushEvent.Initialize(), which overwrites the field without releasing the outgoing semaphore, so
    anyone parked on it was never woken. It now calls Set().
  • Lost cleanup. LogSizeTracker.OnStopped() disposed via a single-shot CAS that silently did
    nothing if a concurrent Signal() won.

Follow-on from the disposal fix

Waking waiters means the retry loops need terminal checks, or they spin where they used to hang:
AllocatorBase gains IsDisposed/ThrowIfDisposed(), applied at 16 sites.

Where the release happens also matters. Releasing flushEvent makes parked waiters runnable, and a
waiter's epoch.Resume() calls Acquire(), which drains pending epoch actions — including the
page-close callbacks that dereference allocator state. It is therefore released in
StopSizeTrackerForDispose(), which both Dispose() overrides call before reclaiming anything,
rather than in AllocatorBase.Dispose(): ObjectAllocatorImpl.Dispose() clears objectPages
before calling base.Dispose(), and a page-close callback drained against a null objectPages is
escalated to FailFast. It also goes before Stop(wait: true), so a resizer parked on flushEvent
cannot keep that wait spinning.

That narrows the window rather than closing it, and the XML doc says so: a thread can still begin a
fresh allocation at any point during teardown and drain a callback on its way in. Disposing an
allocator with operations in flight is unsafe by contract and must be prevented by the owner, which
is why the resizer — the one actor this class owns — is stopped and waited for here.

ShiftBeginAddress also returns rather than proceeding to the head shift, whose OnPagesClosed
would free pages concurrently with teardown; the check sits after the wait loop so it covers every
exit, including the flush-completed break and the noFlush path.

Signal coalescing

InternalUpsert/InternalRMW/InternalDelete signal the size tracker on every object-bearing
record while over budget, and NeedToWaitForClose signals on every page-turn retry. All of them
carry one bit — "go look at the size" — so they are coalesced behind a pending flag. ResizerTask
must capture, then clear, then sample: clearing first lets a signaller's wakeup be consumed by the
capture rather than the wait, parking the resizer for the full ResizeTaskDelaySeconds with work
outstanding.

Measured under sustained pressure (8 threads, 200k records, over budget for ~95% of samples):
~245,000 signal attempts collapse to ~1,900 signals, a ~125x reduction and ~21 MB less gen-0
garbage.
Wall time is unchanged — this is a GC-pressure and contention result, not a throughput
one.

Tests

CompletionEventTests covers the capture discipline, the stale-capture contract, disposal waking
both blocking and async waiters without throwing, Set() waking where Initialize() stranded, and
the coalescing order. LogSizeTrackerSignalCoalescingTests drives sustained over-budget pressure
with the resizer running and asserts its premise, progress, and coalescing; removing the guard takes
signals to ~236,000 and fails it.

Ted Hart added 2 commits October 1, 2026 19:12
CompletionEvent signals by allocating a fresh SemaphoreSlim, swapping it in, releasing the retired one, and abandoning it. That design is kept -- it delegates sync waits, async waits, cancellation and timeouts to SemaphoreSlim -- but it had several bugs around its edges, and one of its callers signalled far more often than it needed to.

LogSizeTracker signalled on every object-bearing record while over budget, which is the steady state for a memory-capped instance: InternalUpsert, InternalRMW and InternalDelete all call IncrementSize on the insert path. A resizePending flag now coalesces that, so an over-budget record update costs one read of a line written only on the signal edge and on resizer wakeups. Measured at 16 threads, that path goes from 5.8M to 3052M updates/s.

ResizerTask must capture the event generation, then clear the flag, then sample sizes. Clearing before capturing loses wakeups: a signaller in that window has its signal consumed by the capture rather than the wait, while the flag stays latched, parking the resizer for the full ResizeTaskDelaySeconds with work outstanding and stalling the allocation retry loop NeedToWaitForClose drives through Signal().

Dispose() disposed the semaphore and nulled the field. SemaphoreSlim.Dispose() does not pulse, so threads parked in the infinite waits -- ShiftBeginAddress, TsavoriteLog.AllocateBlock, the ALLOCATE_FAILED retry -- were never woken; a capture taken beforehand threw ObjectDisposedException from inside an RMW with its epoch suspended; and a wait on the live field threw NullReferenceException. It now releases the live generation and CAS-installs a permanently signaled tombstone, so every waiter present and future falls through and its own retry loop can observe the owner's shutdown state. Set() refuses to replace the tombstone, which would otherwise un-signal a disposed event and double-release it.

Because disposal now wakes waiters, the retry loops need terminal checks or they would spin where they used to hang: AllocatorBase gains IsDisposed/ThrowIfDisposed, and the checks are added to WaitToRetryNow, ShiftBeginAddress, HandleOperationStatus's ALLOCATE_FAILED case, both AllocateBlock overloads, AllocateBlockPartial, and the ten async enqueue loops, where an already-completed await does not yield and would spin at full CPU. ShiftBeginAddress returns rather than falling through, because its head shift is uncapped and the OnPagesClosed that follows would free pages concurrently with the teardown Dispose is already performing. AllocatorBase.Dispose now releases flushEvent waiters before tearing down the epoch and buffer pool they would otherwise touch on the way out.

AllocatorBase.ResetCore called flushEvent.Initialize(), which overwrites the field without releasing the outgoing semaphore, so anyone parked on it was stranded -- every later Set() signals the new instance. It now calls Set().

Garnet.client had its own copy of the type whose Set() did dispose the retired semaphore, giving GarnetClient a reachable ObjectDisposedException in the window between capturing the event and awaiting it. The duplicate is deleted in favour of the shared type via InternalsVisibleTo.
The existing coverage exercised the over-budget allocation path only with the resizer stopped. The case the coalescing exists for is the opposite one: the resizer running but unable to keep up, so the heap stays over budget while inserters keep arriving. Two paths then ask for a signal on every attempt -- every object-bearing record update via IncrementSize, and every page-turn retry via NeedToWaitForClose -> Signal, the latter spinning for as long as a thread is blocked waiting for eviction.

Eight threads upserting 200,000 records with 4KB heap objects into a 64KB budget holds the store over budget for ~95% of sampled intervals and finishes in about 1.2s. In that state roughly 245,000 signal attempts collapse to about 1,900 actual signals, a ~125x suppression worth roughly 21MB of gen-0 garbage. Wall time is unchanged, so this is a GC-pressure and contention result rather than a throughput one.

The test asserts the premise it depends on (pressure really was sustained), that every inserter completes (the allocation path makes progress rather than livelocking), and that signals stay well under one per ten records. Removing the coalescing guard takes that last figure to ~236,000 and fails the assertion, so it is a regression test rather than a tautology.

LogSizeTracker gains an internal ResizerSignalCount for this, incremented only on the path that genuinely signals, so the coalesced path is unaffected.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Shutdown wakeups can race allocator teardown, and test timeout failures bypass sampler cleanup.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity

Open (2)
What changed in this PR

This PR consolidates completion signaling shared by Garnet’s client and Tsavorite, fixes waiter wakeups, and reduces size-tracker signal allocations.

Changes:

  • Replace the client’s duplicate event with the shared implementation.
  • Add disposal-aware retry checks and coalesce resizer signals.
  • Extend event-contract tests and add sustained-pressure coverage.
File Description
libs/​storage/​Tsavorite/​cs/​test/​test.recordops/​LogSizeTrackerSignalCoalescingTests.cs Tests coalescing under sustained memory pressure.
libs/​storage/​Tsavorite/​cs/​test/​CompletionEventTests.cs Covers disposal, reset signaling, and capture ordering.
libs/​storage/​Tsavorite/​cs/​src/​core/​Utilities/​CompletionEvent.cs Releases waiters on disposal and preserves captures.
libs/​storage/​Tsavorite/​cs/​src/​core/​TsavoriteLog/​TsavoriteLog.cs Adds disposal checks to allocation retries.
libs/​storage/​Tsavorite/​cs/​src/​core/​TsavoriteLog/​TsavoriteLog.Chunked.cs Adds a chunked-allocation disposal check.
libs/​storage/​Tsavorite/​cs/​src/​core/​Tsavorite.core.csproj Grants the client access to shared internals.
libs/​storage/​Tsavorite/​cs/​src/​core/​Index/​Tsavorite/​Implementation/​HandleOperationStatus.cs Checks disposal after allocation waits.
libs/​storage/​Tsavorite/​cs/​src/​core/​Index/​Common/​LogSizeTracker.cs Coalesces signals with capture-before-clear ordering.
libs/​storage/​Tsavorite/​cs/​src/​core/​Allocator/​AllocatorBase.cs Updates reset signaling and shutdown handling.
libs/​client/​NetworkWriter.cs Uses Tsavorite’s shared event type.
libs/​client/​CompletionEvent.cs Removes the divergent client implementation.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread libs/storage/Tsavorite/cs/src/core/Allocator/AllocatorBase.cs Outdated
Releasing flushEvent makes parked waiters runnable, and a waiter's epoch.Resume() calls Acquire(), which drains pending epoch actions when drainCount > 0. Those actions include the page-close callbacks that dereference allocator state, so where the release happens decides what that drain can touch.

Releasing it in AllocatorBase.Dispose() was too late for the object allocator: ObjectAllocatorImpl.Dispose() clears objectPages and the page values before calling base.Dispose(), and a page-close callback drained against a null objectPages is escalated to FailFast by OnPagesClosedWorker. Move the release into StopSizeTrackerForDispose(), which both Dispose() overrides call before reclaiming anything, so waiters are woken while that state is still intact. It goes before Stop(wait: true) as well, so a resizer parked on flushEvent cannot keep that wait spinning.

This narrows the window rather than closing it, and the XML doc now says so: a thread can still begin a fresh allocation at any point during teardown and drain a callback on its way in. Disposing an allocator with operations in flight is unsafe by contract and has to be prevented by the owner, which is why the resizer -- the one actor this class owns -- is stopped and waited for here.

ShiftBeginAddress had a second route past the disposal check: the flush-completed break skipped it and went on to the uncapped head shift. The check now sits after the wait loop so it covers every exit, including the noFlush path.

The sustained-pressure test leaked its sampler thread when an inserter timed out: the assertion threw before stopSampling was set, so the sampler kept reading tracker while TearDown cleared it, replacing the intended timeout failure with a NullReferenceException. Stop and join it in a finally, and make it a background thread so it can never hold up the test host.
@badrishc
Badrish Chandramouli (badrishc) merged commit b949396 into main Oct 2, 2026
5 checks passed
@badrishc
Badrish Chandramouli (badrishc) deleted the tedhar-completion-event branch October 2, 2026 18:52
x@01 (x-at-01) added a commit to webc-fork/garnet that referenced this pull request Oct 4, 2026
…n missing key (microsoft#2192), ZADD XX INCR null (microsoft#2197), no empty object from object RMW (microsoft#2194), fresh ObjectOutput per key HCOLLECT/ZCOLLECT (microsoft#2200), CompletionEvent disposal + LogSizeTracker coalesce (microsoft#2198), dynamic test ports (microsoft#2193)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants