Bound LightEpoch waiter semaphore signals to prevent SemaphoreFullException by TedHartMS · Pull Request #2139 · microsoft/garnet

Ted Hart (TedHartMS) · 2026-09-16T01:25:17Z

Problem

LightEpoch.Release() signalled waiterSemaphore once per epoch release for as long as any thread was waiting for a table entry, but each waiter consumes exactly one signal:

entry = kInvalidIndex;
if (waiterCount > 0)
    waiterSemaphore.Release();

The two rates are unrelated and wildly asymmetric, because ProtectAndDrain() hops through SuspendResume() on every refresh while a waiter exists:

if (waiterCount > 0)
    SuspendResume();   // Suspend -> Release -> +1 signal

So a handful of waiters causes every protected thread to emit a signal on every refresh. waiterSemaphore is new SemaphoreSlim(0) (maxCount int.MaxValue), so once the count reaches int.MaxValue every Release() executed while a waiter exists throws SemaphoreFullException.

The throw happens after the slot is freed, so the epoch table stays consistent — but the exception unwinds into the caller: client sessions die in ProcessMessages (and Dispose() throws again through StorageSession.Dispose() → SuspendResume()), and the object-collect / expired-key-deletion tasks exit permanently. Nothing decrements the count, so only a process restart recovers.

Reported in production as ~47 h clean, then 24,622 SemaphoreFullExceptions over 4 days at a steady 126–414/hour. Trigger is sustained concurrency above kTableSize = max(128, ProcessorCount * 2).

Fix

Track outstanding signals in pendingWaiterSignals and only take a reservation while it is below waiterCount:

void SignalWaiter()
{
    var pending = pendingWaiterSignals;
    while (pending < waiterCount)
    {
        var prev = Interlocked.CompareExchange(ref pendingWaiterSignals, pending + 1, pending);
        if (prev == pending)
        {
            waiterSemaphore.Release();
            return;
        }
        pending = prev;
    }
}

The reservation is taken before the semaphore release and returned after a successful Wait(), so the semaphore's count is capped by the peak number of concurrent waiters — O(threads) rather than unbounded — and the overflow is unreachable.

The cap is against the peak rather than the instantaneous count: a waiter can claim the freed slot on its re-probe and decrement waiterCount while a reservation taken for it is still in flight, briefly leaving a signal outstanding with no waiter present. Such a leftover is consumed by the next waiter, which decrements and re-probes, so leftovers drain instead of accumulating.

I used an explicit counter rather than the waiterSemaphore.CurrentCount < waiterCount check suggested in the issue: that check is a racy read-then-act under concurrent releases, and it leaves stale signals behind that accumulate across contention bursts.

Liveness. Suppressing a signal cannot lose a wakeup:

A waiter only blocks when the semaphore's count is zero. A signal is only withheld when a wake is already queued for every waiter, so at least one is inbound.
Waiters are released FIFO and re-probe the whole table before blocking again, so whichever one wakes finds the freed slot.
The read of pendingWaiterSignals follows the release-store that publishes the slot as free, so a waiter woken by an already-outstanding signal sees it.
Dispose() sets the waiterCount MSB, making it negative, so no signal is issued and waiters unwind through cts — unchanged behaviour.

SignalWaiter() is NoInlining so the CAS loop stays out of Release()'s inlined body. Under contention this is also slightly cheaper than before, since it skips SemaphoreSlim.Release() (which takes an internal lock) whenever a signal is already pending.

Background task resilience

ObjectCollectTaskAsync, ExpiredKeyDeletionScanTaskAsync and CompactionTaskAsync each wrapped their whole loop in one catch (Exception) that logged and returned, so a single transient fault disabled that maintenance task for the process lifetime. The per-cycle work is now guarded and retried on the next interval; Task.Delay already provides the backoff.

Since all three loops became structurally identical, the shared loop is extracted into StoreWrapper.RunMaintenanceLoopAsync. That removes the triplication and gives one seam where the retry contract can be tested with fault injection rather than a live server and real timers.

Compaction also no longer logs a spurious error on shutdown: cancellation is caught as OperationCanceledException and suppressed, matching the other two.

Tests

libs/storage/Tsavorite/cs/test/test.epoch/WaiterSignalTests.cs — the helper fills every table slot, confirms the surplus threads are parked on the semaphore, and only then releases the churn, so oversubscription is deterministic rather than timing-dependent.

SignalCountStaysBoundedWhenTheTableIsOversubscribed — asserts the outstanding signal count stays within the waiter bound across 270k+ releases.
EveryThreadCompletesWhenTheTableIsOversubscribed — liveness guard against a lost wakeup, with far more threads than slots so nearly every release is a suppression candidate.
SignalsDrainAfterContentionEnds — asserts signals do not accumulate across three successive bursts.

Verified these catch the bug by temporarily reverting the fix: 4,652 unconsumed signals against a bound of 136 in ~0.4 s of contention, and 858 leaked after a single burst. The liveness test passes either way by design — it guards this change rather than the original bug.

test/standalone/Garnet.test/MaintenanceLoopTests.cs — covers the retry contract directly with an injected fault:

AFailedCycleIsLoggedAndTheLoopKeepsRunning — first cycle throws; logged once at Error (not Critical), and a later cycle runs.
TheLoopKeepsRunningWhenEveryCycleFails — every cycle throws; the loop is still running on the third and never logs terminally.
CancellationEndsTheLoopWithoutFaultingOrLogging — the task completes successfully and logs nothing.
ACycleCancelledDuringShutdownIsNotLoggedAsAFailure — a cycle that throws because of cancellation is shutdown, not a failure.

Verified these catch a regression too: reverting the helper to a single terminal catch fails the two retry tests.

Validation

Suite	Result
`Tsavorite.test.epoch` (net10.0 + net8.0)	18/18 each
`Tsavorite.test` (full)	342 passed, 26 skipped
`Garnet.test` RespTests	354/354
`Garnet.test` MaintenanceLoopTests	4/4
`BasicLockTests` / `ExpiredKeyDeletionTests` / `HashCollect`	all pass
`dotnet build` (full solution)	0 warnings, 0 errors
`dotnet format --verify-no-changes` (both solutions)	clean

Docs

website/docs/dev/tsavorite/epochprotection.md gains a Waiting for an epoch table entry section covering the probe strategy, the waiterCount re-probe that closes the releaser race, the peak-waiter signal bound and its self-draining leftovers, and Dispose handling. It also corrects the existing claim that "If the epoch table is full, the rest of the threads will keep yielding", which describes the spin-wait replaced in #1543.

LightEpoch.Release() signalled waiterSemaphore once per epoch release for as long as any thread was waiting for a table entry, but each waiter consumes exactly one signal. ProtectAndDrain() hops through SuspendResume() on every refresh while a waiter exists, so releases vastly outnumber waiters under contention and the semaphore's count climbed monotonically. At int.MaxValue SemaphoreSlim.Release() began throwing SemaphoreFullException out of every epoch release, killing client sessions and permanently stopping the object collect and expired key deletion tasks until the process was restarted. Track outstanding signals in pendingWaiterSignals and only signal while it is below waiterCount. The reservation is taken before the semaphore release and returned after a successful wait, so the semaphore's count can never exceed the number of waiters. Suppressing a signal cannot strand a waiter: a signal is only withheld when every current waiter already has a wake pending, and a woken waiter re-probes the whole table before blocking again. Also stop the two background tasks in StoreWrapper from terminating for the lifetime of the process on an unexpected exception. The per-cycle work is now guarded so a failed cycle is logged and retried on the next interval.

Copilot

🟡 Changes recommended

The documented waiter invariant is inaccurate, and the new background-task retry behavior lacks regression coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Bounds LightEpoch semaphore signals to prevent overflow and keeps maintenance tasks alive after transient failures.

Changes:

Adds bounded waiter-signal accounting and concurrency tests.
Makes collection and expiration tasks retry failed cycles.
Documents epoch-table waiting behavior.

File summaries

File	Description
`LightEpoch.cs`	Adds bounded signal reservations.
`LightEpoch.TestHooks.cs`	Exposes waiter metrics to tests.
`WaiterSignalTests.cs`	Tests signal bounds and liveness.
`StoreWrapper.cs`	Retries failed maintenance cycles.
`epochprotection.md`	Documents waiter signaling.

Review details

Suppressed comments (1)

libs/storage/Tsavorite/cs/src/core/Epochs/LightEpoch.cs:572

The claimed pendingWaiterSignals <= waiterCount invariant is racy. After the slot is published, a waiter may acquire it at line 686 and leave the slow path while this signaler has already incremented pendingWaiterSignals but has not yet released the semaphore; once that waiter decrements waiterCount, pending can exceed the current waiter count. Please base the explanation on the actual peak-waiter bound and account for stale, self-draining reservations.

        /// The increment is what reserves the right to signal, so it always precedes its
        /// <see cref="SemaphoreSlim.Release()"/> and the matching decrement always follows a successful wait.
        /// The semaphore's count is therefore never above <see cref="pendingWaiterSignals"/>, which this loop
        /// holds at or below <see cref="waiterCount"/>.

Files reviewed: 5/5 changed files
Comments generated: 3
Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

CompactionTaskAsync wrapped its whole loop in a single catch, so any exception from a compaction cycle ended the task for the lifetime of the process. Guard the per-cycle work instead, so a failed cycle is logged and retried on the next interval, matching the object collect and expired key deletion tasks. Cancellation is now caught as OperationCanceledException and suppressed rather than logged as an error, so shutdown no longer reports a spurious failure.

Correct the documented bound on pendingWaiterSignals. A waiter can claim the freed slot on its re-probe and decrement waiterCount while a reservation taken for it is still in flight, so the cap is against the peak number of concurrent waiters, not the instantaneous count. A leftover signal is consumed by the next waiter, which decrements and re-probes, so leftovers drain rather than accumulate. Any such bound prevents the overflow; the stronger invariant the comments and docs asserted does not hold. Extract the shared retry loop used by the compaction, object collect and expired key deletion tasks into RunMaintenanceLoopAsync. The three loops were identical after the resilience change, and a single seam makes the contract testable so it cannot silently regress to a terminal catch. Add MaintenanceLoopTests covering that contract with fault injection: a failed cycle is logged as an error and the loop runs again, repeated failures keep retrying without a terminal log, cancellation ends the loop without faulting or logging, and a cycle cancelled during shutdown is not logged as a failure.

…ter-semaphore-fix

Records that comparing the semaphore's CurrentCount against waiterCount is a read-then-act with no atomicity against the Release that follows, and that the resulting surplus is never reclaimed, so the reservation counter is the bound that holds.

…ter-semaphore-fix

Gabriel Tarpian (gabrieltarpian25) · 2026-09-17T10:20:43Z

Many thanks for taking a look at this one. In the meantime, could you please confirm this changes in the conf file will fix this issue?
"ThreadPoolMaxThreads": 96,
"NetworkSendThrottleMax": 32

If not, could you please let us know if there is an actual workaround for this situation? Ideally, from conf file for existing customers and in a future release we will update to latest version that will have this fix.

Thank you!

Ted Hart (TedHartMS) · 2026-09-18T04:28:39Z

Gabriel Tarpian (@gabrieltarpian25) — From CoPilot: Short answer: ThreadPoolMaxThreads is a valid stopgap, but use ~120 rather than 96. NetworkSendThrottleMax does nothing for this bug — leave it at 8.

Why the thread cap works

kTableSize = max(128, ProcessorCount * 2). Running on a 24-core box, kTableSize = 128. An epoch entry is held only inside a protected region, at most one per thread, and every holder is a .NET thread-pool worker — across every run I did, distinct holder thread IDs exactly equalled peak ThreadPool.ThreadCount (68/96/120/121/215/420), including under BGSAVE. So capping workers below 128 means TryAcquireEntry always succeeds, ReserveEntryWait never runs, waiterCount stays 0, and Release() never touches the semaphore.

Instrumenting storeEpoch under the load that reproduces the bug (48 cores, also kTableSize 128; 512 connections, BGSAVE, 30 s):

`ThreadPoolMaxThreads`	peak pool	peak slots	peak waiters	peak outstanding signals
unset	420	128/128	272	617,796
96	96	96/128	0	0
120	120	120/128	0	0

Uncapped accumulates ~37.5k signals/s, reaching int.MaxValue in ~16 h — consistent with your ~47 h onset.

Why 120 over 96

A session parks a worker thread in GarnetTcpNetworkSender.Throttle() whenever a client drains slowly. That blocking is outside epoch protection, so it burns worker threads without consuming slots — which is exactly what the cap is rationing. With 96 slow-draining connections plus 256 normal ones (60 s):

`ThreadPoolMaxThreads`	throughput	min free workers
unset	113k ops/s (pool grew to 125)	ample
120	47k ops/s	2
96	11k ops/s	5

Both values prevent the bug equally; 96 just throttles the server harder. Pick the highest value safely under 128 — 120 leaves 8 spare slots, which was never encroached in any run.

Two caveats: ThreadPoolMinThreads (Redis io-threads) set above the max throws at startup; and the cap only binds because .NET dispatches socket completions to worker threads, so it would not hold under DOTNET_ThreadPool_UseWindowsThreadPool=1.

Why `NetworkSendThrottleMax` is irrelevant

It bounds outstanding sends per session, which has no bearing on how many threads sit inside an epoch region. With it at 32 and no thread cap, the bug reproduces unchanged — that is the 272-waiter / 617k-signal row above. Raising it also costs ~4x per-session send memory, and saeaStack = new(2 * ThrottleMax) is evaluated before ThrottleMax is assigned, so the SAEA pool stays at 16 regardless.

Verified against v1.1.8: GarnetServer.cs and LightEpoch.cs are identical on both paths, so this applies directly to your deployment.

Ted Hart (TedHartMS) · 2026-09-20T19:46:12Z

Status update: conflicts resolved, and a full answer to the review finding on the signal bound

Two things were outstanding on this PR: a conflicted merge with main, and an open 🟡 Changes recommended verdict questioning the correctness of the bound this PR relies on. Both are addressed below. The short version: the reviewer was right about the commit they reviewed, and that exact defect was already corrected in 254b5ede before the verdict was posted. There is no residual hole.

1. Merge conflict with `main`

Resolved at d9b9075. The conflict surface was a single hunk in libs/server/StoreWrapper.cs (CompactionTaskAsync).

The cause is a benign convergence: #2110 ("Fix root causes behind recurring Garnet .NET CI flakes") landed on main and independently added the same OperationCanceledException suppression to CompactionTaskAsync that this PR had already written. Both sides were fixing the same defect; git could not tell. The resolution keeps this PR's RunMaintenanceLoopAsync form, whose outer catch (OperationCanceledException) when (token.IsCancellationRequested) already provides exactly what #2110 added, and carries over #2110's more descriptive comment wording. Verified afterwards:

CompactionTaskAsync calls DoCompactionAsync exactly once — the auto-merged region left that call outside the conflict markers, so a naive resolution could easily have duplicated it.
CommitTaskAsync retained Fix root causes behind recurring Garnet .NET CI flakes #2110's suppression verbatim (StoreWrapper.cs:693-695). It is structurally distinct (it has a replica branch) and this PR never touched it, so it was left alone.
The comment wording adopted from Fix root causes behind recurring Garnet .NET CI flakes #2110 mentions primary-only task suspension. That is accurate here too, not just for CommitTask: CompactionTask, ObjectCollectTask and ExpiredKeyDeletionTask are all TaskPlacementCategory.Primary (TaskManager/TaskType.cs:76-78), so SuspendPrimaryOnlyTasksAsync() cancels them on replica transition.

On merge strategy: this repo has required_linear_history on main, and the API confirms squash is the only enabled merge method (allow_squash_merge=true, allow_merge_commit=false, allow_rebase_merge=false). The whole branch collapses to one commit on merge, so merge commits inside the branch never reach main and linear history is unaffected. Merging was preferred over rebasing because this PR has resolved review threads anchored to existing commits, which a force-push would detach.

2. The review finding on `pendingWaiterSignals <= waiterCount`

The claimed pendingWaiterSignals <= waiterCount invariant is racy. After the slot is published, a waiter may acquire it and leave the slow path while this signaler has already incremented pendingWaiterSignals but has not yet released the semaphore; once that waiter decrements waiterCount, pending can exceed the current waiter count.

This is correct, and it was a real documentation defect — option (b) in the framing: the instantaneous invariant does not hold, and the bound that actually prevents SemaphoreFullException is a peak-waiter bound.

The verdict was submitted against 2c9b9177, the first commit on the branch. At that commit the field doc did assert the instantaneous claim:

"The semaphore's count is therefore never above pendingWaiterSignals, which this loop holds at or below waiterCount."

"holds at or below waiterCount" is not true instant-to-instant, for precisely the interleaving described. This was found and corrected in 254b5ede ("Address PR review: precise signal bound, tested maintenance loop"), which predates the verdict being posted but postdates the commit it was computed against. The current wording on SignalWaiter (LightEpoch.cs:586-590) states the peak bound and names the same interleaving the reviewer describes:

"...so both are capped by the peak number of concurrent waiters. The cap is against the peak rather than the instantaneous count: a waiter can claim the freed slot on its re-probe and decrement waiterCount while a reservation taken for it is still in flight, leaving a signal outstanding with no waiter present. The next waiter consumes that signal, decrements, and re-probes, so leftovers drain."

So the finding is real, already remediated, and the verdict is stale rather than outstanding.

Why the peak bound is the right one, and why it is sufficient

Let S be the semaphore count, P = pendingWaiterSignals, W = waiterCount.

(i) S <= P always. The CAS increment of P (LightEpoch.cs:605-610) is what authorises a signal, and it strictly precedes its waiterSemaphore.Release(). The matching decrement (:711) strictly follows a successful Wait() (:707). So at any instant releases <= increments and decrements <= waits, giving S = releases - waits <= increments - decrements = P.

(ii) P <= peak concurrent W. P only rises via a CAS that succeeded against the guard pending < waiterCount, so immediately after any increment P <= W as sampled at that moment, and therefore P <= max(W). P never rises by any other route.

Together: S <= P <= peak waiters <= threads in the process. SemaphoreSlim(0) has maxCount = int.MaxValue, so the overflow that motivated this PR is unreachable by many orders of magnitude. Any O(threads) bound is sufficient; the peak bound is one.

The key property is that a stale reservation occupies budget rather than adding to it. If leftovers push P up, further increments are suppressed until W exceeds P again, so leftovers cannot accumulate across contention bursts — they are consumed by the next waiter, which decrements and re-probes. This is exactly the property the CurrentCount < waiterCount check suggested in #2135 lacks, and is why this PR uses an explicit counter.

No residual hole (not (c)), and no lost wakeups. A waiter only blocks when S == 0; a signal is only suppressed when a wake is already queued for every waiter, so at least one is inbound. Waiters are released FIFO and re-probe the entire table before blocking again, so whichever wakes finds this slot or a later one. The read of P in SignalWaiter follows the release-store that publishes the slot as free (:572), so a waiter woken by an already-outstanding signal observes the free slot. During Dispose the MSB of W makes it negative, so no signal is issued and waiters unwind through cts.

One deliberate wording note: the bound guarantees bounded and non-accumulating, not "drains to zero". SignalsDrainAfterContentionEnds asserts both counters stay <= threadCount across three bursts; it does not assert zero, and the docs do not claim it.

3. Regression coverage for the background-task retry

The review also noted the new StoreWrapper retry behaviour lacked coverage. test/standalone/Garnet.test/MaintenanceLoopTests.cs was added in 254b5ede and covers it deterministically — fault injection straight into RunMaintenanceLoopAsync, no live server and no real timers:

Test	Asserts
`AFailedCycleIsLoggedAndTheLoopKeepsRunning`	a throwing cycle is logged and the loop runs the next cycle
`TheLoopKeepsRunningWhenEveryCycleFails`	repeated failures never terminate the loop
`CancellationEndsTheLoopWithoutFaultingOrLogging`	shutdown is clean and silent
`ACycleCancelledDuringShutdownIsNotLoggedAsAFailure`	a cycle cancelled mid-flight is not misreported as an error

These were verified to be genuinely load-bearing rather than vacuously green: reverting RunMaintenanceLoopAsync to a single terminal catch turns AFailedCycleIsLoggedAndTheLoopKeepsRunning and TheLoopKeepsRunningWhenEveryCycleFails red. The same method was applied to the epoch fix — reverting SignalWaiter produces 4,652 unconsumed signals against a bound of 136 in about 0.4s.

4. Validation, and one inherited breakage to be aware of

On the merge result: Tsavorite.test.epoch 18/18, MaintenanceLoopTests 4/4, and both dotnet format --verify-no-changes checks (Garnet.slnx and Tsavorite.slnx) clean.

⚠️ CI on this PR will be red for a reason inherited from main, not from this PR. Tsavorite.test.csproj fails to compile on pristine origin/main (5c6e595):

TestBase.cs(46,9): error CS0234: The type or namespace name 'TestUtils'
  does not exist in the namespace 'Garnet.test'  [Tsavorite.test.csproj]

Root cause: Tsavorite.test.csproj links exactly one file out of Garnet.test — TestBase.cs (Tsavorite.test.csproj:14) — and #2151 added a Garnet.test.TestUtils.EnsurePortSlotResolved() call to TestBase.cs while TestUtils.cs is not linked into that project. Garnet.test.cluster links both files and so is unaffected.

This blocks every PR, not just this one, and it also blocks Tsavorite.test.epoch, which project-references Tsavorite.test. The 18/18 epoch result above was obtained by guarding that one call behind a compilation symbol locally; that workaround was reverted and is not part of this branch, since the fix belongs with #2151 rather than buried in an unrelated PR. No upstream fix appears to be in flight yet.

Separately, RespTests.MultipleClientsUnblockAndAddTest and ClusterVectorSetTests.VectorSetMigrateManyBySlot are known flakes on main unrelated to this change; re-running the failed jobs on an identical commit previously went green at 226 passed / 0 failed.

microsoft#2145), LightEpoch semaphore bound (microsoft#2139)

Copilot AI balanced review requested due to automatic review settings September 16, 2026 01:25

Copilot started reviewing on behalf of Ted Hart (TedHartMS) September 16, 2026 01:26 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Comment thread libs/server/StoreWrapper.cs Outdated

Comment thread libs/storage/Tsavorite/cs/src/core/Epochs/LightEpoch.cs

Comment thread website/docs/dev/tsavorite/epochprotection.md Outdated

Ted Hart (TedHartMS) added 5 commits September 15, 2026 18:49

Merge remote-tracking branch 'origin/main' into tedhar-lightepoch-wai…

0d521d3

…ter-semaphore-fix

Merge remote-tracking branch 'origin/main' into tedhar-lightepoch-wai…

21a6dcb

…ter-semaphore-fix

Ted Hart (TedHartMS) requested a review from Badrish Chandramouli (badrishc) September 16, 2026 21:50

Badrish Chandramouli (badrishc) approved these changes Sep 18, 2026

View reviewed changes

Merge branch 'main' into tedhar-lightepoch-waiter-semaphore-fix

d9b9075

Ted Hart (TedHartMS) mentioned this pull request Sep 20, 2026

Tsavorite.test fails to compile on main: TestBase.cs references Garnet.test.TestUtils, which is not linked into Tsavorite.test.csproj #2161

Closed

Ted Hart (TedHartMS) added 2 commits September 21, 2026 09:18

Merge branch 'main' into tedhar-lightepoch-waiter-semaphore-fix

ac1cf0f

Merge branch 'main' into tedhar-lightepoch-waiter-semaphore-fix

da6aa18

Badrish Chandramouli (badrishc) approved these changes Sep 22, 2026

View reviewed changes

Ted Hart (TedHartMS) added 2 commits September 21, 2026 23:11

Merge branch 'main' into tedhar-lightepoch-waiter-semaphore-fix

e556781

Merge branch 'main' into tedhar-lightepoch-waiter-semaphore-fix

e2817df

Ted Hart (TedHartMS) merged commit 0654725 into main Sep 22, 2026
227 checks passed

Ted Hart (TedHartMS) deleted the tedhar-lightepoch-waiter-semaphore-fix branch September 22, 2026 07:36

Ted Hart (TedHartMS) mentioned this pull request Sep 23, 2026

Tighten up SetupFixtures and move TestBase to Tsavorite; handle cancellation race condition in maintenance loops #2162

Merged

x@01 (x-at-01) added a commit to webc-fork/garnet that referenced this pull request Oct 4, 2026

merge: upstream 2026-09-22 — failed checkpoint not reported as success (

6a1fb90

microsoft#2145), LightEpoch semaphore bound (microsoft#2139)

Navigation Menu

Repository navigation

Bound LightEpoch waiter semaphore signals to prevent SemaphoreFullException - #2139

Bound LightEpoch waiter semaphore signals to prevent SemaphoreFullException#2139
Ted Hart (TedHartMS) merged 11 commits into
mainfrom
tedhar-lightepoch-waiter-semaphore-fix

Ted Hart (TedHartMS) commented Sep 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gabriel Tarpian (gabrieltarpian25) commented Sep 17, 2026

Uh oh!

Ted Hart (TedHartMS) commented Sep 18, 2026 •

edited

Loading

Uh oh!

Ted Hart (TedHartMS) commented Sep 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Ted Hart (TedHartMS) commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Background task resilience

Tests

Validation

Docs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gabriel Tarpian (gabrieltarpian25) commented Sep 17, 2026

Uh oh!

Ted Hart (TedHartMS) commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why the thread cap works

Why 120 over 96

Why NetworkSendThrottleMax is irrelevant

Uh oh!

Ted Hart (TedHartMS) commented Sep 20, 2026

Status update: conflicts resolved, and a full answer to the review finding on the signal bound

1. Merge conflict with main

2. The review finding on pendingWaiterSignals <= waiterCount

Why the peak bound is the right one, and why it is sufficient

3. Regression coverage for the background-task retry

4. Validation, and one inherited breakage to be aware of

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Ted Hart (TedHartMS) commented Sep 16, 2026 •

edited

Loading

Ted Hart (TedHartMS) commented Sep 18, 2026 •

edited

Loading

Why `NetworkSendThrottleMax` is irrelevant

1. Merge conflict with `main`

2. The review finding on `pendingWaiterSignals <= waiterCount`