Repository navigation
Stop a dropped connection or an expired test certificate from wedging the test host - #2204
Merged
Merged
Conversation
… the test host Diagnosing a wedged Garnet.test run surfaced three defects that each turn a transient or environmental problem into a hang, or into a silent loss of coverage. GarnetClient left callers awaiting replies that could never arrive. DisposeOffset handled every task type except LongAsync and LongCallback, so a pending Task<long> was never faulted, and Dispose tore down the socket without draining the requests still holding a slot. Connection teardown and the client's own disposal are independent -- a connection can die while Disposed is still false -- so a request issued in between owned a slot nobody would complete. Dispose now drains, closing that window: a request registered before disposal is drained, and one registered after hits the existing Disposed check. Draining twice is a no-op, because the loop and ConsumeTcsOffset advance together. An expired TLS test certificate does not fail a test, it hangs the host. Under TLS 1.3 the client completes its half of the handshake before the server validates the client certificate, so ConnectAsync succeeds and the first request is the one that dies, the server having closed without a RESP error because writing one mid-handshake would be a protocol violation. TestCertificateExpiryTests reports that in milliseconds together with the renewal steps, and warns 30 days ahead. The certificates carry a one-year validity, so the deadline recurs. PinnedRunSpanningTwoBlocksLeasesBoth probed its ports and only then reserved them, while leasing from a private directory that hides concurrent test hosts. Another host could bind or lease the chosen blocks in between, failing the run; and whenever its single candidate was busy the pre-check ignored the test outright, losing the coverage. It now walks up to eight boundaries, skipping Docker-reserved and out-of-range blocks, and ignores only when every candidate is contended, naming what it tried.
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Client teardown can still duplicate completions or leave requests unresolved, and allocator coverage can be skipped without contention.
Review effort: Balanced
Findings: 3
Open (4)
What changed in this PR
Improves Garnet client teardown and test reliability, complementing the certificate renewal in #2202.
Changes:
- Drains outstanding requests during disposal and handles pending
longresults. - Adds certificate validity and 30-day renewal checks.
- Retries candidate port boundaries when contention prevents reservation.
| File | Description |
|---|---|
test/standalone/Garnet.test/TestPortAllocatorFailureTests.cs |
Adds retries to the cross-block lease test. |
test/standalone/Garnet.test/TestCertificateExpiryTests.cs |
Checks certificate validity and renewal lead time. |
libs/client/GarnetClient.cs |
Extends disposal draining and completion handling. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Connection teardown and client disposal drain independently and neither waits for the other, because NetworkHandler.DisposeImpl returns as soon as its guard is already held. Two drains could therefore retire the same slot, completing one request twice and advancing the cursor past another. DrainOutstandingTasks now holds a lock and re-reads the shared cursor each turn instead of copying it into a local, which a completion that re-enters the drain would leave stale. DisposeOffset retires a slot before completing it, as reply processing already does, so a completion that re-enters teardown cannot find the slot still live. Completion is also guarded: a callback is user code, and one that throws previously stranded the requests queued behind it and abandoned the rest of Dispose. AwaitPreviousTaskAsync now ends on disposal. A request can be allocated a task ID and then have its slot retired by a drain before it publishes, which advances the slot past that task so IsNext never comes true again. The TaskType.None branch only yields and never throws, so the existing catch could not end that wait and the caller spun until the process exited. PinnedRunSpanningTwoBlocksLeasesBoth searched a fixed window above the host's own block, which runs off the end when the host sits near the top of the range and falls entirely inside the Docker reservation for some placements, reporting nothing tried on an idle host. It now walks the whole block range, skips its own block, and spends its attempt budget only on eligible boundaries. GarnetClientDisposalTests covers all of this against a listener that accepts and stays silent, so no reply can arrive and every completion comes from teardown. That makes the paths deterministic and mirrors the failure that motivated them: a server that goes mute with a request in flight.
Vasileios Zois (vazois)
approved these changes
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Diagnosing a wedged
Garnet.testrun surfaced several defects that each turn a transient or environmental problem into a hang, or into a silent loss of coverage. None of them is the expired certificate itself — that is fixed by #2202, which this branch merges — but together they are why the expiry took the whole suite down instead of failing one test.A caller could be left awaiting a reply that can never arrive
GarnetClient.DisposeOffsethandled every task type exceptLongAsyncandLongCallback, so a pendingTask<long>was never faulted on teardown.CheckLengthandGarnetClientProcessRepliesboth handlelongTcs; only this path had been missed.Disposealso tore down the socket and the writer without draining the requests still holding a slot. Connection teardown and the client's own disposal are independent — the handler's drain can run whileDisposedis stillfalse— so a request issued in between owned a slot nobody would complete.Disposenow drains too.Because those two paths are gated by separate guards (
disposeCounton the handler,disposedon the client), they can also run concurrently, and two drains over one slot would complete the same request twice and advance the cursor past another. So the drain is serialized, and re-reads the shared cursor each turn rather than copying it into a local that a re-entered drain would leave stale.DisposeOffsetretires its slot before completing it, asProcessRepliesalready does, and completion is guarded so that a throwing callback cannot strand the requests behind it or abandon the rest of teardown.Finally,
AwaitPreviousTaskAsyncnow ends on disposal. A request can be allocated a task ID and have its slot retired by a drain before it publishes, which advances the slot past that task soIsNextnever comes true again; theTaskType.Nonebranch only yields and never throws, so the existingcatchcould not end that wait.Observed originally as an async
Taskthat never completes, with no Garnet frames on any stack: both handlers disposed, and the reply with nowhere to go.An expired test certificate hangs the host instead of failing a test
Under TLS 1.3 the client completes its half of the handshake before the server validates the client certificate, so
ConnectAsyncsucceeds and the first request is the one that dies. The server closes without a RESP error — deliberately, since writing one mid-handshake would be a protocol violation — so the client has nothing to react to. The host wedges until the hang detector kills it and reports a crash, naming no cause.TestCertificateExpiryTestsreports it in milliseconds instead, with the renewal steps in the failure message, and warns 30 days ahead. The certificates carry a one-year validity, so this deadline recurs.A port-allocator test could be skipped by block placement alone
PinnedRunSpanningTwoBlocksLeasesBothprobed its ports and only then reserved them, while leasing from a private directory that deliberately hides concurrent test hosts, so a real host could bind or lease the chosen blocks in between. Its fixed search window above the host's own block could also run off the end of the range, or fall entirely inside the Docker reservation, reporting nothing tried on a completely idle host.It now walks the whole block range with wraparound, skips the host's own block, spends its attempt budget only on eligible boundaries, retries when a reservation loses a race, and reports how many boundaries were actually attempted.
Tests
GarnetClientDisposalTestsruns against a listener that accepts and then stays silent, so no reply can arrive and every completion comes from teardown. That makes these paths deterministic, and mirrors the failure that motivated them — a server that goes mute with a request in flight.DisposeWithRequestsInFlightCompletesEveryCaller, covering theTask<long>path as well.ConcurrentTeardownCompletesEachRequestExactlyOnce, which starts the connection drop and the disposal together on aBarrierand asserts the completion count is exactly the number of requests issued, so a double-completion fails.ThrowingCallbackDoesNotStrandOtherRequests.Each is bounded and reports how many requests were left pending, because the failure being guarded against otherwise exceeds any timeout.
Validation
Garnet.test: 1314 passed, 0 failed, 3 skipped (pre-existing skips), repeated across runs.ownBlock508 and 421, pinned viaGARNET_TEST_PORT_BASE): both now reserve a straddling run instead of ignoring. Also verified it retries when the first candidate's ports are deliberately held, where the previous code silently skipped.