Skip to content

Stop a dropped connection or an expired test certificate from wedging the test host - #2204

Merged
Ted Hart (TedHartMS) merged 4 commits into
mainfrom
tedhar-flaky-tests-100526
Oct 6, 2026
Merged

Ted Hart (TedHartMS) merged 4 commits into
mainfrom
tedhar-flaky-tests-100526

Conversation

@TedHartMS

@TedHartMS Ted Hart (TedHartMS) commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Diagnosing a wedged Garnet.test run surfaced several defects that each turn a transient or environmental problem into a hang, or into a silent loss of coverage. None of them is the expired certificate itself — that is fixed by #2202, which this branch merges — but together they are why the expiry took the whole suite down instead of failing one test.

A caller could be left awaiting a reply that can never arrive

GarnetClient.DisposeOffset handled every task type except LongAsync and LongCallback, so a pending Task<long> was never faulted on teardown. CheckLength and GarnetClientProcessReplies both handle longTcs; only this path had been missed.

Dispose also tore down the socket and the writer without draining the requests still holding a slot. Connection teardown and the client's own disposal are independent — the handler's drain can run while Disposed is still false — so a request issued in between owned a slot nobody would complete. Dispose now drains too.

Because those two paths are gated by separate guards (disposeCount on the handler, disposed on the client), they can also run concurrently, and two drains over one slot would complete the same request twice and advance the cursor past another. So the drain is serialized, and re-reads the shared cursor each turn rather than copying it into a local that a re-entered drain would leave stale. DisposeOffset retires its slot before completing it, as ProcessReplies already does, and completion is guarded so that a throwing callback cannot strand the requests behind it or abandon the rest of teardown.

Finally, AwaitPreviousTaskAsync now ends on disposal. A request can be allocated a task ID and have its slot retired by a drain before it publishes, which advances the slot past that task so IsNext never comes true again; the TaskType.None branch only yields and never throws, so the existing catch could not end that wait.

Observed originally as an async Task that never completes, with no Garnet frames on any stack: both handlers disposed, and the reply with nowhere to go.

An expired test certificate hangs the host instead of failing a test

Under TLS 1.3 the client completes its half of the handshake before the server validates the client certificate, so ConnectAsync succeeds and the first request is the one that dies. The server closes without a RESP error — deliberately, since writing one mid-handshake would be a protocol violation — so the client has nothing to react to. The host wedges until the hang detector kills it and reports a crash, naming no cause.

TestCertificateExpiryTests reports it in milliseconds instead, with the renewal steps in the failure message, and warns 30 days ahead. The certificates carry a one-year validity, so this deadline recurs.

A port-allocator test could be skipped by block placement alone

PinnedRunSpanningTwoBlocksLeasesBoth probed its ports and only then reserved them, while leasing from a private directory that deliberately hides concurrent test hosts, so a real host could bind or lease the chosen blocks in between. Its fixed search window above the host's own block could also run off the end of the range, or fall entirely inside the Docker reservation, reporting nothing tried on a completely idle host.

It now walks the whole block range with wraparound, skips the host's own block, spends its attempt budget only on eligible boundaries, retries when a reservation loses a race, and reports how many boundaries were actually attempted.

Tests

GarnetClientDisposalTests runs against a listener that accepts and then stays silent, so no reply can arrive and every completion comes from teardown. That makes these paths deterministic, and mirrors the failure that motivated them — a server that goes mute with a request in flight.

  • DisposeWithRequestsInFlightCompletesEveryCaller, covering the Task<long> path as well.
  • ConcurrentTeardownCompletesEachRequestExactlyOnce, which starts the connection drop and the disposal together on a Barrier and asserts the completion count is exactly the number of requests issued, so a double-completion fails.
  • ThrowingCallbackDoesNotStrandOtherRequests.

Each is bounded and reports how many requests were left pending, because the failure being guarded against otherwise exceeds any timeout.

Validation

  • Full Garnet.test: 1314 passed, 0 failed, 3 skipped (pre-existing skips), repeated across runs.
  • Expiry guard negative-tested against the expired certificate: fails in 100 ms with the regeneration hint, versus a six-minute wedge.
  • Allocator fix verified at the two host-block placements that previously skipped silently (ownBlock 508 and 421, pinned via GARNET_TEST_PORT_BASE): both now reserve a straddling run instead of ignoring. Also verified it retries when the first candidate's ports are deliberately held, where the previous code silently skipped.

… the test host

Diagnosing a wedged Garnet.test run surfaced three defects that each turn a transient or environmental problem into a hang, or into a silent loss of coverage.

GarnetClient left callers awaiting replies that could never arrive. DisposeOffset handled every task type except LongAsync and LongCallback, so a pending Task<long> was never faulted, and Dispose tore down the socket without draining the requests still holding a slot. Connection teardown and the client's own disposal are independent -- a connection can die while Disposed is still false -- so a request issued in between owned a slot nobody would complete. Dispose now drains, closing that window: a request registered before disposal is drained, and one registered after hits the existing Disposed check. Draining twice is a no-op, because the loop and ConsumeTcsOffset advance together.

An expired TLS test certificate does not fail a test, it hangs the host. Under TLS 1.3 the client completes its half of the handshake before the server validates the client certificate, so ConnectAsync succeeds and the first request is the one that dies, the server having closed without a RESP error because writing one mid-handshake would be a protocol violation. TestCertificateExpiryTests reports that in milliseconds together with the renewal steps, and warns 30 days ahead. The certificates carry a one-year validity, so the deadline recurs.

PinnedRunSpanningTwoBlocksLeasesBoth probed its ports and only then reserved them, while leasing from a private directory that hides concurrent test hosts. Another host could bind or lease the chosen blocks in between, failing the run; and whenever its single candidate was busy the pre-check ignored the test outright, losing the coverage. It now walks up to eight boundaries, skipping Docker-reserved and out-of-range blocks, and ignores only when every candidate is contended, naming what it tried.
Copilot AI balanced review requested due to automatic review settings October 6, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Client teardown can still duplicate completions or leave requests unresolved, and allocator coverage can be skipped without contention.

Review effort: Balanced
Findings: 3 High severity · 1 Medium severity

Open (4)
What changed in this PR

Improves Garnet client teardown and test reliability, complementing the certificate renewal in #2202.

Changes:

  • Drains outstanding requests during disposal and handles pending long results.
  • Adds certificate validity and 30-day renewal checks.
  • Retries candidate port boundaries when contention prevents reservation.
File Description
test/​standalone/​Garnet.test/​TestPortAllocatorFailureTests.cs Adds retries to the cross-block lease test.
test/​standalone/​Garnet.test/​TestCertificateExpiryTests.cs Checks certificate validity and renewal lead time.
libs/​client/​GarnetClient.cs Extends disposal draining and completion handling.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread libs/client/GarnetClient.cs
Comment thread libs/client/GarnetClient.cs
Comment thread libs/client/GarnetClient.cs Outdated
Comment thread test/standalone/Garnet.test/TestPortAllocatorFailureTests.cs Outdated
Connection teardown and client disposal drain independently and neither waits for the other, because NetworkHandler.DisposeImpl returns as soon as its guard is already held. Two drains could therefore retire the same slot, completing one request twice and advancing the cursor past another. DrainOutstandingTasks now holds a lock and re-reads the shared cursor each turn instead of copying it into a local, which a completion that re-enters the drain would leave stale.

DisposeOffset retires a slot before completing it, as reply processing already does, so a completion that re-enters teardown cannot find the slot still live. Completion is also guarded: a callback is user code, and one that throws previously stranded the requests queued behind it and abandoned the rest of Dispose.

AwaitPreviousTaskAsync now ends on disposal. A request can be allocated a task ID and then have its slot retired by a drain before it publishes, which advances the slot past that task so IsNext never comes true again. The TaskType.None branch only yields and never throws, so the existing catch could not end that wait and the caller spun until the process exited.

PinnedRunSpanningTwoBlocksLeasesBoth searched a fixed window above the host's own block, which runs off the end when the host sits near the top of the range and falls entirely inside the Docker reservation for some placements, reporting nothing tried on an idle host. It now walks the whole block range, skips its own block, and spends its attempt budget only on eligible boundaries.

GarnetClientDisposalTests covers all of this against a listener that accepts and stays silent, so no reply can arrive and every completion comes from teardown. That makes the paths deterministic and mirrors the failure that motivated them: a server that goes mute with a request in flight.
@TedHartMS
Ted Hart (TedHartMS) merged commit ab4044f into main Oct 6, 2026
168 of 169 checks passed
@TedHartMS
Ted Hart (TedHartMS) deleted the tedhar-flaky-tests-100526 branch October 6, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants