Repository navigation
Tighten up SetupFixtures and move TestBase to Tsavorite; handle cancellation race condition in maintenance loops - #2162
Merged
Conversation
TestBase.cs is linked into Garnet.test.cluster and Tsavorite.test via <Compile Include>. Calling Garnet.test.TestUtils.EnsurePortSlotResolved() from its global SetUpFixture made that shared file depend on TestUtils.cs, which only Garnet.test and Garnet.test.cluster compile, so Tsavorite.test failed with CS0234 and blocked the seven projects that reference it. Move the call into a [SetUpFixture] in each of the two assemblies that own TestUtils. Both fixtures are in namespace Garnet.test, because GlobalUnhandledExceptionHandling already occupies the global namespace in both assemblies and NUnit permits one SetUpFixture per namespace. Also extend the ci.yml tsavorite paths-filter to cover TestBase.cs. The filter matched only libs/storage/Tsavorite/**, so a change to the linked file skips every Tsavorite job rather than failing it.
Ted Hart (TedHartMS)
requested review from
Badrish Chandramouli (badrishc),
kevin-montrose and
Vasileios Zois (vazois)
as code owners
September 20, 2026 20:25
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes correctly restore project layering, preserve port-slot behavior, and close the CI coverage gap.
Review effort: Balanced
Findings: None
What changed in this PR
Fixes the Tsavorite test build while preserving eager port-slot validation in Garnet test assemblies.
Changes:
- Moves Garnet-specific setup out of shared
TestBase.cs. - Adds namespace-scoped setup fixtures to standalone and cluster tests.
- Expands CI path filtering for linked Tsavorite test sources.
| File | Description |
|---|---|
test/standalone/Garnet.test/TestProjectSetup.cs |
Eagerly resolves standalone test port slots. |
test/standalone/Garnet.test/TestBase.cs |
Removes the Garnet-only dependency from shared code. |
test/cluster/Garnet.test.cluster/TestProjectSetup.cs |
Eagerly resolves cluster test port slots. |
.github/workflows/ci.yml |
Runs Tsavorite CI when linked TestBase.cs changes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Main fixed the Tsavorite.test build break in #2159 by gating the EnsurePortSlotResolved() call behind a GARNET_TEST_UTILS symbol defined by the two projects that compile TestUtils.cs. This branch removes the call from shared TestBase.cs instead, so the guarded block goes away entirely. Keep the GARNET_TEST_UTILS define in both projects: it marks which projects compile TestUtils.cs, which stays useful for source shared with projects that do not. Its comment now describes that rather than pointing at an #if that no longer exists.
Vasileios Zois (vazois)
approved these changes
Sep 21, 2026
Fixes #2160. ArePortsFree probed one of ten TestPortAssignment values plus the cluster node range, so a server stranded on a non-default assignment left the slot reading as free and auto committed to a slot the owning project could not bind. Claim time now enumerates the ports from TestPortAssignment itself, via SlotPorts, so a new assignment is covered without a second edit. Fixes #2165. IsPortFree bound IPAddress.Loopback, answering "is 127.0.0.1:P free?" where the caller asks "is port P free?". Tests bind both loopbacks, every Dns.GetHostAddresses result, and IPAddress.Any, so a listener on any of those was invisible. Both wildcards are now probed, each with ExclusiveAddressUse, which is load-bearing: under default socket options a wildcard bind does not conflict with a specific-address listener, so probing the wildcards without it reports 127.0.0.1 free and loses the one case the loopback probe did catch. The widest gap closed is IPAddress.Any, which the cluster tests bind, making a stranded cluster node the least detectable case. Socket.OSSupportsIPv6 gates the second bind so a host with IPv6 disabled reads free rather than refusing every slot. ListenerIsDetectedOnEveryBoundAddressKind binds port 0 and probes the port the OS assigns, so it covers all four address kinds without touching a test project port and runs under the unset-variable configuration CI uses. Also records the PR protocol in copilot-instructions.md: issues and pull requests are created only on request, and consolidating smaller changes into one pull request is encouraged when its description covers each change.
The background maintenance loops and the gossip loop each guard a cycle with an exception filter that declines once cancellation is requested. When a cycle fails for its own reasons at the same moment shutdown begins, the filter declines and the failure is no longer a cycle failure to anyone: the outer handler recognizes cancellation by exception type, so a non-cancellation exception skips it and is reported as the terminal, task-ending fault. Classify the failure inside the handler instead of in the filter, so a cycle failure is always handled as one and cancellation simply ends the loop quietly. MaintenanceLoopTests gains a case that cancels inside the cycle before throwing, making the window deterministic. It fails against the filter with a Critical entry and passes with the fix.
…pFixture Tsavorite.test link-compiled TestBase.cs out of Garnet.test, so the Tsavorite tree could not be built on its own and an edit to that shared Garnet file broke the Tsavorite build. TestBase now lives in the Tsavorite test tree in namespace Tsavorite.test, and the two Garnet test projects reference Tsavorite.test rather than compiling the file. They resolve the type through a single aliased global using, which imports only that type so Tsavorite's TestUtils cannot collide with Garnet's. TestBase.cs also carried a global-namespace SetUpFixture installing the unhandled-exception handlers. NUnit discovers a SetUpFixture only in the assembly that compiles it, so behind a project reference that handler would no longer run. It is now a method on TestBase called from one TestProjectSetup per test assembly, covering the same three assemblies as before. GARNET_TEST_UTILS is no longer needed by either csproj, and the ci.yml tsavorite path filter no longer needs an entry for a Garnet file, since the Tsavorite tree is now self-contained.
kevin-montrose
approved these changes
Sep 23, 2026
Ted Hart (TedHartMS)
added a commit
that referenced
this pull request
Sep 27, 2026
Merges origin/main d2d71f6 (8 commits). Git reported zero textual conflicts; all 211 files auto-merged. One semantic break required a fix. Main's #2162 and #2173 moved TestBase from Garnet.test to Tsavorite.test (libs/storage/Tsavorite/cs/test/TestBase.cs) and removed the Garnet ProjectReference from the Tsavorite test projects. Seven test files added by this branch imported TestBase via "using Garnet.test;" and stopped compiling with CS0246. That line is removed from each; no replacement using is needed because all seven declare namespaces nested under Tsavorite.test, so TestBase now resolves by enclosing-namespace lookup. test.recovery/V7DownlevelRecoveryTests.cs test.recovery/LogGeometryVerificationTests.cs test.recovery/ObjectRecoveryUndoReflushTests.cs test.recovery/ObjectSizeBoundaryDiskIOTests.cs test.recovery/ObjectSizeBoundaryRecoveryTests.cs test.recovery/SnapshotBoundarySectorTests.cs test.recordops/ObjectReadOnlyFlushTraceTests.cs Note that dotnet build Garnet.slnx reports 0 warnings and 0 errors while this break is present, because Garnet.slnx does not contain the Tsavorite test projects. Verifying a merge requires building both solutions. This merge also brings in 917e0e1 (#2172), which fixes checkpoint abort abandoning in-flight flushes. Verified: build 0 warnings / 0 errors on both solutions; dotnet format --verify-no-changes clean on both; Tsavorite.test.recovery 575 passed; Tsavorite.test.recordops 379 passed; CheckpointFailureTests 10 passed.
x@01 (x-at-01)
added a commit
to webc-fork/garnet
that referenced
this pull request
Oct 4, 2026
…intenance loop cancellation (microsoft#2162), github-actions bump (microsoft#2169), Azure Benchmark Orchestration Tools (microsoft#2132)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2160
Fixes #2165
Test-infrastructure layering and fixture cleanup, plus a pre-existing product race found along the way. Relates to #2161 (closed by #2159, whose
#if GARNET_TEST_UTILSguard this replaces).1. TestBase moves into Tsavorite
Tsavorite.test.csprojlink-compiledtest/standalone/Garnet.test/TestBase.cs. The Tsavorite tree therefore could not be built on its own, and #2151's edit to that shared file brokeBuild Tsavoriteonmain.TestBasenow lives atlibs/storage/Tsavorite/cs/test/TestBase.csinnamespace Tsavorite.test; 85 Tsavorite files dropusing Garnet.test;.Garnet.testandGarnet.test.clustertake aProjectReferenceonTsavorite.testinstead of link-compiling the file.<Using Include="Tsavorite.test.TestBase" Alias="TestBase" />in each of the twotest/*/Directory.Build.props. The alias imports only that type, soTsavorite.test.TestUtilscannot collide withGarnet.test.TestUtils.<Compile Include>or<ProjectReference>underlibs/storage/Tsavorite/escapes the tree any more.2. SetUpFixtures tightened
TestBase.csalso declared a global-namespaceGlobalUnhandledExceptionHandling[SetUpFixture]. NUnit discovers a SetUpFixture only in the assembly that compiles it, so that handler would have been silently lost once the file moved behind a project reference.It is now
TestBase.InstallUnhandledExceptionHandlers(), invoked from oneTestProjectSetupper assembly —Tsavorite.test,Garnet.test,Garnet.test.cluster, exactly the three that had it before. The two Garnet fixtures also keep the eager port-slot resolution #2151 introduced.Two consequences:
GARNET_TEST_UTILSis removed from both csprojs; nothing needs it now.ci.ymltsavoritepath filter no longer needs aTestBase.csentry —libs/storage/Tsavorite/**covers it by construction rather than by a rule someone must remember to keep in sync.3. Port slot probe answered the wrong question twice
ArePortsFreedecides whetherGARNET_TEST_PORT_SLOT=automay claim a slot. Both holes let it report an occupied slot free, soautocommitted to a doomed slot instead of moving to the next one.TestPortAssignmentvalues plus the cluster band. Ports are now enumerated from the enum itself (SlotPorts), so a new assignment is covered without editing the probe.autotest port slot probe only checks loopback, so a stranded non-loopback listener reads as free #2165 — it boundIPAddress.Loopbackonly, so a listener on any other address was invisible. Both wildcards are now probed, withExclusiveAddressUse = true.The exclusive flag is load-bearing rather than incidental.
TcpListener.ExclusiveAddressUsedefaults tofalse, and under default options a wildcard bind does not conflict with a specific-address listener — so probing both wildcards without it reports127.0.0.1free, a strict regression on the one case the loopback probe did catch.AnydefaultAnyexclusiveIPv6AnyexclusiveLoopback(old probe)127.0.0.1::10.0.0.0Row 3 is the widest hole and is not what #2165 was filed about:
ClusterConfigTestsandClusterTestContextbindIPAddress.Any, so a stranded cluster node — the listener most likely to survive a crashed run — holds the port on every interface, and the loopback probe read it as free.Socket.OSSupportsIPv6gates the second bind so a host with IPv6 disabled reads free rather than refusing all seven slots. Onlyautoprobes, so CI pays nothing.4. Cancellation race in the maintenance and gossip loops
Pre-existing on
mainand unrelated to the above; it surfaced here as a one-off CI flake after themainmerge. This is the residue of a bug #2139 already mostly fixed — before that PR each loop wrapped its whole body in a single terminalcatch, so any cycle fault disabled that task for the lifetime of the process. #2139 added the per-cycle guard and is not in question; what remains is the narrow window where the guard itself declines.Both guards were exception filters:
catch (Exception) when (!token.IsCancellationRequested). If cancellation arrives between the cycle throwing and the filter being evaluated, the filter declines, and because the exception is not anOperationCanceledExceptionit skips the cancellation handler as well and is reported as terminal —LogCritical(… "The task won't be resumed.")inStoreWrapper.RunMaintenanceLoopAsync,LogWarning("GossipMain terminated with error …")inClusterManager.GossipMainAsync.Both now classify inside the handler rather than in the filter, so a cycle failure is never handed to outer handlers that recognize cancellation only by exception type.
MaintenanceLoopTests.ACycleFailingAsShutdownBeginsIsNotLoggedAsTerminalcancels inside the cycle and then throws, making the window deterministic; it fails against the old shape with aCriticalentry while the four existing tests pass.GossipMainAsyncgets no equivalent test: it has no seam for "fail exactly as cancellation lands", and the only way to build one fightsExceptionInjectionHelper's shutdown parking suspension, which exists because parking across shutdown hangs the test process.Validation
Tsavorite.slnx,Garnet.slnx(Debug)Tsavorite.test342/368 — identical to in-repoTsavorite.test·Tsavorite.test.epochGarnet.testPortSlot + MaintenanceLoopGarnet.test.clusterClusterConfig + gossip · ClusterNegativeTestsGarnet.test.collectionsGARNET_TEST_PORT_SLOT=99OneTimeSetUpin both assemblies, unwrapping intactdotnet format --verify-no-changes, both solutionsExtraction check:
libs/storage/Tsavorite/copied out with only the repo-root build files it resolves upward (Directory.Build.props,Directory.Packages.props,Version.props,.editorconfig,Garnet.snk) and notest/directory present. Those five are shared infrastructure thatTsavorite.corealready required; theTestBase.cslink was the only source dependency.