Skip to content

Tighten up SetupFixtures and move TestBase to Tsavorite; handle cancellation race condition in maintenance loops - #2162

Merged
Ted Hart (TedHartMS) merged 6 commits into
mainfrom
tedhar-fix-tsavorite-test-build
Sep 23, 2026
Merged

Ted Hart (TedHartMS) merged 6 commits into
mainfrom
tedhar-fix-tsavorite-test-build

Conversation

@TedHartMS

@TedHartMS Ted Hart (TedHartMS) commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2160
Fixes #2165

Test-infrastructure layering and fixture cleanup, plus a pre-existing product race found along the way. Relates to #2161 (closed by #2159, whose #if GARNET_TEST_UTILS guard this replaces).

1. TestBase moves into Tsavorite

Tsavorite.test.csproj link-compiled test/standalone/Garnet.test/TestBase.cs. The Tsavorite tree therefore could not be built on its own, and #2151's edit to that shared file broke Build Tsavorite on main.

  • TestBase now lives at libs/storage/Tsavorite/cs/test/TestBase.cs in namespace Tsavorite.test; 85 Tsavorite files drop using Garnet.test;.
  • Garnet.test and Garnet.test.cluster take a ProjectReference on Tsavorite.test instead of link-compiling the file.
  • Garnet test projects resolve the type through one <Using Include="Tsavorite.test.TestBase" Alias="TestBase" /> in each of the two test/*/Directory.Build.props. The alias imports only that type, so Tsavorite.test.TestUtils cannot collide with Garnet.test.TestUtils.
  • No <Compile Include> or <ProjectReference> under libs/storage/Tsavorite/ escapes the tree any more.

2. SetUpFixtures tightened

TestBase.cs also declared a global-namespace GlobalUnhandledExceptionHandling [SetUpFixture]. NUnit discovers a SetUpFixture only in the assembly that compiles it, so that handler would have been silently lost once the file moved behind a project reference.

It is now TestBase.InstallUnhandledExceptionHandlers(), invoked from one TestProjectSetup per assembly — Tsavorite.test, Garnet.test, Garnet.test.cluster, exactly the three that had it before. The two Garnet fixtures also keep the eager port-slot resolution #2151 introduced.

Two consequences:

  • GARNET_TEST_UTILS is removed from both csprojs; nothing needs it now.
  • The ci.yml tsavorite path filter no longer needs a TestBase.cs entry — libs/storage/Tsavorite/** covers it by construction rather than by a rule someone must remember to keep in sync.

3. Port slot probe answered the wrong question twice

ArePortsFree decides whether GARNET_TEST_PORT_SLOT=auto may claim a slot. Both holes let it report an occupied slot free, so auto committed to a doomed slot instead of moving to the next one.

The exclusive flag is load-bearing rather than incidental. TcpListener.ExclusiveAddressUse defaults to false, and under default options a wildcard bind does not conflict with a specific-address listener — so probing both wildcards without it reports 127.0.0.1 free, a strict regression on the one case the loopback probe did catch.

holder Any default Any exclusive IPv6Any exclusive Loopback (old probe)
127.0.0.1 FREE INUSE FREE INUSE
::1 FREE FREE INUSE FREE
0.0.0.0 INUSE INUSE FREE FREE
non-loopback IPv4 FREE INUSE FREE FREE

Row 3 is the widest hole and is not what #2165 was filed about: ClusterConfigTests and ClusterTestContext bind IPAddress.Any, so a stranded cluster node — the listener most likely to survive a crashed run — holds the port on every interface, and the loopback probe read it as free. Socket.OSSupportsIPv6 gates the second bind so a host with IPv6 disabled reads free rather than refusing all seven slots. Only auto probes, so CI pays nothing.

4. Cancellation race in the maintenance and gossip loops

Pre-existing on main and unrelated to the above; it surfaced here as a one-off CI flake after the main merge. This is the residue of a bug #2139 already mostly fixed — before that PR each loop wrapped its whole body in a single terminal catch, so any cycle fault disabled that task for the lifetime of the process. #2139 added the per-cycle guard and is not in question; what remains is the narrow window where the guard itself declines.

Both guards were exception filters: catch (Exception) when (!token.IsCancellationRequested). If cancellation arrives between the cycle throwing and the filter being evaluated, the filter declines, and because the exception is not an OperationCanceledException it skips the cancellation handler as well and is reported as terminal — LogCritical(… "The task won't be resumed.") in StoreWrapper.RunMaintenanceLoopAsync, LogWarning("GossipMain terminated with error …") in ClusterManager.GossipMainAsync.

Both now classify inside the handler rather than in the filter, so a cycle failure is never handed to outer handlers that recognize cancellation only by exception type.

MaintenanceLoopTests.ACycleFailingAsShutdownBeginsIsNotLoggedAsTerminal cancels inside the cycle and then throws, making the window deterministic; it fails against the old shape with a Critical entry while the four existing tests pass. GossipMainAsync gets no equivalent test: it has no seam for "fail exactly as cancellation lands", and the only way to build one fights ExceptionInjectionHelper's shutdown parking suspension, which exists because parking across shutdown hangs the test process.

Validation

Tsavorite.slnx, Garnet.slnx (Debug) 0 warnings, 0 errors
Tsavorite tree extracted and built standalone builds clean; Tsavorite.test 342/368 — identical to in-repo
Tsavorite.test · Tsavorite.test.epoch 342/368 · 18/18
Garnet.test PortSlot + MaintenanceLoop 30 passed, 1 skipped
Garnet.test.cluster ClusterConfig + gossip · ClusterNegativeTests 17/17 · 62/62
Garnet.test.collections 119/119
GARNET_TEST_PORT_SLOT=99 still fails at OneTimeSetUp in both assemblies, unwrapping intact
dotnet format --verify-no-changes, both solutions clean

Extraction check: libs/storage/Tsavorite/ copied out with only the repo-root build files it resolves upward (Directory.Build.props, Directory.Packages.props, Version.props, .editorconfig, Garnet.snk) and no test/ directory present. Those five are shared infrastructure that Tsavorite.core already required; the TestBase.cs link was the only source dependency.

TestBase.cs is linked into Garnet.test.cluster and Tsavorite.test via
<Compile Include>. Calling Garnet.test.TestUtils.EnsurePortSlotResolved()
from its global SetUpFixture made that shared file depend on TestUtils.cs,
which only Garnet.test and Garnet.test.cluster compile, so Tsavorite.test
failed with CS0234 and blocked the seven projects that reference it.

Move the call into a [SetUpFixture] in each of the two assemblies that own
TestUtils. Both fixtures are in namespace Garnet.test, because
GlobalUnhandledExceptionHandling already occupies the global namespace in
both assemblies and NUnit permits one SetUpFixture per namespace.

Also extend the ci.yml tsavorite paths-filter to cover TestBase.cs. The
filter matched only libs/storage/Tsavorite/**, so a change to the linked
file skips every Tsavorite job rather than failing it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changes correctly restore project layering, preserve port-slot behavior, and close the CI coverage gap.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes the Tsavorite test build while preserving eager port-slot validation in Garnet test assemblies.

Changes:

  • Moves Garnet-specific setup out of shared TestBase.cs.
  • Adds namespace-scoped setup fixtures to standalone and cluster tests.
  • Expands CI path filtering for linked Tsavorite test sources.
File Description
test/​standalone/​Garnet.test/​TestProjectSetup.cs Eagerly resolves standalone test port slots.
test/​standalone/​Garnet.test/​TestBase.cs Removes the Garnet-only dependency from shared code.
test/​cluster/​Garnet.test.cluster/​TestProjectSetup.cs Eagerly resolves cluster test port slots.
.github/​workflows/​ci.yml Runs Tsavorite CI when linked TestBase.cs changes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Main fixed the Tsavorite.test build break in #2159 by gating the
EnsurePortSlotResolved() call behind a GARNET_TEST_UTILS symbol defined by
the two projects that compile TestUtils.cs. This branch removes the call
from shared TestBase.cs instead, so the guarded block goes away entirely.

Keep the GARNET_TEST_UTILS define in both projects: it marks which projects
compile TestUtils.cs, which stays useful for source shared with projects
that do not. Its comment now describes that rather than pointing at an #if
that no longer exists.
@TedHartMS Ted Hart (TedHartMS) changed the title Fix broken Tsavorite.test build: move EnsurePortSlotResolved out of shared TestBase.cs Replace the GARNET_TEST_UTILS guard with per-project SetUpFixtures, and fix the CI filter gap that hid the Tsavorite break Sep 21, 2026
Fixes #2160. ArePortsFree probed one of ten TestPortAssignment values plus
the cluster node range, so a server stranded on a non-default assignment left
the slot reading as free and auto committed to a slot the owning project could
not bind. Claim time now enumerates the ports from TestPortAssignment itself,
via SlotPorts, so a new assignment is covered without a second edit.

Fixes #2165. IsPortFree bound IPAddress.Loopback, answering "is 127.0.0.1:P
free?" where the caller asks "is port P free?". Tests bind both loopbacks,
every Dns.GetHostAddresses result, and IPAddress.Any, so a listener on any of
those was invisible. Both wildcards are now probed, each with
ExclusiveAddressUse, which is load-bearing: under default socket options a
wildcard bind does not conflict with a specific-address listener, so probing
the wildcards without it reports 127.0.0.1 free and loses the one case the
loopback probe did catch. The widest gap closed is IPAddress.Any, which the
cluster tests bind, making a stranded cluster node the least detectable case.
Socket.OSSupportsIPv6 gates the second bind so a host with IPv6 disabled reads
free rather than refusing every slot.

ListenerIsDetectedOnEveryBoundAddressKind binds port 0 and probes the port the
OS assigns, so it covers all four address kinds without touching a test
project port and runs under the unset-variable configuration CI uses.

Also records the PR protocol in copilot-instructions.md: issues and pull
requests are created only on request, and consolidating smaller changes into
one pull request is encouraged when its description covers each change.
The background maintenance loops and the gossip loop each guard a cycle with
an exception filter that declines once cancellation is requested. When a cycle
fails for its own reasons at the same moment shutdown begins, the filter
declines and the failure is no longer a cycle failure to anyone: the outer
handler recognizes cancellation by exception type, so a non-cancellation
exception skips it and is reported as the terminal, task-ending fault.

Classify the failure inside the handler instead of in the filter, so a cycle
failure is always handled as one and cancellation simply ends the loop quietly.

MaintenanceLoopTests gains a case that cancels inside the cycle before throwing,
making the window deterministic. It fails against the filter with a Critical
entry and passes with the fix.
@TedHartMS Ted Hart (TedHartMS) changed the title Replace the GARNET_TEST_UTILS guard with per-project SetUpFixtures, and fix the CI filter gap that hid the Tsavorite break Tighten up SetupFixtures and move TestBase to Tsavorite; handle cancellation race condition in maintenance loops Sep 23, 2026
…pFixture

Tsavorite.test link-compiled TestBase.cs out of Garnet.test, so the Tsavorite
tree could not be built on its own and an edit to that shared Garnet file broke
the Tsavorite build.

TestBase now lives in the Tsavorite test tree in namespace Tsavorite.test, and
the two Garnet test projects reference Tsavorite.test rather than compiling the
file. They resolve the type through a single aliased global using, which imports
only that type so Tsavorite's TestUtils cannot collide with Garnet's.

TestBase.cs also carried a global-namespace SetUpFixture installing the
unhandled-exception handlers. NUnit discovers a SetUpFixture only in the
assembly that compiles it, so behind a project reference that handler would
no longer run. It is now a method on TestBase called from one TestProjectSetup
per test assembly, covering the same three assemblies as before.

GARNET_TEST_UTILS is no longer needed by either csproj, and the ci.yml tsavorite
path filter no longer needs an entry for a Garnet file, since the Tsavorite tree
is now self-contained.
@TedHartMS
Ted Hart (TedHartMS) merged commit 70958de into main Sep 23, 2026
447 of 449 checks passed
@TedHartMS
Ted Hart (TedHartMS) deleted the tedhar-fix-tsavorite-test-build branch September 23, 2026 18:23
Ted Hart (TedHartMS) added a commit that referenced this pull request Sep 27, 2026
Merges origin/main d2d71f6 (8 commits). Git reported zero textual
conflicts; all 211 files auto-merged.

One semantic break required a fix. Main's #2162 and #2173 moved TestBase
from Garnet.test to Tsavorite.test (libs/storage/Tsavorite/cs/test/TestBase.cs)
and removed the Garnet ProjectReference from the Tsavorite test projects.
Seven test files added by this branch imported TestBase via
"using Garnet.test;" and stopped compiling with CS0246. That line is removed
from each; no replacement using is needed because all seven declare
namespaces nested under Tsavorite.test, so TestBase now resolves by
enclosing-namespace lookup.

  test.recovery/V7DownlevelRecoveryTests.cs
  test.recovery/LogGeometryVerificationTests.cs
  test.recovery/ObjectRecoveryUndoReflushTests.cs
  test.recovery/ObjectSizeBoundaryDiskIOTests.cs
  test.recovery/ObjectSizeBoundaryRecoveryTests.cs
  test.recovery/SnapshotBoundarySectorTests.cs
  test.recordops/ObjectReadOnlyFlushTraceTests.cs

Note that dotnet build Garnet.slnx reports 0 warnings and 0 errors while
this break is present, because Garnet.slnx does not contain the Tsavorite
test projects. Verifying a merge requires building both solutions.

This merge also brings in 917e0e1 (#2172), which fixes checkpoint abort
abandoning in-flight flushes.

Verified: build 0 warnings / 0 errors on both solutions; dotnet format
--verify-no-changes clean on both; Tsavorite.test.recovery 575 passed;
Tsavorite.test.recordops 379 passed; CheckpointFailureTests 10 passed.
x@01 (x-at-01) added a commit to webc-fork/garnet that referenced this pull request Oct 4, 2026
…intenance loop cancellation (microsoft#2162), github-actions bump (microsoft#2169), Azure Benchmark Orchestration Tools (microsoft#2132)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants