Repository navigation
Allocate test ports dynamically from a non-ephemeral range - #2193
Conversation
…nched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists.
Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test.
The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration.
Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail.
…ests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Pinned reservations can overlap across lease blocks, and bind-failure diagnostics misreport cluster, pinned, and non-conflict failures.
Review effort: Balanced
Findings: 1
Open (4)
What changed in this PR
Introduces dynamic, non-ephemeral test-port allocation to prevent cross-process collisions and Linux ephemeral-port flakes.
Changes:
- Adds leased 32-port block allocation and migrates standalone/cluster setup.
- Enforces exclusive Windows binds and improves bind diagnostics.
- Adds hash mixing, allocator tests, and updated developer guidance.
| File | Description |
|---|---|
test/standalone/Garnet.test/TestUtils.cs |
Integrates dynamic ports and failure reporting. |
test/standalone/Garnet.test/TestProjectSetup.cs |
Reserves ports during assembly setup. |
test/standalone/Garnet.test/TestPortAllocatorTests.cs |
Tests allocator invariants and probing. |
test/standalone/Garnet.test/TestPortAllocatorFailureTests.cs |
Tests contention, exhaustion, and pinning. |
test/standalone/Garnet.test/TestPortAllocator.cs |
Implements block leasing and allocation. |
test/standalone/Garnet.test/PortSlotTests.cs |
Removes obsolete slot tests. |
test/standalone/Garnet.test/NetworkTests.cs |
Uses the reserved alternate port. |
test/standalone/Garnet.test/GarnetServerTcpTests.cs |
Extends bind tests across platforms. |
test/standalone/Garnet.test/ConnectionLimitTests.cs |
Uses the reserved alternate port. |
test/standalone/Garnet.test.vectorset/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/standalone/Garnet.test.scripting/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/standalone/Garnet.test.rangeindex/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/standalone/Garnet.test.extensions/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/standalone/Garnet.test.complexstring/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/standalone/Garnet.test.collections/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/standalone/Garnet.test.acl/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/standalone/BfTreeInterop.test/TestProjectSetup.cs |
Migrates setup to dynamic reservation. |
test/cluster/Garnet.test.cluster/TestProjectSetup.cs |
Reserves the cluster port run. |
test/cluster/Garnet.test.cluster/Garnet.test.cluster.csproj |
Links the shared allocator source. |
test/cluster/Garnet.test.cluster/ClusterTestContext.cs |
Exposes dynamically reserved cluster ports. |
test/cluster/Garnet.test.cluster/ClusterPortBandTests.cs |
Removes obsolete fixed-band tests. |
test/cluster/Garnet.test.cluster.vectorsets/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.replication/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.replication.vectorsets/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.replication.tls/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.replication.rangeindex/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.replication.disklesssync/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.replication.asyncreplay/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.multilog/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.multilog.diskless/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.migrate/TestProjectSetup.cs |
Migrates cluster setup. |
test/cluster/Garnet.test.cluster.migrate.rangeindex/TestProjectSetup.cs |
Migrates cluster setup. |
libs/storage/Tsavorite/cs/test/UtilityTests.cs |
Tests hash avalanche and distribution. |
libs/storage/Tsavorite/cs/src/core/Utilities/Utility.cs |
Adds full-width hash mixing. |
libs/server/Servers/GarnetServerTcp.cs |
Adds Windows-exclusive binds and diagnostics. |
.github/skills/add-garnet-command/SKILL.md |
Removes obsolete port-slot setup guidance. |
.github/copilot-instructions.md |
Documents automatic test-port allocation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
ValidationBeyond CI, this branch was run at
Release matters here specifically because The port allocator itself was exercised under the condition it exists for: seven test projects started in parallel from one checkout took seven distinct blocks, each lease naming its project and PID, with listeners spanning 16544-32096 — all inside the allocatable range. Both behavioral changes were verified by mutation rather than by assertion alone:
Not covered
|
Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite.
Conflicts were in the scan-iterator page-read path and the test harness. main #2181 widened the page-read API: int readPage -> long, int devicePageOffset -> long, and out CountdownEvent -> ref (the frame's event is now reused rather than replaced). This branch had added a CircularDiskReadBuffer parameter to the same signatures, so every declaration and override conflicted; resolved by keeping the added parameter and taking main's widening and ref semantics. That widening also reverted a narrowing this branch had made: PageAsyncReadResult.page, GetPageIndexForPage, GetLogicalAddressOfStartOfPage, GetFileOffsetOfPage and BufferAndLoad were int-based here but long at the merge base and in main. Restored to long -- main widened them deliberately, and a page index must not be bounded by int for large logs. main #2193 replaced this branch's GARNET_TEST_PORT_SLOT scheme with TestPortAllocator, which reserves ports dynamically from a non-ephemeral range. Took main's mechanism wholesale and kept this branch's V7CheckpointFixture compile item alongside main's new TestPortAllocator one. Remaining conflicts were additive: CreateGarnetServer parameters and a configuration.md section, where both sides' additions were kept.
…n missing key (microsoft#2192), ZADD XX INCR null (microsoft#2197), no empty object from object RMW (microsoft#2194), fresh ObjectOutput per key HCOLLECT/ZCOLLECT (microsoft#2200), CompletionEvent disposal + LogSizeTracker coalesce (microsoft#2198), dynamic test ports (microsoft#2193)


Why
Test ports were assigned per sub-project from a fixed base of 33278, optionally shifted by
GARNET_TEST_PORT_SLOT. That base sits inside Linux's default outbound-connection range (net.ipv4.ip_local_port_range, 32768-60999), so the kernel could hand a test's port to aconnect()in the window between the port being observed free and the server binding it. Probing cannot close that window; only staying out of the range can. Windows and macOS use 49152 as the ephemeral floor and were never affected - which is why these flakes were Linux-only.A second, independent defect hid port conflicts on Windows:
GarnetServerTcpsetSO_REUSEADDRunconditionally, which on Windows lets two live servers bind the same port. The OS then split incoming connections between them, so a conflict surfaced as plausible-but-wrong results instead of a bind failure.What changed
Port allocation. Ports now come from
[16384, 32768)- below every supported platform's ephemeral floor, and above the crowded region holding both developer services (6379, 5432, 11211, 8080) and ports the suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process. The OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and ports probed second, because software outside the suite can hold ports while holding no lease.Blocks are searched from a start derived from the checkout and project name, so a project returns to the same ports run after run and
netstatoutput stays interpretable. On collision the search advances by a process-derived stride. The stride is forced odd andBlockCountis a power of two, so the walk visits every block exactly once - completing it proves no block is free, rather than exhausting a retry budget.Windows bind semantics. Windows now sets
SO_EXCLUSIVEADDRUSE, which refuses the second bind; Unix paths are unchanged.GarnetServerTcpTestspreviously skipped on Windows because the behavior it asserts did not hold there; it now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts.Hashing.
Utility.GetHashCodeWithMixapplies a MurmurHash3fmix64finalizer toHashBytes.HashBytesis a multiply-accumulate, and multiplication propagates information only toward higher bits, leaving its low bits with almost no entropy. Since the block index is cut from those low bits, the finalizer's right shifts are what make a modulo-power-of-two reduction safe.UtilityTestsasserts the avalanche property directly.Removals.
GARNET_TEST_PORT_SLOTis gone and nothing needs setting. Concurrency is no longer capped at seven checkouts.TestPortAssignment,ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration.Special cases
test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite.TestPortAllocatorTestsreads that base out of the Python file, so moving either side without the other fails as a test.AccessDeniedfor ports excluded by Hyper-V, WSL, or Docker, notAddressAlreadyInUse.base + n; block alignment keeps that run from straddling a boundary.GARNET_TEST_PORT_BASEpins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed.Diagnostics
GarnetServerTcp.Startnow reports the endpoint and the OS's reason when a bind fails, neither of which the rawSocketExceptionnames.TestUtils.OnTearDownadds the allocation context, so a conflict reads as software outside the suite rather than a defect in Garnet or the test.Tests
TestPortAllocatorTests- allocation behavior and Docker carve-out sync.TestPortAllocatorFailureTests- blocks held by another host, pinned base ports, full exhaustion. Leases redirect to a private directory so the tests don't block concurrent runs in other checkouts.UtilityTests- avalanche property ofGetHashCodeWithMix.GarnetServerTcpTests- now runs on all platforms.