Skip to content

Allocate test ports dynamically from a non-ephemeral range - #2193

Merged
Ted Hart (TedHartMS) merged 9 commits into
mainfrom
tedhar-port-selection
Oct 3, 2026
Merged

Ted Hart (TedHartMS) merged 9 commits into
mainfrom
tedhar-port-selection

Conversation

@TedHartMS

Copy link
Copy Markdown
Contributor

Why

Test ports were assigned per sub-project from a fixed base of 33278, optionally shifted by GARNET_TEST_PORT_SLOT. That base sits inside Linux's default outbound-connection range (net.ipv4.ip_local_port_range, 32768-60999), so the kernel could hand a test's port to a connect() in the window between the port being observed free and the server binding it. Probing cannot close that window; only staying out of the range can. Windows and macOS use 49152 as the ephemeral floor and were never affected - which is why these flakes were Linux-only.

A second, independent defect hid port conflicts on Windows: GarnetServerTcp set SO_REUSEADDR unconditionally, which on Windows lets two live servers bind the same port. The OS then split incoming connections between them, so a conflict surfaced as plausible-but-wrong results instead of a bind failure.

What changed

Port allocation. Ports now come from [16384, 32768) - below every supported platform's ephemeral floor, and above the crowded region holding both developer services (6379, 5432, 11211, 8080) and ports the suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process. The OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and ports probed second, because software outside the suite can hold ports while holding no lease.

Blocks are searched from a start derived from the checkout and project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a process-derived stride. The stride is forced odd and BlockCount is a power of two, so the walk visits every block exactly once - completing it proves no block is free, rather than exhausting a retry budget.

Windows bind semantics. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; Unix paths are unchanged. GarnetServerTcpTests previously skipped on Windows because the behavior it asserts did not hold there; it now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts.

Hashing. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, leaving its low bits with almost no entropy. Since the block index is cut from those low bits, the finalizer's right shifts are what make a modulo-power-of-two reduction safe. UtilityTests asserts the avalanche property directly.

Removals. GARNET_TEST_PORT_SLOT is gone and nothing needs setting. Concurrency is no longer capped at seven checkouts. TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration.

Special cases

  • Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file, so moving either side without the other fails as a test.
  • The probe binds the IPv4 and IPv6 wildcards separately and exclusively: one dual-mode socket handles IPv4-mapped addresses differently per platform.
  • A failed probe deliberately ignores the socket error code - Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse.
  • Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary.
  • GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed.

Diagnostics

GarnetServerTcp.Start now reports the endpoint and the OS's reason when a bind fails, neither of which the raw SocketException names. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than a defect in Garnet or the test.

Tests

  • TestPortAllocatorTests - allocation behavior and Docker carve-out sync.
  • TestPortAllocatorFailureTests - blocks held by another host, pinned base ports, full exhaustion. Leases redirect to a private directory so the tests don't block concurrent runs in other checkouts.
  • UtilityTests - avalanche property of GetHashCodeWithMix.
  • GarnetServerTcpTests - now runs on all platforms.

…nched hash

On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared
SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is
permissive: it lets a socket bind an address and port another socket already
holds. Two live servers therefore both bound the same port and the OS split
incoming connections between them, so a port conflict surfaced as plausible but
wrong results rather than as a bind failure. Windows now sets
SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are
unchanged. GarnetServerTcpTests covered this but was skipped on Windows because
the behavior it asserts did not hold there. It now runs on every platform, and
its TIME_WAIT rebind case confirms exclusive use does not block restarts.

Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes.
HashBytes is a multiply-accumulate, and multiplication propagates information
only toward higher bits, so its low bits carry almost no entropy: flipping one
input bit changes some output bits with probability 1.000 rather than 0.5. The
finalizer's right shifts move entropy back down, which makes it safe to reduce
the result modulo a power of two. UtilityTests asserts the avalanche property
directly, since that, rather than the hash value, is why the method exists.
Test ports were fixed per sub-project from a base of 33278, shifted by an
optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and
sits inside Linux's default outbound-connection range (32768-60999,
net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port
to a connect() between the moment the port was observed free and the moment the
server bound it. Probing cannot close that window; only staying out of the range
can. The slot mechanism encoded 49152 as the ephemeral floor, which is the
Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was
already unsafe on Linux. Windows and macOS were never affected, which is why the
flakes were Linux-only.

Ports now come from [16384, 32768), below every supported platform's ephemeral
range and above the crowded region that holds both ports a developer is likely to
be running (6379, 5432, 11211, 8080) and ports this suite itself uses
(10000-10002 Azurite). Each test host leases one aligned 32-port block, held by
an exclusively locked file for the life of the process, which is what keeps
concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a
block frees itself. The lease is taken first and the ports probed second, because
software outside the suite, or a server stranded by a killed run, holds ports
while holding no lease.

Blocks are searched from a start derived from the checkout and the project name,
so a project returns to the same ports run after run and netstat output stays
interpretable. On collision the search advances by a stride derived from the
process, so two hosts that start at the same block diverge rather than walking in
lockstep. The stride is forced odd and the block count is a power of two, which
together make the walk visit every block exactly once: completing it proves no
block is free rather than exhausting a retry budget. The hash is
Utility.GetHashCodeWithMix, because the index is cut from the low bits that
HashBytes alone leaves unmixed.

This replaces slots entirely. Concurrency is no longer capped at seven
checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and
TestPortAssignment, ClusterPortAssignment, and the band-width invariants they
required are deleted - projects are identified by assembly name, so adding one
needs no central registration.

Special cases, each commented where it lives:
- Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py,
  which allocates from its own fixed base and can run alongside this suite.
  TestPortAllocatorTests reads that base out of the Python file so moving either
  side without the other fails as a test.
- The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One
  dual-mode socket handles IPv4-mapped addresses differently per platform, and a
  shared bind succeeds alongside a listener on a specific address under that
  wildcard.
- A failed probe deliberately ignores the socket error code: Windows reports
  AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not
  AddressAlreadyInUse.
- Cluster nodes keep contiguous ports because node n binds base + n; block
  alignment keeps that run from straddling a boundary.
- GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall
  opening. Pinned ports are still leased and probed.

GarnetServerTcp.Start now reports the endpoint and the operating system's reason
when a bind fails, which the raw SocketException names neither of.
TestUtils.OnTearDown adds the allocation context, so a conflict reads as software
outside the suite rather than as a defect in Garnet or in the test.
The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer
exists; test hosts now claim their ports without configuration.
Reserve's error paths decide whether a run stops clearly or proceeds on ports it
does not own, and none of them ran in an ordinary suite: a block another host
holds, a pinned base port, and every block exhausted. A regression in any of
them would surface as the silent cross-checkout corruption the allocator exists
to prevent.

Leases in these tests are redirected to a private directory. Holding blocks in
the shared one would stop concurrent runs in other checkouts from starting,
which is the very failure under test.

Writing them found a real defect in the first draft: two tests took the same
port from a deterministic helper, and because a reservation holds its lease for
the life of the process while binding nothing, the second saw ports that read as
free inside a block that was already leased. The helper now checks both, which
is the same distinction the allocator draws.

BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the
decision and the wording can be asserted without making a real test fail.
…ests

The fixture built its second listener from TestUtils.TestPort + 1. That is the
port TestUtils.AlternateTestPort returns, and it is inside the block this test
host leases, so the behavior is unchanged - but the arithmetic only works
because the reservation covers two ports, which the expression does not say.
Naming the property states the dependency the fixture actually has.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Pinned reservations can overlap across lease blocks, and bind-failure diagnostics misreport cluster, pinned, and non-conflict failures.

Review effort: Balanced
Findings: 1 High severity · 3 Medium severity

Open (4)
What changed in this PR

Introduces dynamic, non-ephemeral test-port allocation to prevent cross-process collisions and Linux ephemeral-port flakes.

Changes:

  • Adds leased 32-port block allocation and migrates standalone/cluster setup.
  • Enforces exclusive Windows binds and improves bind diagnostics.
  • Adds hash mixing, allocator tests, and updated developer guidance.
File Description
test/​standalone/​Garnet.test/​TestUtils.cs Integrates dynamic ports and failure reporting.
test/​standalone/​Garnet.test/​TestProjectSetup.cs Reserves ports during assembly setup.
test/​standalone/​Garnet.test/​TestPortAllocatorTests.cs Tests allocator invariants and probing.
test/​standalone/​Garnet.test/​TestPortAllocatorFailureTests.cs Tests contention, exhaustion, and pinning.
test/​standalone/​Garnet.test/​TestPortAllocator.cs Implements block leasing and allocation.
test/​standalone/​Garnet.test/​PortSlotTests.cs Removes obsolete slot tests.
test/​standalone/​Garnet.test/​NetworkTests.cs Uses the reserved alternate port.
test/​standalone/​Garnet.test/​GarnetServerTcpTests.cs Extends bind tests across platforms.
test/​standalone/​Garnet.test/​ConnectionLimitTests.cs Uses the reserved alternate port.
test/​standalone/​Garnet.test.vectorset/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​standalone/​Garnet.test.scripting/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​standalone/​Garnet.test.rangeindex/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​standalone/​Garnet.test.extensions/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​standalone/​Garnet.test.complexstring/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​standalone/​Garnet.test.collections/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​standalone/​Garnet.test.acl/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​standalone/​BfTreeInterop.test/​TestProjectSetup.cs Migrates setup to dynamic reservation.
test/​cluster/​Garnet.test.cluster/​TestProjectSetup.cs Reserves the cluster port run.
test/​cluster/​Garnet.test.cluster/​Garnet.test.cluster.csproj Links the shared allocator source.
test/​cluster/​Garnet.test.cluster/​ClusterTestContext.cs Exposes dynamically reserved cluster ports.
test/​cluster/​Garnet.test.cluster/​ClusterPortBandTests.cs Removes obsolete fixed-band tests.
test/​cluster/​Garnet.test.cluster.vectorsets/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.replication/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.replication.vectorsets/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.replication.tls/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.replication.rangeindex/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.replication.disklesssync/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.replication.asyncreplay/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.multilog/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.multilog.diskless/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.migrate/​TestProjectSetup.cs Migrates cluster setup.
test/​cluster/​Garnet.test.cluster.migrate.rangeindex/​TestProjectSetup.cs Migrates cluster setup.
libs/​storage/​Tsavorite/​cs/​test/​UtilityTests.cs Tests hash avalanche and distribution.
libs/​storage/​Tsavorite/​cs/​src/​core/​Utilities/​Utility.cs Adds full-width hash mixing.
libs/​server/​Servers/​GarnetServerTcp.cs Adds Windows-exclusive binds and diagnostics.
.github/​skills/​add-garnet-command/​SKILL.md Removes obsolete port-slot setup guidance.
.github/​copilot-instructions.md Documents automatic test-port allocation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/standalone/Garnet.test/TestPortAllocator.cs Outdated
Comment thread test/standalone/Garnet.test/TestUtils.cs
Comment thread test/standalone/Garnet.test/TestUtils.cs Outdated
Comment thread test/standalone/Garnet.test/TestUtils.cs Outdated
@TedHartMS

Copy link
Copy Markdown
Contributor Author

Validation

Beyond CI, this branch was run at 76da5beb2 on a dedicated Ubuntu 24.04 host (16 cores) and on Windows, across both frameworks and both configurations:

net8.0 net10.0
Debug Linux: 7566 tests, 0 failed Windows + WSL2: 0 failed
Release covered by CI (all cells green) Linux: 7541 tests, 0 failed

Release matters here specifically because Debug.Assert is compiled out, so the ScratchBuffer and epoch invariants are unguarded and cluster timing windows shift; asyncreplay, replication.tls, and multilog all passed at 105/105, 105/105, and 106/106.

The port allocator itself was exercised under the condition it exists for: seven test projects started in parallel from one checkout took seven distinct blocks, each lease naming its project and PID, with listeners spanning 16544-32096 — all inside the allocatable range.

block-0005: Garnet.test.complexstring   block-0354: Garnet.test.scripting
block-0063: Garnet.test.acl             block-0474: Garnet.test.rangeindex
block-0124: Garnet.test.extensions      block-0491: Garnet.test.vectorset
block-0249: Garnet.test.collections

Both behavioral changes were verified by mutation rather than by assertion alone:

  • Reverting the SO_EXCLUSIVEADDRUSE change makes StartRejectsAnAlreadyListeningEndpoint fail on both IPv4 and IPv6 — i.e. two Garnet servers really did both bind one port on Windows before this.
  • Deleting the fmix64 finalizer makes MixAvalanchesEveryOutputBit report an input bit changing an output bit with probability 1.000 instead of ~0.5.

Not covered

  • macOS and FreeBSD: the SO_REUSEPORT branch is untouched by this change, but no run exercised it.
  • One unrelated pre-existing bug surfaced during Linux validation: NativeStorageDevice.cs:1283 enforces the Windows MAX_PATH - 11 limit on Linux as well, where PATH_MAX is 4096. A checkout path of 250 characters fails 9 recovery tests. Being addressed separately; out of scope here.

Review of the allocator found four defects, all in paths that only run when
something has already gone wrong.

ReservePinned leased only the block containing its first port. A pinned base is
deliberately not block-aligned, so a run can cross into the next block: base
16415 with two ports leases block 0 while port 16416 sits unleased in block 1.
Another host could lease that block and be handed an overlapping run, both
probes passing because neither side had bound yet. Every intersected block is
now leased. IntersectedBlocks also skips ports outside the allocatable range,
where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for
port 40000, neither of which names a block.

The teardown report fired for every bind failure but always diagnosed a port
conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is
a platform-reserved range, and AddressNotAvailable, which is an address absent
from the machine; neither is another process taking the port. The report now
parses the socket error and stays silent unless it is AddressAlreadyInUse.

The report read the standalone port field, which ClusterTestContext.ReservePorts
never sets, so every cluster bind failure claimed the host had reserved no ports
while it held an eight-port run. The allocator now records each reservation and
the report looks up the run covering the failing port, which also lets it name
the project that holds it.

The report asserted that a reserved port cannot have been taken by the OS. That
is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run
above it, where the kernel really can assign the port to an outbound connection
between the probe and the bind - the one collision pinning opts into. Pinned
runs outside the range are now described as exactly that, and an endpoint the
test built itself is reported as carrying no allocator guarantee at all.

Each fix is covered by a test verified by reverting the fix: restoring the
single-block lease reports the tail of a straddling run as still available, and
removing the error-code gate reports an AccessDenied failure as software outside
the suite.
@TedHartMS
Ted Hart (TedHartMS) merged commit fc97f95 into main Oct 3, 2026
448 of 449 checks passed
@TedHartMS
Ted Hart (TedHartMS) deleted the tedhar-port-selection branch October 3, 2026 02:54
Ted Hart (TedHartMS) added a commit that referenced this pull request Oct 3, 2026
Conflicts were in the scan-iterator page-read path and the test harness.

main #2181 widened the page-read API: int readPage -> long, int
devicePageOffset -> long, and out CountdownEvent -> ref (the frame's event
is now reused rather than replaced). This branch had added a
CircularDiskReadBuffer parameter to the same signatures, so every
declaration and override conflicted; resolved by keeping the added
parameter and taking main's widening and ref semantics.

That widening also reverted a narrowing this branch had made:
PageAsyncReadResult.page, GetPageIndexForPage, GetLogicalAddressOfStartOfPage,
GetFileOffsetOfPage and BufferAndLoad were int-based here but long at the
merge base and in main. Restored to long -- main widened them deliberately,
and a page index must not be bounded by int for large logs.

main #2193 replaced this branch's GARNET_TEST_PORT_SLOT scheme with
TestPortAllocator, which reserves ports dynamically from a non-ephemeral
range. Took main's mechanism wholesale and kept this branch's
V7CheckpointFixture compile item alongside main's new TestPortAllocator one.
Remaining conflicts were additive: CreateGarnetServer parameters and a
configuration.md section, where both sides' additions were kept.
x@01 (x-at-01) added a commit to webc-fork/garnet that referenced this pull request Oct 4, 2026
…n missing key (microsoft#2192), ZADD XX INCR null (microsoft#2197), no empty object from object RMW (microsoft#2194), fresh ObjectOutput per key HCOLLECT/ZCOLLECT (microsoft#2200), CompletionEvent disposal + LogSizeTracker coalesce (microsoft#2198), dynamic test ports (microsoft#2193)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants