Bound per-connection memory: adaptive network buffer budget, session buffer caps, and connection admission - #2157

Adds the three --network-buffer-* options to the configuration table,
warns instead of silently correcting an inverted size pair, and covers
defaults, overrides, power-of-2 rounding, the inverted pair and invalid
values in GarnetServerConfigTests.

Co-authored-by: Copilot <[email protected]>

A RESP session sizes SessionParseState.rootBuffer to the largest array
arity the client has ever sent and keeps it for the life of the
connection. One high-arity command therefore enlarges the session
permanently, and the cost scales with connection count. The same ratchet
exists in ScratchBufferBuilder, whose Reset() zeroes the offset but never
releases the backing array.

Add BufferShrinkPolicy, a shared cap-plus-hysteresis helper, and apply it
at both sites. A buffer above the cap is released only after the
configured number of consecutive batches that did not need the extra
capacity, so a session with recurring large requests keeps its buffer.

Both caps are configurable and default-on:
  --session-parse-state-max-retained-args (default 1024)
  --session-scratch-buffer-max-retained-size (default 64k)

Measured end to end with a forced compacting GC, 200 sessions each
issuing one 20,000-argument EXISTS followed by sustained single-key
work: 327 KB/session retained before, 98 KB/session after.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The batch-boundary shrink releases a pinned array, so anything holding a
reference into it across that boundary would dangle rather than merely
read stale bytes. Two paths cross the boundary by design: a MULTI..EXEC
window, which re-parses queued commands from the network buffer, and
WATCH, which copies key bytes into the transaction scratch allocator so
they outlive the receive buffer.

Add tests that force the shrink to fire inside both windows.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

A response element larger than the send buffer cannot be written
atomically. The chunking helper for this already exists and SCAN uses it,
but several commands still wrote unbounded user data through the atomic
path, where TryWriteBulkString cannot make progress and SendAndReset
throws, killing the session mid-response. KEYS fails today with any key
larger than the 128 KB default send buffer.

Give the shared WriteBulkString helper an atomic-first fallback: write in
place, retry once on a flushed buffer, and only chunk an element that
cannot fit an empty buffer. Small elements keep exactly today's path, so
multi-element replies pay nothing.

Route KEYS, COMMAND GETKEYS, COMMAND GETKEYSANDFLAGS and the pub/sub
channel and pattern echoes through it.

Two related defects found while covering this:

ClusterSession.SendAndReset silently did nothing on an empty buffer, so
a caller's retry loop spun forever rather than failing. It now throws, as
the RESP session already does.

MainStoreOps.GETDEL asserted its output was inline immediately after the
RMW, before the spill-to-pooled-memory case is handled. Any value large
enough to spill tripped it, so GETDEL of a large value failed on every
debug build.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The fallback added for over-sized response elements flushed before
retrying, but SendAndReset throws when there is nothing to send. That is
exactly the over-sized case where the element is the first thing written
to the response, so the chunking path was unreachable there and the
session still died. Every earlier test emitted an array header first,
which left the buffer non-empty and masked it.

Flush only when output is actually buffered, otherwise chunk directly.

CLIENT GETNAME is the reachable instance: the name is the whole response,
so a name larger than the send buffer killed the session. Give the ASCII
bulk-string helper the same fallback and route CLIENT GETNAME through it.

Move both fallbacks to NoInlining helpers so the 49 call sites of the
inlined fast path keep their previous code size.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

Every session was given twelve HdrHistograms when its connection was accepted,
before it had recorded anything. At two significant digits each rents a 32 KB
array from the shared pool, so a connection cost 384 KB of latency monitoring
whether or not it ever issued a command. A dump of a server holding several
thousand connections showed 887 MB in these arrays.

Allocate the histograms for a latency type on the first value recorded into it.
A session typically records into three of the six types, and an idle connection
records into none, so it now pays nothing.

Add --latency-monitor-precision to size the histograms. It defaults to 2, which
is what the histograms were built with, so accuracy is unchanged unless it is
lowered. Dropping it to 1 shrinks each histogram several-fold in exchange for
reporting percentiles to 10% rather than 1% resolution.

Fix three defects on the release path that lazy allocation widens the window
for. LongHistogram.Return() now drops its reference to the array it hands back,
so a later record throws rather than silently writing into an array the pool has
re-rented to another session. The session's pending work is cancelled before its
histograms are released rather than after. GarnetLatencyMetricsSession publishes
its metrics as unavailable before releasing them, and the hot path reads the
field once and checks it.

Dropping the array reference exposed a use-after-return in GarnetClient, which
copied its histogram after Dispose had already handed the array back; it now
reports no metrics instead of reading a recycled array.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The precision knob was only covered by tests that constructed histograms
directly, so a broken binding between the command line and the allocation would
have gone unnoticed and silently degraded resolution.

Assert the size the server reports through LATENCY HISTOGRAM, the size of the
per-session histograms reached through the live sessions, and that the option
parses to 2 by default and rejects values outside the 0-5 range HdrHistogram
accepts.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

A session records its latency after the reply has been written, so a test that
read the histograms as soon as the reply arrived could find none allocated yet.
Poll until one appears.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The buffer pool's byte ceiling bounds only its idle free list. Buffers checked
out by live connections are bounded by nothing, so their footprint is
connections times per-connection size, which is what grew unchecked with
thousands of clients.

Add NetworkBufferBudget, which divides a configured total by the live buffer
count and publishes the largest base size that keeps the two consistent. It is
a division rather than a control loop: buffer sizes move in factor-of-two
steps, so any controller with a narrower deadband cannot converge and
limit-cycles between two size classes. Growth therefore requires the quotient
to reach twice the published target, matching the step it actuates.

One budget is shared by every listener. Each endpoint previously built its own
pool and was handed the whole ceiling, so with N endpoints the effective limit
was N times the configured value. Only server listener pools participate;
replication and migration keep their own unbounded behaviour so that a sync
burst cannot drive live client connections down.

The target is published but consumed by no allocation site yet. Landing the
signal on its own makes a drift in the live buffer count observable in
isolation, which matters because the target is budget divided by that count:
a count that ratchets up does not degrade gracefully, it collapses the target
to the floor permanently. The count rides exactly the two paths that already
maintain the pool's live byte accounting and nothing else.

Defaults are inert: 1 GB against a 128 KB buffer size means adaptation does not
begin until 8,192 live buffers, and the target clamps at the configured size so
it can only ever lower it. --network-buffer-memory-budget 0 disables it.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The budget was published but consumed by nothing. Wire it to the receive side:
a new connection's buffer starts at the published target rather than the
configured size, and a grown buffer shrinks back toward that target.

Only the base size is governed. Demand-driven doubling is untouched, so a
connection that needs a large buffer still reaches it; pressure changes what a
connection starts and settles at, never what it is allowed to grow to.

The allocation size and the shrink floor move together. Clamping only the
allocation would leave a connection allocated below the configured size unable
to shrink at all, because the shrink path both floors at that size and returns
early when the buffer is already at or under it.

A pool cannot recycle a buffer outside its size classes, so NetworkBufferSettings
gains an explicit minimum. Deriving it from the three configured sizes, as
before, gives no way to express "recycle down to 16 KB while new connections
still start at 128 KB", and a clamped buffer would have been dropped rather
than pooled. The per-level idle entry count derived from a byte budget is
capped in the same change, since a smaller minimum size class otherwise raises
it proportionally; the cap sits above the value today's defaults produce, so it
is inert until the two are combined.

Budget accounting is now inert when disabled rather than merely unread. The
disabled budget is a shared singleton, so unbudgeted pools were accumulating a
count on it that was meaningless but would have become misleading the moment
anything read it.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

GarnetSaeaBuffer is the single allocation point behind both
GarnetTcpNetworkSender call sites, so clamping there covers the send
side entirely. Send buffers never grow, so an oversized response is
chunked through whatever buffer it was given, and every consumer reads
the length off the pool entry rather than off the configured setting --
which is what makes the size safe to adapt.

The clamp uses the separate, higher send floor so that a shrunken send
buffer does not push custom raw string responses into repeated
ArrayPool rentals.

Adds an end-to-end test that a 1 MB value and a key larger than the send
buffer both survive while the budget is pinned at its floor, and tightens
the shrink assertion now that both directions adapt. Neutering the clamp
fails that assertion.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The residual left after a fully consumed request is zero, so sizing the
replacement from it always resolved to the base. A connection issuing one
oversized request per batch then re-grew through every intermediate size class
on every request. Floor the replacement at the largest poolable size at both
the socket and TLS transport sites; the idle hysteresis path still takes it
down to the base once the traffic no longer needs the capacity.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

Release drops the counts array while the reply-processing thread may still be
recording, so the record path tolerates a released array and the copy path reads
the field once instead of testing IsReturned and then dereferencing.

Also make the quiesce-threshold test observe empty windows: it recorded a value
and never cleared it, so the windows it asserted over were not empty and a
threshold that never restarted its run still passed.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

A TLS connection decrypts into a second receive buffer that grows through
its own code path, so the above-max release there was unexercised while its
plain-socket twin was pinned.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The above-max branch releases a buffer the pool cannot recycle. Flooring the
replacement at the largest poolable size matches origin/main and saves a
connection that needs the capacity on every request from re-growing through
every size class, but it must not apply while the budget is binding: the target
is derived from a buffer count, so a connection parked at the maximum costs the
same single unit as one at the floor and the budget gets no feedback from it.
Under pressure the replacement is now the adapted base, which the pool keeps
and the budget can see.

Also capture the counts array once in ClearCounts and CopyCountsInto, read the
client call count off the snapshot the percentiles come from, and record the
single-read contract on the field itself.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

The plaintext and TLS receive paths gate the above-max release
independently. Only the plaintext site was exercised, so the TLS gate at
NetworkHandler.cs:801 could be removed with the whole fixture still
green. Parameterize the test over both transports.

Guard the counts array in CopyCountsInto, matching the single-read
invariant documented on the field and the five sibling capture sites.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

`ScratchBufferCapDoesNotChangeLargeArgumentResults` compared a per-session
memory figure across two arms and failed intermittently. Two measurements show
it cannot discriminate: swapping the arms moves each figure by about 130 KB per
session and flips the sign of the difference, because the arm that runs second
also measures the residue of the server instance the first one disposed; and
releasing the buffer at every reset rather than at the checkpoint -- the churn
such an assertion claims to catch -- moves it by less than that, because churn
leaves garbage the forced collection reclaims rather than retention.

The test keeps both arms and the reply validation they drive, which is what its
name has always claimed, and the remarks state why no magnitude is asserted.

`NetworkBufferBudget` documents the per-direction floors as the third reason the
budget is a target rather than a ceiling: once the target reaches them, the
aggregate grows with the connection count again.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7ebed6a-7162-459d-849f-6a9d3a31eb30

Comparing capacity across iterations cannot see the pathology the test
exists to catch: a policy that released on every batch regrows before the
next sample, so consecutive post-boundary capacities are equal and the
churn counts as zero. Sampling before and after each boundary counts every
release.

Co-authored-by: Copilot <[email protected]>

The setting is HdrHistogram significant digits: relative accuracy at every
magnitude, not a fixed time unit. Stating it that way, with the memory each
step costs, lets an operator pick a value without knowing HdrHistogram.

Co-authored-by: Copilot <[email protected]>

The receive buffer grows to fit and maxReceiveBufferSize bounds only what a
connection retains, so a chunk far above it must still be accepted. Dribbled
in small slices so the request sits partially buffered across many receive
passes, where the above-max shrink runs against a live partial request.

Also drop a qualifier from the latency precision help text.

Co-authored-by: Copilot <[email protected]>

Earlier edits flipped the BOM on seven files, which showed up as a
one-line change to each file header with no content behind it.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: a2361ed7-3bec-4fb8-9a6f-b4408448c160

A receive releases its buffer in UpdateNetworkBuffers, which runs after
Process has already written the reply, so a client that has read its
response can still be ahead of the release. Sampling BPSTATS once at
that point read the pre-release numbers on a loaded runner and failed
with the full oversized buffer still live.

Poll until the counters settle. The wait uses only the out-of-band INFO
connection, so it does not hand the connection under test the extra
receive that would let a buffer released one pass too late off the hook:
suppressing the immediate release still fails the assertion.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: a2361ed7-3bec-4fb8-9a6f-b4408448c160

The network and transport receive buffers ran the same three-regime shrink
policy over their own occupancy and countdown, written out twice. Extract it
as TryPlanReceiveBufferShrink, which owns the countdown bookkeeping and the
budget's shrink accounting, and leave each caller with nothing but the buffer
swap. Rename both entry points to TryShrink*, and give the transport buffer a
ShrinkTransportReceiveBuffer mirroring ShrinkNetworkReceiveBuffer.

Drop the cached configuredReceiveBufferSize/configuredSendBufferSize fields
and read them off networkBufferSettings, as maxReceiveBufferSize already is on
the same paths.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: a2361ed7-3bec-4fb8-9a6f-b4408448c160

BaseReceiveBufferSize is read on every receive, so reaching the configured size
through networkBufferSettings puts a dependent load on that path.
BDN Network.BasicOperations.InlinePing measures it at +1.1ns (89.8 -> 90.9) on
net10 and +1.9ns (85.7 -> 87.6) on net8, the net8 range clearing baseline
entirely. Restoring the fields returns both to baseline (89.8 / 85.8).

The shrink consolidation is unaffected and stays.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: a2361ed7-3bec-4fb8-9a6f-b4408448c160

Every other method on the session captures the metrics field once before
dereferencing it. ReclaimQuiescedHistograms re-read it for the null check
and again inside the loop, so it relied on the dispose lock for that
safety where its peers do not.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: a2361ed7-3bec-4fb8-9a6f-b4408448c160

Replayed procedures grow both scratch allocators, not just the transaction
one: the prepare phase copies watched keys into the transaction allocator,
and the procedure builds its arguments from the session allocator.

The session allocator is reset at the start of every procedure rather than
when one ends, so a completed procedure leaves it oversized with a non-zero
offset, and a shrink checkpoint declines in that state. Reset it at the
replay boundary, as the network batch boundary does, and only outside a
transaction, whose slices are still live.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: a2361ed7-3bec-4fb8-9a6f-b4408448c160