feat(operator): default system namespace memory via LimitRange - #3494

This branch gave linstor-controller a 10Gi memory limit, justified in
the values and pinned in cluster_test.yaml on the grounds that the
namespace default was 4Gi, half the -Xmx8G baked into the image.

Both halves of that have moved. The default is now 32Gi, four times the
heap ceiling rather than half of it, so nothing about the JVM requires a
per-chart limit to sit above the namespace default any more. And #3824
argues the opposite direction for the same component -- extending the
controller's existing limit-free exemption to linstor-satellite, using
that exemption as its precedent. Keeping the 10Gi here would remove the
precedent that PR reasons from, and whichever of the two merged second
would silently invalidate the other's rationale.

So linstor policy belongs to one PR, and it is not this one. The three
files go back to their state on main; this branch is now confined to
the operator and the installer chart.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

docs/operations/system-memory-limits.md merged ahead of the code it
describes, so main documents two installer knobs and a 4Gi default that
exist nowhere outside this branch. An operator following the shipped page
today sets a value that silently does nothing.

The page is corrected to the default this branch ships and gains the
three behaviours it had no account of at all: that a namespace holding a
request above the ceiling is raised rather than skipped, that a raised
ceiling only comes back down after the scan has been continuously empty
for a grace period and why that delay has to exist, and that a namespace
another LimitRange already governs is left alone. Each names the log line
to grep for, since none of the three is visible in the object.

The ceiling rationale is also stated more honestly: the scan reads
declared requests and never a container's real resident set, which is the
reason the default sits far above any plausible working set rather than
close to one.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

The static-pod note said containers built from an on-disk manifest are
unaffected "in both directions". True of the container, which the
kubelet runs from disk whatever the API says, but the mirror pod is an
ordinary API object that passes admission like any other, so a static
pod declaring a memory request above the ceiling with no limit of its
own has its mirror pod rejected -- leaving the container running with no
representation in the API. Worth naming, since kube-system is in scope
by design. The scan does cover it: a mirror pod is a live Pod.

The ReplicaSet exclusion was justified on live ones being copies of
their Deployment's template. That does not reach an ownerless
ReplicaSet, which at replicas 0 has neither pods to be seen through nor
a Deployment template standing in for it. It stays out of scope --
nothing here ships a bare ReplicaSet -- but as a gap left open rather
than one the stated reason closes.

The delete guard's comment claimed the enabled apply always adopts and
stamps a same-named foreign object. It does not when that object's spec
already matches, because then the apply is skipped entirely. The
outcome is the same one the guard gives on purpose, but it is reached by
the apply not running.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

… too

The grace period was documented as a property of a raised ceiling coming
back down, which is where it started, but it applies to any namespace
defaulting more than the configured limit -- and a namespace sitting at
a previously higher configured value is indistinguishable, on the
object, from one the operator raised. So lowering the knob reaches those
namespaces within the grace period rather than on the next reconcile,
which is worth stating before an operator concludes the setting did not
take. The page now also shows how to read the countdown off the objects.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

…riod

The grace period guarded only the branch where the scan came back empty,
which left the same defect standing in the branch where it finds
something smaller than the ceiling already in place.

A namespace raised to 16Gi for a pod visible only as a live Pod, with a
second oversized pod at 8Gi beside it, loses the larger pod to a node
drain. The scan still finds the 8Gi one, so the empty-scan guard never
engages, and the ceiling was written straight down to 8Gi -- rejecting
the larger pod's recreation exactly as writing the configured limit
would have. A blocker smaller than the standing ceiling is not evidence
that a larger one is gone.

So the rule is now about direction rather than about why: the ceiling
the evidence justifies is computed first, and any move that lowers it
goes through the grace period, whichever of the three reasons prompted
it -- an empty scan, a smaller blocker, or the configured limit itself
being lowered beneath a standing ceiling. The eventual drop lands on
what the scan justifies rather than on the configured limit, so a
smaller blocker that is genuinely still there keeps the ceiling it
needs.

The raise is also no longer logged when it changes nothing. A namespace
holding a permanently oversized pod re-justifies the same ceiling on
every reconcile, and announcing that put a line in the log for every
such namespace on every HelmRelease status change.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Two imprecisions in the page written alongside the grace period. It said
a lowered limit reaches previously-raised namespaces "within" the grace
period, which is backwards: the grace period is a floor, not a ceiling,
and since reconciles are event-driven the drop lands on the first one
after it elapses. And the withdraw path was described only as leaving
the namespace alone, without saying that it hands the namespace to the
administrator's policy in full, including any admission rejection that
policy implies -- which is theirs, and would have happened without
Cozystack, but is worth naming where an operator is being told the
operator stopped writing.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

… scope

The --help text still carried two claims that values.yaml and the
operations page had already narrowed, and it is the one surface an
operator reads without opening the tree.

Both were wrong in the direction that costs the reader something. "A
namespace where another LimitRange already defaults or caps container
memory is left alone" described the guard before it was widened, so an
administrator holding a container memory min, a maxLimitRequestRatio or
any Pod-scoped bound would read it, expect the LimitRange to be applied
there anyway, and get a namespace with no memory.max for a reason the
help says does not apply. "Removes any previously created one" is the
universal removal claim, and removal is scoped to namespaces an active
Package still targets.

Two comments corrected while here. constrainsMemory said it switched on
the item's own maps "rather than a hand-picked few", which contradicts
the hand-picked list immediately below it; what is actually true is that
a field added upstream has to be added there too, which is the reason
it has its own table test. And memoryDefaultSettledAt claimed the count
of raised namespaces falls to none once the oversized requests are
gone, which stopped being true when the scan started keeping Failed
pods: a carcass Kubernetes never collects holds both the raise and that
namespace's per-reconcile pod List open.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

The pod half of the scan was skipped once a namespace defaulted container
memory at exactly the configured limit, on the argument that the shape
the scan looks for could no longer be admitted there. That argument
proves such a pod cannot come to exist. It does not prove one cannot be
attempted, and a failed attempt leaves nothing behind: a workload built
from a custom resource whose request grows above the limit after its
namespace settled was rejected permanently, with no artefact anywhere
that could justify raising the ceiling again. The premise was not sound
even for existing pods, since LimitRanger reads LimitRanges through an
informer and the shape is admissible in the window before the plugin
sees the apply. The skip bought one namespaced List out of six and paid
for it with a silent permanent-rejection mode, so it is gone along with
memoryDefaultSettledAt and the two tests that pinned it. One of those
tests carried the counterexample in its own fixture: a settled 4Gi
LimitRange beside an 8Gi pod, asserted invisible.

The coexistence guard identified its own object by name. The name is not
reserved, so an administrator's LimitRange called
cozystack-system-defaults was skipped by the guard and then overwritten
by the forced apply, and because LimitRangeSpec.Limits is an atomic list
that replaced every item in it, taking their max and cpu policy along.
Ownership is now the managed-by label, which is what the delete path has
always used.

That leaves a same-named object with no label ambiguous: ours with the
label stripped, or an administrator's. The two mistakes are not
symmetric, so the ambiguous object is treated as foreign and left
entirely alone rather than adopted. Re-stamping it with the label goes,
and its test is replaced, because the same path reaches an
administrator's object and adopting it means the disable path later
deletes their policy. Declining to delete is no longer silent either.

Also: containerMemoryDefault now takes the last matching item rather
than the first, which is what LimitRanger applies, so a multi-item spec
no longer reports a ceiling the cluster is not using.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

…eriod

The raise-and-grace machinery cannot work: LimitRanger writes the
raised limit onto the very container that justified the raise, so at
that container's next admission it carries a limit, the scan skips
containers that have one, and the raise has erased its own evidence.
The ceiling then drops on schedule and the workload is rejected
permanently. Any such machinery approximates admission logic ahead of
admission, and that is incomplete by construction: a
LimitRange default is one number for a whole namespace and cannot mean
"the configured limit, or this container's request, whichever is
larger". Only per-pod conditional defaulting expresses that, which is a
mutating webhook and not this.

The justification for the machinery had also expired. It was designed
when the default was 4Gi, where withholding was self-perpetuating for
VPA-managed namespaces. At 32Gi nothing in the tree approaches the
ceiling -- the largest static memory request is 2Gi -- and VPA sets
requests and limits together, so those containers carry their own limit
and were never blockers.

So a namespace holding a container that requests more than the
configured limit with no limit of its own is now left with no default at
all, and anything written there earlier is withdrawn. It fails in the
safe direction, which is the whole argument: a namespace with no
memory.max is a namespace behaving as it did in every release before
this, while a default below a request is a namespace whose workload
cannot be admitted. It needs no stored state and no timer, because
withholding is not a decision that has to survive between reconciles --
once the oversized request is gone the default is applied again on the
next pass.

Gone with it: the grace period, the clear-since annotation and its
clamp, the injected clock, containerMemoryDefault, raisedCeiling, and
fourteen tests. The scan stays, because deciding whether to apply or
withhold is what it was always for. The foreign-LimitRange ownership
guard, the disable path and the 32Gi default are untouched.

The residual sharp edge is stated rather than machinery: a workload
whose only evidence is a live Pod, built from a custom resource with no
template this scan can read, can still be missed if the scan runs while
that Pod is absent, and applying the default then rejects its
recreation. At 32Gi that is unreachable in this tree. It becomes
reachable if an operator lowers the knob below such a workload's
request, which the flag help, values.yaml and the operations page now
say plainly.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Kubernetes accepts memory keys on PVC-scoped LimitRange items when a
storage bound is also present, but those quantities cannot affect pod
admission. Restrict the foreign-policy guard to Container and Pod items.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Record terminal Job UIDs while scanning templates, then ignore only
Failed pods owned by those Jobs. Failures from unfinished controllers and
ownerless pods remain evidence because they may still be recreated.

Also remove the stale constructor comment that still described raising a
namespace ceiling.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

List LimitRanges by the ownership label once per Package reconcile when
the feature is disabled, and remove managed objects even in namespaces
outside the current Package target set. Skip the per-namespace path while
disabled and keep cleanup failures non-fatal to namespace reconciliation.

Update the flag help, installer guidance, and operations documentation to
describe the global cleanup boundary.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Keep the parser inputs tied to the constants registered as flag defaults,
but compare the results with literal 32Gi and 32Mi expectations. A shipped
default can no longer move while the defaults case stays green.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Replace three stale descriptions of the removed ceiling-raise behavior with
the current withhold and withdraw semantics. Historical explanations of why
the raise design was rejected remain intact.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-By: LLM

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-By: LLM

Preserve foreign same-name objects and keep create-time SSA non-force.

Only ignore failed pods controlled by terminal Jobs.

Make the failure-containment test inject Apply errors for real.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

An admitted Pod loses the distinction between explicit and defaulted
limits. Its disappearance cannot prove a lower default is safe for
its replacement. Keep that decision on the Namespace until an
administrator acknowledges the particular target, without waiving
current blockers or foreign policy ownership.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

Foreign ownership short-circuited the read-error fixtures before their
interceptors ran. Require the injected failure to occur and preserve
both policy and Namespace state, including a pending acknowledgement.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

A foreground-deleting Package can survive long enough to reconcile
after cleanup retires its default. It must not reinstate that policy
or consume an acknowledgement while its components are being removed.
An active Package sharing the namespace can still manage the default.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>