test(e2e): container lane for the srv1-srv3 nodes, plus three storage fixes it exposed - #3961

The sandbox boots three QEMU guests and Cozystack runs KubeVirt inside
them, so a tenant worker sits at L3: Oracle host, runner VM, srv VM,
worker. Running the srv nodes as containers deletes one of those levels
and puts the worker at L2.

Measured 2026-08-25 on two identical 16-OCPU EPYC 9J14 hosts: worker VMI
Ready to CAPI Machine.status.nodeRef was 20-35s on the container lane
against 76-92s on the QEMU lane, same host and same install, and the
container lane was flat across the kvm_amd vgif module parameter where
the QEMU lane moved ~18%. The same spike did NOT reproduce #3513's 1710s
stall in any configuration, so this rests on wall-clock and on removing a
runner property we do not control, not on a demonstrated fix for that
issue.

Sits beside the QEMU lane and is wired into no workflow, so nothing about
the current lane changes. Addressing, CPU and memory mirror
e2e-prepare-cluster.bats exactly, leaving the MetalLB pool and
e2e-post-install-prep.sh untouched.

Three properties of container mode drive the design, each of which fails
quietly rather than loudly if ignored:

  * machine.kernel.modules is a silent no-op (kernel_module_spec.go
    returns early on ModeContainer with no error and no event), so the
    host must load openvswitch and zfs first. Both the host script and
    the BATS file assert it rather than trusting it.
  * Docker gives a privileged container its own tmpfs /dev, seeded once
    at creation, so /dev/kvm works but ZFS zvols created later never
    appear and linstor-csi fails ControllerPublishVolume. The compose
    file binds the host devtmpfs, which is also why `talosctl cluster
    create docker` can never carry LINSTOR.
  * There is no ARP VIP, so .11 stands in for the QEMU lane's .10 and
    the install takes its endpoint from COZY_APISERVER_ENDPOINT,
    defaulting to the unchanged QEMU value.

Provisioning is two steps because the halves run in different places.
The nodes are siblings on the runner's docker daemon rather than nested
in the sandbox, since those host binds must resolve against the real
host devtmpfs; only that half needs a docker socket.

The lane gives up DRBD, and with it more than the replicated
StorageClass: linstor-csi couples RWX to DRBD, so worker disks are RWO
and live migration is unavailable here. That is why the QEMU lane must
keep nightly coverage. kube-ovn still runs the real OVS dataplane and
LINSTOR keeps real ZFS pools and the production `local` StorageClass.

Requires linstor drbd.enabled=false, without which every satellite
CrashLoops and no PVC binds.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

A tenant Kubernetes worker disk is imported from a compressed Talos image
(openstack-amd64.raw.xz, packages/apps/kubernetes/templates/cluster.yaml),
which the CDI importer decompresses and converts onto a block volume. At
CDI's built-in default of 600M that conversion is OOMKilled at 99.8-100%
progress -- right at the end -- and CDI then retries from scratch, so the
DataVolume cycles ImportInProgress 99.98% -> ImportScheduled 99.97%
indefinitely and no worker ever boots.

The failure shape is the problem as much as the failure: nothing reports
an error, the cluster looks healthy, and the DataVolume reports ~100%
throughout. Measured 2026-08-25 on a three-node cluster; 4Gi completed
the same import with zero restarts, on two subsequent runs.

Set it on the CDI CR rather than on CDIConfig. The CDI operator
reconciles CDIConfig.spec from this CR, so patching CDIConfig directly is
silently reverted and .status.defaultPodResourceRequirements stays at the
old value -- which is a trap worth knowing when debugging this live.

A limit is a ceiling, not a reservation, so raising it costs nothing
until an import actually needs the memory; the request is deliberately
left small so concurrent imports still pack onto a node.

The package had no tests directory and hack/package.mk defines no `test`
target, so add both -- hack/helm-unit-tests.sh only runs a package whose
`make -n test` resolves, and a suite added without that line would be
collected by nothing and skipped in silence.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Without --strict-topology, external-provisioner passes every topology
segment as `requisite` and only *prefers* the node the scheduler picked,
so when that node is out of space linstor-csi may legally provision the
volume somewhere else. For a node-pinned volume -- allowRemoteVolumeAccess
false, which is what the cluster-default `local` class uses -- the result
is a PV whose node affinity contradicts the pod's other volumes, and the
pod is then unschedulable forever:

  pv "pvc-..." node affinity doesn't match node "srv1": no matching NodeSelectorTerms

It bites hardest on CDI imports, because an importer pod mounts two
volumes: the disk, and a ~21 GiB scratch volume CDI creates only AFTER
the pod is scheduled. The scheduler therefore sizes the node against the
disk alone, can place two importers where both disks plus both scratch
volumes do not fit, and the second scratch is placed elsewhere.

The failure shape is what makes it worth fixing rather than documenting:
nothing errors, the cluster looks healthy, and the DataVolume sits at
ImportScheduled indefinitely.

Reproduced and verified by measurement 2026-08-25 on a three-node
cluster. Forcing both importers onto one node with 45 GiB free -- enough
for two 20 GiB disks but not their two 21.2 GiB scratch volumes -- put
the scratch PV on a different node than its disk and hung the import with
no error anywhere. With the flag set, the same conditions leave the
scratch PVC Pending and the pod reporting the actionable "1 node(s) did
not have enough free storage"; freeing space let it bind and the import
completed on its own. A silent permanent deadlock becomes a legible,
self-correcting wait.

Note the scheduler was not working from stale data: CSIStorageCapacity
reported srv1 at 4.82 GiB against LINSTOR's 4.60 GiB and ZFS's 4.60G. It
had the right numbers and still could not prevent this, because the
scratch volume does not exist when the placement decision is made.

Verified for node-pinned volumes. For `replicated` the PV is remotely
accessible so the constraint is satisfiable wherever the pod lands; that
case is reasoned rather than measured, hence the switch.

The patch restates the whole argument list because `args` has no
patchMergeKey and a strategic merge replaces it, so a piraeus-operator
bump could silently drop a newly added upstream flag. A test pins the
argument count as a tripwire that forces a re-check.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

The prepull step enumerates images by walking every map that carries a
`containers` key across the kubeovn, linstor and cert-manager renders.
That sweep also reaches the pod templates embedded in the LinstorCluster
CR, which are strategic-merge fragments rather than real PodSpecs.

Every such fragment in the tree had carried an image until the
csi-provisioner entry added in this branch, which overrides only `args`
and so has no image of its own to pull. yq emitted a bare `null` for it,
which reached the DaemonSet as a container with an empty image:

  The DaemonSet "e2e-image-prepuller" is invalid:
  spec.template.spec.containers[4].image: Required value

Drop nulls in the filter, and refuse a literal `null` in the prepull
script itself. The apply error names a container index and names neither
the filter nor the chart behind it, so the guard is what keeps the next
occurrence from costing the same diagnosis.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

The container lane landed as an invocable make target that no workflow
referenced, so CI kept running QEMU exclusively and the lane had no run
history on a real runner. Add `E2E (container lane, informational)`,
which installs and runs the same Chainsaw selection against Talos nodes
in containers, on a second runner and against the same commit.

Informational on purpose: the job is absent from e2e-report's `needs`,
so it cannot move the required "E2E Tests" status and a red there does
not block a merge. Every measurement behind the lane was taken on
hand-built spike stands; none on a CI runner.

A separate job rather than a `lane:` matrix leg on `e2e`, because
e2e-report reads `needs.e2e.outputs.soft_red`. A matrixed job publishes
one output map across all legs, so that value would come from whichever
leg finished last and the node-join soft-red decision would turn
nondeterministic. Duplicating a few steps is the cheaper mistake.

Two gaps had to close for the lane to run unattended. The host script
created zpools and loaded modules as the invoking user, which is root on
a dev box but an unprivileged user on a runner, so it now escalates per
command -- never around docker, or the compose project ends up owned by
a different user than teardown runs as. And COZY_APISERVER_ENDPOINT was
read inside the sandbox by nothing that set it; docker exec starts a
fresh environment, so the Makefile passes it through explicitly. Unset
expands to empty and the `:-` default keeps the QEMU lane on .10.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Pass an E2E storage-class value through Make and Chainsaw so the container lane runs the same suite with local RWO disks. Keep the QEMU replicated default and omit only remote-propagation and RWX assertions in local mode.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Apply the remaining phase-0 local-only requirements before workloads start: pin CDI to RWO/block, put VictoriaLogs on local storage before monitoring is enabled, and cap node allocatable to the container limits.

Add focused regression and live assertions while preserving the QEMU lane replicated defaults.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Accept the Ki, Mi, and Gi forms Kubernetes uses when canonicalizing Node allocatable memory. Cover the exact 24476Mi value produced by the container lane after kubelet reservations.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Signed-off-by: Myasnikov Daniil <[email protected]>

Signed-off-by: Myasnikov Daniil <[email protected]>

Signed-off-by: Myasnikov Daniil <[email protected]>

Signed-off-by: Myasnikov Daniil <[email protected]>

Signed-off-by: Myasnikov Daniil <[email protected]>

Signed-off-by: Myasnikov Daniil <[email protected]>

Make the normal container prepare path tear down both Docker volumes and
host zpools before startup, so retries cannot adopt LINSTOR state from a
partial attempt.

Keep pool cleanup failures visible while still removing the sandbox, and
document the sparse 200 GB tradeoff.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Exercise the System-to-CustomConfig OIDC transition on the existing latest-version cluster and scope inter-suite draining to the cluster being removed.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Prove PUT, GET, content integrity, and DELETE through each newly granted BucketAccess before provisioning database backup workloads.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Replace snapshot-based masked waits with a fresh-list gate that requires current generations, detects late releases, and fails immediately on terminal Flux reconciliation.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Fan Bats files out as independent Make prerequisites and share a bounded four-slot job pool with Helm and controller tests in pull request CI.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

Propagate stale-environment and storage-reclamation failures so a later suite cannot adopt a dirty sandbox.

Keep backing files unless ZFS state is proven detached, give bounded cleanup stages enough outer time, and pin the privileged Talos image by digest.

Assisted-By: GPT-5 <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>