docs(vm-instance): document attaching a VM to an external VLAN - #3196

## What this PR does

Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (`bridge`, `macvlan`, `ipvlan`,
...), and nothing guarantees a node has them where Multus looks. On k3s
they live under `/var/lib/rancher/k3s/data/cni` while Multus and Cilium
install into `/opt/cni/bin`, which is left without a `bridge` binary. A
`bridge`-type NAD then fails with `failed to find plugin "bridge" in
path [/opt/cni/bin]`, and today you work around it by copying the binary
in by hand. #3196 documents one such pattern (attaching a VM to an
external VLAN) that this unblocks.

This bundles 14 of the reference plugins into the `multus-cni` image and
adds an `install-cni-plugins` init container that installs them into the
host `/opt/cni/bin`, mirroring the existing `install-multus-binary` init
container.

## Where it applies, and how to turn it off

The new `networking.stageCniPlugins` value defaults to on for both
bundles. Talos puts six binaries in that directory -- `bridge`,
`firewall`, `host-local`, `loopback` and `portmap` stripped from the
same upstream release, plus `flannel`, which is a different project --
so the other ten are missing there exactly as on generic Linux and a NAD
naming `macvlan`, `ipvlan` or `vlan` fails on either platform. Four of
those six are names this change stages; `loopback` is deliberately not
staged and `flannel` was never in the reference set. The four names that
overlap are rewritten on every Multus pod recreation with the same
upstream version -- Talos pins `containernetworking/plugins` `v1.9.1` in
`siderolabs/pkgs`, and so does this image -- so it is not a downgrade of
a system component. A marker in the image Dockerfile records the Talos
release that pin was compared against, and a test holds it to the
release the platform ships. `isp-hosted` ships no multus at all and is
unaffected. Both defaults can be overridden either way.

If you run a generic-Linux cluster and manage `/opt/cni/bin` yourself,
this changes behaviour on upgrade. Multus starts installing 14 plugins
into that directory every time its pod is recreated, replacing files of
the same name, so a hand-placed or locally patched plugin there will not
survive. Set `networking.stageCniPlugins: false` to decline, and set it
**before** the upgrade that first enables staging: nothing here removes
or restores a plugin, so declining afterwards protects what is still
yours and does not give back what was already replaced. Changing the
value edits the Multus DaemonSet's pod template, so Multus rolls node by
node, and while a node's daemon is down pod creation and deletion on
that node fail for the length of the restart. The four plugins the
package never touches are `loopback`, `dhcp`, `dummy` and `tap`.

Where staging runs it is unconditional, not only-when-absent, because
nothing here ever revisits a plugin it has written. Installing only when
absent would pin each node to whichever version arrived first, and a
later `CNI_PLUGINS_VERSION` bump, including one fixing a binary that
runs as root on every node, would reach zero existing nodes. The
directory is not ours alone and the decision does not assume it is:
kube-ovn copies `loopback`, `portmap` and `macvlan` there from its own
image, two of which this change also writes, node provisioning may place
the set, and `/opt/cni/bin` is also the CNI bin directory on RKE2.
Overwriting is what delivering a version requires, and
`networking.stageCniPlugins: false` is how an operator who owns those
files declines it.

## Notes

- The manifest change lives in `patches/customize-deployment.patch` so
it survives `make update`, and the release rewrite in that target runs
after the patch rather than before it, so the line the patch adds lands
on the same reference as the two it already had. The patch seeds that
line with the upstream image reference while the committed template
carries the pinned cozystack digest, because the package's `image:`
target rewrites every `multus-cni` image line to one digest. So it does
not reverse-apply against the committed template. That `make update`
reproduces the tree is verified forward from pristine upstream instead.
- `/opt/cni/bin` stays shared where staging is on. kube-ovn installs
`portmap` and `macvlan` there and has been seen to do so with a plain
copy onto the live path, which truncates in place. Each plugin is
installed under a temporary name and swapped in with a rename, atomic
within the mount, so the kubelet never execs a half-written binary. The
Dockerfile records which kube-ovn release the version pin was checked
against, and `hack/cni-plugins-staging-contract.bats` fails when that
marker and the vendored release part ways. That check fires on the next
kube-ovn bump and compares the marker with the vendored release; it does
not compare the two `CNI_PLUGINS_VERSION` pins. Moving the marker is the
acknowledgement that someone looked -- so read kube-ovn's
`dist/images/Dockerfile.base` before moving it, because a silent
divergence there makes `portmap` and `macvlan` flap between the two
DaemonSets on every pod recreation.
- The init container runs unprivileged, with capabilities dropped and no
mount propagation. Writing into a hostPath needs root inside the
container and nothing on the host, which is how cilium's
`install-cni-binaries` copies plugin binaries into the same directory on
the same nodes. `Bidirectional` propagation governs mounts made inside a
container, this script makes none, and asking for it is what would force
`privileged`. The tests pin the absence.
- `loopback` is excluded because replacing it is what a node cannot
survive: the runtime calls it for every sandbox, and Cilium and kube-ovn
already install it. `dhcp` needs a host daemon this package does not
run. `tap` and `dummy` have no consumer here. `portmap` is included
because a NAD conflist may chain it for hostPort.
- The plugin version is pinned via `ARG CNI_PLUGINS_VERSION` (v1.9.1)
and the release tarball is verified against a per-arch SHA-256 before
extraction. These are root-executed node binaries, so a tag alone is not
enough. An unknown or empty `TARGETARCH` fails the build.
- The init container skips staging when the image has no `/cni-plugins`,
rather than crashlooping the DaemonSet node-wide. The manifest and the
image digest are re-pinned on different schedules, so a tree taken
between a Dockerfile change and the next release bake can carry one
without the other. A plugin it cannot copy is reported on stderr and
skipped, for the same reason. `multusConfigFile: auto` makes
`00-multus.conf` the primary CNI config and there is no
`cleanupConfigOnExit`, so on a node where Multus has already run, an
init container that never completes leaves the shim's socket gone with
that config still primary, and the node stops scheduling anything that
goes through CNI; host-network pods still start. Failing on a copy error
would therefore leave the node worse off than before this change, which
only ever left it without one plugin. The cost of continuing is that the
pod still reports Ready, so the summary is also written to
`/dev/termination-log`, which the kubelet surfaces in `kubectl describe
pod` regardless of the exit code.
- The bundled plugins add roughly 67 MiB uncompressed per architecture
to an image pulled by every node, including installs where staging is
off. Measured by extracting the staged set from the v1.9.1 release
tarball.

## Testing

`packages/core/platform/tests/bundles_multus_staging_test.yaml` pins the
delivery point: both variant defaults, both explicit overrides, and that
multus is still shipped in each bundle at all. That last one because
multus is now included by hand in two branches rather than by the shared
helper.

`hack/multus-install-cni-plugins.bats` extracts the init container's
script from the rendered DaemonSet and runs it. It pins that every
plugin lands executable, that a replacement swaps the inode rather than
truncating in place, that temp files never leak, that a missing or empty
directory skips instead of failing, and that a plugin which cannot be
installed is reported on stderr and skipped rather than failing the
container. Both halves of that last one are asserted, because exit 0
alone would still pass with the report deleted.

`hack/cni-plugins-staging-contract.bats` covers what the rendered
manifest cannot show: that the Dockerfile stages the intended plugins
and only those, copies them into the final stage, and that nothing there
removes them again, that the tarball is verified before extraction, and
that the version marker matches the vendored kube-ovn release.
helm-unittest pins both states of the value, including off, which is
what most installs run, plus the quoted-string forms a values file can
produce. An explicit `null` is covered too: it is how a values file
clears the key, and the chart folds it onto its own default rather than
refusing it as an unreadable spelling.

The feature is inert until the image is re-baked. All three `image:`
lines still point at the pre-change digest, which has no `/cni-plugins`,
so on `main` after merge a generic install renders the init container,
logs `no /cni-plugins in this image; skipping staging`, and installs
nothing until release-prep re-pins. That is the intended flow
(first-party digests are release-managed, not hand-edited), and the
guide documents the log line, but anyone testing this on `main` will
meet it first and the pod will look healthy.

No CI job stages plugins on a node. CI boots Talos, where the value is
off, and `isp-full-generic` appears in no e2e workflow. The mechanics
are covered by the bats above, which run the real script from the real
manifest and also under `dash`, the shell in the runtime image, but
nothing demonstrates it end to end on a node.

Defaulting it on for the generic bundle is a deliberate maintainer
decision taken with that gap known, not a side effect of the diff.
Shipping it off would leave the bug in place on the only platform where
the feature does anything, and would not add coverage either, since the
code stays unexecuted in CI both ways. What makes the risk acceptable:
the init container skips rather than crashlooping when the image carries
no plugins, each plugin is swapped in with an atomic rename instead of
being written onto the live path, the tarball is checksummed before
extraction, and declining is a value rather than disabling the package.
And a plugin it cannot install is reported and skipped rather than
failing the pod, so the worst case leaves the node exactly as it was
before this change rather than unable to schedule.

### Screenshots

N/A, no UI changes.

### Release note

```release-note
feat(multus): install the reference CNI plugins (bridge, macvlan, ipvlan, ...) into /opt/cni/bin on generic-Linux installs, where nodes do not otherwise provide them. On by default on every platform: Talos ships five of the fourteen, and the four that overlap are rewritten with the same upstream version. On generic clusters this replaces same-named files in /opt/cni/bin every time the Multus pod is recreated, so a locally managed plugin there will not survive. Set networking.stageCniPlugins: false to decline.
```


BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI plugins
in /opt/cni/bin on existing nodes, Talos included -- four of the names
it writes are ones the Talos node image also carries. Opt out before
upgrading: declining afterwards does not restore what was replaced.









<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Multus can now stage reference CNI plugins into `/opt/cni/bin`, adding
missing plugins and safely replacing matching versions.
- Added the `stageCniPlugins` setting, with platform-specific defaults
and explicit opt-out support.
  - Invalid configuration values now produce clear rendering errors.
  - Improved plugin installation diagnostics and failure reporting.

- **Documentation**
- Expanded guidance for CNI prerequisites, plugin staging behavior,
troubleshooting, rollout impact, and configuration.
  - Clarified Multus image reference handling in platform documentation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->