feat(cni): multi-NIC KubeVirt VMs — a Multus delegate mode and a NAD per binding - #27

The two fields multi-attach needs (docs/multi-attach.md), added before the
plumbing that reads them.

VPCBinding.allowForwarding is a GRANT. from_pod runs a source-address RPF check
per interface — the source must resolve, in `locals`, to the very veth it was
emitted on — and a router bridging two VPCs violates it by definition: it emits on
the B interface with an A source. Lifting the check is not a convenience toggle.
A pod's SecurityGroup identity is keyed on that source IP, so a pod exempt from
RPF can impersonate any member of its VPC and inherit its groups. That capability
belongs to whoever owns the network, and a VPCBinding is already the object the
VPC's owner authors (creating one requires the `export` verb on the VPC), so the
grant lands at the right level of authority and shows up in RBAC.

Port.forwarding is what the CNI derives from it, and it is deliberately SEPARATE
from Port.gateway even though both end up as PORT_F_GATEWAY in the ports map.
Gateway means "this is the VPC's .1 egress leg" and is what desiredGateways reads
to program gateways[vni]. A tenant firewall is not its VPC's door; reusing the
field would have silently programmed one as the VPC's egress gateway. They share
a datapath flag, not a meaning.

Generated code regenerated with `make generate` — 4 files, additive only
(conversion both ways, applyconfig for the two specs, openapi). Note for anyone
repeating it from a Windows checkout: `make generate` cannot run there. The
generators derive the Go package name from the filesystem path and emit
`package D:\cozyplane\...`, and core.autocrlf gives the Makefile a CRLF shebang
so make dies with env: 'bash\r' first. Run it in a Linux container against an LF
export of the index.

Assisted-By: Claude Opus 5 (1M context) <[email protected]>
Signed-off-by: Loïc Fontaine <[email protected]>

Implements docs/multi-attach.md. A pod took exactly one VPC, which is enough for
a workload that LIVES in a network and useless for one whose job IS the boundary
between two.

The contract is a JSON-list annotation; the single-VPC annotation keeps working
verbatim and carrying both is an error rather than a precedence rule. Order is
part of it: entry 0 is the PRIMARY attachment — it owns the pod's one fabric
handle, so it is what status.podIP resolves to and what kubelet probes reach, and
it alone gets the default route. Secondary entries get only an onlink route to
their own VPC CIDR. That is not symmetry for its own sake: N default routes would
have the kernel pick an egress interface by whatever metric it assigned, and an
on-link /32 to 169.254.1.1 installed on two interfaces would leave it choosing one
of them for both. onlink needs no route to the hop at all, which is exactly why
the gateway leg has always used it.

The forwarding grant is the half that makes a firewall actually work. from_pod's
source RPF drops a routed packet at its ORIGIN veth — a VM routing A→B emits on B
with an A source and locals[{netB, srcA}] misses. Lifting that check is the right
to impersonate any member of the VPC, since SecurityGroup identity is keyed on the
source IP, so it is read from the VPCBinding (authored by whoever holds `export`
on the VPC) and never requested by the pod. It lands on the new Port.forwarding,
deliberately NOT on Port.gateway: gateway is what desiredGateways reads to program
gateways[vni], and a tenant router is not its VPC's door. They share a datapath
flag, not a meaning.

Static addressing comes with it. A Port's name IS its address claim and was only
ever allocated by walking the CIDR from .2, so nothing that had to be reachable at
a known address could say so. A requested address is claimed exactly; AlreadyExists
is a hard error, because quietly handing back a different address is the failure
the field exists to remove.

Two things that only break once a pod has more than one attachment, fixed here
rather than left to be found:

  - Persistent Ports were selected by {vpc, vm-name} and the first match taken. A
    two-NIC VM has two, so the choice was arbitrary and its interfaces would swap
    addresses across restarts. The NIC index is now part of the selector.
  - rebuildVeth demanded a fabric route on every VPC veth and errored without one.
    A secondary leg has none. It now treats the route's PRESENCE as the signal for
    "this is the primary leg" — and specifically not the gateway flag, because a
    forwarding attachment carries that flag and may still be primary, so testing
    the flag would silently skip rebuilding a bridge that does exist and leave the
    pod unreachable on its own status.podIP after an agent restart.

Host veth naming keeps index 0 byte-identical to before. Pods created without
multi-attach have veths under the old name, and a DEL reconstructing a different
one would strand their links and map entries.

bpf/overlay.c is untouched, as the design predicted: ports/locals are already
keyed per interface and PORT_F_GATEWAY already means exactly this.

Assisted-By: Claude Opus 5 (1M context) <[email protected]>
Signed-off-by: Loïc Fontaine <[email protected]>

Found by running the thing. Multi-attach reused PORT_F_GATEWAY to grant a
tenant router the source-RPF exemption it needs, on the argument that the
datapath already implemented exactly those semantics. It does not. The gateway
flag also makes from_pod stamp GW_MARK, and to_pod skips the east-west
SecurityGroup gate on GW_MARK — so granting allowForwarding silently disabled
policy for the one workload whose traffic most needs policing.

Measured on a live 3-node cluster before the fix: with the grant, the forwarder
reached a peer on a port no rule allowed; without it, denied. The design doc
listed several is_gw side effects and missed this one.

PORT_F_FORWARD (bit 30) now carries tenant forwarding, separately, and the two
diverge exactly where they should:

  - both lift the RPF check in from_pod, because emitting a source you do not
    own IS routing;
  - only a packet whose source is genuinely foreign gets marked, so the
    router's OWN traffic keeps taking the ordinary east-west path, groups and
    all;
  - the mark is FWD_MARK (bit 19), not GW_MARK. It buys passage through the
    destination's ISOLATION check and nothing else. TUN_F_FORWARD carries it
    across a Geneve hop, so real VNIs are now < 2^22 rather than 2^23 — four
    million, against an allocator that starts at 100.

A forwarded packet is then judged by the destination's SecurityGroups as a
NORTH-SOUTH source, through the same ns_sg_admit the fabric bridge and floating
IPs already use. That is not a shortcut: this VPC holds no identity for an
address it does not own, so the east-west group test cannot judge it — srcnet
is 0, the source bitmap empty, and a grouped destination would deny everything
with no rule able to allow it. A tenant writes `from: {cidr: 10.10.0.0/24}` and
means exactly this. An ungrouped destination still passes, so forwarding works
out of the box and tightens the moment the destination joins a group.

Verified on the cluster, four cases: grant off -> dropped at the forwarder's own
veth; grant on, destination ungrouped -> delivered; grant on, destination
grouped with no rule -> DROPPED (the fix); grant on, plus a matching cidr rule
-> delivered on the rule's port and dropped on another.

Two things the compiler and the loader taught, recorded so they are not
re-learned:

  - the verifier rejected the first attempt with "dereference of modified ctx
    ptr R2 off=8 disallowed" — clang had folded one skb->mark test into a
    VARIABLE ctx offset. A ctx field must be loaded at a constant offset. One
    hoisted read of the mark leaves clang no such choice, and is cheaper.
  - the veth alias is the rebuild record, and it encoded only `gw`. Without a
    `fwd` key a granted leg would lose the flag on the first agent restart and
    start dropping its own transit traffic on the RPF check, hours after the
    change that caused it. The key is optional on read, so an alias written by
    an earlier release still parses — as fwd=0, which is what it was.

The regenerated object is byte-different from the committed one beyond this
change: it was built with Debian clang 19.1.7, and CI already treats the object
bytes as clang-sensitive (that job is continue-on-error). The generated Go
bindings are unchanged, which is the part that guards the map and struct ABI.

Assisted-By: Claude Opus 5 (1M context) <[email protected]>
Signed-off-by: Loïc Fontaine <[email protected]>

setGWCondition appended raw metav1.Condition values, where its FloatingIP
twin already used meta.SetStatusCondition. Two consequences:

  - LastTransitionTime was never set. metav1.Condition marks it required, so
    kubectl renders a blank transition time, anything computing a condition's
    age is wrong, and the object would be rejected outright were this kind
    ever CRD-served again — where the schema enforces it.
  - Two calls with the same Type produced two entries rather than updating
    one. Harmless today only because the status is rebuilt from scratch each
    reconcile and every Type is set exactly once; a fragile thing to rely on.

gwStatusEqual now matches conditions by type instead of by index, the shape
fipStatusEqual already had. Ordering belongs to meta.SetStatusCondition now,
and a comparison that depended on it would report a spurious difference — and
so rewrite status forever — the day a condition is added or reordered.
Timestamps stay out of the comparison: they change on every rebuild, and the
added convergence test pins that a settled gateway performs no further write.

Assisted-By: Claude Opus 5 (1M context) <[email protected]>
Signed-off-by: Loïc Fontaine <[email protected]>

Completes the firewall story multi-attach starts. Off-VPC traffic from a VPC pod
is delivered to `gateways[vni]` with its destination intact, so whatever holds
that entry receives the VPC's egress by construction and can route it on. Until
now only cozyplane's own gateway pod could hold it — claimed by addGatewayLeg,
restricted to the agent's namespace and to the VPC's reserved .1 — so there was
no way to say "my appliance is the door", and the capability sat there unusable
by the workload it was for.

`VPCGateway.spec.appliance.podSelector` says it. The VPCGateway is already the
VPC's one declared boundary, so designating what serves as that boundary belongs
on it rather than on a new kind. The controller resolves the selector, finds the
selected workload's Port IN THIS VPC — a multi-attached appliance has one per VPC
and only the local leg can be its door — and moves spec.gateway onto it.
`desiredGateways` does the rest, unchanged.

Nothing else changes. No CNI change, no datapath change: the door was always a
Port flag, and this only decides which Port carries it.

Being the door is about RECEIVING, and it is deliberately not the same grant as
sending. Emitting a source you do not own stays VPCBinding.allowForwarding,
authored by whoever holds `export` on the VPC. A firewall wants both, and they
are granted by different people — which is the point.

Details that are easy to get wrong and are covered by tests:

  - re-pointing the appliance must MOVE the door. gateways[vni] holds one entry;
    leaving two Ports flagged makes the winner whichever agent resynced last.
  - dropping spec.appliance must release it, or the workload keeps receiving the
    VPC's egress after being told to stop.
  - a selector matching nothing is an ordinary state on the way up (the appliance
    may not be scheduled yet) and must report, not wedge the reconcile. Selecting
    a workload that is not ATTACHED to this VPC is a different mistake and says
    so, or the operator edits the wrong field.
  - GatewayReconciler spawns no gateway pod while an appliance is declared: two
    claimants would race for the same map entry.
  - the appliance's pod is normally replaced under it (a Deployment, a VM), so
    Ports and Pods are watched and the door follows.

Verified on a live 3-node cluster: two VPCs, an appliance with a leg in each,
declared the door of both. The controller picked the correct per-VPC leg for each
gateway, and the door followed a full replacement of the appliance pod.

Assisted-By: Claude Opus 5 (1M context) <[email protected]>
Signed-off-by: Loïc Fontaine <[email protected]>

…per binding

Implements lllamnyp#26 (design). Stacked on lllamnyp#25, which is stacked on lllamnyp#24 and lllamnyp#17.

Multi-attach gives a POD several VPCs. It cannot give a VM several VPCs: a
guest only sees NICs KubeVirt put in its domain, KubeVirt only declares NICs
named in spec.networks, and that field admits exactly `pod: {}` and `multus:
{}`. An interface added to a virt-launcher pod without a matching entry is
configured, addressed, and never bridged into the guest. Multus is not one way
to give a VM a second NIC — it is the only way.

Delegate mode, discriminated by `vpc` in the plugin config (a NAD's, never the
cluster conflist's, so the primary path is untouched):

- builds exactly ONE attachment, never primary, on args.IfName. cmdAdd
  otherwise ignores args.IfName and rebuilds the whole annotation list.
- cmdDel removes only its own interface and Port. The annotation path's DEL
  enumerates a name space and releases the pod's FabricIP claims; running that
  for a secondary NIC would take eth0 and the pod's underlay identity with it.
- host veths take a disjoint name space — a letter where the indexed form puts
  a digit. The primary's DEL finds links by RECONSTRUCTING every name in its
  space rather than by consulting state, so a shared space would let it name,
  and destroy, a delegated interface. Disjoint spaces make that impossible by
  construction rather than by care.
- the delegated Port carries sdn.cozystack.io/ifname. Selecting on {pod, VPC}
  alone would match both Ports of a VM holding two NICs on one VPC, so tearing
  down one would release the other's address while its interface is still up.
- LabelVMNIC's two value spaces stay disjoint for the same reason: a decimal
  index from the annotation path, an interface name from a delegate.

The NAD is generated per VPCBinding, in the binding's namespace. Multus
resolves networkName in the POD's namespace and a VPCBinding is exactly the
object saying "pods here may attach to that VPC", so the shim lands where
attachment is already authorized and adds NO new authorization surface. Owned
by the binding, so revocation GCs it. Read as unstructured and gated on
RESTMapping, like the persistent-Port controller's VMI.

It is not a grant, and that is tested: the CNI still resolves the VPCBinding,
so a hand-written NAD naming another namespace's VPC gets nothing.

Static addressing stays in sdn.cozystack.io/networks, in an entry naming the
Multus interface. net[0-9]+ becomes reserved — the primary invocation skips
those entries, the delegate looks up its own. Attachment indices are therefore
assigned over the SURVIVING entries, so a delegated entry listed first cannot
shift index 0 (the fabric handle, the default route, status.podIP) onto a leg
that is not primary.

Two fixes the delegate path makes reachable:

- addPodAddrRoute enabled IPv6 on a hardcoded "eth0" rather than on the link it
  was configuring, so a v6 secondary or delegated leg armed the wrong
  interface and left its own address tentative.
- a delegated DEL deletes the veth pair. The annotation path relies on the pod
  netns going away; a delegate can be DELed while the sandbox lives, and
  leaving the interface would fail an ADD retry on an existing name.

Validated on a 3-node kind cluster with Multus installed: NADs generated from
both bindings, a pod with eth0 in one VPC (annotation) and net1 in another
(NAD), the pinned address and MAC taken exactly, one default route, east-west
over the delegated NIC same-node and cross-node, a duplicate pinned address
refused, a hand-written NAD in an unbound namespace refused with
"no VPCBinding ... authorizes attaching to VPC (default-deny)" and no Port
claimed, and a hand-run delegated DEL removing net1 while eth0 kept routing.
test/vpc-e2e.sh gains that phase, skipped where Multus is absent.

Assisted-By: Claude Opus 5 (1M context) <[email protected]>
Signed-off-by: Loïc Fontaine <[email protected]>

Signed-off-by: Loïc Fontaine <[email protected]>