Skip to content

Commit ffe54ad

Browse files
fix(site-router): close five defects from an independent review pass
None of these overlap the previous round. DataVolume.spec is immutable in CDI, and the appliance digest moves on every release because the image package stays in the root Makefile build: list. So `helm upgrade` on an installed instance rendered a new source under the same `<release>-boot` name, which is an update to an immutable spec. Every release would have broken every installed site-router. The chart now re-emits the live DataVolume's spec verbatim, the way packages/apps/vm-disk already does for the same reason (#2985, docs/storage-immutability.md). Preserving is silent for the digest and loud for storageClass, deliberately. The digest is a platform stamp, so failing the render on it would swap a broken upgrade for a blocked one, and the lifecycle this chart documents already says a running gateway keeps the appliance it was created with. storageClass is operator input, so a change that would silently not happen fails instead — which also answers the immutability caveat raised against this field in the previous round. programNamespaceRoutes recorded the new gateway IP before writing the entries that IP claims. The routes annotation is shared across the namespace, and that record is the only way a later reconcile tells this gateway's entries from a co-tenant's; mergeRoutes needs both the current and previous IP to migrate them. A failure between the two writes left the old entry looking foreign, so mergeRoutes returned RouteConflict forever and reconcileDelete could not identify it either. The record now follows the write, which fails safe: a crash in the new window just migrates the same entries again. renderTunnelIngressFilter degraded an empty destination set to a source-only accept. A jumped-chain accept is a terminal verdict, so that forwards a valid-source packet to any destination including the internet, which is the exact hole the destination constraint exists to be. The guard against reaching it was a comment asserting the controller could never resolve an empty set on a pushing path, and it could: the set is this namespace's live Pod IPs and Service ClusterIPs, and in a namespace holding only the gateway, the tunnel Service disappearing between the LoadBalancer read and the Service list empties it. Empty now emits no new-flow accept at all, and the controller records a Warning so a tunnel that forwards nothing says why. build-vyos had no fork path. The registry login above it is correctly skipped on a fork, but nothing took its place, so PUSH stayed on and a fork PR touching the appliance pushed with a read-only token and took finalize and e2e down with it. It exports to an OCI archive like every other image job, under the pr-oci-* name e2e-fork.yaml globs for. The negative-security e2e asserted Cilium POLICY_DENIED for a foreign tenant pod and for the kubernetes API ClusterIP. Those comments describe rules admitting the whole pod and service CIDRs; the implementation admits one /32 per Pod IP and ClusterIP owned by this namespace, so both packets die at the guest and never reach the veth where Cilium enforces. The assertions failed because the isolation is stronger than they assumed. They assert the guest counter now, and hubble_dropped says why it has no caller left. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent b5c812f commit ffe54ad

11 files changed

Lines changed: 421 additions & 46 deletions

File tree

‎.github/workflows/pull-requests.yaml‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -610,13 +610,25 @@ jobs:
610610
env:
611611
DOCKER_CONFIG: ${{ runner.temp }}/.docker
612612
- name: Build VyOS router image (qcow2 + containerDisk)
613-
run: make -C packages/system/vyos-router-image image
613+
run: |
614+
mkdir -p _out/oci
615+
make -C packages/system/vyos-router-image image
614616
env:
615617
DOCKER_CONFIG: ${{ runner.temp }}/.docker
616618
BUILDER: ${{ steps.buildx.outputs.name }}
617619
IMAGE_TAG: pr-${{ github.event.pull_request.number }}-${{ github.event.pull_request.head.sha }}
618620
PUBLISH_VERSIONED: '0'
619621
PUBLISH_FLOATING: '0'
622+
# Same fork handling as the per-package build job, and it was missing
623+
# here. The login step above is correctly skipped on a fork, but nothing
624+
# took its place: PUSH stayed on, so a fork PR touching the appliance or
625+
# its consumers pushed with a read-only token, failed, and took finalize
626+
# and e2e down with it. Export to an OCI archive instead and let the
627+
# privileged e2e-fork.yaml workflow_run push it by digest. Empty on
628+
# same-repo PRs → normal push path; setting it also forces PUSH/LOAD off
629+
# in common-envs.mk (#3257). Absolute, because `make -C <pkg>` makes
630+
# buildx resolve a relative dest against the package directory.
631+
OCI_EXPORT_DIR: ${{ github.event.pull_request.head.repo.fork && format('{0}/_out/oci', github.workspace) || '' }}
620632
- name: Capture VyOS patch fragment
621633
run: |
622634
mkdir -p _out/fragments
@@ -631,6 +643,16 @@ jobs:
631643
name: pr-patch-fragment-vyos
632644
path: _out/fragments/vyos.patch
633645
if-no-files-found: error
646+
# Fork PR only: the containerDisk was exported rather than pushed
647+
# (OCI_EXPORT_DIR above). The name has to match e2e-fork.yaml's
648+
# `pattern: pr-oci-*` or the archive is built and then ignored.
649+
- name: Upload OCI image archives (fork PR)
650+
if: ${{ github.event.pull_request.head.repo.fork }}
651+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
652+
with:
653+
name: pr-oci-vyos
654+
path: _out/oci/*.oci.tar
655+
if-no-files-found: error
634656

635657
# Reassemble the unified pr.patch from every fragment, then build the installer
636658
# LAST: `make -C packages/core/installer image` runs `flux push artifact

‎hack/e2e-chainsaw/site-router/_probe-lib.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,16 @@ transfer_from_b() {
135135
}
136136

137137
# hubble_dropped <dst> <reason-substring>
138+
#
139+
# NO CURRENT CALLER, deliberately kept. The two cases that used it (a foreign
140+
# tenant pod, the kubernetes API ClusterIP) now assert the guest counter instead:
141+
# TUNNEL-INGRESS admits one /32 per Pod IP and Service ClusterIP owned by the
142+
# tenant namespace rather than the whole pod/service CIDRs, so both packets die
143+
# at the guest and never reach the pod veth where Cilium enforces. Asserting a
144+
# POLICY_DENIED record there failed BECAUSE the isolation is stronger than the
145+
# assertion assumed. Exercising the Cilium layer needs a probe originating inside
146+
# the pod network, which this suite does not have; this helper is what that probe
147+
# would use.
138148
# Assert a Cilium DROPPED flow toward <dst> matching <reason> from the
139149
# cilium-monitor capture taken over the probe window (POLICY_DENIED maps to
140150
# cilium-monitor's "Policy denied"). Hubble need not be enabled.

‎hack/e2e-chainsaw/site-router/chainsaw-test.yaml‎

Lines changed: 29 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -702,11 +702,12 @@ spec:
702702
-o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
703703
API_CLUSTERIP=$(kubectl -n default get svc kubernetes -o jsonpath='{.spec.clusterIP}')
704704
# A POD-NETWORK pod outside tenant-test (another tenant / a system
705-
# namespace). Must be pod-network (podIP != hostIP): a host-network
706-
# pod's IP is a node address, which the guest TUNNEL-INGRESS drops
707-
# before Cilium — this case must exercise the Cilium pod-identity
708-
# policy, so filter those out and take the first pod-network pod in
709-
# any other namespace.
705+
# namespace). Still filtered to podIP != hostIP, but no longer to
706+
# steer the drop to Cilium — the guest drops every foreign address
707+
# now (see case 2). The filter stays so the probe target is a real
708+
# cross-tenant pod address rather than a node address, which case 3
709+
# already covers separately and which would make this case a
710+
# duplicate rather than the cross-tenant one it is meant to be.
710711
OTHER_POD_IP=$(kubectl get pods -A \
711712
-o jsonpath='{range .items[*]}{.status.podIP}{" "}{.status.hostIP}{" "}{.metadata.namespace}{"\n"}{end}' \
712713
| awk -v ns="$NAMESPACE" '$1!="" && $1!=$2 && $3!=ns {print $1; exit}')
@@ -743,11 +744,23 @@ spec:
743744
"guest_counter_flat TUNNEL-INGRESS default-action" \
744745
tcp "$BACKEND_IP" 8080
745746
746-
# 2. Another tenant's pod → Cilium tenant isolation (POLICY_DENIED):
747-
# dest is in the pod CIDR so the guest dest-constraint admits it;
748-
# the pod-layer identity policy is what drops it.
747+
# 2. Another tenant's pod → guest TUNNEL-INGRESS dest-constraint.
748+
# This asserted a Cilium POLICY_DENIED record until the
749+
# destination set was narrowed: the guest admits one /32 per Pod
750+
# IP and Service ClusterIP owned by THIS namespace, not the whole
751+
# pod CIDR, so a foreign pod IP matches no accept and dies at the
752+
# guest default-action. It never reaches the pod veth, Hubble
753+
# never records it, and the old assertion failed precisely
754+
# BECAUSE the isolation is stronger than it assumed.
755+
#
756+
# What that costs: the tunnel can no longer exercise Cilium's
757+
# pod-identity policy as a second layer, because nothing survives
758+
# the guest to reach it. Cross-tenant isolation is asserted here
759+
# at the layer that actually enforces it. The Cilium layer needs
760+
# its own probe from inside the pod network, which is not this
761+
# suite.
749762
assert_dropped "another tenant's pod" \
750-
"hubble_dropped $OTHER_POD_IP POLICY_DENIED" \
763+
"guest_counter_incremented TUNNEL-INGRESS default-action" \
751764
tcp "$OTHER_POD_IP" 8080
752765
753766
# 3. Node address → guest TUNNEL-INGRESS dest-constraint (node/join
@@ -756,15 +769,14 @@ spec:
756769
"guest_counter_incremented TUNNEL-INGRESS default-action" \
757770
tcp "$NODE_IP" 10250
758771
759-
# 4. Kubernetes API ClusterIP → Cilium (reserved:kube-apiserver
760-
# identity denies the routed source). TUNNEL-INGRESS rule 20 admits
761-
# the service CIDR, so the guest forwards it; Cilium then denies it
762-
# at the kube-apiserver identity. The ClusterIP is DNAT'd to the
763-
# apiserver endpoint BEFORE the policy check, so the drop record
764-
# carries the "kube-apiserver" identity, not the ClusterIP — match
765-
# on that identity rather than $API_CLUSTERIP.
772+
# 4. Kubernetes API ClusterIP → guest TUNNEL-INGRESS dest-constraint,
773+
# for the same reason as case 2. The old comment described a rule
774+
# admitting the whole service CIDR; the implementation admits only
775+
# ClusterIPs owned by this namespace, and the kubernetes Service
776+
# lives in `default`. So the packet is dropped at the guest and
777+
# produces no Cilium record to match on.
766778
assert_dropped "kubernetes API" \
767-
"hubble_dropped kube-apiserver POLICY_DENIED" \
779+
"guest_counter_incremented TUNNEL-INGRESS default-action" \
768780
tcp "$API_CLUSTERIP" 443
769781
770782
# 5. 169.254.0.0/16 incl. metadata → guest TUNNEL-INGRESS

‎internal/controller/siterouter/reconciler.go‎

Lines changed: 30 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -578,22 +578,38 @@ func (r *SiteRouterReconciler) programNamespaceRoutes(ctx context.Context, inst
578578
}
579579
return fmt.Errorf("merge routes for namespace %s: %w", inst.namespace, err)
580580
}
581-
if err := r.rememberRouteGatewayIP(ctx, inst, gatewayIP); err != nil {
582-
return fmt.Errorf("remember route gateway IP: %w", err)
583-
}
584-
if ns.Annotations[routesAnnotation] == merged {
585-
return nil // already programmed; nothing to apply
581+
if ns.Annotations[routesAnnotation] != merged {
582+
apply := &corev1.Namespace{
583+
ObjectMeta: metav1.ObjectMeta{
584+
Name: inst.namespace,
585+
Annotations: map[string]string{routesAnnotation: merged},
586+
},
587+
}
588+
apply.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("Namespace"))
589+
if err := r.Patch(ctx, apply, client.Apply, client.FieldOwner(routesFieldOwner), client.ForceOwnership); err != nil {
590+
return fmt.Errorf("apply routes annotation on namespace %s: %w", inst.namespace, err)
591+
}
586592
}
587593

588-
apply := &corev1.Namespace{
589-
ObjectMeta: metav1.ObjectMeta{
590-
Name: inst.namespace,
591-
Annotations: map[string]string{routesAnnotation: merged},
592-
},
593-
}
594-
apply.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("Namespace"))
595-
if err := r.Patch(ctx, apply, client.Apply, client.FieldOwner(routesFieldOwner), client.ForceOwnership); err != nil {
596-
return fmt.Errorf("apply routes annotation on namespace %s: %w", inst.namespace, err)
594+
// Ownership is recorded only AFTER the entries it claims are in the
595+
// namespace, and the order is the whole point. This annotation is the only
596+
// way a later reconcile can tell which entries in a shared annotation are
597+
// this gateway's, and mergeRoutes needs BOTH the current and the previous IP
598+
// to migrate them when the gateway pod is replaced.
599+
//
600+
// Recording first loses that. Write the new IP, fail or crash before the
601+
// namespace patch, and the next reconcile reads the new IP as "previous":
602+
// the entry still keyed to the old one now looks like a co-tenant
603+
// site-router's, so mergeRoutes returns RouteConflict on every attempt and
604+
// reconcileDelete cannot identify it either. Nothing recovers that without
605+
// hand-editing the namespace.
606+
//
607+
// This order fails safe instead. A crash between the patch and this write
608+
// leaves the previous IP recorded, so the next reconcile migrates the same
609+
// entries again and converges — mergeRoutes is idempotent under a repeated
610+
// migration, which is what TestProgramNamespaceRoutes_GatewayIPChange covers.
611+
if err := r.rememberRouteGatewayIP(ctx, inst, gatewayIP); err != nil {
612+
return fmt.Errorf("remember route gateway IP: %w", err)
597613
}
598614
return nil
599615
}

‎internal/controller/siterouter/reconciler_test.go‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import (
2222
ctrl "sigs.k8s.io/controller-runtime"
2323
"sigs.k8s.io/controller-runtime/pkg/client"
2424
"sigs.k8s.io/controller-runtime/pkg/client/fake"
25+
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
2526
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
2627

2728
"github.com/cozystack/cozystack/internal/siterouter/denyset"
@@ -779,3 +780,129 @@ func TestClassify_RecordsEveryFailure(t *testing.T) {
779780
})
780781
}
781782
}
783+
784+
// TestProgramNamespaceRoutes_GatewayIPChange is the write-ordering regression.
785+
//
786+
// The routes annotation is shared by every site-router in the namespace, and the
787+
// only thing distinguishing this gateway's entries from a co-tenant's is the gw
788+
// address. When the gateway pod is replaced, mergeRoutes needs BOTH the new IP
789+
// and the previously recorded one to migrate those entries.
790+
//
791+
// Recording the new IP before writing the entries loses that. If the namespace
792+
// patch then fails, the next reconcile reads the new IP as "previous", the entry
793+
// still keyed to the old one looks like somebody else's, and mergeRoutes returns
794+
// RouteConflict forever with reconcileDelete unable to identify it either.
795+
// Nothing short of hand-editing the namespace recovers it.
796+
func TestProgramNamespaceRoutes_GatewayIPChange(t *testing.T) {
797+
newFixture := func(t *testing.T, failPatch bool) (*SiteRouterReconciler, *instance) {
798+
t.Helper()
799+
hr := siteRouterHRWithValues(t, "demo", map[string]interface{}{
800+
"remoteCIDRs": []interface{}{"172.31.0.0/16"},
801+
})
802+
hr.Annotations = map[string]string{routeGatewayIPAnnotation: "10.244.0.5"}
803+
ns := &corev1.Namespace{
804+
ObjectMeta: metav1.ObjectMeta{
805+
Name: "tenant-test",
806+
Annotations: map[string]string{routesAnnotation: `[{"dst":"172.31.0.0/16","gw":"10.244.0.5"}]`},
807+
},
808+
}
809+
// The replacement gateway pod, on a different IP.
810+
gateway := gwPod("virt-launcher-"+releasePrefix+"demo-fghij", "demo", "10.244.0.6")
811+
812+
scheme := runtime.NewScheme()
813+
if err := clientgoscheme.AddToScheme(scheme); err != nil {
814+
t.Fatalf("add client-go scheme: %v", err)
815+
}
816+
if err := helmv2.AddToScheme(scheme); err != nil {
817+
t.Fatalf("add helm-controller scheme: %v", err)
818+
}
819+
b := fake.NewClientBuilder().WithScheme(scheme).WithObjects(hr, ns, gateway, cozystackConfigMap())
820+
if failPatch {
821+
b = b.WithInterceptorFuncs(interceptor.Funcs{
822+
Patch: func(ctx context.Context, c client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error {
823+
if _, isNS := obj.(*corev1.Namespace); isNS {
824+
return errors.New("simulated apiserver failure writing the namespace annotation")
825+
}
826+
return c.Patch(ctx, obj, patch, opts...)
827+
},
828+
})
829+
}
830+
r := &SiteRouterReconciler{Client: b.Build(), Scheme: scheme, ManagementCIDR: "10.244.0.0/16"}
831+
832+
live := &helmv2.HelmRelease{}
833+
if err := r.Get(context.Background(), types.NamespacedName{Namespace: hr.Namespace, Name: hr.Name}, live); err != nil {
834+
t.Fatalf("get HelmRelease: %v", err)
835+
}
836+
values, err := decodeValues(live)
837+
if err != nil {
838+
t.Fatalf("decode values: %v", err)
839+
}
840+
return r, &instance{hr: live, name: "demo", namespace: "tenant-test", values: values, gatewayPod: gateway}
841+
}
842+
843+
t.Run("a failed namespace write leaves the previous owner recorded", func(t *testing.T) {
844+
r, inst := newFixture(t, true)
845+
846+
if err := r.programNamespaceRoutes(context.Background(), inst); err == nil {
847+
t.Fatal("expected the namespace patch failure to propagate")
848+
}
849+
850+
hr := &helmv2.HelmRelease{}
851+
if err := r.Get(context.Background(), types.NamespacedName{Namespace: "tenant-test", Name: releasePrefix + "demo"}, hr); err != nil {
852+
t.Fatalf("get HelmRelease: %v", err)
853+
}
854+
if got := hr.Annotations[routeGatewayIPAnnotation]; got != "10.244.0.5" {
855+
t.Fatalf("route owner = %q, want the PREVIOUS gateway IP 10.244.0.5 still recorded; "+
856+
"recording %q before the entries it claims strands the old entry permanently", got, got)
857+
}
858+
})
859+
860+
t.Run("the migration converges when the namespace write succeeds", func(t *testing.T) {
861+
r, inst := newFixture(t, false)
862+
863+
if err := r.programNamespaceRoutes(context.Background(), inst); err != nil {
864+
t.Fatalf("program routes: %v", err)
865+
}
866+
867+
ns := &corev1.Namespace{}
868+
if err := r.Get(context.Background(), types.NamespacedName{Name: "tenant-test"}, ns); err != nil {
869+
t.Fatalf("get namespace: %v", err)
870+
}
871+
if got := routeSet(t, ns.Annotations[routesAnnotation])["172.31.0.0/16"]; got != "10.244.0.6" {
872+
t.Errorf("route should have migrated to the replacement gateway, got gw %q", got)
873+
}
874+
875+
hr := &helmv2.HelmRelease{}
876+
if err := r.Get(context.Background(), types.NamespacedName{Namespace: "tenant-test", Name: releasePrefix + "demo"}, hr); err != nil {
877+
t.Fatalf("get HelmRelease: %v", err)
878+
}
879+
if got := hr.Annotations[routeGatewayIPAnnotation]; got != "10.244.0.6" {
880+
t.Errorf("route owner = %q, want the new gateway IP 10.244.0.6 once the entries carry it", got)
881+
}
882+
})
883+
884+
t.Run("a repeated migration is idempotent, so a crash before the owner write converges", func(t *testing.T) {
885+
// The state the first subtest leaves behind: entries already migrated,
886+
// owner still recording the old IP. Running again must not conflict.
887+
r, inst := newFixture(t, false)
888+
if err := r.programNamespaceRoutes(context.Background(), inst); err != nil {
889+
t.Fatalf("first pass: %v", err)
890+
}
891+
// Put the owner annotation back to the old IP, as a crash between the two
892+
// writes would leave it, and reconcile again.
893+
inst.hr.Annotations[routeGatewayIPAnnotation] = "10.244.0.5"
894+
if err := r.Update(context.Background(), inst.hr); err != nil {
895+
t.Fatalf("rewind owner annotation: %v", err)
896+
}
897+
if err := r.programNamespaceRoutes(context.Background(), inst); err != nil {
898+
t.Fatalf("second pass must converge rather than conflict: %v", err)
899+
}
900+
ns := &corev1.Namespace{}
901+
if err := r.Get(context.Background(), types.NamespacedName{Name: "tenant-test"}, ns); err != nil {
902+
t.Fatalf("get namespace: %v", err)
903+
}
904+
if got := routeSet(t, ns.Annotations[routesAnnotation])["172.31.0.0/16"]; got != "10.244.0.6" {
905+
t.Errorf("route should still point at the replacement gateway, got gw %q", got)
906+
}
907+
})
908+
}

‎internal/controller/siterouter/vyospush.go‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,10 @@ const (
9898
// certificate to pin, so the channel that carries the token is not verified.
9999
reasonAPITLSUnverified = "APITLSUnverified"
100100

101+
// reasonNoTunnelDestinations marks an instance whose destination set resolved
102+
// empty, which now means the guest accepts no new decrypted flows at all.
103+
reasonNoTunnelDestinations = "NoTunnelDestinations"
104+
101105
// reasonTunnelNotConfigured marks an instance that carries no usable tunnel
102106
// because peer.address or the PSK resolved empty. The schema requires
103107
// peer.address, but `required` only means the key is present and its default
@@ -593,6 +597,20 @@ func (r *SiteRouterReconciler) resolveInputs(ctx context.Context, inst *instance
593597
if err != nil {
594598
return render.Inputs{}, err
595599
}
600+
// The renderer now refuses to emit any new-flow accept without destinations,
601+
// which is the right failure but an invisible one: the tunnel comes up and
602+
// carries only return traffic. The set is supposed to be non-empty on every
603+
// pushing path (the chart's LoadBalancer Service always holds a ClusterIP in
604+
// this namespace), so reaching this means that invariant broke — say which
605+
// way rather than leaving a tunnel that mysteriously forwards nothing.
606+
if len(tenantCIDRs) == 0 && len(remoteCIDRs) > 0 {
607+
log.FromContext(ctx).Info("no tunnel destinations resolved; the guest will accept no new decrypted flows",
608+
"instance", inst.name, "namespace", inst.namespace)
609+
if r.Recorder != nil {
610+
r.Recorder.Event(inst.hr, corev1.EventTypeWarning, reasonNoTunnelDestinations,
611+
"no Pod IP or Service ClusterIP resolved in this namespace, so the gateway accepts no new decrypted flows; this clears itself once the tunnel Service and the tenant's workloads are present")
612+
}
613+
}
596614

597615
in := render.Inputs{
598616
ManagementCIDR: r.ManagementCIDR,

0 commit comments

Comments
 (0)