A tenant two levels below root can't reach the etcd it inherits from tenant-root. Its tenant egress policy allows etcd only in its direct parent's namespace.
packages/apps/tenant/templates/networkpolicy.yaml builds the list of ancestor namespaces by splitting .Release.Namespace on -. That is the parent's namespace. Children of root are named tenant-<name>, not tenant-root-<name>, so for tenant-test-cplane, whose release lives in tenant-test, the list is just tenant-test and never includes tenant-root. The namespace still carries namespace.cozystack.io/etcd: tenant-root, so a managed Kubernetes cluster in that tenant points its apiserver at root's etcd and gets dropped.
Seen in e2e run 37010089138 (#4713). The tenant apiserver logs dial tcp 10.244.0.134:2379: i/o timeout (also .132 and .136, the tenant-root/etcd-* pods), then Error creating leases: error creating storage factory: context deadline exceeded, and the control plane never comes up. The live tenant-test-cplane-egress allows etcd in tenant-test only. Root's own tenant-root-ingress-etcd would accept the traffic. Clusters in a depth-1 tenant, tenant-test, pass in the same run.
The vminsert and ingress rules in the same template use the same name-splitting. #2171 (ingress at depth 2) is closed, and #2912 added the full ancestor label chain to tenant namespaces. Deriving the ancestors from those tenant.cozystack.io/* labels, or from _namespace, instead of from the name would cover all three rules.
A tenant two levels below root can't reach the etcd it inherits from
tenant-root. Its tenant egress policy allows etcd only in its direct parent's namespace.packages/apps/tenant/templates/networkpolicy.yamlbuilds the list of ancestor namespaces by splitting.Release.Namespaceon-. That is the parent's namespace. Children of root are namedtenant-<name>, nottenant-root-<name>, so fortenant-test-cplane, whose release lives intenant-test, the list is justtenant-testand never includestenant-root. The namespace still carriesnamespace.cozystack.io/etcd: tenant-root, so a managed Kubernetes cluster in that tenant points its apiserver at root's etcd and gets dropped.Seen in e2e run 37010089138 (#4713). The tenant apiserver logs
dial tcp 10.244.0.134:2379: i/o timeout(also .132 and .136, thetenant-root/etcd-*pods), thenError creating leases: error creating storage factory: context deadline exceeded, and the control plane never comes up. The livetenant-test-cplane-egressallows etcd intenant-testonly. Root's owntenant-root-ingress-etcdwould accept the traffic. Clusters in a depth-1 tenant,tenant-test, pass in the same run.The
vminsertand ingress rules in the same template use the same name-splitting. #2171 (ingress at depth 2) is closed, and #2912 added the full ancestor label chain to tenant namespaces. Deriving the ancestors from thosetenant.cozystack.io/*labels, or from_namespace, instead of from the name would cover all three rules.