gateway: a second tenant Gateway steals the platform namespaces from the first #3869

The platform-wide gateway.attachedNamespaces list is rendered into every tenant's TenantGateway, not only the publishing one, and the controller patches the attach label onto each of those namespaces without checking whether another TenantGateway already owns it. With two tenants owning Gateways, each reconcile flips the label and the platform routes detach from whichever Gateway lost the last write.

Mechanism

Two independent halves. Line references are where each sits on main; the properties, not the lines, are what this issue is about, and the reproduction below checks the properties directly.

The list is a single cluster-scoped value with no per-tenant filter. packages/core/platform/templates/apps.yaml:127 writes gateway.attachedNamespaces into the shared _cluster dict as a comma-joined gateway-attached-namespaces string, defaulting from packages/core/platform/values.yaml:491-501 (cozy-cert-manager, cozy-dashboard, cozy-keycloak, cozy-system, cozy-harbor, cozy-bucket, cozy-kubevirt, cozy-kubevirt-cdi, cozy-monitoring). Every tenant's gateway module reads that same dict, and packages/extra/gateway/templates/tenantgateway.yaml copies the whole string into spec.attachedNamespaces with no condition on which tenant is rendering (the split at :35-41 and the emission at :94-99). The invariant to check is the rendered output, not those lines: rendering the module for any tenant namespace should not produce another tenant's platform namespaces.

The controller takes the label without checking who holds it. ensureNamespaceLabels (internal/controller/tenantgateway/reconciler.go:978-1012) walks tgw.Spec.AttachedNamespaces and calls patchNamespaceGatewayLabel for each. That function sets the label with no comparison against an existing value (reconciler.go:1027):

ns.Labels[namespaceGatewayLabel] = ownerNamespace

and overwrites the ownership annotation the same way at :1031.

The garbage-collect pass immediately below (:992-1010) does check ownership, but only in the other direction: it skips a namespace whose cozystack.io/gateway-attached-by annotation names a different TenantGateway, so it will not remove a foreign label. It has no bearing on the patch loop that already overwrote it.

The label is what the port-443 listeners select on (internal/controller/tenantgateway/renderers.go:106-118) and what collectHostnameClaims uses to decide which namespaces may attach (reconciler.go:300-307). So flipping it moves the platform namespaces wholesale from one Gateway to the other.

Reproducing it

Render the tenant gateway module for a tenant that is not the publishing one:

helm template cozystack packages/extra/gateway --namespace tenant-alice \
  --set gatewayClassName=cilium \
  --set-json '_namespace={"host":"alice.example.com"}' \
  --set-json '_cluster={"gateway-attached-namespaces":"cozy-dashboard,cozy-keycloak,cozy-system"}' \
  --show-only templates/tenantgateway.yaml

The result carries apex: "alice.example.com" and the full platform list:

spec:
  apex: "alice.example.com"
  attachedNamespaces:
    - "cozy-dashboard"
    - "cozy-keycloak"
    - "cozy-system"

On a cluster, set gateway: true on a second tenant alongside the root tenant, then watch the label:

kubectl get namespace cozy-dashboard --output jsonpath='{.metadata.labels.namespace\.cozystack\.io/gateway}{"\n"}{.metadata.annotations.cozystack\.io/gateway-attached-by}{"\n"}'

It alternates between the two tenant namespaces as the two controllers reconcile. The dashboard HTTPRoute's .status.parents follows it, reporting a namespace-selector refusal against whichever Gateway currently does not hold the label.

Impact

Requires an explicit opt-in on a second tenant — tenant.spec.gateway: true, which is off by default and never auto-enabled — so a default installation is unaffected, and so is the common shape where only the root tenant publishes. That is the reason this is not urgent.

It is also the reason it is bad when it happens: the operator enabling a Gateway on a second tenant is doing exactly the supported thing, and the failure lands on the first tenant's platform services rather than on the tenant that changed. The symptom is a dashboard, Keycloak or Harbor route reporting a namespace-selector refusal on a Gateway that was working, with no change to that tenant, and it oscillates rather than settling — which makes it read as a Cilium or Gateway API problem rather than an ownership collision. The comment at packages/core/platform/values.yaml:485-487 states the opposite guarantee ("Tenants not on this list cannot attach a route to somebody else's Gateway, which prevents hostname hijacking across tenants"), which is true of the list's intent and false of how it is distributed.

Fix shape

The chart and the controller each need one half, and either alone leaves a hole.

Chart side: spec.attachedNamespaces should be emitted only for the tenant that actually publishes the platform services, whatever shape the template has by then. The discriminator the rest of the platform already uses is publishing.ingressName (default tenant-root), so the cluster-wide list should reach a TenantGateway only when the rendering namespace matches it, and any other tenant should render an empty list. A tenant that genuinely needs extra attached namespaces should get them from a per-tenant value rather than from the cluster-wide one. The test that fixes the shape is a render assertion — for a non-publishing namespace, spec.attachedNamespaces is absent — which stays valid however the template is restructured.

Controller side: make the label write refuse a takeover instead of performing one. patchNamespaceGatewayLabel should leave the namespace alone and surface a condition when the existing cozystack.io/gateway-attached-by annotation names a different TenantGateway, matching the takeover-guard contract already used for the Gateway, the Issuer and the redirect route (reconciler.go:641-650 is the pattern). That turns a silent oscillation into a legible "namespace already attached to another Gateway" on the second tenant's TenantGateway status.

Both halves are needed. Fixing only the chart leaves the controller willing to take a label from another Gateway the moment two TenantGateways ever name the same namespace; fixing only the controller leaves every non-publishing tenant emitting a claim that now fails loudly instead of silently.