Skip to content

Nested tenants (depth > 1) cannot reach root tenant ingress due to incomplete egress policy #2171

Closed

Description

@mvgijssel

Problem

Tenants nested more than one level deep cannot access services in the root tenant (e.g., the root ingress controller). Traffic to the root ingress LoadBalancer IP times out because the egress policy only allows access to the immediate parent's services, not the full ancestry chain.

Example Hierarchy

tenant-root (has ingress at 192.168.50.100)
    └── tenant-prod (can reach tenant-root ingress ✅)
            └── tenant-prod-argocd (cannot reach tenant-root ingress ❌)

Symptoms

Pods in tenant-prod-argocd trying to curl api.enigma.vgijssel.nl (which resolves to tenant-root's ingress IP) timeout:

* Host api.enigma.vgijssel.nl:443 was resolved.
* IPv4: 192.168.50.100
*   Trying 192.168.50.100:443...
[timeout]

Meanwhile, the same curl from tenant-prod or tenant-root succeeds.

Root Cause

The CiliumClusterwideNetworkPolicy template in packages/apps/tenant/templates/networkpolicy.yaml (lines ~56-68) generates egress rules by splitting .Release.Namespace:

{{- $parts := splitList "-" .Release.Namespace }}
{{- range $i, $v := $parts }}
{{- if ne $i 0 }}
- matchLabels:
    cozystack.io/service: ingress
    "k8s:io.kubernetes.pod.namespace": {{ join "-" (slice $parts 0 (add $i 1)) }}
{{- end }}
{{- end }}

For tenant-prod-argocd (deployed in namespace tenant-prod):

  • .Release.Namespace = "tenant-prod"
  • $parts = ["tenant", "prod"]
  • Generated egress: only tenant-prod
  • Missing: tenant-root

The template assumes the namespace name encodes the full hierarchy, but tenant-prod doesn't contain information about its parent tenant-root.

Additional Evidence

The namespace labels also only include immediate parent, not full ancestry:

# tenant-prod labels (correct - has both)
tenant.cozystack.io/tenant-prod: ""
tenant.cozystack.io/tenant-root: ""

# tenant-prod-argocd labels (missing tenant-root)
tenant.cozystack.io/tenant-prod: ""
tenant.cozystack.io/tenant-prod-argocd: ""
# ❌ Missing: tenant.cozystack.io/tenant-root: ""

Related Issue

This is the same class of bug as #1970 (nested Fluent Bit cannot send logs to parent VLogs), where the egress policy is missing rules for parent services.

Suggested Fix

The namespace labels already track ancestry via tenant.cozystack.io/* labels. The egress policy template could iterate through all tenant.cozystack.io/* labels on the parent namespace instead of trying to parse the namespace name.

Alternatively, ensure child namespaces inherit all tenant.cozystack.io/* labels from their entire ancestry chain, then use those labels to generate egress rules.

Workaround

Manually add a CiliumNetworkPolicy in the affected namespace:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-to-root-ingress
  namespace: tenant-prod-argocd
spec:
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            cozystack.io/service: ingress
            k8s:io.kubernetes.pod.namespace: tenant-root

Environment

  • Cozystack version: 1.0.2

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    on Mar 7, 2026
  2. added
    area/networkingIssues or PRs related to networking (ingress, gateway, vpn, metallb, kube-ovn)
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    triage/acceptedIndicates an issue is ready to be actively worked on
    on Apr 27, 2026
  3. added a commit that references this issue on Jul 3, 2026
    1baf126
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/networkingIssues or PRs related to networking (ingress, gateway, vpn, metallb, kube-ovn)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions