Repository navigation
Nested cluster Fluent Bit cannot send logs to parent VLogs: missing vlogs in tenant egress policy #1970
Description
Problem
Fluent Bit in nested Kubernetes clusters (Kamaji) cannot send logs to VLogs in the parent cluster. Connection attempts timeout:
[error] [upstream] connection #145 to tcp://10.43.81.27:9428 timed out after 10 seconds
[error] [output:http:http.0] no upstream connections available to vlogs-generic.tenant-root.svc:9428
Root Cause
The CiliumClusterwideNetworkPolicy for tenant egress (<tenant>-egress) explicitly allows traffic to vminsert in parent tenants for metrics, but does not include vlogs for logs.
In packages/apps/tenant/templates/networkpolicy.yaml (lines 43-55):
{{- if ne (include "tenant.name" .) "tenant-root" }}
- toEndpoints:
{{- if hasPrefix "tenant-" .Release.Namespace }}
{{- $parts := splitList "-" .Release.Namespace }}
{{- range $i, $v := $parts }}
{{- if ne $i 0 }}
- matchLabels:
"k8s:app.kubernetes.io/name": "vminsert" # ← metrics work
"k8s:io.kubernetes.pod.namespace": {{ join "-" (slice $parts 0 (add $i 1)) }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}There is no equivalent rule for vlogs (app.kubernetes.io/name: vlogs), so Cilium blocks the traffic.
Impact
- Metrics from nested clusters → parent cluster: ✅ Works (vminsert allowed)
- Logs from nested clusters → parent cluster: ❌ Blocked (vlogs not allowed)
Fix
Add vlogs to the tenant egress policy, mirroring the vminsert rule.
Activity
[-]Fluent Bit in nested Kubernetes cluster cannot send logs to VLogs in parent cluster[/-][+]Nested cluster Fluent Bit cannot send logs to parent VLogs: missing vlogs in tenant egress policy[/+]fix(tenant): allow egress to vlogs in parent tenants
dosubot commentedon Feb 3, 2026
Hi Aleksei Sviridkin (@lexfrei)! I'm Dosu and I’m helping the cozystack team.
Your root cause analysis is spot on: the Fluent Bit output config in the monitoring-agents Helm chart uses
Host vlogs-generic.tenant-root.svcwithout the.cozy.localsuffix, which breaks cross-cluster service discovery for nested clusters. There’s no built-in logic to append.cozy.localautomatically, so you’ll need to override the output host in your Helm values for nested clusters to use the full FQDN, likeHost vlogs-generic.tenant-root.svc.cozy.local—this will be rendered into the ConfigMap by the chart and is compatible with Flux reconciliation as long as you manage it via values files or overlays, not manual patches (source).For the network side, there’s currently no Cilium policy in the repo that allows ingress to VLogs on port 9428 from tenant workloads. You’ll want to add a
CiliumNetworkPolicy(or clusterwide policy) that selects the VLogs pods intenant-rootand allows ingress on TCP/9428 from the identities/labels representing your Kamaji tenant workloads. Place this manifest in thepackages/system/cilium-networkpolicyHelm chart so it’s managed by Flux and reconciles automatically (source).If you need a concrete example or want to discuss label selectors for the policy, let me know! For more background, see cozystack/cozystack#938 and cozystack/cozystack#1221.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
fix(tenant): allow egress to vlogs in parent tenants