Skip to content

Commit cdff5ed

Browse files
mattia-eleuterimattia-eleuteri
committed
feat(monitoring): route tenant logs to the tenant's own VictoriaLogs
A tenant that runs its own monitoring stack gets a VictoriaLogs instance, but the node fluent-bit sends every container log to global.target, so that instance stays empty. The tenant chart already names the stack a namespace reports to in namespace.cozystack.io/monitoring, the label the per-tenant vmagent selects on, so the log path keys off the same label. The routed record leaves the default outputs (keep false): a tenant with its own monitoring holds its logs, the root store no longer does. The re-tagged record re-enters the filter chain, where the Match * "Add tenant" cannot overwrite the key, so a per-target Set labels it with the tenant it is sent to. One rewrite_tag carries every rule, keeping a single emitter however many tenants are routed. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 0a2054e commit cdff5ed

4 files changed

Lines changed: 200 additions & 0 deletions

File tree

‎packages/system/monitoring-agents/templates/_tenant-log-routes.tpl‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{{- /*
2+
Tenant log routes for the node fluent-bit, as JSON {"<target>": ["<namespace>", ...]}.
3+
4+
The tenant chart sets namespace.cozystack.io/monitoring on every tenant
5+
namespace to the nearest tenant (itself or an ancestor) that runs its own
6+
monitoring stack, so the label names the VictoriaLogs a namespace's logs
7+
belong to. A namespace is routed when that label is set and names a tenant
8+
other than global.target; an unlabelled namespace, an empty label, or one
9+
pointing at global.target stays on the default outputs.
10+
11+
lookup results are not part of the digest helm-controller compares, so a
12+
namespace gaining, changing or losing the label does not re-render this
13+
release by itself. The tenantlogrouting reconciler in cozystack-controller
14+
(internal/controller/tenantlogrouting) forces an upgrade of this release
15+
whenever any namespace's label changes, so a new selection here that reads
16+
anything beyond that label must be taught to the reconciler too. Offline
17+
renders (helm template, dry-run) get no namespaces back from lookup and route
18+
nothing.
19+
*/ -}}
20+
{{- define "monitoring-agents.tenantLogRoutes" -}}
21+
{{- $routes := dict -}}
22+
{{- $namespaces := lookup "v1" "Namespace" "" "" -}}
23+
{{- range $ns := ($namespaces.items | default list) -}}
24+
{{- $target := index ($ns.metadata.labels | default dict) "namespace.cozystack.io/monitoring" | default "" -}}
25+
{{- if and $target (ne $target $.Values.global.target) -}}
26+
{{- $_ := set $routes $target (append (get $routes $target | default list) $ns.metadata.name) -}}
27+
{{- end -}}
28+
{{- end -}}
29+
{{- toJson $routes -}}
30+
{{- end -}}

‎packages/system/monitoring-agents/tests/fluent-bit_tenant_log_routing_offline_test.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
suite: fluent-bit routes no tenant logs when lookup sees no cluster
2+
3+
# No kubernetesProvider: lookup returns nothing, as it does under helm template
4+
# and dry-run. The config must then carry no routing at all, rather than a
5+
# rewrite_tag with no Rule or an output with an empty host.
6+
templates:
7+
- charts/fluent-bit/templates/configmap.yaml
8+
release:
9+
name: monitoring-agents
10+
namespace: cozy-monitoring
11+
set:
12+
global:
13+
target: tenant-root
14+
tests:
15+
- it: renders no rewrite_tag filter, tenant output or tenant label override
16+
asserts:
17+
- notMatchRegex:
18+
path: data["fluent-bit.conf"]
19+
pattern: 'rewrite_tag|Match tenant\.|Set tenant'

‎packages/system/monitoring-agents/tests/fluent-bit_tenant_log_routing_test.yaml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
suite: fluent-bit routes tenant logs by the namespace monitoring label
2+
3+
# The node fluent-bit sends a namespace's container logs to the VictoriaLogs
4+
# named by its namespace.cozystack.io/monitoring label, when that label names a
5+
# tenant other than global.target. These cases pin which namespaces get a rule
6+
# and an output, which are skipped, and that a routed record is labelled with
7+
# the tenant it lands in rather than with global.target.
8+
templates:
9+
- charts/fluent-bit/templates/configmap.yaml
10+
release:
11+
name: monitoring-agents
12+
namespace: cozy-monitoring
13+
set:
14+
global:
15+
target: tenant-root
16+
kubernetesProvider:
17+
scheme:
18+
"v1/Namespace":
19+
gvr:
20+
version: v1
21+
resource: namespaces
22+
namespaced: false
23+
objects:
24+
- kind: Namespace
25+
apiVersion: v1
26+
metadata:
27+
name: cozy-system
28+
- kind: Namespace
29+
apiVersion: v1
30+
metadata:
31+
name: tenant-empty
32+
labels:
33+
namespace.cozystack.io/monitoring: ""
34+
- kind: Namespace
35+
apiVersion: v1
36+
metadata:
37+
name: tenant-root
38+
labels:
39+
namespace.cozystack.io/monitoring: tenant-root
40+
- kind: Namespace
41+
apiVersion: v1
42+
metadata:
43+
name: tenant-inherits-root
44+
labels:
45+
namespace.cozystack.io/monitoring: tenant-root
46+
- kind: Namespace
47+
apiVersion: v1
48+
metadata:
49+
name: tenant-a
50+
labels:
51+
namespace.cozystack.io/monitoring: tenant-a
52+
- kind: Namespace
53+
apiVersion: v1
54+
metadata:
55+
name: tenant-a-child
56+
labels:
57+
namespace.cozystack.io/monitoring: tenant-a
58+
- kind: Namespace
59+
apiVersion: v1
60+
metadata:
61+
name: tenant-b
62+
labels:
63+
namespace.cozystack.io/monitoring: tenant-b
64+
tests:
65+
- it: routes every namespace of a monitored tenant through one rewrite_tag filter
66+
asserts:
67+
- matchRegex:
68+
path: data["fluent-bit.conf"]
69+
pattern: 'Name rewrite_tag\n\s+Match kube\.\*\n\s+Rule \$kubernetes_namespace_name \^\(tenant-a\|tenant-a-child\)\$ tenant\.tenant-a\.\$TAG false\n\s+Rule \$kubernetes_namespace_name \^\(tenant-b\)\$ tenant\.tenant-b\.\$TAG false\n'
70+
71+
- it: writes one output per monitored tenant, to that tenant's vlinsert
72+
asserts:
73+
- matchRegex:
74+
path: data["fluent-bit.conf"]
75+
pattern: 'Match tenant\.tenant-a\.\*\n\s+Host vlinsert-generic\.tenant-a\.svc'
76+
- matchRegex:
77+
path: data["fluent-bit.conf"]
78+
pattern: 'Match tenant\.tenant-b\.\*\n\s+Host vlinsert-generic\.tenant-b\.svc'
79+
80+
- it: labels a routed record with the tenant it is sent to
81+
asserts:
82+
- matchRegex:
83+
path: data["fluent-bit.conf"]
84+
pattern: 'Match tenant\.tenant-a\.\*\n\s+Set tenant tenant-a\n'
85+
- matchRegex:
86+
path: data["fluent-bit.conf"]
87+
pattern: 'Match tenant\.tenant-b\.\*\n\s+Set tenant tenant-b\n'
88+
89+
- it: leaves unlabelled, empty-labelled and global.target namespaces on the default outputs
90+
asserts:
91+
- notMatchRegex:
92+
path: data["fluent-bit.conf"]
93+
pattern: 'cozy-system|tenant-empty|tenant-inherits-root'
94+
- notMatchRegex:
95+
path: data["fluent-bit.conf"]
96+
pattern: 'tenant\.tenant-root\.|\(tenant-root\)|Host vlinsert-generic\.\.svc|tenant\.\.'
97+
98+
- it: keeps global.target on the default outputs
99+
asserts:
100+
- matchRegex:
101+
path: data["fluent-bit.conf"]
102+
pattern: 'Match kube\.\*\n\s+Host vlinsert-generic\.tenant-root\.svc'
103+
104+
- it: follows global.target when it names a tenant that is otherwise routed
105+
set:
106+
global:
107+
target: tenant-a
108+
asserts:
109+
- notMatchRegex:
110+
path: data["fluent-bit.conf"]
111+
pattern: 'tenant\.tenant-a\.'
112+
- matchRegex:
113+
path: data["fluent-bit.conf"]
114+
pattern: 'Rule \$kubernetes_namespace_name \^\(tenant-inherits-root\|tenant-root\)\$ tenant\.tenant-root\.\$TAG false'
115+

‎packages/system/monitoring-agents/values.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -408,6 +408,19 @@ fluent-bit:
408408
json_date_format iso8601
409409
header AccountID 0
410410
header ProjectID 0
411+
{{- range $target, $_ := include "monitoring-agents.tenantLogRoutes" . | fromJson }}
412+
[OUTPUT]
413+
Name http
414+
Match tenant.{{ $target }}.*
415+
Host vlinsert-generic.{{ $target }}.svc
416+
port 9481
417+
compress gzip
418+
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date
419+
format json_lines
420+
json_date_format iso8601
421+
header AccountID 0
422+
header ProjectID 0
423+
{{- end }}
411424
filters: |
412425
[FILTER]
413426
Name kubernetes
@@ -473,6 +486,29 @@ fluent-bit:
473486
Rename keep_kubernetes_pod_name kubernetes_pod_name
474487
Rename keep_kubernetes_namespace_name kubernetes_namespace_name
475488
Rename keep_kubernetes_container_name kubernetes_container_name
489+
{{- /*
490+
Routed records are re-tagged with keep false: they leave the default
491+
outputs, so a tenant that runs its own monitoring holds its logs and the
492+
root store no longer does. rewrite_tag sits last so the re-tagged record
493+
already carries the kube.* processing above; the record then re-enters
494+
the chain under its new tag, where the Match * "Add tenant" is a no-op
495+
(Add never overwrites), hence the per-target Set.
496+
*/}}
497+
{{- $routes := include "monitoring-agents.tenantLogRoutes" . | fromJson }}
498+
{{- if $routes }}
499+
[FILTER]
500+
Name rewrite_tag
501+
Match kube.*
502+
{{- range $target, $namespaces := $routes }}
503+
Rule $kubernetes_namespace_name ^({{ $namespaces | sortAlpha | join "|" }})$ tenant.{{ $target }}.$TAG false
504+
{{- end }}
505+
{{- range $target, $_ := $routes }}
506+
[FILTER]
507+
Name modify
508+
Match tenant.{{ $target }}.*
509+
Set tenant {{ $target }}
510+
{{- end }}
511+
{{- end }}
476512
scrapeRules:
477513
etcd:
478514
enabled: false

0 commit comments

Comments
 (0)