feat(monitoring): switch VLogs to cluster mode for secure tenant log isolation #2000

Problem

PR #1971 proposes adding NetworkPolicy egress to VLogs pods so that nested cluster Fluent Bit instances can forward logs to the parent tenant's VLogs. However, VLogs in single-node mode exposes both write (/insert/*) and read (/select/*) endpoints on the same port (9428). If we open NetworkPolicy egress to VLogs, a child tenant would be able to query /select/* and read ALL logs stored in the parent tenant's VLogs — including logs from other tenants.

This is why PR #1971 cannot be merged as-is (see kvaps' comment about the security concern).

Note on metrics: vminsert is architecturally write-only — it has no query endpoints. The existing NetworkPolicy egress to vminsert is secure. This issue is VLogs-specific.

Current State

• VLogs is deployed via the deprecated VLogs CRD (operator.victoriametrics.com/v1beta1) in single-node mode
• victoria-metrics-operator is at v0.55.0
• VictoriaLogs image is v1.17.0
• The VLogs CRD will become read-only after operator v0.61.0

Proposed Solution

Switch VLogs from single-node to cluster mode using the VLCluster CRD (introduced in operator v0.59.0):

• vlinsert — write-only component, accepts logs from tenants
• vlselect — read-only component, handles log queries
• vlstorage — storage backend

With this architecture, tenant NetworkPolicy egress would target only vlinsert pods (write-only by design, with -select.disable flag). Tenants can ingest logs but cannot read them. This mirrors the existing vminsert/vmselect separation for metrics.

Prerequisites

1. Upgrade victoria-metrics-operator from v0.55.0 to v0.59.0+ (current stable: v0.67.0)
2. Migrate from deprecated VLogs CRD (v1beta1) to VLCluster CRD (v1)
3. Update VictoriaLogs image to a cluster-capable version

This also addresses the upcoming VLogs CRD deprecation.

Alternative

Deploy vmauth (VictoriaMetrics auth proxy) in front of single-node VLogs, routing only /insert/* paths to tenants. Simpler but adds an extra component without solving the CRD deprecation.

References

• [tenant] fix: allow egress to vlogs in parent tenants #1971 — PR that surfaced this issue (tenant log forwarding NetworkPolicy)
• Nested cluster Fluent Bit cannot send logs to parent VLogs: missing vlogs in tenant egress policy #1970 — Related bug report