Skip to content

Commit d5fc723

Browse files
committed
feat(monitoring): bound every traces store by disk usage
The distributed tracing design requires each traces backend to be disk-bounded, not only age-bounded, so a full volume cannot block ingest. retentionDiskUsageBytes was optional, so a store without it filled its volume and stopped taking spans; with a shared-central store that stops ingest for every tenant at once. A store without a byte cap now drops its oldest day once its filesystem is 80% full. VictoriaTraces refuses to start with both limits set, so a byte cap, when given, replaces the percentage one. It always keeps the last two days, which the values text now says. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 2d10565 commit d5fc723

6 files changed

Lines changed: 91 additions & 15 deletions

File tree

‎packages/extra/monitoring/README.md‎

Lines changed: 10 additions & 10 deletions
Large diffs are not rendered by default.

‎packages/extra/monitoring/values.schema.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -351,7 +351,7 @@
351351
"type": "string"
352352
},
353353
"retentionDiskUsageBytes": {
354-
"description": "Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. \"8GB\" or \"8GiB\"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so \"8Gi\" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value.",
354+
"description": "Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. \"8GB\" or \"8GiB\"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so \"8Gi\" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. Unset, the store drops its oldest day once its filesystem is 80% full instead, so every store is disk-bounded. Either way VictoriaTraces keeps the last two days, so spans arriving faster than the cap allows for two days can still fill the disk.",
355355
"type": "string"
356356
},
357357
"retentionPeriod": {
@@ -382,7 +382,7 @@
382382
}
383383
},
384384
"tracingCentralHost": {
385-
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: with retentionDiskUsageBytes set the oldest spans of all of them are evicted together, without it the disk fills for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
385+
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space or limits the rate per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together, and a tenant sending faster than the cap allows for the two days VictoriaTraces always keeps can fill the disk for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
386386
"type": "boolean",
387387
"default": false
388388
},

‎packages/extra/monitoring/values.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,12 +106,12 @@ logsStorages:
106106
## @field {string} retentionPeriod="14d" - Retention period for traces.
107107
## @field {quantity} storage="10Gi" - Persistent volume size.
108108
## @field {string} storageClassName="replicated" - StorageClass used to store the data.
109-
## @field {string} [retentionDiskUsageBytes] - Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value.
109+
## @field {string} [retentionDiskUsageBytes] - Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. Unset, the store drops its oldest day once its filesystem is 80% full instead, so every store is disk-bounded. Either way VictoriaTraces keeps the last two days, so spans arriving faster than the cap allows for two days can still fill the disk.
110110

111111
## @param {[]TracingStorage} tracingStorages - Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. A tenant with shared-central tracing (the Tenant's tracingCentral) gets the collector without an entry, which then exports to the shared store; entries still listed keep their store and datasource, so traces stored before the switch stay readable until the entry is removed.
112112
tracingStorages: []
113113

114-
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: with retentionDiskUsageBytes set the oldest spans of all of them are evicted together, without it the disk fills for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
114+
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space or limits the rate per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together, and a tenant sending faster than the cap allows for the two days VictoriaTraces always keeps can fill the disk for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
115115
tracingCentralHost: false
116116

117117
##

‎packages/system/monitoring-rd/cozyrds/monitoring.yaml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/system/monitoring/templates/vtraces/vtraces.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@
1212
export. The entries still listed here stay rendered: the switch belongs to the
1313
parent tenant and the stores to this one, and a VTSingle takes its PVC with it.
1414

15+
Every store is disk-bounded, not only age-bounded, so a full PVC cannot
16+
block ingest: without a byte cap it drops its oldest day once its filesystem
17+
is 80% full. VictoriaTraces refuses to start with both limits, so only one is
18+
rendered. It always keeps the last two days, which a flood faster than that
19+
can still push past the limit.
20+
1521
The readiness-gate healthCheckExprs on VTCluster/VTSingle stay unconditional:
1622
Flux evaluates them only against CRs present in the release inventory, so with
1723
nothing rendered they are inert.
@@ -34,6 +40,9 @@ spec:
3440
retentionPeriod: {{ .retentionPeriod | quote }}
3541
{{- with .retentionDiskUsageBytes }}
3642
retentionMaxDiskSpaceUsageBytes: {{ . | quote }}
43+
{{- else }}
44+
extraArgs:
45+
retention.maxDiskUsagePercent: "80"
3746
{{- end }}
3847
# Stamped on the storage PVC so the monitoring post-delete cleanup hook can
3948
# select it by this label (same release-scoped cleanup as logs/metrics).
@@ -70,6 +79,9 @@ spec:
7079
retentionPeriod: {{ .retentionPeriod | quote }}
7180
{{- with .retentionDiskUsageBytes }}
7281
retentionMaxDiskSpaceUsageBytes: {{ . | quote }}
82+
{{- else }}
83+
extraArgs:
84+
retention.maxDiskUsagePercent: "80"
7385
{{- end }}
7486
replicaCount: 2
7587
resources: {}

‎packages/system/monitoring/tests/vtraces_tracingstorages_required_test.yaml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,70 @@ tests:
123123
path: spec.retentionMaxDiskSpaceUsageBytes
124124
value: "4GB"
125125

126+
# The design requires every traces backend to be disk-bounded, not only
127+
# age-bounded, so a full PVC cannot block ingest. Without an explicit byte cap
128+
# the store drops its oldest day once its filesystem is 80% full. The two
129+
# flags are mutually exclusive: VictoriaTraces refuses to start with both.
130+
- it: bounds a cluster-mode store at 80% of its filesystem without a byte cap
131+
templates:
132+
- templates/vtraces/vtraces.yaml
133+
set:
134+
tracingStorages:
135+
- name: generic
136+
mode: cluster
137+
retentionPeriod: "14d"
138+
storage: 10Gi
139+
asserts:
140+
- equal:
141+
path: spec.storage.extraArgs["retention.maxDiskUsagePercent"]
142+
value: "80"
143+
- notExists:
144+
path: spec.storage.retentionMaxDiskSpaceUsageBytes
145+
146+
- it: bounds a single-mode store at 80% of its filesystem without a byte cap
147+
templates:
148+
- templates/vtraces/vtraces.yaml
149+
set:
150+
tracingStorages:
151+
- name: generic
152+
mode: single
153+
retentionPeriod: "14d"
154+
storage: 10Gi
155+
asserts:
156+
- equal:
157+
path: spec.extraArgs["retention.maxDiskUsagePercent"]
158+
value: "80"
159+
- notExists:
160+
path: spec.retentionMaxDiskSpaceUsageBytes
161+
162+
- it: sets no percentage cap on a cluster-mode store that has a byte cap
163+
templates:
164+
- templates/vtraces/vtraces.yaml
165+
set:
166+
tracingStorages:
167+
- name: generic
168+
mode: cluster
169+
retentionPeriod: "14d"
170+
storage: 10Gi
171+
retentionDiskUsageBytes: "8GB"
172+
asserts:
173+
- notExists:
174+
path: spec.storage.extraArgs
175+
176+
- it: sets no percentage cap on a single-mode store that has a byte cap
177+
templates:
178+
- templates/vtraces/vtraces.yaml
179+
set:
180+
tracingStorages:
181+
- name: generic
182+
mode: single
183+
retentionPeriod: "14d"
184+
storage: 10Gi
185+
retentionDiskUsageBytes: "4GB"
186+
asserts:
187+
- notExists:
188+
path: spec.extraArgs
189+
126190
- it: renders a Jaeger-typed Grafana datasource at vtselect:10471 for cluster mode
127191
templates:
128192
- templates/vtraces/grafana-datasource.yaml

0 commit comments

Comments
 (0)