docs(observability): add a distributed tracing guide - #3772

…3775)

## What this PR does

Adds the opt-in shared-central traces topology from the distributed
tracing design (cozystack/community
`design-proposals/distributed-tracing`, rollout item 4): a tenant can
send its spans to one VictoriaTraces store in tenant-root instead of
running its own, with both write and read isolation between the tenants
that share it.

**Switching it on.** A tenant opts in with the new Tenant field
`tracingCentral`, which needs the tenant's own Monitoring. The tenant
chart publishes it to the tenant's monitoring charts as
`_namespace.tracingCentral` in the `cozystack-values` Secret, which the
apps API does not let an application write, and renders the two network
paths the topology needs under the same condition. tenant-root decides
whether it hosts the store, with the new Monitoring field
`tracingCentralHost` next to a cluster-mode `tracingStorages` entry
named `generic`.

**Isolation.** VictoriaTraces takes the tenant account from the
`AccountID`/`ProjectID` headers and does no authorization of its own, so
other tenants reach the shared store only through a vmauth in
tenant-root. Each central tenant gets a VMUser that pins an account
derived from its namespace name and UID, for both its collector's writes
and its Grafana reads, and vmauth overrides any account a client sends.
The collector and Grafana log in with credentials the chart generates
and keeps. The tenant egress rules allow only pods carrying the
collector's or Grafana's label to reach the vmauth proxy port, and a pod
that carries such a label without the tenant's credentials gets only 401
from it; the vmauth internal routes, whose `/metrics` lists every
tenant's user, are served on a port tenants cannot reach. The
tenant-root namespace itself is inside the trust boundary, as it already
is for the metrics of tenants that rely on a parent's monitoring: its
workloads reach the store directly, and one that learns a tenant's
account ID can read and write that account, and whoever tenant-root
admits to its Grafana reads the store through a datasource that bypasses
vmauth (with the System oidc mode, tenant-root's own groups, which
already hold the same access level in every tenant under it); the
`tracingCentralHost` description says so.

**Sharing the store.** Each tenant's VMUser is capped at four requests
in flight per vmauth replica, reads and writes together, and each
collector replica exports with two senders, so more collector replicas
do not raise a tenant's throughput; above the cap vmauth queues for up
to 10s and answers 429, which the exporter retries. The store has one
disk and one retention for all of its tenants and nothing reserves space
per tenant; `tracingCentralHost` states that, since hosting is
tenant-root's decision. The vmauth runs two replicas spread across nodes
under a PodDisruptionBudget, so draining one node does not stop every
central tenant at once. Every traces store is also disk-bounded now, as
the design requires: without `retentionDiskUsageBytes` it drops its
oldest day once its filesystem is 80% full. That alone does not keep one
tenant from filling the shared disk, since VictoriaTraces always keeps
the last two days.

**States that would drop spans silently are refused, one race aside.** A
live render fails, with a message naming the prerequisite, when a tenant
turns central on while tenant-root hosts nothing, and when tenant-root
stops hosting while central tenants remain; tenant-root's render also
refuses `tracingCentralHost` without a store to host. The second check
does not wait on a VMUser that is already being deleted, since its
tenant is leaving. Each check reads the other side's objects when it
renders, so a tenant turning central on while tenant-root stops hosting
can pass both and export to a vmauth that is gone; the
`tracingCentralHost` description names this race and its recovery. At
runtime, tenant-root's Monitoring release is not Ready while the vmauth
has failed. These checks run on a render, so removing the tenant-root
Monitoring altogether is not refused, and from then on every Monitoring
render of a central tenant fails until it turns central off; the
`tracingCentralHost` description says to turn central off on the tenants
first. Turning central on keeps the tenant's own local stores and their
datasources, so traces stored before the switch stay readable until the
tenant removes them; turning it off again leaves the spans written
meanwhile in the shared store, unreadable from the tenant until it turns
central on again, and spans sent while the collector switches back may
be dropped; a deleted tenant's spans likewise stay until retention drops
them. Right after a tenant turns central on, or after its credentials
Secret is re-created, spans its collector sends before the vmauth has
loaded the new login get a 401, which the exporter does not retry, so
they are dropped; the tracing guide in #3772 covers it.

**Where this departs from the design.** The design has the per-tenant
collector inject the account headers and adds exactly one egress rule.
Here the account is pinned by the tenant's VMUser at the vmauth and the
collector sends none, because the collector runs in the tenant namespace
and whatever runs with its pod label could otherwise choose an account;
vmauth is also what the design already requires for reads, so both
directions go through it. That needs a second egress rule, for the
tenant's Grafana to reach the vmauth, beside the collector's. The
per-tenant write rate the design gives the collector is not in this PR:
no OpenTelemetry distribution has a rate-limiting processor, so #4648
builds the collector with one and holds each central tenant to a rate
tenant-root publishes. Until it lands, a tenant writing faster than the
disk cap allows over two days can fill the disk the other tenants share.

**Consent is per host, not per tenant.** `tracingCentralHost` lets every
tenant that sets `tracingCentral` in, nested tenants included, and each
gets its own VMUser and its own request cap; there is no allow-list. A
tenant-root that needs to limit who shares its store keeps
`tracingCentralHost` off.

**Upgrade.** Tenants that do not opt in render the same objects as
before, and their `cozystack-values` Secret is unchanged. The monitoring
charts render the same objects too, with one exception: a VTSingle or
VTCluster without `retentionDiskUsageBytes` gains
`retention.maxDiskUsagePercent: "80"` in its `extraArgs`, so its spec
changes and its storage pods restart once, and a store already more than
80% full drops its oldest days when it starts. The inner monitoring
HelmRelease's values gain the new `tracingCentralHost: false` default,
as with any new field, and its `healthCheckExprs` gain a `VMAuth` entry,
which matches nothing outside a hosting tenant-root. Both change the
HelmRelease spec, so every tenant's monitoring release upgrades once,
rendering the same objects. A tenant-root that has a cluster-mode
`generic` store gets no vmauth until it sets `tracingCentralHost`.

**Testing.** helm-unittest covers the three charts the change touches
(tenant, monitoring, extra monitoring) and every guard was checked by
mutation; a bats test covers what helm-unittest cannot, that the
generated password and the collector's rollout checksum agree on a first
install. On a test cluster: a spoofed `AccountID` from a tenant is
overridden by vmauth, one tenant cannot read another's spans, the vmauth
starts under the real tenant network policies (and stays in init without
its apiserver label), the guards refuse as described, and the rendered
collector config passes `otelcol validate` in the pinned image. There is
no end-to-end test yet; one that covers cross-tenant isolation is
planned as a follow-up.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: covered by #3772, the tracing guide
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#64
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The Tenant gains the user-facing field `tracingCentral` (bool, default
false), which the trigger map sends to terraform-provider-cozystack: its
hand-written Tenant schema and model need the field. The follow-up is
cozystack/terraform-provider-cozystack#64. The Monitoring gains
`tracingCentralHost` (bool, default false); the provider's
`cozystack_tenant_module` is a marker without values, so it needs no
change. The website reference pages for both are generated from the
READMEs on release. The tracing user guide in #3772 covers the
per-tenant backend only. A central-mode section in the tracing guide of
#3772 (hosting and opting in, the shared disk and retention, turning
central off on the tenants before tenant-root stops hosting, and the
recovery from the opt-in/stop-hosting race) also takes the lifecycle
details the `tracingCentral` description no longer carries.

### Release note

```release-note
feat(monitoring,tenant): add an opt-in shared-central traces store. A tenant sets tracingCentral to send its spans to one VictoriaTraces store in tenant-root instead of its own, and tenant-root opts in to hosting it with tracingCentralHost on its Monitoring. Tenants sharing the store are isolated on write and read through a vmauth in tenant-root, and each is capped in the requests it can hold in flight.
```

)

## What this PR does

Closes the part of the distributed tracing design that #3775 left open:
the per-tenant collector rate-limits its tenant, so that a noisy or
hostile tenant cannot exhaust a shared backend (cozystack/community
`design-proposals/distributed-tracing`, Security and §2).

**Why the disk cap is not enough.** #3775 bounds every traces store by
disk usage, but VictoriaTraces always keeps the last two days, so a
tenant writing faster than the cap allows over two days fills the disk
every tenant on the shared store writes to. Only a per-tenant write rate
closes that.

**The collector is built here.** No upstream OpenTelemetry distribution
or processor limits throughput, and neither the open-source vmauth nor
VictoriaTraces limits a tenant's rate. Elastic's EDOT Collector
registers Elastic's `ratelimitprocessor` among its default components,
but it ships under the Elastic License 2.0 and carries far more
components than the chart uses. The collector is therefore built with
the OpenTelemetry Collector Builder from the upstream components the
chart already uses, at the same v0.136.0, plus that `ratelimitprocessor`
(Apache-2.0). Its v0.8.2 is the release built against collector
v0.135.0, the last one before v0.137.0, and it builds against v0.136.0
unchanged. The image is versioned by the collector release with a
cozystack respin counter (`v0.136.0-cozystack.0`), runs as the upstream
image's user, and is built with the other package images.

**How the rate is set and enforced.** The platform sets the rate with
`monitoring.tracingCentralTenantBytesPerSecond`, and it reaches every
tenant in `_cluster` through the `cozystack-values` Secret. A change
there is a change in the values every monitoring HelmRelease reads, so
Flux upgrades each of them; a rate read with `lookup` at render time
would never reach a tenant whose release is already in sync. The
platform writes the key only for a positive rate, because every release
in every tenant reads that Secret, and refuses to render on a value that
is not a whole number of bytes per second in plain decimal digits, up to
1 TiB/s ("1Mi", 1.5, "0100"), rather than reading it as the default.
Without it, or on a value that is not a positive number, the collector
holds the tenant to the default of about 97 KiB/s, at which one tenant
writes at most the disk cap of a default 10Gi store over two days,
counted uncompressed; the store keeps spans compressed, so they take
less than that. A larger store needs the rate raised with it. Each
central tenant's collector splits the rate across its replicas and
refuses data above it before the batch processor, so the application
sees the refusal, and right after the memory limiter, ahead of the
sampler, which adds a tracestate to every span it keeps, and of the
tenant attribute added to every resource, so it counts a request as the
receiver admitted it. The rate is therefore counted before sampling, and
the store keeps only the sampled share of it. Over OTLP/HTTP it gets a
429, which OTLP exporters retry. Over OTLP/gRPC the limiter answers
RESOURCE_EXHAUSTED without RetryInfo, which the OTLP specification makes
final, so a gRPC exporter drops those spans; the Tenant's
`tracingCentral`, the Monitoring `tracingCollector` and the platform key
tell central tenants to export over HTTP. The limiter sees the requests
the receiver admits, so in central mode the receiver caps a request at 3
MiB over both OTLP/HTTP and gRPC, decompressed for HTTP. The limiter
counts a request as the collector re-encodes it, which can be larger
than it arrived (13% for a span carrying only its IDs, the smallest
valid one, and up to 1.4x for spans with empty-valued attributes), so
burst is at least 5 MiB, above 1.4 times the cap, since a request larger
than burst never passes. Each collector pod starts with a full bucket,
so a pod start lets through above the rate up to 5 MiB or one second of
the pod's share of the rate, whichever is larger, and the chart refuses
to render a central tenant's collector with more than 10 replicas.

**Only the collector can write.** The rate holds only if the collector's
login is the only way the tenant writes. Each central tenant now has two
VMUsers on its one account: the collector's, which may only write, and
its Grafana datasource's, which may only read, each with its own Secret.
With one login for both, any user the tenant admits to its Grafana, a
Viewer included, could POST through the datasource proxy to `../insert`,
which vmauth cleans before it picks a route, and write into the shared
store past the limiter. The tenant's Grafana pod may reach the vmauth
too, so its image is the platform's, like the collector's, and only the
`auth.ini` key is mounted from the Secret a tenant names in
`oidc.customConfig.secretRef`: an image of the tenant's choosing with
the collector's Secret mounted would otherwise write past the limiter as
well.

**The collector image is the platform's.** The apps API defaults a
Monitoring spec from its schema but does not prune keys the schema
leaves out, and the Monitoring chart hands its values to the inner chart
as they are, so a tenant could set `tracingCollector.image` and run any
collector it liked, one without the limiter included. That key is no
longer passed through.

**Trade-offs.** The limiter is a vendor component, not an OpenTelemetry
one, and Elastic marks it as in development; it moves with Elastic's
releases, so bumping the collector means picking the matching release of
it (v0.74.0 builds against collector v1.67.0), and collector v0.136.0 is
a year old. It logs each refused request at error level, so a tenant
held at its limit produces steady log lines in its own collector.
Several tenants writing at their limits together can still exceed the
cap within two days; the rate bounds each tenant, not their sum. The
limiter keeps its state in each pod's memory, so a tenant admin who
restarts its collector pods over and over gains, per pod start, up to 5
MiB or one second of the pod's share of the rate, whichever is larger;
closing that needs a limiter with shared state (the processor's
gubernator mode), which is out of scope here. A sender whose connection
stays on one replica gets that replica's share of the rate. In central
mode the receiver refuses a request larger than 3 MiB: over OTLP/HTTP
with a 400, which senders do not retry, where the receiver default is 20
MiB, and over gRPC, where the default is 4 MiB. Changing the rate
upgrades every release in every tenant once, since it travels in the
Secret they all read. Until CI builds the image,
`tracingCollector.image` in `values.yaml` still names the upstream
distribution, which has no rate limiter, so a central-mode collector
from an unbuilt tree does not start; CI and releases stamp the built
image, and the image repository in the build registry has to be public
before a cluster can pull it.

**Upgrade.** Collectors in both modes move to the built image, which
carries the same components at the same version; central collectors also
roll for the new processor. The replica cap fails the render of a
central tenant's collector with more than 10 replicas, while a collector
writing to the tenant's own store keeps any replica count; the collector
has shipped only in the v1.7.0 alphas. Each central tenant's existing
VMUser loses its read route, and a second VMUser and Secret appear for
Grafana, whose datasource moves to the new login in the same release, so
the tenant's trace reads may fail until vmauth loads the new user.
Nothing else changes for clusters that do not set the rate: the platform
writes no new key, so no other release upgrades.

**Testing.** helm-unittest covers the platform writing the rate into
`_cluster` only for a positive whole number and refusing a malformed
one, numbers Helm reads as float64 included, the collector's limiter
(the configured rate, the split across replicas, burst, the receiver's
request cap in central mode and not in local mode, the default when
unset and on a word, zero or a negative value, its place right after the
memory limiter, with and without redaction, its absence in local mode),
the replica cap for a central tenant's collector and its absence for a
local one, the image key no longer passing through, the two logins (the
collector's with only the write route, Grafana's with only the read
route, separate Secrets and passwords), the Grafana image not passing
through, and only `auth.ini` being mounted from a `secretRef` Secret for
a central tenant; every term was checked by mutation. On a test cluster,
with the chart's rendered VMAuth, VMUsers, VTCluster, Grafana datasource
and collector: with one login for both, a Grafana Viewer's POST to
`../insert` through the datasource proxy landed a span in the shared
store; with the two logins vmauth answered `400 missing route` for every
encoding of the traversal tried, reads went through, and the collector's
login could write but not read. With the built collector at 100%
sampling and the receiver capped at 4 MiB, a fresh pod accepted a
request just under that cap made of 53-byte spans and one made of
ID-only spans, refused a larger one with a 400, and answered 429 once
the bucket was drained; the 53-byte spans it accepted were read back
through the read-only login; the 3 MiB cap the chart renders has not
been run on a cluster. A pod mounting a Secret through the chart's
`secretRef` volume as rendered for a central tenant got only `auth.ini`,
and one naming the collector's Secret, which has no such key, did not
start. The design's Testing section also asks for a sustained span flood
against a disk-capped store to show that the oldest spans are evicted
and ingest is not blocked; there is no end-to-end lane for
shared-central tracing yet, so that flood test is a follow-up, like the
cross-tenant isolation test #3775 names. On a test cluster, under the
network policies the tenant chart renders, the built image validated the
rendered config in both modes and wrote and read through the shared
store as before; a tenant given a small limit got 200 for its first
OTLP/HTTP requests and 429 for the rest of a burst, with the processor
logging each refusal, while another tenant under the default rate got
200 for all of the same requests.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#729
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The new platform key `monitoring.tracingCentralTenantBytesPerSecond`
belongs in the hand-written `platform-package.md` reference on the
website, tracked in cozystack/website#729. The Monitoring schema and the
Tenant's `tracingCentral` change only descriptions; the provider models
neither, so it needs nothing. The central-mode section #3772 adds to the
tracing guide covers the rate.

### Release note

```release-note
feat(monitoring): hold each shared-central tracing tenant to a write rate. The platform sets the rate with monitoring.tracingCentralTenantBytesPerSecond, and each central tenant's collector refuses spans above it, which bounds what one tenant writes into the store every tenant shares. The traces collector is now built by cozystack from the upstream components plus Elastic's rate-limit processor, and its image can no longer be set from a Monitoring application. A central tenant's Grafana reads the shared store with a login that cannot write, and its image can no longer be set from a Monitoring application.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Central tracing now applies a configurable per-tenant write-rate limit
shared across collector replicas. The default is approximately 97 KiB
per second, and collector replica counts are limited to 10.
* Each starting replica can temporarily exceed its rate share by up to 5
MiB, or by one second of that share when it is larger. Tenants share
central storage, so combined writes can fill the store and cause older
traces to be evicted.

* **Documentation**
* OTLP/HTTP is recommended for central tracing: excess traffic is
retried over HTTP, while OTLP/gRPC traffic may be dropped.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->