Implement distributed tracing (OTLP) in the monitoring stack #3761

Summary

Implement distributed tracing — the third observability signal alongside metrics and logs — per the accepted design proposal cozystack/community#38 (design-proposals/distributed-tracing/README.md).

Load-bearing decisions from the design:

• Traces are pushed (not scraped), so the OTLP Collector lives inside the tenant namespace (the per-tenant trust boundary); zero NetworkPolicy change on the default path.
• Pluggable backend behind an OTLP + Grafana-datasource seam: VictoriaTraces is the default (no new operator — VTCluster/VTSingle already ship in victoria-metrics-operator), with Tempo/Jaeger selectable.
• Enablement follows the metrics model (platform-configured for supported engines), not a per-app toggle; the OTLP endpoint is platform-decided.

Phased implementation

• Backend + Grafana datasource — tracingStorages → VTCluster/VTSingle in the monitoring stack + traces datasource + readiness gate + helm-unittest + schema. (first PR — see below)
• OTLP Collector — per-tenant in-namespace OpenTelemetry Collector (plain Deployment + ConfigMap; no operator exists), tenant identity injection / sampling / rate-limiting; enable the VictoriaTraces OTLP/gRPC listener.
• Engine integrations (one PR per engine) — ClickHouse (re-enable opentelemetry_span_log + add exporter) → NATS/Kafka/RabbitMQ/MariaDB/Redis (client / agent) → Postgres (pg_tracing).
• Shared-central topology (opt-in) — narrow Cilium egress rule (collector → central backend) + vmauth for read isolation.
• Docs — enablement guide under docs/observability/.

Confirm-before-implementation

• Grafana traces datasource type (Jaeger-compatible surface vs Tempo Query-frontend-compatible API) and the exact vtselect/vtsingle service name, port, and query path — verify against the pinned VictoriaTraces version on a live cluster.
• VictoriaTraces is pre-GA upstream; the first cut may ship on Tempo and switch to VictoriaTraces once GA.

Original request

Supersedes/fulfils #679 (Add traces on monitoring stack).