E2E flake: snapshot-validation-webhook Deployment CrashLoopBackOff cascades install-wide when cert-manager is loaded #3203

What broke

packages/system/snapshot-controller's bundled snapshot-validation-webhook Deployment enters CrashLoopBackOff on E2E sandbox installs under moderate cert-manager load, cascading into a full-install failure. All flux HelmReleases depending on cozy-snapshot-controller/snapshot-controller — piraeus-operator-crds, piraeus-operator, linstor, linstor-scheduler, and (through cozystack-basics) roughly 20 downstream operators (postgres-operator, mariadb-operator, mongodb-operator, opensearch-operator, redis-operator, rabbitmq-operator, metallb, multus, metrics-server, monitoring-agents, etc.) — sit in DependencyNotReady until the outer install-wait times out at 15m and the whole sandbox install fails.

Observed in several 2026-07-04 E2E runs, most recently on #3197 attempt 1 (crust-gather artifact cozyreport-3197.zip).

Root cause

packages/system/snapshot-controller/charts/snapshot-controller/templates/deployment_validation_webhook.yaml (vendored from piraeus-charts/snapshot-controller v3.0.5) hard-codes the webhook probes without any startup grace:

livenessProbe:
  httpGet:
    path: /readyz
    port: https
    scheme: HTTPS
readinessProbe:
  httpGet:
    path: /readyz
    port: https
    scheme: HTTPS

No initialDelaySeconds, no startupProbe, no values.yaml override. The kubelet defaults kick in: initialDelaySeconds=0, periodSeconds=10, failureThreshold=3 — the container has ~30 seconds from Started to become healthy on :8443 before Killing container ... failed liveness probe. On a loaded sandbox that is not enough. Two things add up:

1. cert-manager Certificate issuance is slow under load. The webhook mounts snapshot-validation-webhook-tls (populated by cert-manager). On the failing run the pod hit FailedMount: secret ... not found at start, sat for ~8 minutes waiting on cert-manager, and only received Updated current TLS certificate (from certwatcher.go:129) at 20:19:56 — 8m 18s after the pod was scheduled.

2. The webhook binary needs its own startup time on top. Even after the secret is populated, the container's Exit Code: 2 and one-log-line output indicate the process is exiting before the HTTPS listener is up. With initialDelaySeconds: 0, the liveness probe fires immediately and racks up three consecutive connection refused failures long before the binary has a chance to bind :8443.

Combined: 7 restarts, each Started → Exit Code 2 in ≤30s, BackOff phase, Deployment marked Failed, HR snapshot-controller Failed, cascade.

Timeline from the failing run (excerpts, cozy-snapshot-controller/snapshot-validation-webhook-57bbf5ff7d-mnmtz)

20:11:38  Scheduled
20:11:38  Warning FailedMount     secret "snapshot-validation-webhook-tls" not found
...       Container restart cycle × 7, each Started → Exit Code 2 in ~23s
20:19:56  certwatcher.go:129] Updated current TLS certificate
20:20:19  Container exit (still Exit Code 2)
20:22:10  Helm upgrade abort:
          failed early due to stalled resources:
          [Deployment/cozy-snapshot-controller/snapshot-validation-webhook status: 'Failed']

Downstream cascade at t = 15m54s in that run:

timed out waiting for the condition on helmreleases/linstor
timed out waiting for the condition on helmreleases/linstor-scheduler
timed out waiting for the condition on helmreleases/piraeus-operator
timed out waiting for the condition on helmreleases/piraeus-operator-crds
timed out waiting for the condition on helmreleases/mariadb-operator
timed out waiting for the condition on helmreleases/metallb
timed out waiting for the condition on helmreleases/mongodb-operator
timed out waiting for the condition on helmreleases/metrics-server
timed out waiting for the condition on helmreleases/monitoring-agents
timed out waiting for the condition on helmreleases/multus
timed out waiting for the condition on helmreleases/objectstorage-controller
timed out waiting for the condition on helmreleases/opensearch-operator
timed out waiting for the condition on helmreleases/postgres-operator
timed out waiting for the condition on helmreleases/rabbitmq-operator
timed out waiting for the condition on helmreleases/redis-operator
timed out waiting for the condition on helmreleases/reloader
timed out waiting for the condition on helmreleases/securitygroup-controller
timed out waiting for the condition on helmreleases/snapshot-controller
timed out waiting for the condition on helmreleases/application-definition-crd

Why other E2E runs sometimes pass

Non-deterministic — depends on how fast cert-manager (and cainjector, now bumped in #3199) can service the concurrent burst of Certificate CRs on install. When cert issuance lands under the ~30s liveness window, the webhook comes up on the first try. When it doesn't, cascade. Observed today: 3 SUCCESS / 2 FAILURE across recent PRs at similar-baseline main tips.

Blast radius

Any PR that triggers a fresh cozystack install into the E2E sandbox is at risk. Not specific to any particular chart change — the failure lives in the snapshot-validation-webhook Deployment shape itself. E2E flakiness bug attractor, at minimum.

Proposed fix direction

Introduce a patches/*.diff on the vendored chart following the cozystack convention (e.g. packages/system/vertical-pod-autoscaler/patches/cert-manager-admission-tls.diff). Two shapes worth considering:

1. startupProbe — the k8s-native pattern for slow-startup containers. Grants a large budget for startup (e.g. periodSeconds: 10, failureThreshold: 60 = 10-minute startup window) and does not gate on it once the container is Started. Would cover both the cert-mount wait and the binary startup.
2. initialDelaySeconds + failureThreshold bump on livenessProbe — simpler edit, but the delay has to cover worst-case cert-manager latency (multiple minutes on a loaded sandbox) which makes the value fragile.

Option 1 is the k8s-native answer. Happy to draft a PR against this shape.

Filed together with a companion PR (link forthcoming) that carries the actual fix.