test(e2e): external-dns liveness kill-loop flakes external-dns-inmemory (missing startupProbe) #3459

Describe the bug

The external-dns/external-dns-inmemory Chainsaw E2E case flakes under cluster load. The external-dns container enters a liveness-probe kill-loop: it does not bind its /healthz server on :7979 before the livenessProbe starts failing, so kubelet restarts it, and the cycle repeats. The Deployment never becomes Ready, the wrapping HelmReleases time out, and the Chainsaw case fails.

Root cause is a missing startupProbe combined with a tight livenessProbe (initialDelaySeconds 10, periodSeconds 10, failureThreshold 2 — roughly a 30s budget). Under CI-cluster load (apiserver/etcd latency) the external-dns informer source-cache sync takes longer than that budget, so a healthy-but-slow start is killed as if it were unhealthy.

This is not a regression of any specific feature PR: in the same E2E run the kubernetes-previous and kubernetes-latest cases pass; only external-dns-inmemory fails, and the external-dns chart is orthogonal to those changes.

Environment

• Cozystack version: main (observed in Release E2E run 30254691571)
• Provider: CI (Release E2E workflow)

To Reproduce
Run the Release E2E suite while the cluster is under load. The external-dns/external-dns-inmemory Chainsaw case fails.
Reference failing run: https://github.com/cozystack/cozystack/actions/runs/30254691571/job/89984573704

Expected behaviour
external-dns tolerates a slow first start (informer cache sync) without being killed by the liveness probe; its Deployment becomes Ready and external-dns-inmemory passes.

Actual behaviour
Chainsaw assertion fails on the wrapping HelmRelease:

HelmRelease/tenant-test/test
status.(conditions[?type == 'Ready'])[0].status: Expected "True", got "Unknown"  (actual status: {})

Failure cascade (from the run's cozyreport artifact):

pod external-dns: lastState.terminated exitCode 137 (SIGKILL, reason Error — not OOMKilled), restartCount 5
  -> Deployment external-dns never Ready
    -> HR test-system (cozy-external-dns): InstallFailed — timeout waiting for Deployment ... 'InProgress'
      -> HR test (external-dns):        InstallFailed — timeout waiting for HR test-system 'InProgress'
        -> chainsaw external-dns-inmemory: FAIL

The container log stops right after client init and is terminated ~29s later, before the health server binds:

level=info msg="Created Kubernetes client https://10.96.0.1:443"
level=info msg="Received SIGTERM. Terminating..."

Logs

# external-dns container (previous instance), full output before it was killed:
config: { ... Provider:inmemory ... MetricsAddress::7979 ... }
GitCommitShort=unknown, GoVersion=go1.25.4, ExternalDNS/v20251114-v0.20.0
Instantiating new Kubernetes client
Using inCluster-config based on serviceaccount-token
Created Kubernetes client https://10.96.0.1:443
Received SIGTERM. Terminating...        # ~29s later; kubelet restart from liveness failure

# kubelet events on the pod:
Readiness probe failed: Get "http://<pod-ip>:7979/healthz": dial tcp <pod-ip>:7979: connect: connection refused
Liveness probe failed:  Get "http://<pod-ip>:7979/healthz": dial tcp <pod-ip>:7979: connect: connection refused
Container external-dns failed liveness probe, will be restarted

Additional context

The probe is templated from .Values.livenessProbe / .Values.readinessProbe in packages/system/external-dns/charts/external-dns/templates/deployment.yaml (around lines 160-163 / 198-201); the template renders no startupProbe, so a slow start has no grace period distinct from the liveness check.

Suggested fix, following the precedent in #3204 (fix(snapshot-controller): add startupProbe as defense-in-depth against slow HTTPS bind on snapshot-validation-webhook): add a startupProbe on /healthz:7979 with a generous failureThreshold, so the informer cache sync can complete before the liveness probe begins evaluating; alternatively, relax the liveness initialDelaySeconds / failureThreshold.

Checklist

• I have checked the documentation
• I have searched for similar issues (found fix(snapshot-controller): add startupProbe as defense-in-depth against slow HTTPS bind on snapshot-validation-webhook #3204 as a same-class precedent for a different component; no existing external-dns duplicate)
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs