flake(tests): clickhouse e2e misses the endpoint Service after the server pod restarts on Address already in use #3852

What happens

The base clickhouse e2e case fails on verify-endpoint-service: the chendpoint-clickhouse-test Service is not created within the 3-minute assert budget, so the case ends with actual resource not found. Seen once so far, on run 31873137335 (2026-08-15, branch fix/cozytest-negated-assertions), while the two backup cases of the same suite passed on the same run and the whole suite passed on the runs before and after. Filed for tracking with the mechanism read out of the captured diagnostics, since a single occurrence is not enough to call it a stable flake.

Timeline from the run (all from the job log)

• 10:42:39 ClickHouse/test applied; the HelmRelease is Ready two seconds later.
• 10:42:41 assert on Service chendpoint-clickhouse-test starts, budget 3 minutes.
• 10:43:11 the chi-clickhouse-test-clickhouse-0-0-0 pod's clickhouse container starts and immediately logs Address already in use for every listen port (8123, 9000, 9004, 9005) plus Host not found for all three keeper hosts, so the server never comes up healthy.
• 10:43:31 readiness and liveness probes fail; kubelet kills the container (Container clickhouse failed liveness probe), it restarts.
• 10:44:32 the previous instance receives Terminated.
• 10:45:41 the assert times out. The pod has restarts=1, the CHI operator shows one CreateCompleted for a Service and one UpdateFailed for a Service, and chendpoint-clickhouse-test does not exist.

Mechanism, and what is proven versus taken

Two things are measured from the log. First, the pod restarted once because the clickhouse process could not bind its ports: Address already in use on every listener means a previous clickhouse process was still holding the sockets inside the same pod when the new one started, so this looks like a restart racing its own predecessor rather than a config error. Second, the keeper hostnames did not resolve at all during that window, so even a clean start would have had no ZooKeeper.

What is taken rather than measured is the link from that pod restart to the missing endpoint Service. The operator's UpdateFailed on a Service in the same window is the plausible bridge: the CHI reconciler creates the endpoint Service as part of the host reconcile, and a reconcile that fails on Service update while the pod is crash-looping would leave the endpoint uncreated. Whether that update failure was caused by the unhealthy pod or is independent needs the operator log for that reconcile, which the catch block did not capture (it captures pod previous-instance logs and events, and the operator's own log only through the events it emits).

The suite's own diagnostics were partly starved: the previous-logs capture hit COZY_PREVLOG_MAX=12 with 3 restarted containers left uncaptured, on a cluster where 15 containers had restarted, so the operator pod itself may be among the missing.

What would confirm it

A second occurrence with the operator log for the reconcile at hand. Until then, the load correlation is worth noting: this run happened during a wave of six parallel e2e workflows on the same host, in the same window in which the node-join failures of #3513 and the install-phase failures of #3851 also cluster, and the Address already in use shape (an old process outliving its container's restart) is exactly what a starved node produces.