Repository navigation
keycloak starts before its CNPG database has a ready endpoint, and the failure surfaces as no route to host #3570
Description
Summary
A platform install failed with the keycloak release never becoming ready, and the pod log says No route to host while connecting to its database. The network is fine. The connection went to a ClusterIP whose service had no ready endpoint yet, and under kube-proxy replacement that returns EHOSTUNREACH, which client libraries report as a routing failure. Because cozy-system/application-definition-crd waits on cozy-keycloak/keycloak, and most of the platform waits on that, one client racing its own database stalls the whole install.
Mechanism
packages/system/cilium/values.yaml sets kubeProxyReplacement: true and no kube-proxy DaemonSet ships in the platform, so every ClusterIP on the host cluster is programmed by the cilium agent. A ClusterIP with no ready backend has nothing to translate to, and connecting to it fails with EHOSTUNREACH rather than ECONNREFUSED.
The timing is in the artifacts. In the failing run the CNPG Cluster object keycloak-db was created at 12:46:25Z (Created resource via patch: {"gvk":"postgresql.cnpg.io/v1, Kind=Cluster","name":"keycloak-db"}), and keycloak reported java.net.NoRouteToHostException: No route to host at 12:46:59Z, thirty four seconds later, long before a database pod could be serving.
The same shape shows up harmlessly elsewhere, which is what makes it a pattern rather than one bad chart. On a different run grafana logged dial tcp 10.96.30.70:5432: connect: no route to host at 03:37:11Z and 03:37:15Z; 10.96.30.70 is tenant-root/grafana-db-rw, whose only backend grafana-db-1 started at 03:37:03Z and reached Ready at 03:37:29Z. That client lost the race by eighteen seconds, restarted, and recovered.
The cilium agents were not the problem, and that was checked rather than assumed: on the same run all three agents reached Ready at 03:24:31Z, 03:24:46Z and 03:24:31Z with a restart count of zero, thirteen minutes before the first failure.
One caveat on the error mapping. The link from "no ready backend" to EHOSTUNREACH rests on two independent timing pairs rather than on reading the cilium source, so treat it as strongly supported rather than proven.
What is not established
Why the keycloak retries did not converge in the roughly twenty minutes of budget left after the database came up. The race explains the first failures. It does not by itself explain a release that stays failed for the rest of the install, and I could not settle that from the artifacts available.
Closing that needs a run capturing: the target service's EndpointSlice over time rather than only its final state, which currently shows ready: true after the fact and hides the empty window; the client pod's logs across every restart; cilium-dbg service list and cilium-dbg bpf lb list from the client's node at the moment of failure; and the database readiness transition next to the client error in one place, since today they have to be stitched together from separate files.
Directions
Two, and I am not picking between them. Gate the client on the database having an endpoint, through a dependency or an init container that waits, so it never starts into an empty service. Or make the client tolerate an empty service, so the release converges on retry instead of spending its timeout.
Activity
lexfrei commentedon Aug 5, 2026
A fourth instance turned up the same day, and it weakens one thing I wrote above. Run
31021522273failed during install withdial tcp 10.96.4.25:5432: connect: no route to hostat16:57:15Zand16:57:23Z. The client is grafana again, connecting to its own Postgres ClusterIP, and it restarted seven times in ten seconds, at16:57:09,:10,:11,:11,:15,:18and:19.What separates this run from the one described above is that it did not recover. The install ended with
timed out waiting for the condition on helmreleases/monitoring, and the same forseaweedfsandtenant-root. So the split I drew, keycloak as the blocking case and grafana as the harmless one, does not survive a second sample: the same client with the same failure shape sat inside a release that never converged.I am not claiming grafana's race caused the monitoring timeout. The database could have been slow for its own reasons and I did not check that. What the second sample does establish is narrower and still worth having: the shape is not harmless in one release and fatal in another by nature, so "which client hits it" is the wrong axis to sort by. The open question above, why the retries fail to converge once the endpoint appears, is what decides whether an instance blocks, and it now has two examples instead of one.
lexfrei commentedon Aug 6, 2026
A fifth instance, and it is the same scheduled job as the one quoted above as harmless. Nightly
31067501623onmain, 2026-08-06, failed atInstall Cozystack into sandboxwithdial tcp 10.96.29.196:5432: connect: no route to host, grafana again,restarts=3on both of its pods, ending the install atConfigure Tenant and wait for applications (exit 1). The run cited above at03:37:11Zwith10.96.30.70was the previous night's nightly on the same branch, so two consecutive runs of one scheduled job produced the recovering form and then the blocking one. No new mechanism here, and no cause claimed; the point is that neither the client nor the branch varies between those two outcomes, which leaves the unanswered question above as the only thing that separates them.lexfrei commentedon Aug 18, 2026
Two more instances of the class, one of them the first with a hardware record.
Run 32025422447 (2026-08-17, a PR branch run):
Configure Tenant and wait for applicationsfails withdial tcp 10.96.35.244:5432: connect: no route to host, chainsaw suites never start.Run 32117285553 (2026-08-18, manually dispatched Nightly on main): same step,
dial tcp 10.96.253.31:5432: connect: no route to host, 20m02s into the tenant wait, suites never start.The second run carries the runner identity record that now ships in cozyreport (
sandbox-host/runner-identity.txt): VM.Standard.E4.Flex, AMD EPYC 7J13 (Milan), us-ashburn-1, FAULT-DOMAIN-2, 16 OCPUs / 128 GB. That makes it the first instance of this class where the hardware the run landed on is known rather than assumed, so from here on the instances can be split by shape and region as well as by client and branch.lexfrei commentedon Aug 18, 2026
I pulled the cozyreport artifact for the six runs listed in this thread and checked, for each one, whether the client reconnected after its Service received an endpoint. It did, in all six. That moves the issue somewhere else, so here is the evidence.
The EHOSTUNREACH mapping is now proven rather than strongly supported. At cilium v1.19.5,
__sock4_xlate_fwdinbpf/bpf_sock.creturns-EHOSTUNREACHwhensvc->count == 0andenable_no_service_endpoints_routableis set, andcil_sock4_connectturns that intoSYS_REJECTwith the errno preserved. The flag is not taken from upstream defaults: the collectedcilium-configreadsenable-no-service-endpoints-routable: "true"next tokube-proxy-replacement: "true"in five of the six bundles, and the sixth (31003636724) collected no configmaps at all because its crust-gather was truncated, so for that one run the flag is unobserved rather than assumed. So the connect fails synchronously inside the cgroup hook, no packet ever leaves the pod, and Go renders the errno asconnect: no route to host. The caveat in the body can come out.Every instance recovered on the next container start. The Ready transition of
grafana-db-1and the write to thegrafana-db-rwEndpoints object land in the same second in every run, every client failure is before that second and every successful start is after it.The same race runs in parallel against five to eight other Services in every one of these runs, with the same outcome. alertmanager against alerta, grafana-operator against grafana-service, kube-apiserver against the aggregated APIServices, kube-ovn-pinger against coredns on
10.96.0.10:53. None of them cost a run anything, which is what you would expect from a start-order race that resolves in twenty to forty seconds.What did end each install is something else, and four of the six are one already filed cause.
timed out waiting for the condition on vmalertmanagers/alertmanagertimeout waiting for: [Cluster/cozy-keycloak/keycloak-db status: 'InProgress']timed out waiting for the condition on vmclusters/longtermtimeout waiting for: [VLCluster/tenant-root/generic status: 'InProgress']timeout waiting for: [VLCluster/tenant-root/generic status: 'InProgress']The three router IP cases are the silent mode #3740 describes. In 31003688705 the pod IP of
keycloak-db-2is 10.244.0.188 on srv3, which is srv3'sCiliumInternalIP; its startup probe was refused 61 times between 12:48:14 and 12:58:14 withrestartCount: 0, while postgres inside that same container loggedstarted streaming WAL from primaryat 12:49:17 and completed a restartpoint at 12:55:25. Pod to pod worked, host to pod did not, the CNPG cluster stayed at one ready instance out of two, and the keycloak release timed out on the Cluster object rather than on anything the application did.31003636724 is the same collision seen from the other direction, which also explains the
i/o timeoutvariant that looked like a separate shape. The cdi-operator pod IP is 10.244.0.119 on srv1, srv1'sCiliumInternalIP. Its discovery request to10.96.0.1:443was translated by socket LB to its own node's apiserver at192.168.123.11:6443, and the cilium conntrack entry readsTxFlagsSeen=0x02 RxFlagsSeen=0x00. The SYN left, the reply was addressed to an address the node considers local, and the client sat there for its full 30 second discovery timeout.31067501623 is the loud mode.
vmselect-longterm-0was refused four sandboxes withputEndpointIdInvalid "IP ipv4:10.244.0.156 is already in use"and never got a pod IP; 10.244.0.156 is srv1'sCiliumNode.spec.ingress.ipv4, andcilium-dbg endpoint liston srv1 shows it held byreserved:ingress.For the last two I checked the negative as well, since attribution by elimination is worth little on its own. I compared every cilium infrastructure address, router and ingress on all three nodes, against every pod IP in both snapshots. There is no collision in either run, so #3740 is not present there and the LINSTOR provisioning failure stands alone.
That leaves this issue with the mechanism proven and the open question answered in the negative: in these six runs the retries always converge, so retry convergence is not what separates a run that continues from a run that stops. Whether the original keycloak instance in the body is the same story is worth one more look, because the one keycloak run I have here failed on the database pod being unreachable from its own node rather than on the client's retries. The two directions in the body still stand on their own merit as hygiene: a client that starts into an empty Service produces a restart and a scary log line every install, and nothing about that is desirable even when it costs nothing.
lexfrei commentedon Aug 18, 2026
One loose end from the same analysis, left unattributed on purpose. In run 30971312858 the alertmanager container listens on
[::]:9093from 03:53:19 and takes a SIGTERM at 03:54:06 with exit code 0, yet theKillingevent series for that container stops at 03:41:26 and its count never grows again. The liveness arithmetic points at kubelet (ten failures at five second periods against a 49 second container lifetime), but no collected event supports it, so I am not writing it down as a cause. What would settle it is the kubelet log for that pod over that window, which no bundle carries today. The run's attribution to #3740 does not rest on this either way, since the pod IP equals the node's router IP directly.