bug(apps/mariadb): first boot still overruns the startup probe budget and bricks the datadir #3744

A fresh MariaDB sometimes does not finish mariadb-install-db and the entrypoint's temporary-server pass inside the startup probe budget. The kubelet kills the container in the middle of it, after the system tables exist but before the root grant lands. The restarted entrypoint sees a populated datadir, skips initialization for good, and brings the server up with the install-time root account (local, passwordless) instead of the credentials the operator holds. Every probe after that answers Access denied for user 'root', the pod never goes ready, and the HelmRelease install never finishes.

This is the failure packages/apps/mariadb/templates/mariadb.yaml already describes, and the reason it sets startupProbe.failureThreshold: 30 to lift the budget from 40s to 310s. Everything below happened after that landed, so 310s is not always enough.

Evidence

Run 31330211628, suite mariadb-2-backup-roundtrip, replica mariadb-mariadb-src-1. The container started at 20:59:09 and its log stops at [Note] [Entrypoint]: Initializing database files. Database files initialized never appears, 3306 never opens, and the agent probe answers 500 every 10s from 20:59:30 to 21:04:20:

{"level":"error","logger":"probe.liveness","msg":"error getting SQL client","error":"dial tcp [::1]:3306: connect: connection refused"}

Then Startup probe failed: HTTP probe failed with statuscode: 500, and Container mariadb failed startup probe, will be restarted at 21:04:20. That is 311s after start, against a budget of 20 + (30 - 1) * 10 = 310. The other replica did the same bootstrap in 13s in the same run: init at 20:59:08, Database files initialized at 20:59:10, ready for connections at 20:59:21.

Run 31230672907, suite mariadb, primary mariadb-test-0. State after the kill, in both the current and the previous container instance, once per probe interval:

2026-08-08  2:16:20 41 [Warning] Access denied for user 'root'@'::1' (using password: YES)

Run 31247006881, suites mariadb and mariadb-single. Pod mariadb-single-0 has the whole progression in its events. A single replica takes the operator's exec probe instead of the agent HTTP one, so the overrun reads differently and lands in the same place:

Startup probe failed: command timed out: "bash -c mariadb -u root -p\"${MARIADB_ROOT_PASSWORD}\" -e \"SELECT 1;\"" timed out after 5s
Startup probe failed: ERROR 2002 (HY000): Can't connect to local server through socket '/run/mysqld/mysqld.sock' (2)
Container mariadb failed startup probe, will be restarted
Startup probe failed: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

Why the HelmRelease hangs

The repo currently claims the opposite happens, so: helm-controller v1.5.0 defaults spec.waitStrategy.name to poller, so the install waits on kstatus. kstatus has no rules for k8s.mariadb.com/MariaDB and falls through to its Ready condition fallback, which maps Ready=False to InProgress. The operator holds Ready=False with reason StatefulSetNotReady until the StatefulSet is fully ready, so the install waits for every replica to pass startup and then readiness.

Measured inside run 31330211628: mariadb-single install 20:07:44 to Ready 20:08:35 (51s, one replica), mariadb-test 20:08:58 to 20:09:54 (56s, two replicas), and mariadb-mariadb-src started 20:58:48 and was still running at 21:04:26 with the CR at Ready=False / StatefulSetNotReady and status.replicas: 1.

Ruled out

Storage: the volume attached and mounted clean, ext4 mount at 20:59:00, no DRBD state change and no I/O error on that node between 20:55 and 21:06.

OOM: container status running with an empty lastState, nothing in the kernel ring buffer.

Node contention: the node sat at 2.51 of 8 cores and 12.9 of 23.4 GiB allocatable while the container stalled, so it was bounded by its own cgroup quota.

S3, for the round-trip suite: the failure is at the source instance, bucket provisioning had already reported ready and the backup step is six steps later.

Galera: this chart runs async replication. The CR carries replication.enabled: true and the server logs Plugin 'wsrep-provider' is disabled.

Frequency

83 runs of the pull-request and nightly workflows have a chainsaw report and ran the mariadb suites, spanning 2026-07-21 to 2026-08-10. Four carry a mariadb suite failure, all between 2026-08-07 and 2026-08-09. That clustering is a sampling artefact: pull-request artefacts expire after a few days, so the sample is 70 runs from 2026-08-07 on plus 13 lone nightly runs before that. Against the dense part it is 4 in 70. Neither the pod index nor the suite is fixed. Three failures are the primary and one a replica, three sit in mariadb or mariadb-single and one in mariadb-2-backup-roundtrip.

Not established

Why the stalled bootstrap costs about twenty times the CPU its sibling needed. In run 31330211628 the container held its full 250m quota for the whole 310s window, roughly 77 CPU-seconds, for work the other replica finished in under four. The only other resource near a ceiling was memory, at 94% of the 128Mi limit against the sibling's 81%. Sustained page-cache reclaim in a memory cgroup sitting just under its limit gives exactly that pair of readings, and it does not trigger an OOM kill because what gets reclaimed is file-backed. There is no direct measurement for it: neither PSI nor memory.stat is collected. Raising the default preset for this chart and re-running would settle it.

Without that, both obvious moves (a wider budget, more headroom for first boot) are guesses, so this is a report rather than a patch.

Two smaller things found on the way

examples/backups/mariadb/run-all.sh:135 and :170 allow the HelmRelease 300s to go Ready. By the mechanism above that wait spans each replica's 310s first-boot budget, so a slow but successful first boot still fails the harness. The 600s wait_for_field on the CR right after each of those lines is redundant for the same reason, which is the same observation as #2653.

hack/e2e-chainsaw/mariadb/chainsaw-test.yaml:47 says the HelmRelease "goes Ready as soon as helm applies the manifests -- it never waits for a pod". The three measurements above say otherwise, and the budget reasoning in the comments around it rests on that claim.