Repository navigation
bug(apps/mariadb): the chart's my.cnf is mounted without a .cnf extension and the server never reads it #4836
Description
The MariaDB chart renders its tuned my.cnf, but the server never reads it. On the default t1.nano preset the instance then runs with the stock 128M InnoDB buffer pool inside a 128Mi memory limit, and in one e2e run both source replicas were OOM-killed in the middle of mariadb-2-backup-roundtrip.
The chart passes its config through myCnfConfigMapKeyRef with key: config (packages/apps/mariadb/templates/mariadb.yaml). The operator projects that key into /etc/mysql/conf.d under the same name, so the file lands as /etc/mysql/conf.d/config. MariaDB's !includedir reads only files ending in .cnf on Linux (option files docs), so this file is skipped. The operator's own 0-default.cnf in the same directory is read: its temp-pool shows up as a deprecation warning in the server log.
The run below shows the gap directly. The rendered ConfigMap says innodb_buffer_pool_size=64M, while every server start in both pods logs the default:
[Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m
The same ConfigMap also carries default_authentication_plugin, which is a MySQL option, and the server starts without complaining about it. This is probably also why the old hardcoded innodb_buffer_pool_size=60G never stopped a 128Mi pod from starting.
What happened in the e2e run
Run 37954293087, suite mariadb-2-backup-roundtrip, namespace tenant-root:
- 17:59:52 to 18:01:38 UTC: the BackupJob runs and ends
Succeeded. - 18:01:05:
mariadb-mariadb-src-0containermariadbis terminated withOOMKilled, exit 137 (the kubelet also logs a liveness 500 at the same second). The operator switches primary 0 to 1. - 18:02:46: the to-copy RestoreJob ends
Succeeded. - 18:02:50:
mariadb-mariadb-src-1isOOMKilled, exit 137. At 18:02:51 to 18:02:52 the operator switches primary 1 to 0, and the MariaDB CR goesReady=FalsewithSwitchPrimary. - The suite fails with
($error == null)on its main script. The diagnostics snapshot was taken at 18:02:54.
The script's last check reads the sentinel back from the source through the <cr>-primary Service (examples/backups/mariadb/run-all.sh). My guess is that this read landed inside the second switchover. I can't confirm it: my copy of the job log stops before the script output, so I haven't seen the failing line.
What is and isn't established
The OOM kills are established from the pod status. The my.cnf being skipped is established from the server log against the ConfigMap. I haven't confirmed that the 64M pool from the chart would prevent the OOM, and I don't know what pushed src-1 over the limit at 18:02:50. Other runs on the same day, on different PRs, passed this suite.
A likely fix is renaming the key to something like my.cnf and checking SHOW VARIABLES LIKE 'innodb_buffer_pool_size' on a running instance. Renaming changes the effective config of every existing instance on its next rollout, since every setting in that file starts to apply. For the next red run of this suite it would help to keep the script's stderr, plus the source pods' lastState and memory usage.
Activity
fix(mariadb): mount the chart's my.cnf under a .cnf name (#4843)