Skip to content

bug(apps/mariadb): the chart's my.cnf is mounted without a .cnf extension and the server never reads it #4836

Closed

Description

The MariaDB chart renders its tuned my.cnf, but the server never reads it. On the default t1.nano preset the instance then runs with the stock 128M InnoDB buffer pool inside a 128Mi memory limit, and in one e2e run both source replicas were OOM-killed in the middle of mariadb-2-backup-roundtrip.

The chart passes its config through myCnfConfigMapKeyRef with key: config (packages/apps/mariadb/templates/mariadb.yaml). The operator projects that key into /etc/mysql/conf.d under the same name, so the file lands as /etc/mysql/conf.d/config. MariaDB's !includedir reads only files ending in .cnf on Linux (option files docs), so this file is skipped. The operator's own 0-default.cnf in the same directory is read: its temp-pool shows up as a deprecation warning in the server log.

The run below shows the gap directly. The rendered ConfigMap says innodb_buffer_pool_size=64M, while every server start in both pods logs the default:

[Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m

The same ConfigMap also carries default_authentication_plugin, which is a MySQL option, and the server starts without complaining about it. This is probably also why the old hardcoded innodb_buffer_pool_size=60G never stopped a 128Mi pod from starting.

What happened in the e2e run

Run 37954293087, suite mariadb-2-backup-roundtrip, namespace tenant-root:

  • 17:59:52 to 18:01:38 UTC: the BackupJob runs and ends Succeeded.
  • 18:01:05: mariadb-mariadb-src-0 container mariadb is terminated with OOMKilled, exit 137 (the kubelet also logs a liveness 500 at the same second). The operator switches primary 0 to 1.
  • 18:02:46: the to-copy RestoreJob ends Succeeded.
  • 18:02:50: mariadb-mariadb-src-1 is OOMKilled, exit 137. At 18:02:51 to 18:02:52 the operator switches primary 1 to 0, and the MariaDB CR goes Ready=False with SwitchPrimary.
  • The suite fails with ($error == null) on its main script. The diagnostics snapshot was taken at 18:02:54.

The script's last check reads the sentinel back from the source through the <cr>-primary Service (examples/backups/mariadb/run-all.sh). My guess is that this read landed inside the second switchover. I can't confirm it: my copy of the job log stops before the script output, so I haven't seen the failing line.

What is and isn't established

The OOM kills are established from the pod status. The my.cnf being skipped is established from the server log against the ConfigMap. I haven't confirmed that the 64M pool from the chart would prevent the OOM, and I don't know what pushed src-1 over the limit at 18:02:50. Other runs on the same day, on different PRs, passed this suite.

A likely fix is renaming the key to something like my.cnf and checking SHOW VARIABLES LIKE 'innodb_buffer_pool_size' on a running instance. Renaming changes the effective config of every existing instance on its next rollout, since every setting in that file starts to apply. For the next red run of this suite it would help to keep the script's stderr, plus the source pods' lastState and memory usage.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    kind/flakeCategorizes issue or PR as related to a flaky test
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    on Oct 9, 2026
  2. added a commit that references this issue on Oct 10, 2026
    6144805
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugkind/flakeCategorizes issue or PR as related to a flaky testtriage/needs-triageIndicates an issue needs triage by a maintainer

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions