[Bug] Cannot add a semi-sync replica to existing cluster

Describe the bug
I have a running semi-sync replica cluster with 1 master and 1 replica.
Trying to add a new replica fails since it cannot find binlogs required for replica

2023-06-19  9:47:12 12 [Note] Master connection name: 'mariadb-operator'  Master_info_file: '[email protected]'  Relay_info_file: '[email protected]'
2023-06-19  9:47:12 12 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='mariadb-myns-0.mariadb-myns.myns.svc.cluster.local', master_port='3306', master_log_file='', master_log_pos='4'.
2023-06-19  9:47:12 12 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos
2023-06-19  9:47:12 13 [Note] Master 'mariadb-operator': Slave I/O thread: Start semi-sync replication to master '[email protected]:3306' in log '' at position 4
2023-06-19  9:47:12 14 [Note] Master 'mariadb-operator': Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './[email protected]' position: 4; GTID position ''
2023-06-19  9:47:12 13 [Note] Master 'mariadb-operator': Slave I/O thread: connected to master '[email protected]:3306',replication starts at GTID position ''
2023-06-19  9:47:12 13 [ERROR] Master 'mariadb-operator': Error reading packet from server: Could not find GTID state requested by slave in any binlog files. Probably the slave state is too old and required binlog files have been purged. (server_errno=1236)
2023-06-19  9:47:12 13 [ERROR] Master 'mariadb-operator': Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Could not find GTID state requested by slave in any binlog files. Probably the slave state is too old and required binlog files have been purged.', Internal MariaDB error code: 1236
2023-06-19  9:47:12 13 [Note] Master 'mariadb-operator': Slave I/O thread exiting, read up to log 'FIRST', position 4; GTID position , master mariadb-myns-0.mariadb-myns.myns.svc.cluster.local:3306
2023-06-19  9:47:12 13 [Note] Master 'mariadb-operator': Failed to gracefully kill our active semi-sync connection with primary. Silently closing the connection.

Expected behaviour
New replica initializes properly.

Steps to reproduce the bug

Create cluster with semi-sync replication and 2 replicas
Run the cluster until older binlog file are purged
Try to scale replicas to 3

Environment details:

Kubernetes version: 1.26.5
mariadb-operator version: v0.0.15
Install method: helm

@fchiacchiaretta

Hey @fchiacchiaretta ! thanks for reporting this 🙏🏻

Unfortunately I didn't manage to reproduce the issue, it scaled from 2 to 3 after running this MariaDB for a couple of hours. Could you provide your MariaDB resource?

Run the cluster until older binlog file are purged

How did this happened? Was it somehow triggered by a MariaDB internal process or performed manually? It would be great to understand this first to reproduce it and have a better context. The error 1236 can happen due to multiple reasons and there are some remediations for it that can be potentially applied via spec.myCnf.

cc @grooverdan

New State master_log_file='' looks odd. Was an empty master_log_file actually used in CHANGE MASTER TO?

never mind. See that is standard "CHANGE MASTER" output.

@mmontes11

Hello @mmontes11 ,
thank you for your answer.
Here my MariaDB resource

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-myns
  namespace: myns
spec:
  rootPasswordSecretKeyRef:
    name: mariadb-credentials
    key: root-password
  database: mydb
  username: myuser
  passwordSecretKeyRef:
    name: mariadb-credentials
    key: password
  image:
    repository: mariadb
    tag: "10.6.13"
    pullPolicy: IfNotPresent
  port: 3306
  podSecurityContext:
    runAsUser: 0
  securityContext:
    allowPrivilegeEscalation: false
  livenessProbe:
    exec:
      command:
        - bash
        - -c
        - mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "SELECT 1;"
    initialDelaySeconds: 20
    periodSeconds: 10
    timeoutSeconds: 5
  readinessProbe:
    exec:
      command:
        - bash
        - -c
        - mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "SELECT 1;"
    initialDelaySeconds: 20
    periodSeconds: 10
    timeoutSeconds: 5
  replicas: 2
  replication:
    primary:
      podIndex: 0
      automaticFailover: true
      service:
        type: ClusterIP
    replica:
      waitPoint: AfterCommit
      gtid: CurrentPos
      connectionTimeout: 10s
      connectionRetries: 10
      syncTimeout: 10s
    syncBinlog: true
  resources:
    limits:
      memory: "4096Mi"
      cpu: "2000m"    
    requests:
      memory: "1024Mi"
      cpu: "500m"
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: "kubernetes.io/hostname"
  tolerations:
    - key: "mariadb.mmontes.io/ha"
      operator: "Exists"
      effect: "NoSchedule"
  podDisruptionBudget:
    minAvailable: "66%"
  metrics:
    exporter:
      image:
        repository: prom/mysqld-exporter
        tag: "v0.14.0"
        pullPolicy: IfNotPresent
      resources:
        requests:
          cpu: 50m
          memory: 64Mi
        limits:
          cpu: 300m
          memory: 512Mi
    serviceMonitor:
      prometheusRelease: prometheus
      interval: 10s
      scrapeTimeout: 10s
  myCnf: |
    [mysqld]
    bind-address=0.0.0.0
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    max_allowed_packet=256M
    max_user_connections = 1000
    innodb_log_file_size=3G
    innodb_buffer_pool_size=3G
    collation_server=utf8mb4_general_ci
    init_connect='SET NAMES utf8mb4'
    character_set_server=utf8mb4
    ## Caches and Limits
    ##
    tmp_table_size=32M
    max_heap_table_size=32M
    # Re-enabling as now works with Maria 10.1.2
    query_cache_type=1
    query_cache_limit=4M
    query_cache_size=256M
    thread_cache_size=50
    open_files_limit=65535
    table_definition_cache=4096
    table_open_cache=4096
  volumeClaimTemplate:
    resources:
      requests:
        storage: 50Gi
    storageClassName: default
    accessModes:
      - ReadWriteOnce
  service:
    type: ClusterIP

I did no special operation to trigger binlog purging, but on master pod I see that some files have been purged:

MariaDB [(none)]> show global status like '%binlog%';
+---------------------------------------+--------------------------+
| Variable_name                         | Value                    |
+---------------------------------------+--------------------------+
| Binlog_commits                        | 20759                    |
| Binlog_group_commits                  | 20758                    |
| Binlog_group_commit_trigger_count     | 0                        |
| Binlog_group_commit_trigger_lock_wait | 0                        |
| Binlog_group_commit_trigger_timeout   | 0                        |
| Binlog_snapshot_file                  | mariadb-myns-bin.000010  |
| Binlog_snapshot_position              | 50327739                 |
| Binlog_bytes_written                  | 50337747                 |
| Binlog_cache_disk_use                 | 0                        |
| Binlog_cache_use                      | 20759                    |
| Binlog_stmt_cache_disk_use            | 0                        |
| Binlog_stmt_cache_use                 | 0                        |
| Com_binlog                            | 0                        |
| Com_show_binlog_events                | 0                        |
| Com_show_binlogs                      | 0                        |
| Com_show_binlog_status                | 2                        |
+---------------------------------------+--------------------------+
16 rows in set (0.001 sec)

MariaDB [(none)]> SHOW BINARY LOGS;
+--------------------------+-----------+
| Log_name                 | File_size |
+--------------------------+-----------+
| mariadb-myns-bin.000007 | 286514232  |
| mariadb-myns-bin.000008 |  61618825  |
| mariadb-myns-bin.000009 | 144421966  |
| mariadb-myns-bin.000010 |  50329734  |
+--------------------------+-----------+
4 rows in set (0.000 sec)

MariaDB [(none)]> Bye
root@mariadb-myns-0:/# ls -l /var/lib/mysql/
total 3766964
-rw-rw---- 1 mysql mysql     475136 Jun 20 04:00 aria_log.00000001
-rw-rw---- 1 mysql mysql         52 Jun  7 10:26 aria_log_control
drwx------ 2 mysql mysql      12288 Jun 15 14:41 mydb
-rw-rw---- 1 mysql mysql          9 Jun  7 10:26 ddl_recovery-backup.log
-rw-rw---- 1 mysql mysql      16384 Jun 15 14:41 ddl_recovery.log
-rw-rw---- 1 mysql mysql     396372 May 30 07:32 ib_buffer_pool
-rw-rw---- 1 mysql mysql 3221225472 Jun 20 07:12 ib_logfile0
-rw-rw---- 1 mysql mysql   79691776 May 30 07:32 ibdata1
-rw-rw---- 1 mysql mysql   12582912 Jun  7 10:26 ibtmp1
drwx------ 2 mysql mysql      16384 May 24 10:48 lost+found
-rw-rw---- 1 mysql mysql  286514232 May 29 15:14 mariadb-myns-bin.000007
-rw-rw---- 1 mysql mysql   61618825 May 30 07:32 mariadb-myns-bin.000008
-rw-rw---- 1 mysql mysql  144421966 Jun  7 10:25 mariadb-myns-bin.000009
-rw-rw---- 1 mysql mysql   50327739 Jun 20 07:10 mariadb-myns-bin.000010
-rw-rw---- 1 mysql mysql        108 Jun  7 10:26 mariadb-myns-bin.index
-rw-rw---- 1 mysql mysql          0 May 24 10:48 multi-master.info
drwx------ 2 mysql mysql       4096 Jun  7 13:53 mysql
-rw-r--r-- 1 mysql mysql         15 May 24 10:48 mysql_upgrade_info
drwx------ 2 mysql mysql       4096 May 24 10:48 performance_schema
drwx------ 2 mysql mysql      12288 May 24 10:48 sys

I checked for expire_log_days and binlog_expire_logs_seconds but it seems that neither of them have been set.

I'm trying to reproduce the issue on a test cluster too, I'll be back if I have some additional results.

@grooverdan

@grooverdan we are not providing the MASTER_LOG_FILE, but we do specify MASTER_USE_GTID always. Would this be a problem with the @fchiacchiaretta's configuration?

mariadb-operator/pkg/client/client.go

Line 278 in 3f5e6a3

tpl := createTpl("change-master.sql", `CHANGE MASTER '{{ .Connection }}' TO

I checked for expire_log_days and binlog_expire_logs_seconds but it seems that neither of them have been set.

expire_log_days is actually in the container default. (which I'm open to remove from the defaults).

$ podman run mariadb:10.6 my_print_defaults --mysqld
--socket=/run/mysqld/mysqld.sock
--host-cache-size=0
--skip-name-resolve
--pid-file=/run/mysqld/mysqld.pid
--basedir=/usr
--expire_logs_days=10
--character-set-server=utf8mb4
--collation-server=utf8mb4_general_ci

expire_logs_days=0 in the configuration should override this. Check with select @@expire_logs_days.

So ConfigureReplica is expecting the full binary logs to exist? Is a mariadb-backup stream the entire datadir as a point-in-time copy from the primary to replica if the logs are missing required?

@grooverdan

@grooverdan I can confirm expire_logs_days is set to 10 as per default

MariaDB [(none)]> select @@expire_logs_days;
+--------------------+
| @@expire_logs_days |
+--------------------+
|          10.000000 |
+--------------------+
1 row in set (0.000 sec)

So ConfigureReplica is expecting the full binary logs to exist?

Not quite sure about this. ConfigureReplica performs the following CHANGE MASTER statement:

[Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='mariadb-myns-0.mariadb-myns.myns.svc.cluster.local', master_port='3306', master_log_file='', master_log_pos='4'.
[Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos

which does not take into account the master_log_file at any point. I'm guessing that if master_log_file is empty, it tries to default to a binary log that in this case was deleted. Does this behaviour depend somehow on the GTID? By default we use current_pos but we support specifying slave_pos via spec.replication.replica.gtid = SlavePos

Is a mariadb-backup stream the entire datadir as a point-in-time copy from the primary to replica if the logs are missing required?

Good question, unfortunately I'm not aware of mariabackuip internals. Maybe something to discuss with the MariaDB team?

expire_log_days is actually in the container default. (which I'm open to remove from the defaults).

I will add expire_logs_days here as well to avoid this situations in the future if that makes sense.

mariadb-operator/pkg/builder/statefulset_builder.go

Line 205 in 3f5e6a3

func buildStatefulSetArgs(mariadb *mariadbv1alpha1.MariaDB) []string {

As far as I know, setting up a new replica via mariabackup is the easiest way, docs for reference.

Having a sane default for expire_logs_days is good for production setup, as the PVC would fill up very fast with old binlogs that would be rarely used.

note came across this, MDEV-27437, which comes into play if the mariadb backup donor isn't the same version as replica.

Upon further testing, I found that this issue occurs also when deleting one of the already configured and running replica pods, here the steps I followed to reproduce:

create semi-sync replica cluster with 3 replicas
wait for binlog auto purging
delete one of the replica pods
wait for the deleted replica pod to be restarted

Upon restart, MariaDB properly restarts replication with the correct position and GTID

2023-06-21  9:11:39 3 [Note] Reading Master_info: '[email protected]'  Relay_info:'[email protected]'
2023-06-21  9:11:39 3 [Note] Initialized Master_info from '[email protected]'
2023-06-21  9:11:39 4 [Note] Master 'mariadb-operator': Slave I/O thread: Start asynchronous replication to master 'repl@mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local:3306' in log 'mdb-replica-test-bin.000002' at position 3892
2023-06-21  9:11:39 5 [Note] Master 'mariadb-operator': Slave SQL thread initialized, starting replication in log 'mdb-replica-test-bin.000002' at position 3892, relay log './[email protected]' position: 4; GTID position '0-10-22'
2023-06-21  9:11:39 3 [Note] Started replication for 'mariadb-operator'
2023-06-21  9:11:39 0 [Note] mariadbd: ready for connections.
Version: '10.6.14-MariaDB-1:10.6.14+maria~ubu2004-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2023-06-21  9:11:39 4 [Note] Master 'mariadb-operator': Slave I/O thread: connected to master 'repl@mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local:3306',replication starts at GTID position '0-10-22'

Checking logs on master pod, we can notice that server_id is wrong (stop with id 12, but start with id 1)

2023-06-21  9:11:28 49 [Note] Stop semi-sync binlog_dump to slave (server_id: 12)
2023-06-21  9:11:39 349 [Note] Start binlog_dump to slave_server(1), pos(mdb-replica-test-bin.000002, 3892), using_gtid(1), gtid('0-10-22')

After that, mariadb-operator kicks in and reconfigure replica with ConfigureReplica, stopping Slave Thread, resetting slave position and setting server_id programmatically (10 + podIndex)

2023-06-21  9:12:49 5 [Note] Master 'mariadb-operator': Error reading relay log event: slave SQL thread was killed
2023-06-21  9:12:49 5 [Note] Master 'mariadb-operator': Slave SQL thread exiting, replication stopped in log 'mdb-replica-test-bin.000002' at position 3892; GTID position '0-10-22', master: mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local:3306
2023-06-21  9:12:49 4 [Note] Master 'mariadb-operator': Slave I/O thread exiting, read up to log 'mdb-replica-test-bin.000002', position 3892; GTID position 0-10-22, master mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local:3306
2023-06-21  9:12:49 17 [Note] Master connection name: 'mariadb-operator'  Master_info_file: '[email protected]'  Relay_info_file: '[email protected]'
2023-06-21  9:12:49 17 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local', master_port='3306', master_log_file='mdb-replica-test-bin.000002', master_log_pos='3892'. New state master_host='mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local', master_port='3306', master_log_file='', master_log_pos='4'.
2023-06-21  9:12:49 17 [Note] Previous Using_Gtid=Current_Pos. New Using_Gtid=Current_Pos
2023-06-21  9:12:49 18 [Note] Master 'mariadb-operator': Slave I/O thread: Start semi-sync replication to master 'repl@mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local:3306' in log '' at position 4
2023-06-21  9:12:49 19 [Note] Master 'mariadb-operator': Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './[email protected]' position: 4; GTID position ''
2023-06-21  9:12:49 18 [Note] Master 'mariadb-operator': Slave I/O thread: connected to master 'repl@mdb-replica-test-0.mdb-replica-test.mdb-replica-test.svc.cluster.local:3306',replication starts at GTID position ''

Since replication restarts in log 'FIRST' at position 4 , if binlogs were purged replication fails.

@fchiacchiaretta

@fchiacchiaretta awesome investigation, thanks for the level of detail 💯 🥇

I've managed to reproduce the issue by:

Creating your MariaDB instance
Setting max-binlog-size=4096 and binlog_expire_logs_seconds=10
Ingest some data

As a result, the binary logs got deleted

MariaDB [(none)]> show binary logs;
+-------------------------+-----------+
| Log_name                | File_size |
+-------------------------+-----------+
| mariadb-repl-bin.000013 |   1992007 |
| mariadb-repl-bin.000014 |   1998579 |
| mariadb-repl-bin.000015 |   1989927 |
| mariadb-repl-bin.000016 |   1978851 |
| mariadb-repl-bin.000017 |   1933420 |
| mariadb-repl-bin.000018 |   1937694 |
| mariadb-repl-bin.000019 |    151378 |
| mariadb-repl-bin.000020 |       590 |
+-------------------------+-----------+

At this point, upscaling to 3 results in the following logs in the new replica:

mariadb 2023-06-21 16:45:16 7 [Note] Master 'mariadb-operator': Slave I/O thread: Start semi-sync replication to master '[email protected]:3306' in l
og '' at position 4
mariadb 2023-06-21 16:45:16 8 [Note] Master 'mariadb-operator': Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './mariadb-repl-relay-bin-mariadb@0
02doperator.000001' position: 4; GTID position ''
mariadb 2023-06-21 16:45:16 7 [Note] Master 'mariadb-operator': Slave I/O thread: connected to master '[email protected]:3306',replication starts at
GTID position ''
mariadb 2023-06-21 16:45:16 7 [ERROR] Master 'mariadb-operator': Error reading packet from server: Could not find GTID state requested by slave in any binlog files. Probably the slave state
is too old and required binlog files have been purged. (server_errno=1236)
mariadb 2023-06-21 16:45:16 7 [ERROR] Master 'mariadb-operator': Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Could not find GTID state requested by slave
in any binlog files. Probably the slave state is too old and required binlog files have been purged.', Internal MariaDB error code: 1236
mariadb 2023-06-21 16:45:16 7 [Note] Master 'mariadb-operator': Slave I/O thread exiting, read up to log 'FIRST', position 4; GTID position , master mariadb-repl-0.mariadb-repl.default.svc.c
luster.local:3306
mariadb 2023-06-21 16:45:16 7 [Note] Master 'mariadb-operator': Failed to gracefully kill our active semi-sync connection with primary. Silently closing the connection.

This happens with both MariaDB version 10.11.3 and 10.6.13. Having a look now 👀

I've manually managed to solve the issue by taking a backup from the primary and restoring it on the failing replica. Basically following this guide:

https://mariadb.com/kb/en/setting-up-a-replica-with-mariabackup/

The steps involved were:

Creating an extra PVC to share the backup

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: backup
spec:
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: standard

Stop the mariadb-operator
Edit the StatefulSet to mount the backup PVC

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb-repl
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: mariadb
        args:
        - --log-bin
        - --log-basename
        - mariadb-repl
        - --max-binlog-size=4095
        - --binlog_expire_logs_seconds=10
        volumeMounts:
          name: config
        - mountPath: /backup
          name: backup
      ....
      volumes:
      - name: backup
        persistentVolumeClaim:
          claimName: backup
....

Perform a backup on the primary:

mariabackup --backup --target-dir=/backup --user=root --password=mariadb --host mariadb-repl-0.mariadb-repl.default.svc.cluster.local --port 3306

Add an extra initContainer to restore the backup. This could be improved to conditionally restore the backup if /var/lib/mysql is empty (no PVC)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb-repl
spec:
  replicas: 3
  template:
    spec:
      initContainers:
      - command:
        - sh
        - -c
        - mariabackup --copy-back --target-dir=/backup
        volumeMounts:
        - mountPath: /backup
          name: backup
        - mountPath: /var/lib/mysql
          name: storage
        image: mariadb:10.6.13
        imagePullPolicy: IfNotPresent
      ....
      volumes:
      - name: backup
        persistentVolumeClaim:
          claimName: backup
....

Delete the PVC from the failing replica
Delete the failing replica
Start the mariadb-operator

Logs from the failing replica after doing this:

mariadb 2023-06-21 18:16:09 7 [ERROR] Master 'mariadb-operator': Error reading packet from server: Could not find GTID state requested by slave in any binlog files. Probably the slave state
is too old and required binlog files have been purged. (server_errno=1236)
mariadb 2023-06-21 18:16:09 7 [ERROR] Master 'mariadb-operator': Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Could not find GTID state requested by slave
in any binlog files. Probably the slave state is too old and required binlog files have been purged.', Internal MariaDB error code: 1236
mariadb 2023-06-21 18:16:09 7 [Note] Master 'mariadb-operator': Slave I/O thread exiting, read up to log 'FIRST', position 4; GTID position , master mariadb-repl-0.mariadb-repl.default.svc.c
luster.local:3306
mariadb 2023-06-21 18:16:09 7 [Note] Master 'mariadb-operator': Failed to gracefully kill our active semi-sync connection with primary. Silently closing the connection.
mariadb 2023-06-21 18:16:34 8 [Note] Master 'mariadb-operator': Error reading relay log event: slave SQL thread was killed
mariadb 2023-06-21 18:16:34 8 [Note] Master 'mariadb-operator': Slave SQL thread exiting, replication stopped in log 'FIRST' at position 4; GTID position '', master: mariadb-repl-0.mariadb-r
epl.default.svc.cluster.local:3306
mariadb 2023-06-21 18:16:34 13 [Note] Deleted Master_info file '/var/lib/mysql/master.info'.
mariadb 2023-06-21 18:16:34 13 [Note] Deleted Master_info file '/var/lib/mysql/relay-log.info'.
mariadb 2023-06-21 18:16:34 13 [Note] Semi-sync replication initialized for transactions.
mariadb 2023-06-21 18:16:34 13 [Note] Semi-sync replication enabled on the master.
mariadb 2023-06-21 18:16:34 0 [Note] Starting ack receiver thread
mariadb 2023-06-21 18:16:34 15 [Note] Start binlog_dump to slave_server(11), pos(, 4), using_gtid(1), gtid('')
mariadb 2023-06-21 18:16:34 15 [Note] Start semi-sync binlog_dump to slave (server_id: 11), pos(./mariadb-repl-bin.000001, 4)
mariadb 2023-06-21 18:16:55 20 [Note] Start binlog_dump to slave_server(10), pos(, 4), using_gtid(1), gtid('')
mariadb 2023-06-21 18:16:56 20 [Note] Start semi-sync binlog_dump to slave (server_id: 10), pos(./mariadb-repl-bin.000001, 4)

This would be hard to automate and requires supporting mariabackup, contributions are welcome!

For the time being I think the best thing will be acknowledging this limitation and allow the user to configure the binlog:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-repl
spec:
  ...
  replication:
    ...
    binlog:
       sync: true
       maxSize: 4095 # default to 1GB
       expireSeconds: 86400 # default to 0

The defaults were taken from the oficial MariaDB image:

Checking logs on master pod, we can notice that server_id is wrong (stop with id 12, but start with id 1)

Regarding this, it could be related to the fact that the server_id is ephemeral and set up on the fly by the operator here:

mariadb-operator/pkg/controller/replication/config.go

Line 98 in 3f5e6a3

"server_id": serverId(primaryPodIndex),

mariadb-operator/pkg/controller/replication/config.go

Line 119 in 3f5e6a3

"server_id": serverId(ordinal),

We could use an initContainer and dynamically set the server_id in a configuration time, but it will be worth ensuring that this is related to the problem.

Note the mariabackup needs to be --prepared. This will apply redo log changes made during the backup.

Steps for a mariabackup based provisioning of a replica:

On replica in mariadb container (uninitialized but with volume)

mkdir /var/lib/mysql/.mariabackup
chown -R mysql: /var/lib/mysql
socat -u TCP-LISTEN:4444,reuseaddr stdio | gosu mysql mbstream -x -C /var/lib/mysql/.mariabackup

note obviously any port, I couldn't see a "just pick one" socat option, but all should be free in the container namespace.
other socat options like OPENSSL-LISTEN can also be used too. Requires all the usual tls stuff however. If network encryption is required an openssl enc in the pipeline with a random operator password is probably easier.

on donor, exec in the container

mariabackup --backup --user backupuser --stream=xbstream | socat -u stdio TCP:{replicaIP}:4444

backupuser is the database user with required grants.

on replica

gosu mysql mariabackup --target-dir=/var/lib/mysql/.mariabackup --prepare

read gtid from /var/lib/mysql/.mariabackup/xtrabackup_binlog_info

Not sure if its easy, but if provisioning multiple replicas at this point this directory could be copied. Or multiple destinations in step 2 like:

mariabackup --backup --user dan --stream=xbstream | tee >(socat -u stdio TCP:repl1:4444)  | socat -u stdio TCP:repl2:4445

or more than 2:

mariabackup --backup --user dan --stream=xbstream | tee >(socat -u stdio TCP:repl1:4444) >(socat -u stdio TCP:repl2:4444)  | socat -u stdio TCP:repl3:4445

ref: https://unix.stackexchange.com/questions/28503/how-can-i-send-stdout-to-multiple-commands

on replica

gosu mysql mariabackup --target-dir=/var/lib/mysql/.mariabackup --datadir=/var/lib/mysql/.restore --move-back
rm -rf /var/lib/mysql/.mariabackup
mv /var/lib/mysql/.restore/* /var/lib/mysql/

replica can be started with standard entrypoint now.
using gtid from step 3

SET GLOBAL gtid_slave_pos = "0-1-2";
CHANGE MASTER TO 
   MASTER_HOST="dbserver1", 
   MASTER_PORT=3306, 
   MASTER_USER="repl",  
   MASTER_PASSWORD="password", 
   MASTER_USE_GTID=slave_pos;
START SLAVE;

ref: https://mariadb.com/kb/en/setting-up-a-replica-with-mariabackup/#gtids

This issue is stale because it has been open 60 days with no activity.

We have made some progress by supporting PhysicalBackups: #1266

This will be used for rebuilding replicas in further releases.

This issue is stale because it has been open 60 days with no activity.

This issue was closed because it has been stalled for 30 days with no activity.

To be addressed as part of: #1423

Navigation Menu

Repository navigation

[Bug] Cannot add a semi-sync replica to existing cluster #141

52 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug] Cannot add a semi-sync replica to existing cluster #141

Description

Activity

mmontes11 commented on Jun 19, 2023

grooverdan commented on Jun 19, 2023

fchiacchiaretta commented on Jun 20, 2023

mmontes11 commented on Jun 20, 2023

grooverdan commented on Jun 20, 2023

fchiacchiaretta commented on Jun 20, 2023

mmontes11 commented on Jun 20, 2023

fchiacchiaretta commented on Jun 20, 2023

grooverdan commented on Jun 21, 2023

fchiacchiaretta commented on Jun 21, 2023

mmontes11 commented on Jun 21, 2023

mmontes11 commented on Jun 21, 2023

grooverdan commented on Jun 22, 2023

52 remaining items

github-actions commented on Apr 9, 2025

github-actions commented on Jun 10, 2025

mmontes11 commented on Jun 26, 2025

github-actions commented on Aug 26, 2025

github-actions commented on Sep 28, 2025

mmontes11 commented on Sep 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions