Skip to content

ClickHouse system-bucket backup needs pod restart after first credential projection #3986

Open

Description

@myasnikovdaniil

What is wrong?

ClickHouse with backup.useSystemBucket: true can start before cozy-backups-creds exists. Secret refs are optional, so sidecar starts without S3 values. First BackupJob projects Secret into tenant namespace, but environment of running clickhouse-backup container is not updated and backup still does not complete until pod restart.

This bug predates #3963. Scheme fix from that PR works when Secret existed before ClickHouse pod start.

How to reproduce

  1. Use tenant namespace without cozy-backups-creds.
  2. Create ClickHouse with backup.enabled: true and backup.useSystemBucket: true.
  3. Create BackupJob for it with backupClassName: cozy-default.
  4. Secret appears, but backup does not complete. Restart ClickHouse pod and retry, projected credentials are picked up.

Expected behaviour

First backup and restore should work without manual pod restart.

Acceptance

Make sidecar pick up credentials projected after pod start, or roll and wait for ClickHouse pod after projection. Add ClickHouse e2e which starts without Secret and runs backup and restore through cozy-default.

Update docs/operations/backup-classes.md and projector comment: projected endpoint stays bare, ClickHouse chart adds https://.

Related to #2734, projected Secret lifecycle must stay compatible with long-running ClickHouse sidecar.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    priority/backlogGeneral backlog priority. Lower than priority/important-longterm
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    triage/acceptedIndicates an issue is ready to be actively worked on
    on Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugpriority/backlogGeneral backlog priority. Lower than priority/important-longtermtriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions