feat(platform): add e2e tests for backup altinity strategy for clickhouse - #2600

Port the intended ClickHouse Altinity backup/restore e2e coverage from
#2600 (never-merged hack/e2e-apps/backup-clickhouse.bats) to the
current Chainsaw approach, reworked on origin/main where the Altinity
backup feature and examples/backups/clickhouse already landed.

Two Tests, etcd-suite shaped:
- clickhouse-backup-1-contracts (CI): a backup-enabled ClickHouse, asserting
  the chart<->Altinity-strategy contract against the running cluster — the
  clickhouse-backup sidecar on the ch-backup-api :7171 port and the populated
  <release>-backup-api-auth Secret the strategy Pod authenticates with. Needs
  no reachable object store.
- clickhouse-backup-2-roundtrip: the full Bucket -> BackupJob -> Backup ->
  in-place + to-copy RestoreJob round-trip, driving
  examples/backups/clickhouse/run-all.sh. Gated out of CI
  (CLICKHOUSE_E2E_S3_ROUNDTRIP=1) because kind's self-signed SeaweedFS S3
  endpoint cannot be TLS-validated by the in-Pod sidecar.

Wire clickhouse-application -> "clickhouse clickhouse-backup" in
hack/select-e2e.sh so ClickHouse chart edits exercise the backup suite too.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

Port the intended ClickHouse Altinity backup/restore e2e coverage from
#2600 (never-merged hack/e2e-apps/backup-clickhouse.bats) to the
current Chainsaw approach, reworked on origin/main where the Altinity
backup feature and examples/backups/clickhouse already landed.

Two Tests, etcd-suite shaped:
- clickhouse-backup-1-contracts (CI): a backup-enabled ClickHouse, asserting
  the chart<->Altinity-strategy contract against the running cluster — the
  clickhouse-backup sidecar on the ch-backup-api :7171 port and the populated
  <release>-backup-api-auth Secret the strategy Pod authenticates with. Needs
  no reachable object store.
- clickhouse-backup-2-roundtrip: the full Bucket -> BackupJob -> Backup ->
  in-place + to-copy RestoreJob round-trip, driving
  examples/backups/clickhouse/run-all.sh. Gated out of CI
  (CLICKHOUSE_E2E_S3_ROUNDTRIP=1) because kind's self-signed SeaweedFS S3
  endpoint cannot be TLS-validated by the in-Pod sidecar.

Wire clickhouse-application -> "clickhouse clickhouse-backup" in
hack/select-e2e.sh so ClickHouse chart edits exercise the backup suite too.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

Port the intended ClickHouse Altinity backup/restore e2e coverage from
#2600 (never-merged hack/e2e-apps/backup-clickhouse.bats) to the
current Chainsaw approach, reworked on origin/main where the Altinity
backup feature and examples/backups/clickhouse already landed.

Two Tests, etcd-suite shaped:
- clickhouse-backup-1-contracts (CI): a backup-enabled ClickHouse, asserting
  the chart<->Altinity-strategy contract against the running cluster — the
  clickhouse-backup sidecar on the ch-backup-api :7171 port and the populated
  <release>-backup-api-auth Secret the strategy Pod authenticates with. Needs
  no reachable object store.
- clickhouse-backup-2-roundtrip: the full Bucket -> BackupJob -> Backup ->
  in-place + to-copy RestoreJob round-trip, driving
  examples/backups/clickhouse/run-all.sh. Gated out of CI
  (CLICKHOUSE_E2E_S3_ROUNDTRIP=1) because kind's self-signed SeaweedFS S3
  endpoint cannot be TLS-validated by the in-Pod sidecar.

Wire clickhouse-application -> "clickhouse clickhouse-backup" in
hack/select-e2e.sh so ClickHouse chart edits exercise the backup suite too.

Assisted-By: Claude <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>

…cking the backup e2e round-trip (#3385)

## What this PR does

Gives the ClickHouse `clickhouse-backup` sidecar a way to trust an S3
endpoint whose certificate is signed by a private CA, and uses that to
make the ClickHouse backup/restore e2e round-trip actually run in CI.
This actualizes #2600, whose feature code and example scripts have since
landed on `main` independently, and whose remaining contribution — the
e2e test — was still in the retired BATS format. #2600 can be closed
once this merges.

### 1. `backup.endpointCA` on the ClickHouse chart

The sidecar had **no TLS/CA surface at all**: bucket coordinates and
credentials arrive as `S3_*` env, and nothing let it verify an endpoint
signed by a private CA. Cozystack's own in-cluster SeaweedFS is exactly
that case — it serves `:8333` behind the self-signed "SeaweedFS CA" — so
a ClickHouse release could only ever back up to a publicly-trusted
endpoint. MariaDB (`storage.s3.tls.caSecretKeyRef`) and Postgres (barman
`endpointCA`) have had a CA field all along; this closes the gap for
ClickHouse.

`backup.endpointCA` takes `{name, key}`. When set, the chart mounts that
Secret into the sidecar and points `SSL_CERT_DIR` at it. Empty — the
default — renders no env, no mount and no volume, so existing releases
are byte-for-byte unchanged.

**Why `SSL_CERT_DIR` and not `AWS_CA_BUNDLE`.** Both are honoured by
`altinity/clickhouse-backup:2.7.4`; I measured them against a
self-signed endpoint and public AWS S3 in turn. Go reads `SSL_CERT_DIR`
*in addition to* the system bundle, so both endpoints verify.
`AWS_CA_BUNDLE` *replaces* the SDK's trust pool: the private endpoint
verifies, and public S3 then fails `x509: certificate signed by unknown
authority` — it would silently break a release that also talks to a
public endpoint. `S3_DISABLE_CERT_VERIFICATION` was rejected outright,
since it drops verification rather than trusting one more CA.

### 2. The e2e suite, and what it was really testing

Two Tests added to the existing `hack/e2e-chainsaw/clickhouse/` suite
(in name order, mirroring the `etcd`/`mariadb`/`postgres` shape — see
the layout rationale below for why they live here rather than in a new
directory):

- **`clickhouse-1-backup-contracts`** (CI, `tenant-test`, ~98s): a
backup-enabled ClickHouse asserted against the *running* cluster — the
`clickhouse-backup` sidecar on the `ch-backup-api` `:7171` port, and the
`<release>-backup-api-auth` Secret whose keys the sidecar's
`API_USERNAME`/`API_PASSWORD` actually resolve `valueFrom`. Each is a
silent break if a chart bump renames it. Needs no object store.
- **`clickhouse-2-backup-roundtrip`** (CI, `tenant-root`): `Bucket` →
`Altinity` strategy + `BackupClass` → source ClickHouse with a sentinel
row → `BackupJob` → in-place `RestoreJob` → to-copy `RestoreJob` into a
second instance, driving `examples/backups/clickhouse/run-all.sh` so
test and docs cannot drift.

The round-trip was originally gated behind `CLICKHOUSE_E2E_S3_ROUNDTRIP`
and **skipped in CI** — its `PASS (1.01s)` meant the gate fired, not
that a backup happened. `backup.endpointCA` removes the reason for the
gate, so the suite now adopts the same three preconditions
mariadb/postgres rely on: run in `tenant-root` (an isolated tenant's
Cilium egress allowlist blocks its Pods from reaching `tenant-root`'s
SeaweedFS), target the in-cluster endpoint instead of the `Bucket`'s
external ingress URL (an unroutable placeholder in CI), and trust its
certificate via the copied CA. It also pre-cleans with `cleanup.sh`,
because `tenant-root` is shared and a leftover `Succeeded` `BackupJob`
would satisfy the harness's wait without a backup ever running.

Both backup Tests live in the existing `hack/e2e-chainsaw/clickhouse/`
suite rather than a new suite directory, so no selector mapping is
added: `hack/select-e2e.sh` already derives `clickhouse` both from
`packages/apps/clickhouse/**` (the `*-application` rule) and from
`examples/backups/clickhouse/**` (the harness path), and both paths
therefore select the suite that actually runs the harness. That
alignment is the reason for the layout — a separate `clickhouse-backup/`
directory would have made an edit to the harness select a suite that
cannot exercise it. `hack/select-e2e.sh` and `hack/select-install.sh`
are otherwise untouched (the only change is a comment recording that
invariant), and the existing `backup example harness edit selects its
app suite` case in `hack/select-e2e_test.bats` covers the mapping.

### Verification

- `helm unittest` — 17 pass, including 5 new `endpointCA` cases: the
unset default renders nothing, the env/mount/volume wiring, a custom key
still projected as `ca.crt`, composition with `useSystemBucket`, and
`backup.enabled=false` renders nothing.
- The CA-trust behaviour was measured against the real image, not
assumed: without a CA knob the sidecar reports `x509: certificate signed
by unknown authority`; with `SSL_CERT_DIR` the handshake succeeds.
- `hack/select-e2e_test.bats` (16) and `hack/select-install_test.bats`
(17) pass; `select-install.sh --validate` reports graph OK.
- `make generate` is idempotent, so the pre-commit job's `git diff
--exit-code` stays clean.
- The contracts Test passed in CI on earlier runs of this branch. The
round-trip only became CI-live with this work, so it still needs a green
E2E run before merge — the branch has since been rebased onto current
`main` to get one.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff, file by file. The load-bearing
change is the new `backup.endpointCA` field in
`packages/apps/clickhouse/values.yaml` / `values.schema.json`;
everything else is `hack/e2e-chainsaw/`, `examples/backups/`, one docs
paragraph, and regenerated artifacts, none of which appear in the map.

- [ ] No downstream repository is affected by this change
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
cozystack/terraform-provider-cozystack#18 —
"add, remove or rename a field in an app's `values.schema.json`" reaches
the provider's hand-maintained schema, model and expand/flatten pair, so
the new nested object is not surfaced until it is added there. Opened as
an issue rather than a PR because the natural moment is alongside that
repo's next `api/apps/v1alpha1` pin bump, which is a maintainer call.

Not ticked, with reasons rather than silence: **cozystack/website** —
the trigger is *adding, renaming or removing a package*, and this adds a
field to an existing one. `clickhouse` is already in the website
`Makefile`'s app list and its reference page is generated from this
package's `README.md`, so the regenerated `backup.endpointCA` rows are
carried by the existing release-tag automation with no manual PR.
**cozystack/external-apps-example** — vendors `hack/package.mk` and
`update-crd.sh`, neither of which this PR touches.
**cozystack/ansible-cozystack**, **ccp**, **talm**, **cozyhr**,
**cozy-proxy**, **cozystack-telemetry-server**, **examples** — no
installer values, `hack/` layout, namespace, variant, node-prerequisite,
annotation, label or metric in their trigger rows is touched here.

### Release note

```release-note
feat(clickhouse): add `backup.endpointCA` so the clickhouse-backup sidecar can verify an S3 endpoint whose certificate is signed by a private CA, such as Cozystack's in-cluster SeaweedFS. Leave it unset for publicly-trusted endpoints; existing releases are unaffected.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional private S3 endpoint CA support for ClickHouse
backup/restore sidecars via `backup.endpointCA`, including configurable
Secret name and CA bundle key (default `ca.crt`).
* Sidecars mount the CA and extend trust while leaving public endpoints
unchanged.

* **Documentation**
* Updated ClickHouse chart docs, operational backup guidance, and
backup/restore examples to cover CA setup, overrides, and cleanup.

* **Tests**
* Added helm-unittest and end-to-end coverage validating CA
rendering/mounting behavior and ClickHouse backup/restore workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->