backup-controller CRDs fail to apply: x-cozystack-options is not valid in apiextensions.k8s.io/v1 schemas #2822

What happened

The cozy-backup-controller HelmRelease has been failing on every reconcile since #2778 merged. helm-controller cannot apply the four backups.cozystack.io CRDs:

Helm upgrade failed for release cozy-backup-controller/backup-controller:
failed to create resource: failed to create typed patch object
(/backupjobs.backups.cozystack.io; apiextensions.k8s.io/v1, Kind=CustomResourceDefinition): errors:
  .spec...applicationRef.properties.kind.x-cozystack-options: field not declared in schema
  .spec...backupClassName.x-cozystack-options: field not declared in schema
  .spec...planRef.properties.name.x-cozystack-options: field not declared in schema

As a result the CRDs are never created and the backup-controller pod crashloops:

"if kind is a CRD, it should be installed before calling Start","kind":"Plan.backups.cozystack.io",
"error":"no matches for kind \"Plan\" in version \"backups.cozystack.io/v1alpha1\""

Observed in the v1.5.0-alpha.1 release-cut run (https://github.com/cozystack/cozystack/actions/runs/26877029597, PR #2803): HR with failures: 127 and 156 failed Helm revisions over ~85 minutes.

This is broken on main, not just on the release branch. A green E2E run (https://github.com/cozystack/cozystack/actions/runs/26920127507) carries the identical failure in its cozyreport — same UpgradeFailed, backup-controller pods in CrashLoopBackOff. Backups are entirely non-functional in any build since #2778, including the v1.5.0-alpha.1 assets. Release blocker for v1.5.0.

Why CI stayed green

hack/e2e-install-cozystack.bats has two holes that mask any permanently-failing HelmRelease:

1. The final check echoes but never fails:

   # Fail the test if any HelmRelease is not Ready
   if kubectl get hr -A | grep -v " True " | grep -v NAME; then
     kubectl get hr -A
     echo "Some HelmReleases failed to reconcile" >&2
   fi    # <- no exit 1

2. The HR wait fan-out (kubectl wait ... & piped into sh -ex, ending in wait) — POSIX wait without arguments returns 0 regardless of children's exit codes.

Root cause

#2778 (commits d0c74f9, 4c34e0f) injects the x-cozystack-options vendor extension into the real Kubernetes CRDs under packages/system/backup-controller/definitions/ via hack/inject-cozystack-options.awk, called from hack/update-codegen.sh.

This cannot work for apiextensions CRDs, in two independent ways:

1. Not applyable: JSONSchemaProps is a closed struct — the only vendor extensions it supports are the explicit x-kubernetes-* fields. helm-controller's server-side-apply machinery builds a typed patch object and hard-fails on any unknown key.
2. Not storable: even with a lenient client, kube-apiserver drops unknown fields at decode — the extension would never be persisted or served. The consumer (use-crd-schema.ts in cozystack-ui, which GETs the live CRD object) could never see it through this path.

The same extension works everywhere else because app schemas live in ApplicationDefinitions served by cozystack-api, which controls its own OpenAPI emission. The backups group is the only place where it was put into a vanilla-apiserver CRD.

Proposed fix

Option A (preferred) — move the field→provider mapping into CRD metadata.annotations, which the apiserver preserves:

• Add +kubebuilder:metadata:annotations markers on the four types in api/backups/v1alpha1 (per-CRD target lists currently encoded in update-codegen.sh), e.g. for BackupJob:
  options.cozystack.io/fields=spec.applicationRef.kind:appkind,spec.planRef.name:plan,spec.backupClassName:backupclass
• Delete hack/inject-cozystack-options.awk and the inject_cozystack_options block in hack/update-codegen.sh; regenerate the definitions (extension out of the schema, annotation in metadata).
• cozystack-ui: graft x-cozystack-options into the parsed schema from the annotation in use-crd-schema.ts (it already fetches the full CRD object, and enrichSchemaWithEnums already does client-side grafting for applicationRef.name).

Option B (smaller) — drop the extension from the CRDs entirely and hardcode the three dropdown sources in the UI backup pages, like applicationRef.name already is.

Both options additionally need:

• e2e-install-cozystack.bats must exit 1 when any HelmRelease is not Ready (and/or check the fanned-out kubectl wait exit codes), so a permanently-failing platform HR can never ship green again.

Affected

• packages/system/backup-controller/definitions/backups.cozystack.io_{backupjobs,backups,plans,restorejobs}.yaml
• hack/update-codegen.sh, hack/inject-cozystack-options.awk
• hack/e2e-install-cozystack.bats
• cozystack-ui: apps/console/src/lib/use-crd-schema.ts, backup create pages

Refs: #2778, #2803