Reported in the community Slack: on a 1.6.0 production cluster the velero image is velero/velero:v1.18.1, which added the Queued and ReadyToStart backup phases, but the CRDs in the cluster still dated from the initial install. The API server rejected every phase transition (status.phase: Unsupported value: "Queued"), so no backup could leave New. Backups were silently down for 5 days while the HelmRelease stayed green (Ready=True, UpgradeSucceeded).
The CRDs are frozen at whatever version was first installed, for three stacked reasons:
- Helm never touches the chart's crds/ directory on upgrade, that only works on the first install.
- The generated HelmRelease leaves spec.upgrade.crds at the helm-controller default (Skip). The operator already supports a per-component override (ComponentInstall.UpgradeCRDs -> HelmRelease.spec.upgrade.crds), but the velero PackageSource does not set it.
- The package sets upgradeCRDs: false in values.yaml, disabling the upstream chart's upgrade-crds Job. The comment there justifies it with "CRDs ship via the chart's crds/ directory", which only holds for the initial install.
Fix: add upgradeCRDs: CreateReplace to the velero component in packages/core/platform/sources/velero.yaml, the same fix mongodb-operator got in e9aaee2. The values.yaml comment should be corrected as well.
Velero only extends the phase enums between versions, so CreateReplace is safe here.
Other packages that ship CRDs via a vendored crds/ directory are also on Skip; that audit is worth doing separately.
Reported in the community Slack: on a 1.6.0 production cluster the velero image is velero/velero:v1.18.1, which added the Queued and ReadyToStart backup phases, but the CRDs in the cluster still dated from the initial install. The API server rejected every phase transition (status.phase: Unsupported value: "Queued"), so no backup could leave New. Backups were silently down for 5 days while the HelmRelease stayed green (Ready=True, UpgradeSucceeded).
The CRDs are frozen at whatever version was first installed, for three stacked reasons:
Fix: add upgradeCRDs: CreateReplace to the velero component in packages/core/platform/sources/velero.yaml, the same fix mongodb-operator got in e9aaee2. The values.yaml comment should be corrected as well.
Velero only extends the phase enums between versions, so CreateReplace is safe here.
Other packages that ship CRDs via a vendored crds/ directory are also on Skip; that audit is worth doing separately.