fix(kubernetes, opensearch, foundationdb)!: let tenants pick versions, not images - #4287

## What this PR does

Removes the HTTPCache application.

Telemetry says it is the least installed app we ship: 3 instances in
June, July and August 2026, 4 so far in September, out of 1162-1418
clusters reporting per month. That is not zero, so I am not going to
claim nobody uses it. The maintenance side is what does not hold up.

`make update` in this package is broken. All eight sed lines in the
`update` target patch `images/nginx/Dockerfile`, but the image lives in
`images/nginx-cache/Dockerfile`, so the target fails on the first line.
It is the only package under `packages/apps/` pointing at a Dockerfile
that does not exist. The pins show what that cost us:

```
nginx              1.25.3   -> 1.31.5
ngx_cache_purge    2.5.3    -> 3.0.2
nginx-module-vts   0.2.2    -> 0.2.7
ip2proxy-c         4.1.2    -> 4.3.0
IP2Location-C      8.6.1    -> 8.7.0
ip2location-nginx  8.6.0    -> 8.7.0
ip2proxy-nginx     8.1.1    -> 8.2.0
51Degrees          3.2.21.1 -> 3.2.21.1 (frozen upstream)
```

So we ship an old nginx with third party C modules compiled into it, and
nothing tests it: no chainsaw suite, no `helm unittest` target. I don't
think we should keep carrying that for 4 instances.

Removed: the chart in `packages/apps/http-cache`, its ResourceDefinition
in `packages/system/http-cache-rd`, the values type in
`api/apps/v1alpha1/httpcache`, the `cozystack.http-cache-application`
PackageSource, its line in the `naas` bundle, the `httpcaches` entry in
`cozy:tenant:admin:base`, the dashboard icon, and the entry in the root
`Makefile` build list.

`cozy:tenant:admin:base` enumerates the `apps.cozystack.io` resources a
tenant owner may write, so `httpcaches` comes out of it with the rest of
the application. A cluster that freezes gets the grant back from the
migration instead, which keeps the chart's list tracking the catalog and
leaves no entry behind for a later sweep to read as dead config.

### Existing installations

A cluster that still runs an HTTPCache keeps it, served from a frozen
source. A cluster that never created one keeps nothing.

Without a migration an upgrade would break both.
`templates/sources.yaml` is a bare glob with no keep annotation, so Helm
prunes the http-cache PackageSource; the ArtifactGenerator is owned by
it and goes too, and both ExternalArtifacts go with the generator. Those
are the `chartRef` targets of `hr/http-cache-rd` and of every tenant
`http-cache-<name>` release, so all of them would sit `Ready=False` for
good and `HelmReleaseNotReady` would fire on every cluster with the naas
bundle.

Migration 58 runs as a pre-upgrade hook, before that prune, and branches
on whether any instance exists. Its header comment carries the mechanism
in full; this is the summary.

If one does, it copies the packages `OCIRepository` and pins the copy to
the manifest digest that object resolves at hook time, which is the
release the instances are rendering from right then. `spec.ref.digest`
is used when the installer pinned one; a tag or semver ref goes through
`status.artifact.revision`, whose digest half is the manifest digest.
`status.artifact.digest` is the checksum of the downloaded tarball and
cannot stand in, and a source with neither aborts the migration rather
than publishing a source that points nowhere. Reading the digest live
rather than baking one into the script makes the freeze a no-op by
construction and asks no question about which release was the last to
ship the chart.

The copy takes the url, the pull secret, the interval and any
verification block from the live object, so an air-gapped or mirrored
registry keeps serving exactly what it already served. It is written
with no owner references and no Helm metadata, and with the
`platform.cozystack.io/no-delete` label the platform chart puts on the
original, because it becomes the only chart source those instances have.
The PackageSource is then repointed at the copy, annotated
`helm.sh/resource-policy: keep` and stripped of its release metadata, so
neither Helm nor a later release adopts or prunes it. Helm re-reads that
annotation off the live object when it prunes, which is what makes
disowning from inside a pre-upgrade hook work.

Names stay as they are, deliberately. The ApplicationDefinition inside
that artifact hardcodes its `chartRef`, so a renamed PackageSource would
leave every release pointing at an artifact nobody produces. Because the
names hold, `hr/http-cache-rd` and every tenant release keep the
`chartRef` they already have and nothing goes not ready. What changes is
where the chart comes from: a pinned digest that no later release moves.

The migration also applies `cozy:tenant:admin:http-cache-frozen`, a
ClusterRole labelled `rbac.cozystack.io/aggregate-to-tenant-admin`,
granting the verbs `cozy:tenant:admin:base` used to carry on
`httpcaches`. The hook applies it before the upgrade drops the rule from
the chart, so an owner's access never lapses and nothing changes for
them. Without it a frozen instance would be visible and undeletable by
the person who owns it: the wildcard over `apps.cozystack.io` lives in
`cozy:tenant:base`, which aggregates into `cozy:tenant` and not into
`cozy:tenant:admin`, `cozy:tenant:view:base` is read only, and no role
in the tenant chain carries a verb on `helm.toolkit.fluxcd.io`.

If no instance exists, the migration removes the platform side instead:
the Package first and waited on, because the operator reconciles off an
informer cache and re-creates the release it owns while that Package is
still there, then the release, its ApplicationDefinition and its Helm
storage. Nothing is pinned or granted on a cluster that never used the
application.

Every path is fail-closed. A failed fleet scan, a failed read of the
source OCIRepository, a source with no digest to freeze on, a failed
patch or a failed delete stops the migration before it stamps the
version.

The migration is numbered 58, because main already carries a 57 from the
SeaweedFS volume-size-limit change and its `targetVersion` is already
58. #4287 wants 58 as well, so whichever of the two lands second
renumbers and bumps `targetVersion` with it. main is merged in rather
than rebased, because the branch carries commits that are not mine to
rewrite; the one conflict was `migrations.targetVersion`, resolved as
59, one past this branch's migration number.

### The procedure, written down

`docs/deprecating-applications.md` is new, and is the point of doing
this one carefully. It covers how to choose between migrating instances
onto a successor and freezing them, the chain that makes a bare chart
deletion strand every surviving instance, how the freeze reads its
digest off the cluster's own source, what the migration owes a cluster
on each branch, and where the tenant RBAC grant goes. `AGENTS.md` points
at it so the next retirement starts there rather than rediscovering it.

### If you run HTTPCache

Nothing changes on upgrade. Your instance keeps running and stays
manageable through the Cozystack API and the dashboard, and `kubectl get
httpcaches` keeps working.

What changes is that it is frozen. The chart now comes from the packages
artifact your cluster was already running, pinned by digest, and gets no
further updates from us, including security updates to the nginx build.
There is no replacement in the catalog. Delete the HTTPCache object when
you no longer need it, the way you always did; the migration keeps the
tenant permission that lets you.

### Verification

`helm template` on `packages/core/platform` before and after, for
`isp-full`, `isp-full-generic` and `isp-hosted`. In all three the
differences are the `http-cache` Package and PackageSource going away
and `migrations.targetVersion` moving from 57 to 58, which on a cluster
below 58 also renders the migration hook Job and its RBAC.

The migration suite `hack/migration-58-http-cache-freeze.bats` covers
both branches, the order of every step and each fail-closed path.
Freezing on the tarball checksum instead of the manifest digest,
dropping the `no-delete` label, dropping the `managed-by` null, dropping
the tenant grant, returning from the Package delete without waiting,
freezing unconditionally and granting on a cluster with no instance each
turn the matching test red.
`packages/system/cozystack-basics/tests/clusterroles-tenant-admin-aggregation_test.yaml`
pins the aggregation label the migration's ClusterRole reaches tenant
admins through, so renaming it fails in the chart rather than silently
on the clusters that need the grant.

Green: `helm unittest` for `packages/core/platform` and
`packages/system/cozystack-basics`, `make migrations-target-check`,
`hack/cozystack-version-stamp.bats`, `go test
./internal/controller/cacert/...`, `go build ./...` in the
`api/apps/v1alpha1` module, the console `ServicesTab` and
`sidebar-icons` suites with `tsc --noEmit`, `make rd-presets-check`, and
the bats suites that enumerate packages and image refs.

### Screenshots

The catalog entry and its nginx sidebar icon are gone. Nothing new to
show.

### Downstream repositories

Two repos are affected. No follow-up is open yet, so no box is ticked.

`cozystack/website` carries `content/en/docs/*/networking/http-cache.md`
in eight doc versions and lists `http-cache` in the hardcoded
`NETWORKING` list in its Makefile. The `next` page and the Makefile
entry are what need to go, released versions stay as history.

`cozystack/terraform-provider-cozystack` has a `cozystack_httpcache`
resource and data source. The resource keeps working after this: it is
hand written against its own terraform-plugin-framework model and
imports nothing from here. What breaks, once that repo bumps its
`api/apps/v1alpha1` pin past this release, is
`internal/provider/httpcache_model_test.go`, the per kind schema guard,
which is the only file over there importing this module. The resource
and the guard should go together.

The other repos have no reference to the package.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(http-cache)!: remove the HTTPCache application from the catalog. New instances can no longer be created from the catalog and the `HTTPCache` values type is gone from the `api/apps/v1alpha1` Go module. A platform migration keeps existing instances working: on a cluster that runs one, the application is frozen on the packages artifact that cluster was already rendering from, pinned by digest, and stays manageable through the Cozystack API and the dashboard, but receives no further updates, including security updates to the nginx build. On a cluster with no instance the application is removed outright. There is no replacement in the catalog; delete the HTTPCache object when you no longer need it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Removed**
* Removed the HTTP Cache application and its deployment, configuration,
dashboard metadata, and packaging from the platform.
  * The HTTP Cache application is no longer included in the NaaS bundle.
* HTTP Cache resources and related API definitions are no longer
available.

* **Migration**
* Existing HTTP Cache installations are frozen during migration;
environments without instances are cleaned up.
  * Platform migration target advanced to version 58.

* **Documentation**
* Updated storage immutability documentation to reflect Valkey instead
of HTTP Cache.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…4453)

## What this PR does

This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves
the platform to declarative volume hotplug. It changes how every volume
hotplug on the platform works, so the title carries `!`. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the
kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is
the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at
v1.9.0, and Beta gates are on by default. virt-controller skips
`handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so
dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate
off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so
the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments
next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451
does not know the flag and exits on it at startup. The image is stamped
at release preparation and ships in the same release as this change, so
only a `make apply` from main before then would crash-loop the CSI
controller.

What changes at runtime on an upgrade:

- The VM-level `addvolume` and `removevolume` calls now edit `VM.spec`
directly. virt-controller then makes the VMI hotplug volumes match it,
instead of going through `VM.status.volumeRequests`.
- Volumes hot-plugged through the VM subresource are in `VM.spec`, so
they stay attached.
- During a tenant teardown, when the VM goes before its VMI, the driver
no longer calls the VMI subresource. It waits for the VMI status to drop
the volume, and the garbage collector deletes the VMI with its VM.
Upstream waits up to two minutes. If the VMI is still there after that,
the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its
first pass virt-controller removes every hotpluggable VMI volume that is
missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at
the VMI level (the Role then granted
`virtualmachineinstances/addvolume`), and live migration carries such a
VMI across KubeVirt updates. Without help, a tenant node not recreated
since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the
VM template before anything else changes. Adding a hotplug volume to the
VM spec is a valid live update in KubeVirt, so no VM needs a restart. It
runs as the platform chart's pre-upgrade hook, before the upgrade
re-points `cozystack-packages`. The KubeVirt CR and the tenant
kubernetes charts both render from that source, so the gate and the flag
cannot change before the migration has finished. It only touches VMs
that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user
added at the VMI level, for example with `virtctl addvolume` without
`--persist` from an older virtctl, has the same shape as kubevirt-csi's
old attachments, because virt-api builds both the same way. There is
nothing reliable to tell them apart by, so both are copied. Leaving the
user's volume out would not keep it ephemeral: the first reconcile would
unplug it from the running VM. After the upgrade it is part of `VM.spec`
and attaches again on a restart, and `virtctl removevolume` drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api
refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns
(`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and
removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI,
and a stopped VM builds its next VMI from `VM.spec`, so copying from a
stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted
variants. KubeVirt comes from the platform only on isp-full and
isp-full-generic, so those are covered. On the default variant, where
packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy.
It uses the same rule. On the default variant, run it before upgrading
and recreate each tenant node it lists:

```bash
kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'
```

Empty output means nothing is affected. I checked the query against the
migration's test fixtures only. Neither the query nor the migration has
run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to
60: the file, the version it stamps, `migrations.targetVersion`, the
bats file (its `run_migration` argument and expected `STAMP`) and
`hack/testdata/migration-59-vmi-hotplug`.

A VM with a pending hotplug request at upgrade time may also get stuck.
virt-controller still processes `VM.status.volumeRequests` before the
declarative handler, and it applies them through the VMI subresource,
which virt-api rejects once the old gate is off. The sync then fails
before the VM spec is updated, so nothing clears the request, and every
later sync of that VM fails the same way. A retry through the VM
subresource edits the spec and clears it, which covers kubevirt-csi's
own requests. A one-shot request stays stuck until the VM restarts. I
read this from the v1.9.0 code and haven't reproduced it. Waiting until
this lists nothing before upgrading narrows the window, but a request
made during the rollout can still hit it:

```bash
kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'
```

The two halves live in different HelmReleases, so there is a short
window during the rollout where the driver and the gate disagree. In
that window an unpublish fails and is retried until both are updated.

A tenant cluster whose `kubernetes` HelmRelease is suspended or failing
keeps its old driver for longer. Drivers from before July 2025 attach
through the VMI subresource, so on such a tenant every attach is refused
until its release reconciles. Such a driver can also attach a volume
after the migration has run and before the gate flips. The migration
doesn't see that volume, and the first declarative reconcile unplugs it
from the running VM.

The migration lists the VMs once and patches each one with a
`resourceVersion` test, so a VM that changes in between fails the hook
Job. The Job has `backoffLimit: 3`, and each retry lists again and skips
the VMs already patched. Re-reading only the conflicting VM would avoid
the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With
the old gate gone it is the only hotplug path, and the chart refuses a
render that disables it. I found no other user of `HotplugVolumes` in
the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has
no rule for the VMI-level `removevolume` (#4452), so that call likely
got 403 with or without the gate. I haven't checked that on a cluster.

`hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against
a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left
alone (a VMI that is not Running among them), a second run, a cluster
without KubeVirt and the fail-closed paths. The helm-unittests for the
KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in
`disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The
CSI test pins the flag at false, and the `make update` idempotency test
still passes. None of that exercises hotplug itself. The real test is
e2e with volume attach and detach on a tenant cluster, including a
teardown. No CI lane upgrades from the previous release yet, so an
upgrade with a live VMI-only volume is untested; #4463 tracks that test.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Declarative hotplug volumes are enabled by default for KubeVirt.
* Running virtual machines keep eligible hotplugged volumes and disks
when their configuration is migrated.
* **Bug Fixes**
* The migration avoids adding duplicate volumes or disks and does not
mark the migration complete if required operations fail.
  * The Kubernetes CSI driver no longer uses the VMI hotplug fallback.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->