feat(site-router): add routed site-to-site IPsec gateway app (Phase 1) - #3426

• Fix all pre-merge checks with AI

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

IvanHunters Both MAJORs were real and are fixed, though the first one's root cause turned out to be deeper than the gating change you suggested. Range f0619b4a3..10cb5f302.

The default load path could not work whether or not the image was published. vm-default-images is registered through cozystack.platform.package.optional.default, so the golden PVC was never guaranteed on any install, and f0619b4a3 had already dropped the dependsOn to escape an install-time deadlock — leaving nothing guaranteeing the clone source. Not registering the app until publication would have hidden that rather than fixed it.

The boot disk now imports the digest-pinned appliance containerDisk straight from the registry, per instance, with the reference living in the app chart it belongs to. Worth being straight about the path there, since the history shows it: I first moved the golden into a dedicated package shipped with the app, and reviewing that found three defects in it. CDI populates a DataVolume only at creation, so a shared golden could never be advanced — a new digest would either fail the upgrade on an immutable field or be silently ignored, leaving later gateways booting the previous appliance while its cloud-init contract had moved on. Its hard dependsOn reintroduced exactly the deadlock class f0619b4a3 escaped, because bundles.disabledPackages is per-name and an operator could strand the app by disabling only the golden. And the namespaced Role it needed in cozy-public had no ordering guarantee, since site-router-controller's dependency closure never reaches cozystack-basics. Importing per instance removes all three: nothing shared to go stale, no package to disable, no cross-chart name contract. The cost is one 12Gi import per gateway instead of a shared clone, which is the right trade for an appliance a tenant runs one or two of.

On the placeholder digest — a real one is a merge prerequisite, and that is now mechanical rather than remembered: hack/image-refs-no-placeholder.bats fails the build on any all-zero digest anywhere in packages/, and is red right now on purpose. Worth recording why the shape is dangerous: it passes required and every schema check, so the failure moves from render time to a stuck import at runtime, and nightly-mirror.sh skips it silently because its ownership filter matches the build-registry host while a not-yet-published ref still carries the public one.

Both MINORs are fixed. The deny-set rejection records a Warning Event and the docstring no longer claims status surfaces it, since it returns a hard error and classify stops the pipeline before updateStatus runs. bgp.enabled without localASN records a Warning too — localASN is optional in the schema, so that document is valid and silently rendered no BGP at all.

Your "legible status on the missing-source path" is addressed, but not the way I first tried. I added a controller-side BootImageUnavailable Event and then found it unreachable in the only scenario it existed for: no image means no gateway pod, so pushVyOSConfig returns GatewayPending and classify requeues before updateStatus — which is what called it. It also never set Ready=False as I had written, since this package makes no condition write at all (D9 keeps status off the app CR). Rather than ship dead code I removed it: with a per-instance import there is no missing-source state, and an unresolvable digest now surfaces as ImportFailed on the tenant's own boot DataVolume, in their namespace, where they can actually see it.

On item 8 — the status column did read deferred-to-empirical and item 9 disclosed the CI exclusion, but the item title said "suite passes", which is what invited the reading. It now says authored-not-executed and states the zero executed drop evidence explicitly.

On the port_security operational risk: that trade-off was raised and accepted in the design proposal, cozystack/community#30. §Design records that kube-ovn v1.15.10 cannot express a CIDR in its allowed-address-pair path, so Phase 1 ships the gateway port fully relaxed with scoped port security as follow-up hardening; §Security states the guest source allow-list is therefore mandatory and platform-owned rather than defence in depth, and that Cilium bounds destinations rather than claimed source identity. Worth re-reading there — if that conclusion should change, the proposal is the place to change it.

Also narrowed the RBAC you judged acceptable, since two verbs were dead: list/watch are gone from Secrets, Services, ConfigMaps and Namespaces.

Your recommended dev-cluster run is still outstanding: build-vyos has not produced a digest yet. Its first run failed on an unrelated stale-branch test failure — this branch is 113 commits behind main, and main's naas.yaml gained a bundles.system.enabled guard that the suite's set: blocks did not satisfy, which is fixed in 10cb5f302.

IvanHunters Addressed the actionable code and documentation gaps in f1ded230b and 869bafbea.

• Node and LoadBalancer protection: admission and reconciliation now share live discovery of Node Internal/External IPs and allocated Service external/LoadBalancer IPs, represented as /32s. A topology change that makes an existing remote CIDR unsafe withdraws its programmed route.
• Tenant isolation: the guest filter now admits the pod CIDR plus same-namespace Service ClusterIPs as /32s, not the entire service CIDR. The documentation explicitly assigns cross-tenant pod isolation to Cilium identity policy; the guest filter blocks world/non-cluster forwarding but does not claim to distinguish tenant pod identities.
• Boot-image trust: the tenant-settable image override is removed. Gateways always use the platform’s digest-pinned appliance because the relaxed OVN port relies on guest-enforced controls.
• Route cleanup: route ownership is persisted on the HelmRelease. Cleanup uses persisted/current gateway IPs and never falls back to destination matching, so it cannot remove a sibling’s same-destination route after the pod disappears.
• Route conflicts: sibling instances declaring the same destination now produce RouteConflict, preserve the first route, and stop before guest configuration instead of flapping the namespace annotation.
• Deny-set failures: a newly unsafe CIDR withdraws existing owned routes before returning the hard error, and mediation failures remove stale tunnel/BGP gauges.
• Observability: the controller still has no durable per-instance condition, and the documentation no longer claims one. Typed pending states now emit Warning Events, while a durable status design remains an explicit follow-up.
• Input validation: peer, static-route, and BGP-neighbor fields now receive admission-time hostname/IP/CIDR validation. The deny-set remains specific to remoteCIDRs, which are the values that program namespace return routes.
• Flag documentation: the incorrect SSH references were removed from --management-cidr help and warnings.

The tenant-admin RBAC omission for siterouters was also corrected, and the controller/API RBAC now covers the live Node and Service reads used by deny-set discovery.

Validation passed with the full Go suite, the standalone application API module, all affected Helm unit suites, package generation, and make manifests. The runtime negative-security e2e and durable controller-owned status remain documented follow-ups rather than being claimed as completed here.

IvanHunters rebased this onto current main (branch was 276 commits behind) and fixed what CI actually fails on now, old logs were useless. Your two blockers are untouched, details at the end.

Result is one squashed commit 8fbe38938 on 09dc7f47f. Only two textual conflicts, both in hack/select-e2e.sh, and resolving them turned up something worse than the deferred-suite gap you flagged.

DEFERRED_SUITES="site-router" stripped the name from selector's final output. Main has since grown per-path coverage escalation (#3330). Together they give empty selection, not reduced one: site-router-application depends on kubevirt-cdi, so reverse walk gave kubevirt-cdi and cozystack-basics exactly one suite, and stripping turned that one into zero. Both e2e lanes read empty as "skip chainsaw" and post E2E Tests green.

packages/system/cozystack-basics        21 suites -> EMPTY
packages/system/kubevirt-cdi            21 suites -> EMPTY
packages/system/site-router-controller  21 suites -> EMPTY

Fixed by parking the suite as chainsaw-test.yaml.disabled, which is main's own convention already (hack/e2e-chainsaw/backup/). That drops it out of selector universe and out of select-install.sh round-trip at once, so hack/select-e2e.sh is byte-identical to main now and un-parking is just renaming the file back. One deliberate behaviour change, a packages/apps/site-router change escalates to full suite instead of selecting nothing, which is what #3330 prescribes for a package with no runnable suite.

Three real unit test failures behind your item 5:

1. cozystack-api rbac_test pinned rule count at 16, branch adds name-scoped cozystack ConfigMap read for deny-set. Bumped to 17 and added coverage pinning resourceNames and verbs, so a widening cannot pass under it.
2. Main now errors when a package declares test: and runs zero suites, and site-router-controller chart shipped none. Added rbac and deployment tests, 18 of them, pinning what the RBAC comments claim: Secret get only, Namespace get and patch only, HelmRelease never update or delete, and fail-closed --allow-open-management.
3. EXIT traps in the 4 new bats files. Per docs/agents/e2e-testing.md that count is a ratchet review keeps from growing, so i removed the traps instead of raising it.

Security surface is byte-identical to pre-rebase, checked file by file: denyset.go, discovery.go, rest_siterouter.go, render.go, cnimediation.go and the whole chart including both Cilium policies. rest.go was auto-merged so i verified admission separately rather than trusting the merge, still invoked fail-closed on create and update.

Both your blockers stay open. Placeholder digest is deliberate, hack/image-refs-no-placeholder.bats is red until the appliance image is published and that is what its commit message says it is for, so make unit-tests cannot go green before that ships. Tunnel-ingress destination breadth needs the scoping call, i did not touch it.

Please take another look at the delta.

Correction to my last review: I over-graded it

My previous review filed 11 MAJOR findings. That grading was wrong, and the effect on this PR was to block it on a list far longer than the evidence supports. Sorry for the churn that caused.

I checked my own calibration across 358 reviews rather than guess. The median review I write has 1 MAJOR; 164 have none, and only 14 of 358 have five or more. The round I posted here had 11, the highest I have ever produced. On this PR the trend is worse than the number: across five rounds my MAJOR count went 3, 0, 3, 5, 11, while you closed 31 of the 41 findings carried forward with zero regressions. The code got better and my verdict got harder. That is my failure, not yours.

What actually blocks, in my judgement, is two things:

• internal/controller/siterouter/reconciler.go:1015 — a failed ConfigMap read re-stamps the management ACL to the hard-coded default and locks the controller out of the gateway until the VM is recreated.
• packages/apps/site-router/templates/service.yaml:20 — no loadBalancerClass immutability guard, where extra/ingress implements one for the same platform value.

Everything else in that review is non-blocking. The rule-count bound is a strong should-fix. The rest (the set -e diagnostics in negative-security, the missing test on the dashboard Secret grant, the 30s convergence window, the config.boot skew I could not verify, the management-firewall scope I said was not exploitable, the _ values) are MINOR or notes. Take or leave them.

I also withdraw three findings outright: the uncached LISTs, the unbounded rule product as a re-file, and the install-wait annotations. You had moved those to docs/followups.md as accepted residuals and I re-filed them as MAJOR anyway. That was me re-opening your decision, which is not review. I will not re-file an accepted residual again.

Separately: E2E (in-tree) is green on dcb65895, and the negative-security gate passed with the allowed flow preserved. That closes the acceptance item I had been carrying as unverifiable.

IvanHunters thanks for recalibrating, both blockers are fixed: ConfigMap read error in e7c1664 and loadBalancerClass guard in 5deb9cf.

Step 12 comment says now it is the 30s poll, you were right there (ae26708).

Main is merged back with #4411 in it, rule-count bound stays in followups.md.

Andrei Kvapil (@kvaps) Timofei Larkin (@lllamnyp) API Review Gate needs one of you.

I'm requesting changes on licensing grounds. The problem is with the appliance image this PR builds and publishes, and it blocks the PR regardless of the technical review.

packages/system/vyos-router-image downloads the official VyOS Stream 2026.03 ISO, converts it with build-vyos-image --reuse-iso, adds an overlay, and publishes the result as a containerDisk. make build pushes it to ghcr.io/cozystack/cozystack/vyos-router-disk, and every installation that enables site-router imports it from there. The ISO carries its own EULA at /usr/share/vyos/EULA, and the build leaves the branding unchanged (/etc/os-release still reads NAME="VyOS"). My reading of that EULA, which is not legal advice:

• §2.2: the licensee "shall not unbundle or repackage the Software for distribution". Converting the ISO to a qcow2 and then to a containerDisk is repackaging it for distribution.
• §3.1(g) prohibits "duplicate the Software or publish the Software for others to copy". The only exception §3.1 allows is "a separate custom agreement". Publishing the image to a public registry is exactly this.
• §4: the VyOS trademarks and logo artwork, boot splash included, are all rights reserved, and the EULA "does not permit you to distribute the Software using VyOS trademarks, regardless of whether the Software has been modified".
• §3.1(a) and (c) prohibit "using the Software on behalf of third parties" and "providing use of the Software in a service bureau arrangement". Running this image as a router for tenants is the app's whole use case. So even if the project published nothing, the app's default path would leave every operator who adopts it in breach of the EULA of the image it boots.

The same EULA says the open-source components remain under their own licences (§2, §12), and §4 allows redistribution once every VyOS mark and logo is removed and replaced. That leaves an open-source project three ways forward:

1. Operator-supplied image. Cozystack ships no VyOS binary. The app takes the appliance image as a platform-level value, each operator obtains VyOS under terms that cover their own use, and the controller and chart remain Apache-2.0 code that manages a VyOS-API-compatible appliance.

2. A debranded build from source.

   • Every VyOS mark and logo is replaced, boot splash included.
   • The EULA file and every reference to it are left out.
   • Sources are published as the GPL requires.

   This is closer to what community#30 planned. It also brings back the dependency on the public rolling repository, which the Makefile records dropping the pinned kernel package on 2026-09-08, so someone has to own that build.

3. A non-VyOS appliance (for example FRR, strongSwan and nftables on a plain distribution). It needs no licence carve-outs, but internal/vyos and internal/vyos/render would have to be rewritten against a different target.

Whichever option is chosen, vyos-router-image needs to come out of make build and out of this PR. The build-vyos job also needs to stop pushing the image, which it currently does to the CI registry on every run.

The image-lifecycle section of community#30 should also be amended to record the choice. Its premise, that the platform builds and publishes the image, doesn't hold for the Stream binary, and it doesn't hold reliably for the public rolling repository either.

Timofei Larkin (@lllamnyp) agree on licensing, going with option 2, debranded build from source.

Operator-supplied image doesn't fit here: gateway VM runs with relaxed port security, so image it boots has to be one the platform controls.

Build moves out of this PR to its own repo, cozystack/barerouter. It builds from vyos-build rolling at pinned commit, replaces name and logo artwork on every user-visible surface and leaves EULA file out, and a check fails the build if name or artwork is still in the finished ISO. Each release publishes ISO, SBOM, list of installed packages and manifest of where source of each package is (Debian packages via snapshot.debian.org, VyOS packages via their GitHub repos and vyos-build package-build scripts).

In this PR vyos-router-image stops building from Stream ISO, it takes barerouter release ISO pinned by URL and SHA256 and only adds appliance layer on top. Nothing Stream-based is pushed anymore. Artifact names drop vyos, and default NTP servers move off time*.vyos.net.

Image lifecycle section of cozystack/community#30 will be amended in separate PR.

myasnikovdaniil the answer rules out option 1 but says nothing about option 3. Why not just use FreeBSD for the gateway?

It looks like it would remove most of what makes VyOS painful here:

• Licensing. FreeBSD is BSD-licensed. No EULA to strip and no marks to replace, so there's nothing to debrand and no need for a barerouter fork to own.
• Build ownership. There's no dependency on vyos-build rolling at a pinned commit, whose package repo already dropped the kernel package this Makefile had pinned (2026-09-08). A FreeBSD release plus pinned pkg versions of strongSwan and FRR is a stable base that someone else maintains, and FreeBSD publishes cloud-init VM images we could layer the appliance on.
• Syntax churn. The firewall is pf, whose syntax is stable across releases. The PR currently has to keep the VyOS 1.5 firewall ipv4 ... family behind single-point helpers in case it changes. MSS clamping is scrub max-mss.
• Footprint. The image should be much smaller than a 12Gi import per gateway.

The cost is the one Timofei Larkin (@lllamnyp) named for option 3: internal/vyos and internal/vyos/render get rewritten against a different target. The controller also loses the VyOS HTTP API and would have to push config some other way, such as a small agent in the guest or rendered files plus a reload. That's real work, but it happens once, while the debranded fork has to be maintained on every VyOS release.

Is there a reason to stay on VyOS that I'm missing, like a feature we can't easily get from strongSwan + FRR + pf? If there isn't, I'd rather weigh that rewrite against owning barerouter before the new repo gets created.

Timofei Larkin (@lllamnyp) small correction to my previous comment: image repo is aenix-io/barerouter, not cozystack/barerouter. v0.1.1 is released and this PR now pins its KubeVirt disk by digest, nothing here builds or republishes image anymore. Also I rewrote history into 5 commits.

Aleksei Sviridkin (@lexfrei)

1. Rebased on main, controller Dockerfile is FROM --platform=$BUILDPLATFORM golang:1.27-alpine with main's digest, same as other Go controllers. Full CI with e2e passed on 0e138c4 (https://github.com/cozystack/cozystack/actions/runs/36970845851), last commit 231fd99 only drops stale docs.
2. History rewritten into 5 commits (deny-set admission, gateway API client and renderer, controller, app, e2e), old commit bodies are gone.
3. DECISIONS.md / PLAN.md references and decision/task ids removed everywhere, where reason mattered it is written inline now.
4. PR-BODY.md deleted. In followups.md both stale items removed, image is published and pinned by digest and negative-security step passes in e2e.

Aleksei Sviridkin (@lexfrei) I took allowlist route, arm64 build of barerouter is not planned now. Disk is never run as container: dv.yaml sets no pullMethod, so CDI importer fetches it as data and single-platform manifest imports on node of any arch, only the VM is bound to amd64. VM now has kubernetes.io/arch: amd64 node selector with helm test, and hack/verify-multiarch.sh passes on the pin with the allowlist entry.

Stale tunnel gauge comments in chainsaw are fixed, polling stays as is. PR body now says what the commits after the first five are.

Thanks for moving the appliance to a debranded image. Pinning a barerouter release by digest, with the SBOM, per-package source manifest and written offer published alongside it, resolves the licensing concern I raised earlier. This is the technical review. I'm still requesting changes, but mostly about where this code lives and what it depends on, not about how it's written. Several premises of community#30 have changed since it was accepted, so I think the proposal needs an amendment before this lands anywhere. Proposed points for that amendment are at the end.

1. This belongs next to barerouter, as an external package

Most of the Go code in this PR drives a single appliance. Excluding tests, generated types and comments, there are about 2,900 lines of code. Of those, internal/vyos (API client and parsers of the router's show output), internal/vyos/render, and the bulk of vyospush.go are a configuration-management client for barerouter. The part specific to Kubernetes and Cozystack (namespace route mediation, deny-set, gateway pod discovery and the port-security check) is roughly 400–500 lines. The code imports nothing from the rest of the repository except its own new packages.

The image and the code that drives it also form one contract, and the PR's own documentation says so:

• docs/image-lifecycle.md requires the image and the configuration contract to "advance atomically".
• Moving to a new barerouter release means re-capturing the config-version footer in _helpers.tpl, in the e2e remote-site fixture, and in a chart unit test.
• The renderer's syntax is "validated live against the shipped image", and the parsers scrape command output whose format a rolling rebuild can change.

Today every barerouter release therefore needs a matching PR in this repository. If the client, renderer and controller lived in the barerouter repository, release N would ship client N, tested against a booted image of N.

I'd suggest moving the controller, the client and renderer, the chart, the ApplicationDefinition, the tenant roles and the e2e suite next to barerouter, and shipping them as an opt-in external package: a PackageSource that operators apply, which serves the charts and images from that repository. Concretely:

• Tenant RBAC can ship as ClusterRoles carrying the rbac.cozystack.io/aggregate-to-tenant* labels, instead of editing cozystack-basics.
• cozy-lib: the chart uses a single helper, which can be vendored.
• The generated api/apps/v1alpha1/siterouter types aren't needed for the app to work.
• What this repository has to provide is generic. Every PackageSource on main currently points at cozystack-packages, so serving a catalog app (an ApplicationDefinition plus a chart with libraries) from an external source needs checking and probably some small platform work. That work is worth doing in its own right, and site-router would be its first user.

This also takes care of naming. Cozystack never talks to VyOS here; it talks to barerouter. Code identifiers referring to the software you interoperate with are ordinary referential use, but in this repository, outside the barerouter project, there's no reason for internal/vyos or VyOSClient to exist at all.

2. Site-router-specific admission in the application API server

This is the one part that can't be moved out, which is a good sign it's the wrong shape. rest_siterouter.go, the SiteRouterKind constant, the calls in rest.go, and the cozystack-api ConfigMap grant add a branch to the aggregated API server that runs only for one application kind. community#30 puts this check in the controller, with the rejection reported as a status condition (Reason: InvalidRemoteCIDR). The aggregated API should stay application-agnostic.

followups.md explains why a synchronous 403 was added: controller failures don't reach the Application's readiness, so the tenant never sees the rejection. The fix for that is the generic path the follow-up describes, from controller-owned conditions into Application status, which every app with a controller would benefit from. Please drop the admission code and its RBAC, and let the controller's condition carry the rejection.

3. The CNI coupling

community#30 accepted a dependency on kube-ovn for routed mode. In practice the PR also depends on Cilium, and both matter now that a networking variant without either is in progress (#3149, the cozyplane eBPF CNI). Under that variant:

• The return path disappears. The ovn.kubernetes.io/routes namespace annotation is only acted on by kube-ovn's webhook. Without it, tenant pods send replies for remote sources out their default route.
• Forwarding breaks. The ovn.kubernetes.io/port_security: "false" pod-template annotation has no effect, and the datapath's anti-spoofing drops every packet the gateway forwards from a remote source.
• The deny-set and the security model assume this datapath. The deny-set reads kube-ovn's join network, and the security analysis relies on Cilium's destination containment.
• The bundle gate only implies kube-ovn. It checks bundles.iaas.enabled, which today implies an isp-full* install and therefore kube-ovn. Nothing ties it to the networking variant.

Instances created on a kube-ovn cluster would stop working when that cluster changes networking variant, and there would be no migration path.

I'd suggest putting the CNI-specific part behind a small platform contract that each networking variant implements. Something like "this workload is an authorized forwarder for CIDRs X in this namespace, and traffic to X is routed to it":

• kube-ovn would implement it with the routes annotation and the port-security relaxation, as this PR does now.
• cozyplane would implement it with a scoped forwarder grant and a VPC route that resolves to the workload's identity, not its IP, which is the design sketched in lllamnyp/cozyplane#6. That design also replaces "port security off" with a narrow grant for the declared CIDRs, which is the hardening followups.md already wants.

The site-router controller would consume the contract and stop knowing which CNI it runs on, and its gateway policies would go through whatever policy layer the variant provides. As an external, opt-in package, Phase 1 can still ship as kube-ovn-only before that contract exists, because only operators who choose it are exposed.

4. Before it ships, wherever it lives

• The multi-arch release gate. images/barerouter-disk.tag matches the */*/images/*.tag paths that hack/lib/image-refs.sh enumerates, and hack/verify-multiarch.sh runs in the release and nightly arm64 workflows. The barerouter containerDisk is a single-arch manifest and isn't in hack/multiarch-allowlist. That allowlist is meant for images that can't exist for arm64, so the right fix is an arm64 barerouter build. In an external package the gate doesn't apply, but the arm64 question remains for arm64 clusters.
• Negative-security e2e. followups.md and security-model.md both say the negative-security step has never been reached in a run. community#30 makes those tests the acceptance bar for Phase 1, so one green run that reaches and passes that step is needed.
• followups.md should become issues. It's a good list, and items such as the tenant super-admin being able to rewrite the gateway VM, and the tenant baseline network policy not excluding the gateway, deserve tracking on their own. A file inside the chart isn't where they'll be found.
• Comment volume. Comments are still roughly 45% of reconciler.go, vyospush.go and render.go, and 158 of the 194 lines in values.yaml. Much of it is history of earlier revisions, which belongs in commit messages or the proposal.

Proposed amendments to community#30

1. Image. Replace "built in the monorepo, next to Talos" with a reference to an external, debranded appliance image pinned by digest, and record why: the EULA on VyOS's published images, and the rolling repository dropping pinned packages.
2. Packaging. Site-router, and later site-gateway, ship as external opt-in packages next to the appliance and its client. The core repository only provides generic extension points: catalog apps from external package sources, and controller conditions in Application status.
3. Validation. Restate that validation stays in the values schema and in the controller, with no per-application logic in the API server.
4. CNI. Replace the kube-ovn dependency with a forwarder or route contract that networking variants implement, and note which variants support routed mode.

Timofei Larkin (@lllamnyp) thanks for the review, I will rebuild this instead of patching the PR. Below is architecture I plan to build, I want to agree on it before writing code.

What is built

Standalone operator with own namespaced SiteRouter CRD. One SiteRouter brings up whole gateway (kubevirt vm with barerouter disk pinned by digest, seed and credentials secrets, LoadBalancer service for the tunnel, optional cilium policies) and does everything controller does today: validation against cluster networks, config push, source filter confirmation, return routes. Nothing in it is tied to cozystack, vyos or barerouter, or to any particular CNI. Cozystack side is app wrapper that writes the CR, it will be separate package in some repo with external apps (which one is not decided yet), tenants never create CR themselves.

apiVersion: siterouter.cozystack.io/v1alpha1
kind: SiteRouter
metadata:
  name: office
  namespace: tenant-acme
spec:
  peer:
    address: 203.0.113.10
    auth:
      secretRef: {name: office-psk}
  remoteCIDRs: [192.0.2.0/24]
  resources: {cpu: 2, memory: 2Gi}
status:
  tunnelAddress: 198.51.100.7
  conditions:
    - {type: Validated, status: "True"}
    - {type: Configured, status: "True"}
    - {type: RoutesProgrammed, status: "True"}
    - {type: TunnelEstablished, status: "False"}

For this CR operator creates vm site-router-office, DataVolume site-router-office-boot, secrets site-router-office-cloud-init and site-router-office-api-key, service site-router-office-tunnel and a Forwarder, all owned by SiteRouter with owner references.

Forwarder is how operator talks to the network, it says "this workload is authorized forwarder for these CIDRs, route them to it":

apiVersion: siterouter.cozystack.io/v1alpha1
kind: Forwarder
metadata:
  name: site-router-office
  namespace: tenant-acme
spec:
  workloadSelector:
    matchLabels: {vm.kubevirt.io/name: site-router-office}
  cidrs: [192.0.2.0/24]
status:
  forwarder:
    addresses: [10.244.1.9]
  pendingWorkloads: []

Operator fills cidrs only after engine confirms source filter. Connector (we write them, one per network, outside operator) watches Forwarder, finds gateway pods by workloadSelector, allows them to forward traffic for these CIDRs and routes the CIDRs to them. Then it writes back to status gateway address in tenant network (next hop for operator) and pendingWorkloads, pods created before the route (some networks apply routes only at pod creation). Operator shows pending workloads in RoutesProgrammed condition and never restarts anything. On delete operator removes Forwarder and connector withdraws routes and grant by finalizer. Operator does not know which network is under it, so another network is just new connector.

Where things live

• operator is own repo, which org is first question below
• appliance engine (client, renderer, seed template, image pin) is go module in barerouter repo, so release N of barerouter ships the image and the code that drives it together
• network connectors live outside operator, kube-ovn first and cozyplane later
• nothing from this PR goes to cozystack/cozystack, admission in aggregated api, rbac changes, bundle gate and PackageSources are dropped

What is better than in this PR

• resources have clear owner. Now chart creates vm, disk, service and secrets and controller finds them by naming and lineage labels, there are no owner references and no status. New operator creates what it manages, owns it with owner references and reports health in status.conditions of the CR. Platform specific settings become typed operator config instead of _ values
• flow is easy to follow: validate, connect, apply, confirm, grant and route, observe. Routes are programmed only after engine confirms source filter, this PR does it in opposite order
• validation does not touch api server, static checks are in schema and CEL, cluster network check is in controller and rejection goes to status condition (Validated=False, InvalidRemoteCIDR), as you suggested
• two seams instead of kube-ovn and vyos wired into reconciler. Network seam is Forwarder above. Engine seam is small go interface (render, push, confirm, observe, bootstrap artifacts), so replacing appliance later doesn't touch reconciler or CRD

I checked the network seam on kube-ovn 1.15.10. Forwarding grant (ovn.kubernetes.io/port_security) is applied to running pod immediately in both directions, docs in this PR say opposite. Routes still apply only at pod creation, so connector reports workloads that predate the route.

Also e2e suite already has negative-security step as acceptance gate and it passed in this run on current head, so I port the suite as is.

I have detailed local design and work plan and can share it, but don't want to paste all of it here. Now I need to agree on direction and implementation, then start building.

Questions

1. Where do you see the operator, cozystack org, aenix-io or somewhere else? I'm not attached to answer, can discuss or argue other view. What do you want to see for it to be in cozystack org?
2. Any objections to this shape (operator, Forwarder with connectors, engine seam) before I start?
3. I think architecture and decisions should go to existing design proposal as amendment (plus decision records when a question is closed), and work plan to an epic with checklist of phases in operator repo. Does it work for you?