Skip to content

feat(kubernetes-nodes): let CDI trust a self-hosted Talos image factory - #3797

Merged
Aleksei Sviridkin (lexfrei) merged 1 commit into
cozystack:mainfrom
batonogov:feat/kubernetes-image-factory-ca
Oct 10, 2026
Merged

Aleksei Sviridkin (lexfrei) merged 1 commit into
cozystack:mainfrom
batonogov:feat/kubernetes-image-factory-ca

Conversation

@batonogov

@batonogov Fedor Batonogov (batonogov) commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Adds talos.imageFactoryCA to the kubernetes-nodes chart, so CDI can verify a Talos Image Factory served with a certificate from an internal CA. Fixes #3796.

The gap

talos.imageFactoryURL is documented for self-hosted factories and mirrors in air-gapped, rate-limited or flaky-egress environments. Such an endpoint is normally served by the organisation's own CA, and CDI, which trusts only its system store, refuses the import with x509: certificate signed by unknown authority. The DataVolume never completes and no worker joins.

DataVolume.spec.source.http.certConfigMap exists for exactly this, but the worker DataVolume in templates/nodegroup.yaml renders url alone and no value reaches that field.

The change

One optional value on the existing talos group of a KubernetesNodes pool:

talos:
  imageFactoryURL: "https://factory.example.com"
  imageFactoryCA: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

When set on an HTTP import path (the default and osImage.factory), the chart renders a ConfigMap <release>-talos-factory-ca in the release namespace and points the DataVolume's source.http.certConfigMap at it. A pool on osImage.builtin clones a golden image, builds no HTTP source and gets no ConfigMap. Empty, the default, renders nothing: helm template with the field unset is byte-identical to main, so existing pools keep their KubevirtMachineTemplate name. Setting the CA changes that name, so the pool rolls once.

CDI appends the bundle to the system pool rather than replacing it (x509.SystemCertPool() followed by AppendCertsFromPEM in pkg/importer/http-datasource.go), so a publicly trusted endpoint keeps verifying.

The PEM is taken inline rather than as a ConfigMap name because certConfigMap must live in the DataVolume's namespace, and the dashboard offers no way to create that object by hand.

Fail-closed guards

The value lands verbatim in a ConfigMap that anything able to read the namespace can read, so it must be complete PEM certificate blocks and nothing else, and must never carry private key material. That includes the realistic accident of cat ca.pem key.pem pasted into the field. The rules now live in one shared helper, cozy-lib.tls.validateCACert, which cozy-lib.tls.caCertSecret also calls instead of keeping its own copy. The regexes are unchanged; the failure message names the caller's field.

Testing

  • packages/apps/kubernetes-nodes/tests/talos_image_factory_ca_test.yaml, 13 cases: unset renders nothing, set renders the ConfigMap and a matching certConfigMap, whitespace-only counts as unset, multi-certificate chains, the builtin path, and each guard on the input that trips it.
  • packages/tests/cozy-lib-tests/tests/tls_validatecacert_test.yaml, 6 cases for the shared helper, plus the existing caCertSecret tests updated for the new message.
  • make generate in the package: regenerated values.schema.json, README.md, api/apps/v1alpha1/kubernetesnodes/types.go and the kubernetes-nodes-rd definition are committed.

Not yet exercised end to end against a live internal factory.

Not in this PR

packages/apps/vm-disk/templates/dv.yaml renders source.http.url with the same gap. Left for a follow-up if this shape is accepted.

Screenshots

Not a UI change.

Downstream repositories

Walked the trigger map against the diff:

  • terraform-provider-cozystack: ticked. The diff adds a field to packages/apps/kubernetes-nodes/values.schema.json, which the trigger map names directly. Filed as an issue there.
  • website: not ticked. No package is added, renamed or removed, and the reference page is regenerated from the package README.md, which is regenerated and committed here.
  • Every other repository: no path in the trigger map is touched.

Release note

feat(kubernetes-nodes): add `talos.imageFactoryCA` so CDI can verify a self-hosted Talos Image Factory served with a certificate from an internal CA. Previously the worker image import failed with `x509: certificate signed by unknown authority`. Set the CA in PEM form and the chart wires it into the worker DataVolume; leave it empty and nothing changes.

Summary by CodeRabbit

  • New Features
    • Added an optional setting for providing PEM certificate authorities trusted when importing worker OS images over HTTPS. This supports the default image source and factory images, allowing imports from endpoints that use certificates not already trusted by the system.
    • Certificate authorities are provided as PEM certificate blocks; when configured, they are used for HTTPS image imports. The setting defaults to empty and has no effect on pools using built-in OS images.
  • Bug Fixes
    • Invalid certificate input, including private keys, is rejected.

@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration
  • Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e84208e0-b0cc-459a-b4f1-af7f884f6f20

📥 Commits

Reviewing files that changed from the base of the PR and between 5e88212 and 68cd102.


You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e6a16b29-503e-4058-a738-3b1dc891b0be

📥 Commits

Reviewing files that changed from the base of the PR and between 9bb77e2 and 5e88212.


📒 Files selected for processing (2)
  • packages/apps/kubernetes-nodes/templates/nodegroup.yaml
  • packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.



📝 Walkthrough

Walkthrough

The Kubernetes application adds an optional talos.imageFactoryCA value. Helm validates PEM certificate blocks, renders a ConfigMap for applicable image imports, and references it from the worker DataVolume’s HTTP source. Builtin-image pools do not use this setting.

Changes

Talos Image Factory CA

Layer / File(s) Summary
Shared CA validation
packages/library/cozy-lib/templates/_tls.tpl, packages/tests/cozy-lib-tests/templates/tests/tls-validatecacert.yaml, packages/tests/cozy-lib-tests/tests/tls_validatecacert_test.yaml, packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml
The shared TLS helper validates PEM certificate-block shape and caller-provided field names. The existing CA Secret helper uses the shared validator. Tests cover valid and invalid values, required arguments, and updated error prefixes.
CA value and image-import rendering
api/apps/v1alpha1/kubernetesnodes/types.go, packages/apps/kubernetes-nodes/values.yaml, packages/apps/kubernetes-nodes/values.schema.json, packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml, packages/apps/kubernetes-nodes/README.md, packages/apps/kubernetes-nodes/templates/_helpers.tpl, packages/apps/kubernetes-nodes/templates/image-factory-ca.yaml, packages/apps/kubernetes-nodes/templates/nodegroup.yaml
The API and chart configuration add the optional, empty-default imageFactoryCA value. Helm validates and trims non-empty values, renders a release-namespace ConfigMap for non-clone images, and conditionally sets source.http.certConfigMap on the worker DataVolume.
CA rendering coverage
packages/apps/kubernetes-nodes/tests/talos_image_factory_ca_test.yaml
Tests cover empty, whitespace-only, valid, bundled, and invalid CA values. They check ConfigMap and DataVolume references, builtin-image behavior, and validation errors.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant TalosValues
  participant KubernetesNodesTemplates
  participant ValidateCACert
  participant CAConfigMap
  participant WorkerDataVolume
  TalosValues->>KubernetesNodesTemplates: Provide talos.imageFactoryCA
  KubernetesNodesTemplates->>ValidateCACert: Validate and trim PEM certificate blocks
  ValidateCACert-->>KubernetesNodesTemplates: Return validated CA content
  KubernetesNodesTemplates->>CAConfigMap: Render ca.pem
  KubernetesNodesTemplates->>WorkerDataVolume: Set source.http.certConfigMap
Loading

Merge Risk | ⚪ Minimal · up to 5e882

Merge Risk: ⚪ Minimal · up to 5e882

The optional CA is wired into HTTP image imports without changing the empty-default or builtin-clone paths. The available evidence establishes no material merge-blocking risk.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 9bb77

The additional trust is limited to worker image imports that explicitly configure a CA bundle. Certificate-only checks are preserved, and builtin-image clones do not consume the new setting. No introduced vulnerability was established, but authorization and behavior during CA rotation, removal, or interrupted imports remain unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The direct rendered trust scope is the HTTP-importing worker pool belonging to the release. A CA trusted for its image endpoint can affect the authenticity of worker boot-disk downloads. The existing API already permits image-endpoint selection; no extension of this CA to the control-plane PKI, tenant-wide trust anchor, or builtin-image catalog is demonstrated.

Security Findings and Attack Paths

  • inferred — If an unauthorized identity could change the CA bundle and supply or intercept the image endpoint using a matching certificate, it could influence the imported worker OS. This is a conditional attack path, not a verified finding: effective API and ConfigMap write permissions and runtime TLS handling were not established.

Trust Boundaries and Controls

  • observed — The chart boundary separates certificate-only configuration from persisted trust material and confines the reference to the HTTP source. Namespace placement and release-derived naming establish intended ownership, but do not prove authorization. Supplying arbitrary valid CA certificates remains an explicit trust decision, not something the PEM validator authorizes.

Resilience and Maintainability Implications

  • observed — Stable naming intentionally makes trust rotation an in-place update. The chart does not establish a transaction between trust-object updates and import execution. Missing or deleted ConfigMaps, interruption, retries, concurrent rotation, rollback, and the trust snapshot used by already-running importers remain runtime evidence gaps, rather than demonstrated insecure fallback.

Hardening Proposals

  • proposed — Validate the deployed CDI version's fail-closed behavior for missing and invalid bundles, and exercise rotation and removal during pending and running imports. Pair this with confirmation that CA-object mutation is restricted to identities already authorized to control the pool's image source.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: allowing CDI to trust a self-hosted Talos image factory.
Linked Issues check ✅ Passed Issue #3796 requests optional PEM CA support for worker-image imports from a self-hosted Talos Image Factory. The chart adds talos.imageFactoryCA, validates certificate-only PEM input, renders a rel…
Out of Scope Changes check ✅ Passed The shared cozy-lib.tls.validateCACert helper and caCertSecret delegation support the PEM validation required by #3796. The schema, API type, README, and kubernetes-nodes-rd updates expose and d…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/feature Categorizes issue or PR as related to a new feature labels Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fedor Batonogov (@batonogov) NOT LGTM, but only because main moved under this PR. The need is real: certConfigMap is still not rendered by any app on main, and #3796 is accepted.

The worker DataVolume is no longer in the kubernetes app. On current main packages/apps/kubernetes/templates/cluster.yaml has no source.http at all, and the import now lives in packages/apps/kubernetes-nodes/templates/nodegroup.yaml, which has its own talos.imageFactoryURL. That is also why the branch conflicts in _helpers.tpl, cluster.yaml and the RD. The field, the ConfigMap and the tests need to move to kubernetes-nodes, otherwise the CA would sit in values that no longer reach the import.

While moving it, please don't copy the PEM guards a second time. The two regexes are the same as in cozy-lib.tls.caCertSecret, and a security check kept in two copies will drift. A small validate-only helper in cozy-lib that takes the field name for the error message, used by both, is the shape I'd take. The doc comment above the helper can then lose the part addressed to maintainers.

The commit has Assisted-By: Claude <[email protected]>. The project takes exactly Assisted-by: LLM, no model or vendor name.

CI on this head is from 2026-08-13 and 2026-09-12, before the fork CI lane was fixed on 2026-09-27, so the red there is not caused by the diff. The push after the rebase will get a fresh run.

@batonogov
Fedor Batonogov (batonogov) force-pushed the feat/kubernetes-image-factory-ca branch from 5bb2dd5 to 9bb77e2 Compare October 1, 2026 06:55
@github-actions github-actions Bot added size/XL This PR changes 500-999 lines, ignoring generated files and removed size/L This PR changes 100-499 lines, ignoring generated files labels Oct 1, 2026
@batonogov

Copy link
Copy Markdown
Contributor Author

Reworked on current main per the review: the field, the ConfigMap and the tests moved to kubernetes-nodes (gated to the source.http import path; osImage.builtin clone pools render neither), the PEM guards now live in a single validate-only cozy-lib.tls.validateCACert helper shared with cozy-lib.tls.caCertSecret, and the trailer is Assisted-by: LLM. An empty value renders byte-identical to main (verified by diffing helm template output against upstream/main in clean worktrees).

The five workflow runs on this head sit in action_required pending maintainer approval — could you approve them and take another look?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fedor Batonogov (@batonogov) NOT LGTM, but only because the PR description and title still describe the old version. The code itself is ready.

The three points from my previous review are closed on 9bb77e2. The field and the certConfigMap reference moved to kubernetes-nodes, the PEM checks live once in cozy-lib.tls.validateCACert, and the trailer is Assisted-by: LLM with a matching sign-off.

I checked it on a trial merge with current main. Helm unit tests for kubernetes-nodes and cozy-lib-tests pass, and make generate leaves no diff. With the field unset, helm template output is byte-identical to main, so existing pools keep their KubevirtMachineTemplate name. I also tried three mutations: dropping the certConfigMap line, narrowing the private-key regex to BEGIN PRIVATE KEY, and removing the \A/\z anchors. Each one turns at least one test red. The Kubernetes app no longer renders worker pools, so setting the field on a KubernetesNodes app is the normal path and nothing is missing on the parent side.

The PR body becomes the merge commit message, so it has to match the code that merges. Right now it says:

  • the value goes on the kubernetes chart and templates/cluster.yaml rendered url alone. The change is in kubernetes-nodes, templates/nodegroup.yaml.
  • the guards are kept local and restate caCertSecret, with an offer to extract them. They are extracted now.
  • "11 cases" and "216 passed". The test file has 13 cases now, and the package has 198 tests after the rebase.
  • the regenerated files are api/apps/v1alpha1/kubernetes/types.go and the kubernetes-rd CRD. They are kubernetesnodes/types.go and kubernetes-nodes-rd.
  • the downstream note points at packages/apps/kubernetes/values.schema.json, and the release note starts with feat(kubernetes).

The title has the same problem. The commit says feat(kubernetes-nodes), the title says feat(kubernetes), and the labeler takes the area/* label from the title scope. The terraform issue cozystack/terraform-provider-cozystack#31 also asks for the field on the Kubernetes resource, while the provider already has a kubernetes_nodes model where it belongs.

Not blocking: the field description could say that setting or clearing it rolls the pool once. certConfigMap is part of the DataVolume that is hashed into the KubevirtMachineTemplate name, and in my render the name changed from -c59638 to -59d103 when I set a CA. Rotating the PEM later rolls nothing, because the ConfigMap name stays the same. The neighbouring fields already carry a "Changing it rolls the pool" sentence.

@lexfrei Aleksei Sviridkin (lexfrei) changed the title feat(kubernetes): let CDI trust a self-hosted Talos image factory feat(kubernetes-nodes): let CDI trust a self-hosted Talos image factory Oct 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fedor Batonogov (@batonogov) I updated the title and body to match the rework. The code was ready already, so LGTM.

auto-merge was automatically disabled October 7, 2026 13:49

Head branch was pushed to by a user without write access

@batonogov
Fedor Batonogov (batonogov) force-pushed the feat/kubernetes-image-factory-ca branch from 9bb77e2 to 5e88212 Compare October 7, 2026 13:49

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fedor Batonogov (@batonogov) LGTM again. The last push is only a rebase onto current main. The diff is the same as the one I approved, except the regenerated schema line in kubernetes-nodes-rd. That line now also has what main added to the chart in the meantime, and it still matches values.schema.json. I ran helm unittest on a trial merge with main for kubernetes-nodes, cozy-lib-tests and redis (the other caCertSecret user), all green.

talos.imageFactoryURL is documented for self-hosted factories and mirrors
in air-gapped, rate-limited or flaky-egress environments. Such an endpoint
is normally served with a certificate from an internal CA, and CDI — which
trusts only its system store — refuses the import with "x509: certificate
signed by unknown authority". The DataVolume never completes, no worker
joins, and the release sits in install until it times out.

DataVolume.spec.source.http.certConfigMap exists for exactly this, but the
chart rendered source.http.url alone and no value reached the field.

Add an optional talos.imageFactoryCA taking the CA in PEM form. When set,
the chart renders it into a ConfigMap in the release namespace — CDI
requires certConfigMap to sit alongside the DataVolume — and references it
from the worker DataVolume on the HTTP import path; a pool cloning a
golden image via osImage.builtin builds no source.http and renders
neither. When empty, nothing is rendered and the output is byte-for-byte
what it was.

CDI appends the bundle to the system pool rather than replacing it, so a
release pulling from a publicly-trusted endpoint keeps verifying either
way.

The value is emitted verbatim into a ConfigMap readable by anything that
can read the namespace, so the render fails closed on private key material
and on anything that is not complete PEM certificate blocks. Those guards
live in a new validate-only cozy-lib.tls.validateCACert helper that takes
the caller's field name for its messages, and cozy-lib.tls.caCertSecret is
refactored onto it, so the check exists in exactly one place.

Assisted-by: LLM
Signed-off-by: Fedor Batonogov <[email protected]>
@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/kubernetes-image-factory-ca branch from 5e88212 to 68cd102 Compare October 10, 2026 19:36
@lexfrei
Aleksei Sviridkin (lexfrei) merged commit f0e3e17 into cozystack:main Oct 10, 2026
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/feature Categorizes issue or PR as related to a new feature size/XL This PR changes 500-999 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kubernetes: no way to trust a self-hosted Talos Image Factory served by an internal CA

2 participants