Repository navigation
kubernetes: no way to trust a self-hosted Talos Image Factory served by an internal CA #3796
Description
Problem / motivation
talos.imageFactoryURL is documented for exactly this situation:
Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments.
Following that advice only works if the endpoint carries a publicly-trusted certificate. In a closed environment it usually does not — the factory is served with a certificate from the organisation's own CA. CDI then refuses the import:
tls: failed to verify certificate: x509: certificate signed by unknown authority
The DataVolume never completes, the worker VM never boots, no node joins, and the Kubernetes release sits in install until it times out. The only diagnosis is the importer pod's log inside the tenant namespace, so it reads as a hang rather than a trust failure.
There is no way to configure around it. The DataVolume API has carried source.http.certConfigMap since the beginning, but packages/apps/kubernetes/templates/cluster.yaml renders source.http.url alone, and no chart value reaches that field. A grep for certConfigMap across this repository returns nothing.
So the documented escape hatch for restricted environments is only usable by operators who can also obtain a public certificate for an internal hostname — which is the one thing a closed environment tends to rule out. The practical fallback today is to keep pulling worker images from the public factory over whatever narrow egress exists, which is the dependency imageFactoryURL was added to remove.
Proposed solution
A single optional value on the existing talos group:
talos:
imageFactoryURL: "https://factory.example.com"
imageFactoryCA: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----When set, the chart renders the PEM into a ConfigMap in the release namespace (CDI requires certConfigMap to live alongside the DataVolume) and references it from source.http.certConfigMap. When empty — the default — nothing is rendered and the DataVolume is byte-for-byte what it is today.
Two properties make this safe for existing installs:
- CDI appends the bundle to the system pool rather than replacing it (
x509.SystemCertPool()thenAppendCertsFromPEMover every file in the mounted directory,pkg/importer/http-datasource.go). A release that also talks to a publicly-trusted endpoint keeps verifying. - The field is optional with an empty default, so no rendered object changes for anyone who does not set it.
Taking the PEM as a value rather than the name of a pre-existing ConfigMap keeps the whole configuration inside the application's values, so it works from the dashboard and through GitOps without asking the user to create an object by hand in the tenant namespace.
Scope check
- This affects a single component / package
- I can describe the expected behavior concretely
Alternatives considered
insecureSkipVerify: trueon the http source. CDI supports it, and it is what people reach for today (see KeycloakRealm Resource for cozy Realm Missing After Initial CozyStack Deployment #1215, where the same trust gap is worked around withinsecureSkipVerifyonClusterKeycloak). It drops verification instead of adding one CA, so it trades a broken install for a silent downgrade.- A
certConfigMapname instead of the PEM. Smaller change, and one ConfigMap could serve several clusters, but the user has to create that object themselves — which the dashboard offers no path for. Happy to switch if maintainers prefer it; the plumbing is the same either way. - Plain HTTP from the mirror. What
hack/e2e-talos-image-cache.yamldoes in CI, and it works — but that manifest calls itself "a CI band-aid, not a product artifact" and names a self-hosted factory as the production path. This issue is about making that production path usable. - Serving the internal factory with a publicly-trusted certificate. Requires a public DNS name for an internal service; not available in the environments where
imageFactoryURLmatters most.
Additional context
- Use custom CA for SSL verification in private environments #951 asked for custom CA support in private environments and was closed as stale without being addressed.
- [EPIC] Support air-gapped installation #173 (air-gapped installation epic) was closed the same way, with the note that the work is on hold rather than abandoned.
- KeycloakRealm Resource for cozy Realm Missing After Initial CozyStack Deployment #1215 is the same trust gap in another component, worked around by disabling verification.
The same gap exists in packages/apps/vm-disk/templates/dv.yaml, which also renders source.http.url with no way to supply a CA. Keeping this issue to the kubernetes chart; happy to follow up there separately if this shape is accepted.
I have a working patch for the above (chart value, ConfigMap template, certConfigMap wiring, unit tests, regenerated schema/README/CRD) and will open it as a PR against this issue.
Activity
feat(kubernetes-nodes): let CDI trust a self-hosted Talos image facto…