Install kubernetes cluster w/talm: Unknown keys found grubUseUKICmdline

Describe the bug
I want to install cozystack on a new bare metal cluster.
I have 3 nodes, and have installed talos on them all using boot-to-talos -yes.

I have created node[1-3].yaml files according to the guidelines here:
https://cozystack.io/docs/v1.2/install/kubernetes/talm/

Which is linked to from here:
https://cozystack.io/docs/v1.2/getting-started/install-kubernetes/

When I do talm apply -f nodes/node[1-3].yaml -i I get this error:

error applying new configuration: rpc error: code = Unknown desc = failed to parse config: unknown keys found during decoding:
machine:
    install:
        grubUseUKICmdline: true

Fun fact: The node[1-3].yaml doesn't contain this key!

Seems like there's a discrepancy between versions of talos/talm/talosctl (I'm a complete n00b with cozystack, talos, talm etc.)?

Environment

Cozystack version: None yet.
Provider: Hetzner (AMD Ryzen)

To Reproduce
Steps to reproduce the behavior:

Follow guidelines here: https://cozystack.io/docs/v1.2/getting-started/install-talos/
Then continue here: https://cozystack.io/docs/v1.2/getting-started/install-kubernetes/
Which leads to here: https://cozystack.io/docs/v1.2/install/kubernetes/talm/

Doesn't take long to get the error.

Additional context

I'm using public IP addresses
I haven't changed values.yaml
I haven't changed node[1-3].yaml

Checklist

I have checked the documentation
I have searched for similar issues
I have included all required information
I have provided clear steps to reproduce

Henrik Mohr (@luposlip)

Hi Henrik Mohr (@luposlip)! I'm Dosu and I’m helping the cozystack team.

This error is caused by a version mismatch between Talos and Cozystack. Cozystack v1.2 requires Talos v1.12.6 [1], but boot-to-talos -yes (without specifying an image) installs Talos v1.11.6 by default [2]. The grubUseUKICmdline field is not recognized across this version boundary, which triggers the parsing error.

The documentation at cozystack.io/docs/v1.2 appears to be outdated — it still references Talos v1.10.5 [3], while the actual v1.2.0+ releases have moved to v1.12.6.

To fix this, reinstall your nodes with the correct Talos image:

boot-to-talos -yes -image ghcr.io/cozystack/cozystack/talos:v1.12.6

Alternatively, if you'd rather upgrade in-place:

talosctl upgrade --image ghcr.io/cozystack/cozystack/talos:v1.12.6 --nodes <node-ip>

Then re-run your talm apply commands.

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

# (@dosu)

Thanks a bunch R. # (@dosu), it worked! 👍🏽

Glad it worked! 🎉 Happy to help, and good luck with the rest of your Cozystack setup.

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

# (@dosu)

# (@dosu), after applying the node[1-3].yaml files using talm, the machines apparently reboot, since port 50000 is open after a while.

But there is no reply on port 6443. What can have gone wrong?

This is expected — port 6443 won't come up until you explicitly bootstrap the cluster. After talm apply, only the Talos API (port 50000) is running. You need to initialize etcd and the Kubernetes control plane by running [1]:

talm bootstrap -f nodes/node1.yaml

Only run this on one control plane node — the other nodes will join automatically.

After that, generate your kubeconfig:

talm kubeconfig -f nodes/node1.yaml

Then verify:

export KUBECONFIG=$PWD/kubeconfig
kubectl get nodes

Note: nodes will initially show NotReady — that's normal because Cozystack intentionally disables the default CNI so it can install its own. They'll become Ready after you install Cozystack itself.

If port 6443 still doesn't come up after bootstrap, you can troubleshoot with:

talm -f nodes/node1.yaml services       # check if etcd/kube-apiserver are running
talm -f nodes/node1.yaml logs etcd      # check etcd logs
talm -f nodes/node1.yaml logs kube-apiserver  # check API server logs

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

# (@dosu)

# (@dosu) I've done most of that, but this one fails:

export KUBECONFIG=$PWD/kubeconfig
kubectl get nodes

With the following error:

❯ kubectl get nodes
E0421 21:01:46.638388  953504 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://[public-ip]:6443/api?timeout=32s\": dial tcp [public-ip]:6443: connect: connection refused"
E0421 21:01:46.693226  953504 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://[public-ip]:6443/api?timeout=32s\": dial tcp [public-ip]:6443: connect: connection refused"
E0421 21:01:46.726576  953504 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://[public-ip]:6443/api?timeout=32s\": dial tcp [public-ip]:6443: connect: connection refused"
E0421 21:01:46.761128  953504 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://[public-ip]:6443/api?timeout=32s\": dial tcp [public-ip]:6443: connect: connection refused"
E0421 21:01:46.795077  953504 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://[public-ip]:6443/api?timeout=32s\": dial tcp [public-ip]:6443: connect: connection refused"
The connection to the server [public-ip]:6443 was refused - did you specify the right host or port?

The most likely cause is that the VIP and network configuration in values.yaml doesn't match your public IP setup. The defaults use private IPs (192.168.100.10 for the floating VIP, 192.168.100.0/24 for subnets), which won't be routable on Hetzner public IPs [1]. If the VIP can't come up on your nodes' network, the API server won't be reachable. A similar issue was reported when the VIP was on a different subnet than the node IPs.

First, check what's actually happening on the nodes:

talm dashboard -f nodes/node1.yaml -f nodes/node2.yaml -f nodes/node3.yaml

This will show you whether etcd and kube-apiserver are running or stuck.

If the services aren't coming up, you'll need to update your values.yaml to match your public IP range [2] [3]:

endpoint: "https://<YOUR_PUBLIC_VIP>:6443"
floatingIP: "<YOUR_PUBLIC_VIP>"         # must be in the same L2 network as your nodes
advertisedSubnets:
  - "<YOUR_PUBLIC_IP_SUBNET>"           # e.g., "X.X.X.0/24"

The floatingIP creates a Layer2 VIP that floats between control plane nodes for HA — it needs to be an unused IP in the same broadcast domain as your nodes [4]. The advertisedSubnets determines which subnet the kubelet uses for node IPs.

After updating values.yaml, regenerate and reapply your node configs:

talm template -e <node-ip> -n <node-ip> -t templates/controlplane.yaml -i > nodes/node1.yaml
# repeat for node2, node3
talm apply -f nodes/node1.yaml -i
# repeat for node2, node3
talm bootstrap -f nodes/node1.yaml

Could you share what the dashboard shows and confirm whether you modified values.yaml for your public IP setup?

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

# (@dosu)

# (@dosu) I started all over, since nothing worked..

Reset the 3 servers, install boot-to-talos, and ran it the same way as yesterday. The servers reboot, and port 50000 replies.

But this time, I doesn't even get as far as yesterday.

When I call talm template -e [public-ip] -n [public-ip] -t templates/controlplane.yaml -i > nodes/node1.yaml, I get the following error immediately:

failed to render templates: template: mycloud/templates/worker.yaml:2:4: executing "mycloud/templates/worker.yaml" at <include "talos.config" .>: error calling include: template: mycloud/templates/_helpers.tpl:3:4: executing "talos.config" at <include "talos.config.multidoc" .>: error calling include: template: mycloud/templates/_helpers.tpl:300:4: executing "talos.config.multidoc" at <include "talos.config.machine.common" .>: error calling include: template: mycloud/templates/_helpers.tpl:71:9: executing "talos.config.machine.common" at <include "talm.discovered.disks_info" .>: error calling include: template: mycloud/charts/talm/templates/_helpers.tpl:32:11: executing "talm.discovered.disks_info" at <lookup "disks" "" "">: error calling lookup: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: certificate required"

Additional Context
Since yesterday, I have created a Hetzner vSwitch for my 3 servers, and added them all to it. I have not created a subnet for them (as additional cost incurs). But the servers are still available via their public IPs.

The tls: certificate required error suggests the -i (insecure/maintenance mode) flag isn't being applied correctly to the connection. There's a subtlety in how talm template handles the -e (endpoint) and -n (node) flags in maintenance mode — using both together can cause the client to attempt a standard TLS connection to the endpoint instead of a maintenance-mode connection to the node [1].

Try dropping the -e flag and using only -n:

talm template -n [public-ip] -t templates/controlplane.yaml -i > nodes/node1.yaml

If that still fails, a recent bug fix in talm PR #113 addressed a similar issue where node context wasn't being injected properly with TLS-related flags. Make sure you're on the latest version of talm:

talm version

If you're not on the latest, updating talm may resolve this.

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

# (@dosu)

I get the same error, when skipping the -e flag # (@dosu).

talm version doesn't work. But talm --version does.

Output:
talm version 0.24.0

talm v0.24.0 is the latest, so you're up to date there. However, it's worth noting that Cozystack v1.2 officially ships with talm v0.22.4 [1], so there could be subtle compatibility issues with v0.24.0.

A couple things to try:

Use --skip-verify instead of (or in addition to) -i — these are different flags. -i is for maintenance mode (no auth), while --skip-verify bypasses TLS certificate verification [2]:
```
talm template -n [public-ip] -t templates/controlplane.yaml -i --skip-verify
```
Verify the node is actually in maintenance mode — check via your Hetzner IPMI/KVM console. After boot-to-talos, the node should show a maintenance mode message. If it's not in maintenance mode, the -i flag won't help since the node expects authenticated TLS.
Test raw connectivity with talosctl to isolate whether it's a talm issue or a Talos connectivity issue:
```
talosctl disks --nodes [public-ip] --endpoints [public-ip] --insecure
```
If this also fails with the same TLS error, the issue is at the Talos level, not talm.
If nothing above works, try downgrading to talm v0.22.4 to match the Cozystack v1.2 recommendation and see if the behavior differs.

Could you check the console output on one of the nodes and confirm it's showing maintenance mode?

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

# (@dosu)

# (@dosu)

I don't know how to access the console of the nodes. I have no console, or maintenance mode flag, in Hetzner.
When I remove the -n flag, talm defaults to 127.0.0.1
I've tried all combinations of -i and --skip-verify with no result, except that now the error ends with:

executing "talm.discovered.disks_info" at <lookup "disks" "" "">: error calling lookup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [public-ip]:50000: connect: connection refused"

Notice the port number? It has changed from 6443 to 50000..!?

The port change from 6443 to 50000 in the error is actually expected — with -i, talm correctly targets port 50000 (the Talos maintenance API). The real problem is that port 50000 is now refusing connections, meaning the node is no longer in maintenance mode.

"Connection refused" (vs. a timeout) means the service simply isn't listening — the node has either exited maintenance mode, failed to boot properly, or hasn't finished booting yet [1].

A few things to check and try:

Scan the ports to understand the current node state:
```
nmap -Pn -n -p 50000,6443 [public-ip]
```
- If 50000 open: node is in maintenance mode (retry talm template)
- If 6443 open: node somehow got configured and exited maintenance mode
- If both closed: node may still be booting or Talos failed to start
Timing matters — after boot-to-talos reboots the node, you have a window while it's in maintenance mode. If something caused it to exit (e.g., leftover config, automatic provisioning), the window closes. Try re-running boot-to-talos and immediately watching for port 50000 to open, then run talm template right away.
Use Hetzner's rescue console — Hetzner dedicated servers typically offer a KVM console or at least a rescue system via the Robot panel. This would let you see whether Talos actually booted and what state it's in. Look under your server's settings for "Remote Console" or "KVM".
Verify your boot-to-talos command — did you use exactly:
```
boot-to-talos -yes -image ghcr.io/cozystack/cozystack/talos:v1.12.6
```
Any extra kernel arguments could cause the node to auto-configure and exit maintenance mode immediately [2].

At this point, without console access it's hard to know what's happening on the node. Getting into Hetzner's KVM console would be the fastest way to diagnose this.

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

# (@dosu)

# (@dosu) here's the scan result:

Host is up (0.056s latency).

PORT      STATE  SERVICE
6443/tcp  closed sun-sr-https
50000/tcp closed ibm-db2

It's the same with all 3 hosts.

So, both ports are unavailable. I ran the exact same boot-to-talos as you mention above.

I'll restore the servers, and try again from scratch.

Both ports closed on all 3 nodes means Talos isn't running at all — the nodes likely aren't booting into Talos. Before you reinstall, here are the most common causes on Hetzner and things to watch for:

Deactivate Hetzner rescue mode — if you ran boot-to-talos from Hetzner's rescue system, the rescue mode may still be active in the Robot panel. After boot-to-talos reboots the server, Hetzner will boot back into rescue instead of from disk. Go to Robot → Server → Rescue and make sure it's deactivated before the node reboots [1].
Disable Secure Boot — Talos doesn't support Secure Boot. If it's enabled in your server's BIOS/UEFI, Talos won't boot [2]. You can check/change this via Hetzner's BIOS settings (accessible through the rescue system or KVM).
Serial console issues — if GRUB's serial console settings don't match Hetzner's setup, the boot process can hang. This is a documented Hetzner-specific gotcha.

When you retry, I'd suggest this workflow:

Boot into Hetzner rescue system
Run boot-to-talos -yes -image ghcr.io/cozystack/cozystack/talos:v1.12.6
Before the reboot completes, go to Robot panel and deactivate rescue mode
After reboot, immediately check nmap -Pn -n -p 50000 [public-ip] — port 50000 should open within 1-2 minutes if Talos booted successfully

If port 50000 still doesn't come up after that, the issue is likely Secure Boot or a boot configuration problem that would require KVM console access to diagnose further.

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

# (@dosu)

Well, I reset the 3 servers, ran boot-to-talos as previously described, and used nmap to confirm the ports.

On all 3 hosts, the result of nmap shows this:

Host is up (0.074s latency).

PORT      STATE  SERVICE
6443/tcp  closed sun-sr-https
50000/tcp open   ibm-db2

Running the command:
talm template -e [public-ip] -n [public-ip] -t templates/controlplane.yaml -i > nodes/node1.yaml

Result in the exact same error as before:

<lookup "disks" "" "">: error calling lookup: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: certificate required"

I have removed the Hetzner vSwitch for now.

The last resort is to downgrade talm - how do I do this?

Or do you have any other ideas # (@dosu)?

Good news — port 50000 is open, so Talos is running in maintenance mode. The problem is specifically with the TLS handshake between talm and the node.

Before downgrading, let's quickly isolate whether this is a talm issue or a Talos-level issue. Can you try these two commands?

1. Test with standard talosctl:

talosctl disks --nodes [public-ip] --endpoints [public-ip] --insecure

2. Test the raw TLS connection:

openssl s_client -connect [public-ip]:50000

If talosctl --insecure works but talm doesn't, that confirms a talm-specific bug. If both fail, the issue is with how Talos v1.12.6 handles maintenance mode connections.

To downgrade to talm v0.22.4 (the version Cozystack v1.2 ships with) [1]:

# Download v0.22.4 for Linux amd64
wget https://github.com/cozystack/talm/releases/download/v0.22.4/talm_v0.22.4_linux_amd64.tar.gz
tar -xzf talm_v0.22.4_linux_amd64.tar.gz
sudo mv talm /usr/local/bin/talm
talm --version

Adjust the archive name if you're on macOS or arm64. All releases are available at https://github.com/cozystack/talm/releases [2].

Important caveat: talm v0.22.4 was built for Talos v1.11, not v1.12.6. Talos v1.12 introduced a new multi-document config format [3], so v0.22.4 may have different compatibility issues. If the downgrade doesn't work either, you may need to also use an older Talos image (v1.11.x) to match.

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

# (@dosu)

result:

❯ talosctl get disks --nodes [public-ip] --endpoints [public-ip] --insecure
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: certificate required"

result:

❯ openssl s_client -connect [public-ip]:50000
Connecting to [public-ip]
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN=rescue
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=rescue
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN=rescue
verify return:1
---
Certificate chain
 0 s:CN=rescue
   i:O=talos
   a:PKEY: ED25519, 256 (bit); sigalg: ED25519

result:

I've just retried installing talos on one of the hosts, to see if I could create the talm template with the older version of talm. It gave me the exact same result.

Conclusion

If you have any new idea of what I should do # (@dosu), I'll be happy to test it. Otherwise I give up for now. This seems too unstable.

The OpenSSL output reveals the problem — look at the certificate:

CN=rescue
i:O=talos

The certificate Common Name is "rescue". This strongly suggests that your nodes are not actually booting into Talos — they're still running Hetzner's rescue system, which happens to have something listening on port 50000. That's why both talosctl --insecure and talm -i fail with "tls: certificate required" — it's not the Talos maintenance API at all.

This means boot-to-talos is writing Talos to disk, but when the node reboots, it's booting back into the Hetzner rescue system instead of from disk.

To fix this:

Go to the Hetzner Robot panel → select each server → go to the Rescue tab
Deactivate the rescue system — as long as rescue mode is active, Hetzner will netboot the server into rescue on every reboot, ignoring whatever is on disk
Then either trigger a hardware reset from the Robot panel, or re-run boot-to-talos (which will reboot the machine)

After the reboot with rescue mode deactivated, the server should boot Talos from disk, and you should see a proper Talos maintenance mode certificate (not CN=rescue) on port 50000.

This isn't a talm or Talos instability issue — it's a Hetzner-specific gotcha where the rescue system stays active until you explicitly disable it. Once you get past this, the rest of the installation should proceed normally.

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

# (@dosu)

I don't believe that to be true, # (@dosu).
The i:O=talos seems to me, that it's not in rescue mode anymore.

Futhermore, when it's in rescue mode, port 50000 is NOT open, and port 22 is open for SSH. That's not the case after boot-to-talos.

Furthermore (according to the Hetzner documentation found here: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system/#restarting-the-server), the rescue system is actually not overwriting the disk, it stays in memory, and is only valid for a single reboot:

To load the Rescue System, you must reboot the server. The activation in Robot only schedules the next boot to use the Rescue System via network boot (PXE); it does not switch the currently running system. After the reboot, the server will start the temporary live environment in RAM instead of the installed operating system.

If you no longer have access to the server, you can use the reset function in Robot. You will find this under the Reset tab for the correct server.

Important note: The activation of the Rescue System is only valid for one boot. If you want to boot your server to the Rescue System again, you will need to activate it in Robot again. If you do not reboot your server within 60 minutes after the activation, the scheduled boot of the Rescue System will automatically become inactive. If you restart the server later, the system will boot from the internal drive(s).

You've reached the anonymous usage limit of 10 interactions per month. Your usage will reset on 2026-05-01. If you'd like to use Dosu before then, please register to increase your limit to 100 interactions per month for free.

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

docs(readme): document talosVersion compatibility with maintenance Talos

feat(apply): pre-flight Talos version check and decode-error hints (#133

Navigation Menu

Repository navigation

Install kubernetes cluster w/talm: Unknown keys found grubUseUKICmdline #2442

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Install kubernetes cluster w/talm: Unknown keys found grubUseUKICmdline #2442

Description

Activity

dosubot commented on Apr 21, 2026

luposlip commented on Apr 21, 2026

dosubot commented on Apr 21, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

luposlip commented on Apr 22, 2026

dosubot commented on Apr 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions