Skip to content

Switch /etc to use confext in mutable mode - #3555

Merged
pothos merged 4 commits into
mainfrom
kai/default-confext
Mar 3, 2026
Merged

Switch /etc to use confext in mutable mode#3555
pothos merged 4 commits into
mainfrom
kai/default-confext

Conversation

@pothos

@pothos pothos commented Dec 10, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

This includes a systemd update to 258.2 to have a smaller diff for the backports.

How to use

Wait for verity cert fix for release builds. Update git refs before merging (drop mantle change here by updating it through the regular ref update PR).

Testing done

With mantle PR used here for GitHub Actions and on Jenkins

  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)

  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc. (FROM_A=bincache FROM_B=bincache FILESONLY=1 CUTKERNEL=1 CUTSIGKEYPATH=1 FILE=… ./package-diff 4547.0.0 9999.0.101+kai-default-confext)

  • CI: http://localhost:8080/job/container/job/packages_all_arches/7135/cldsv/

@pothos
pothos requested a review from a team as a code owner December 10, 2025 17:24
@pothos
pothos force-pushed the kai/default-confext branch from 57b08de to 18bda4e Compare December 12, 2025 06:46
@pothos
pothos force-pushed the kai/default-confext branch 3 times, most recently from dc36a2c to 1fff71f Compare December 15, 2025 02:42
@pothos pothos mentioned this pull request Dec 15, 2025
Closed
@github-actions

github-actions Bot commented Dec 15, 2025 •
edited
Loading

Copy link
Copy Markdown

Test report for 4552.0.0+nightly-20251217-2100 / amd64 arm64

Platforms tested : qemu_uefi-amd64 qemu_update-amd64 qemu_uefi-arm64 qemu_update-arm64

🟢 ok bpf.execsnoop; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok bpf.local-gadget; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.basic; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.cloudinit.basic; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.cloudinit.multipart-mime; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.cloudinit.script; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid0.data; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid0.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid1.data; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid1.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.etcd-member.discovery; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.etcd-member.etcdctlv3; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.etcd-member.v2-backup-restore; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.filesystem; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.flannel.udp; Succeeded: qemu_uefi-amd64 (1)

🟢 ok cl.flannel.vxlan; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.instantiated.enable-unit; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.kargs; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.luks; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.indirect; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.indirect.new; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.regular; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.regular.new; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.reuse; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.wipe; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.partition_on_boot_disk; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.symlink; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.translation; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.btrfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.ext4root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.groups; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.once; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.sethostname; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.users; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.xfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.btrfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.ext4root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.users; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.xfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2_1.ext4checkexisting; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2_1.swap; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2_1.vfat; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.install.cloudinit; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.internet; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.locksmith.cluster; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.initramfs.second-boot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.iptables; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.listeners; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.nftables; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.wireguard; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.omaha.ping; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.osreset.ignition-rerun; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.overlay.cleanup; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.swap_activation; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.sysext.boot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.sysext.fallbackdownload # SKIP; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tang.nonroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tang.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.toolbox.dnf-install; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.eventlog; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.nonroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root-cryptenroll; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root-cryptenroll-pcr-noupdate; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root-cryptenroll-pcr-withupdate; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.update.badverity; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.update.payload; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok cl.update.payload-boot-part-too-small; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok cl.update.reboot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.users.shells; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.verity; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok confext.skiprefresh; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.auth.verify; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.groups; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.once; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.resource.local; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.resource.remote; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.resource.s3.versioned; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.security.tls; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.sethostname; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.systemd.enable-service; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.locksmith.reboot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.locksmith.tls; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.selinux.boolean; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.selinux.enforce; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.tls.fetch-urls; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.update.badusr; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok devcontainer.docker; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok devcontainer.systemd-nspawn; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.btrfs-storage; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.containerd-restart; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.enable-service.sysext; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.lib-coreos-dockerd-compat; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.network-openbsd-nc; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.selinux; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.userns; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok extra-test.[first_dual].cl.update.docker-btrfs-compat; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok extra-test.[first_dual].cl.update.payload; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok extra-test.[first_dual].cl.update.payload-boot-part-too-small; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok kubeadm.v1.32.4.calico.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.32.4.cilium.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.32.4.flannel.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.33.0.calico.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.33.0.cilium.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.33.0.flannel.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.34.1.calico.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.34.1.cilium.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.34.1.flannel.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok linux.nfs.v3; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok linux.nfs.v4; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok linux.ntp; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok misc.fips; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok packages; Succeeded: qemu_uefi-amd64 (4); qemu_uefi-arm64 (1); Failed: qemu_uefi-amd64 (1, 2, 3)

                Diagnostic output for qemu_uefi-amd64, run 3 
    L1: " _packages/sys-block/open-iscsi (33.06s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: a2318d6c47ec: Download complete"
    L7: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L8: "cluster.go:125: 3d3086a1439f: Download complete"
    L9: "cluster.go:125: a2318d6c47ec: Pull complete"
    L10: "cluster.go:125: 3d3086a1439f: Pull complete"
    L11: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L12: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L13: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L14: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L15: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L16: " "
                Diagnostic output for qemu_uefi-amd64, run 2 
    L1: " _packages/sys-block/open-iscsi (22.53s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: a2318d6c47ec: Verifying Checksum"
    L7: "cluster.go:125: a2318d6c47ec: Download complete"
    L8: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L9: "cluster.go:125: 3d3086a1439f: Download complete"
    L10: "cluster.go:125: a2318d6c47ec: Pull complete"
    L11: "cluster.go:125: 3d3086a1439f: Pull complete"
    L12: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L13: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L14: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L15: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L16: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L17: " "
                Diagnostic output for qemu_uefi-amd64, run 1 
    L1: " _packages/sys-block/open-iscsi (32.82s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L7: "cluster.go:125: 3d3086a1439f: Download complete"
    L8: "cluster.go:125: a2318d6c47ec: Verifying Checksum"
    L9: "cluster.go:125: a2318d6c47ec: Download complete"
    L10: "cluster.go:125: a2318d6c47ec: Pull complete"
    L11: "cluster.go:125: 3d3086a1439f: Pull complete"
    L12: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L13: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L14: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L15: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L16: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L17: " "
    L18: "  "

🟢 ok sysext.custom-docker.sysext; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.custom-oem; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.disable-containerd; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.disable-docker; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.simple; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok systemd.journal.remote; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok systemd.journal.user; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok systemd.sysusers.gshadow; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

@pothos
pothos force-pushed the kai/default-confext branch from 1fff71f to 6025111 Compare December 16, 2025 08:58
@pothos

pothos commented Dec 16, 2025

Copy link
Copy Markdown
Member Author

I've added the upstream PR links to the commit message:
aef9fd6

@pothos

pothos commented Dec 16, 2025

Copy link
Copy Markdown
Member Author

For less disruption due to emerge version picking I've deleted the unused systemd ebuilds but can add them back and make sure that the mask setting is enough to use the wanted version.

chewi
chewi approved these changes Dec 16, 2025
View reviewed changes

@chewi chewi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume @danzatt will unrevert his changes along with the necessary fix before you merge this.

I haven't reviewed the upstream systemd changes, as I'll leave the experts to do that.

Comment thread build_library/build_image_util.sh
@pothos
pothos force-pushed the kai/default-confext branch from 6025111 to d446fd8 Compare December 17, 2025 12:49
@pothos
pothos force-pushed the kai/default-confext branch 2 times, most recently from cd18daa to 0ea4f07 Compare December 18, 2025 04:31
@pothos pothos mentioned this pull request Dec 19, 2025
Merged
pothos added a commit to flatcar/init that referenced this pull request Dec 19, 2025
@pothos
systemd: Add noop sysupdate transfer config (#139)
860090d 
When the systemd-sysupdate.service runs it fails if there is no config.
Since we don't use it, it's expected to do nothing but this failure is
making problems when the service is started from the timer unit which is
now enabled by default. This service is also used by the sysext-bakery
update configs for drop-in steps that update sysupdate components. There
we let users add a noop config to prevent this service failure.
Since we run into this in Flatcar now and we anyway had users add this
manually, let's just add it directly to Flatcar.

## How to use

Needs new scripts PR. Should then be backported to Alpha and Beta.

## Testing done

Was tested in flatcar/scripts#3555
@github-actions github-actions Bot mentioned this pull request Dec 22, 2025
Closed
@pothos
pothos force-pushed the kai/default-confext branch from 0ea4f07 to 96b98e4 Compare February 2, 2026 15:21
@pothos

pothos commented Feb 4, 2026

Copy link
Copy Markdown
Member Author

No luck with the runners, can someone start a Jenkins build for me? Thanks!

@dongsupark

dongsupark commented Feb 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

@pothos Jenkins CI is running: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/7367/cldsv/

Yeah, GitHub Actions CI seems broken since weeks.

@tormath1

tormath1 commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

@pothos Jenkins CI is running: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/7367/cldsv/

Yeah, GitHub Actions CI seems broken since weeks.

Yeah runners are currently WIP (#3617) - using this PR one can build and test but not all tests are passing.

@dongsupark

dongsupark commented Feb 5, 2026

Copy link
Copy Markdown
Member

@pothos CI failed:

 !!! Fetched file: bootengine-626047d0af61a9351e91d885b945b0f0f49d0e55.tar.gz VERIFY FAILED!
 !!! Reason: Insufficient data for checksum verification
 !!! Got:      
 !!! Expected: BLAKE2B BLAKE2S MD5 RMD160 SHA1 SHA256 SHA3_256 SHA3_512 SHA512 WHIRLPOOL

@dongsupark

dongsupark commented Feb 5, 2026

Copy link
Copy Markdown
Member

Yes, the manifest mismatch error posted above prevented packages steps from passing. That's why bincache artifacts could not be created.

@pothos
pothos force-pushed the kai/default-confext branch from 96b98e4 to 4731566 Compare February 28, 2026 14:32
@pothos
pothos force-pushed the kai/default-confext branch from 4731566 to 74cfecd Compare February 28, 2026 16:53
@pothos

pothos commented Mar 1, 2026

Copy link
Copy Markdown
Member Author

Ok, it should build now, can I get a new Jenkins run? Thanks!

@pothos pothos mentioned this pull request Mar 2, 2026
Merged
@chewi

chewi commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Okay, I've kicked off Jenkins here. The version string is main-9999.0.0-default-confext.

pothos added 4 commits March 3, 2026 17:57
@pothos
acct-group/clock: New package
c339137 
From 7f0d7d0eb24afe14cd62bfa78cb9e139b9f824d1

Signed-off-by: Kai Lueke <[email protected]>
@pothos
sys-apps/systemd: Update to 258.2
58f7884 
Signed-off-by: Kai Lueke <[email protected]>
@pothos
build_sysext: Set daemon reload metadata field
c24579a 
Most sysexts need a daemon reload to get their services (re)started when
systemd-sysext (re)loads extensions after boot. Up to now this was done
through ensure-sysext.service but only when systemd-sysext.service was
restarted. A manual invocation of systemd-sysext did not get the
daemon reload and, e.g., Docker failed to start.
Since the daemon reload is done when at least one extension needs it
and we can say that this is likely true for most of the extensions we
build, we can unconditionally set it for now and see if we need to
make it conditional per sysext later, e.g., when one could only have
plain CLI extensions and no service extensions such as Docker/containerd
or Podman (and even NVIDIA/ZFS require services after load).

Signed-off-by: Kai Lueke <[email protected]>
@pothos
Use systemd-confext instead of custom /etc overlay mount
01eea0f 
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <[email protected]>
@pothos
pothos force-pushed the kai/default-confext branch from 74cfecd to 01eea0f Compare March 3, 2026 08:57
@pothos
pothos merged commit 7072d08 into main Mar 3, 2026
1 of 5 checks passed
@pothos
pothos deleted the kai/default-confext branch March 3, 2026 08:58
@chewi

chewi commented Mar 3, 2026

Copy link
Copy Markdown
Contributor

You didn't need to see the test result? YOLO. 😁 For the record, it passed aside from a known issue.

@pothos

pothos commented Mar 3, 2026

Copy link
Copy Markdown
Member Author

Since they land on bincache I checked them there instead of Jenkins

t-lo added a commit that referenced this pull request Mar 3, 2026
@t-lo
bootengine, init: fix manifest, update version symlinks
35efeb3 
Commit 01eea0f of PR #3555
introduced  build issue for bootengine and coreos-init; the source
tarball filename reflected in the manifest does not correspond to the
filename used in the ebuild. Hence, the checksum verification fails.

Also, the version symlinks were not updated by the original change.

This change uses the correct filename and checksums and updates the
versions.

Signed-off-by: Thilo Fromm <[email protected]>
@t-lo t-lo mentioned this pull request Mar 3, 2026
Merged
@t-lo

t-lo commented Mar 3, 2026

Copy link
Copy Markdown
Member

This broke a few things, unfortunately. #3762 should fix it.

sayanchowdhury pushed a commit that referenced this pull request Mar 4, 2026
@t-lo @sayanchowdhury
bootengine, init: fix manifest, update version symlinks
03d1843 
Commit 01eea0f of PR #3555
introduced  build issue for bootengine and coreos-init; the source
tarball filename reflected in the manifest does not correspond to the
filename used in the ebuild. Hence, the checksum verification fails.

Also, the version symlinks were not updated by the original change.

This change uses the correct filename and checksums and updates the
versions.

Signed-off-by: Thilo Fromm <[email protected]>
@tormath1 tormath1 mentioned this pull request Mar 16, 2026
Closed
3 tasks
@github-actions github-actions Bot mentioned this pull request Mar 22, 2026
Closed
@defo89 defo89 mentioned this pull request Jun 9, 2026
Closed
@hervenicol hervenicol mentioned this pull request Sep 17, 2026
Merged
@lassehoj lassehoj mentioned this pull request Sep 28, 2026
Open

This branch had an error being deployed

1 failed deployment
development — 01eea0f9 Deployed Mar 3, 2026 by pothos via Check if we need to update the SDK #5039
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chewi chewi chewi approved these changes

Assignees

No one assigned

Labels

main

Projects

Flatcar tactical, release planning, and roadmap
Status: Implemented

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@pothos @dongsupark @tormath1 @chewi @t-lo