Skip to content

Commit 1436fee

Browse files
kvapskvaps
authored
[hack] Add host runtime preflight check for standalone containerd/docker (#2371)
## What this PR does Adds a host preflight diagnostic script (`hack/check-host-runtime.sh`) that warns operators when a standalone `containerd.service` or `docker.service` is running on the host alongside the embedded k3s runtime used by the cozystack `generic` variant (k3s / kubeadm on Ubuntu). **Why it matters.** K3s ships its own containerd at `/run/k3s/containerd/containerd.sock` and `/var/lib/rancher/k3s/agent/containerd`, while a system-package containerd or docker uses `/run/containerd/containerd.sock` and `/var/lib/containerd`. The two runtimes do not fight over sockets, so both keep running silently. Over time the standalone one accumulates unpruned images and build cache in `/var/lib/containerd` — enough to fill the root disk, trigger `DiskPressure`, and put `cozystack-api` into an eviction loop. This is silent on day zero and surfaces as a mysterious production incident weeks later. The script exists to warn the next operator before the failure mode surfaces. The script is warning-only — it always exits 0 and never blocks the install. It detects: - `containerd.service` or `docker.service` active via `systemctl is-active` - Standalone runtime sockets at well-known paths, with a fallback that works on hosts without systemd - Standalone data directory sizes via `du -sh` When a warning fires, the HINT names only the detected services and instructs the operator to disable them with `sudo systemctl disable --now <service>`. Reclaiming the data directory is called out separately with an explicit note not to delete it blindly — the data may still be in use. **Entry points.** - `make preflight` runs the script directly (for operators preparing a generic-variant host) - `make unit-tests` now runs `bats-unit-tests` alongside `helm-unit-tests`, auto-discovering every `hack/*.bats` file that is not an e2e test **Test coverage.** `hack/check-host-runtime.bats` (11 cases, run via `hack/cozytest.sh`) covers clean hosts with and without systemd, single-service detection, both services simultaneously, socket-only fallback for both runtimes, glob-expansion regression guard, explicit exit-code-0 assertion, `sudo` prefix assertion, `du` failure robustness, and the "service + socket = exactly one warning" de-duplication invariant. Every test is self-contained with `trap 'rm -rf $STUB_DIR' EXIT` for clean recovery on assertion failure, and has no runtime dependencies beyond bash and core utilities — no python3, no real systemd. Irrelevant on Talos where the container runtime lifecycle is fully managed by the distribution. ### Release note ```release-note [hack] Add `check-host-runtime.sh` and `make preflight` target that warn when a standalone containerd or docker runtime is running alongside the embedded k3s runtime on the cozystack generic variant. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a non-blocking preflight check that detects standalone container runtimes, reports disk-usage estimates, and displays an actionable hint to disable detected services. * **Tests** * Added BATS-based unit tests with comprehensive coverage for the preflight validations and various host scenarios. * **Chores** * Build updated to run the BATS unit tests alongside the existing test suite. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents aa7d1a8 + 44dd002 commit 1436fee

3 files changed

Lines changed: 605 additions & 2 deletions

File tree

‎Makefile‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: manifests assets unit-tests helm-unit-tests
1+
.PHONY: manifests assets unit-tests helm-unit-tests bats-unit-tests preflight
22

33
include hack/common-envs.mk
44

@@ -82,11 +82,38 @@ test:
8282
make -C packages/core/testing apply
8383
make -C packages/core/testing test
8484

85-
unit-tests: helm-unit-tests
85+
unit-tests: helm-unit-tests bats-unit-tests
8686

8787
helm-unit-tests:
8888
hack/helm-unit-tests.sh
8989

90+
# Discover every hack/*.bats file that is NOT an e2e test and run it
91+
# through cozytest.sh. Drop a new *.bats file in hack/ and it is picked
92+
# up automatically on the next `make unit-tests` run.
93+
#
94+
# Caveat: $(wildcard ...) returns space-separated names, so a filename
95+
# containing a literal space would split into multiple tokens here. All
96+
# current bats files use hyphen-separated names; if the project ever
97+
# introduces whitespace-bearing filenames this recipe must be rewritten
98+
# (e.g. to use `find ... -print0 | xargs -0`).
99+
BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))
100+
101+
bats-unit-tests:
102+
@if [ -z "$(BATS_UNIT_FILES)" ]; then \
103+
echo "ERROR: no hack/*.bats unit test files found"; \
104+
exit 1; \
105+
fi
106+
@for f in $(BATS_UNIT_FILES); do \
107+
echo "--- running $$f ---"; \
108+
hack/cozytest.sh "$$f" || exit 1; \
109+
done
110+
111+
# Operator-facing host preflight check. Warns about a standalone
112+
# containerd.service or docker.service running alongside the embedded
113+
# k3s runtime. Safe to run at any time; always exits 0.
114+
preflight:
115+
@hack/check-host-runtime.sh
116+
90117
prepare-env:
91118
make -C packages/core/testing apply
92119
make -C packages/core/testing prepare-cluster

0 commit comments

Comments
 (0)