Skip to content

Commit 12eec02

Browse files
myasnikovdaniilmyasnikovdaniil
committed
ci(tests): run the container lane beside the QEMU lane on every PR
The container lane landed as an invocable make target that no workflow referenced, so CI kept running QEMU exclusively and the lane had no run history on a real runner. Add `E2E (container lane, informational)`, which installs and runs the same Chainsaw selection against Talos nodes in containers, on a second runner and against the same commit. Informational on purpose: the job is absent from e2e-report's `needs`, so it cannot move the required "E2E Tests" status and a red there does not block a merge. Every measurement behind the lane was taken on hand-built spike stands; none on a CI runner. A separate job rather than a `lane:` matrix leg on `e2e`, because e2e-report reads `needs.e2e.outputs.soft_red`. A matrixed job publishes one output map across all legs, so that value would come from whichever leg finished last and the node-join soft-red decision would turn nondeterministic. Duplicating a few steps is the cheaper mistake. Two gaps had to close for the lane to run unattended. The host script created zpools and loaded modules as the invoking user, which is root on a dev box but an unprivileged user on a runner, so it now escalates per command -- never around docker, or the compose project ends up owned by a different user than teardown runs as. And COZY_APISERVER_ENDPOINT was read inside the sandbox by nothing that set it; docker exec starts a fresh environment, so the Makefile passes it through explicitly. Unset expands to empty and the `:-` default keeps the QEMU lane on .10. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 5846267 commit 12eec02

4 files changed

Lines changed: 228 additions & 7 deletions

File tree

‎.github/workflows/pull-requests.yaml‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1210,6 +1210,203 @@ jobs:
12101210
if: always()
12111211
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
12121212

1213+
# ▸ Container lane — the same suite against Talos nodes running as containers
1214+
# instead of QEMU guests, so tenant workloads sit one nesting level
1215+
# shallower (L2 rather than L3).
1216+
#
1217+
# INFORMATIONAL, deliberately. It is absent from `e2e-report`'s `needs`, so
1218+
# whatever it concludes cannot move the required "E2E Tests" status and a red
1219+
# here never blocks a merge. That is the correct posture until the lane has a
1220+
# run history: every measurement behind it was taken on hand-built spike
1221+
# stands, and none on a CI runner.
1222+
#
1223+
# It runs as its own job rather than a `lane:` matrix leg on `e2e` because
1224+
# `e2e-report` reads `needs.e2e.outputs.soft_red`. A matrixed job publishes
1225+
# one output map for all legs, so the value would come from whichever leg
1226+
# finished last and the node-join soft-red decision would turn
1227+
# nondeterministic. Duplicating a few steps is the cheaper mistake.
1228+
#
1229+
# Why a separate runner and not the same one, sequentially: the two lanes
1230+
# each claim 24 vCPU and 72 GiB for their nodes, so they cannot overlap, and
1231+
# running them back to back doubles a job that already spends ~2h in Chainsaw
1232+
# alone. Runner-to-runner variance is real but it is measurable rather than
1233+
# uncontrolled -- `Prepare environment` prints `svm features:` on both lanes,
1234+
# which is what lets results be compared within the vgif-present and
1235+
# vgif-absent strata separately. That stratification, not host identity, is
1236+
# what #3513 was established on.
1237+
e2e-container:
1238+
name: "E2E (container lane, informational)"
1239+
needs: ["finalize"]
1240+
# Same-repo, non-release PRs only. A fork PR has no pushed images to test
1241+
# (see the `e2e` job), and a release promotion PR deliberately does not
1242+
# repeat the suite.
1243+
if: |
1244+
!cancelled()
1245+
&& !github.event.pull_request.head.repo.fork
1246+
&& needs.finalize.result == 'success'
1247+
&& !contains(github.event.pull_request.labels.*.name, 'release')
1248+
runs-on: oracle-vm-32cpu-128gb-x86-64
1249+
timeout-minutes: 215
1250+
permissions:
1251+
contents: read
1252+
packages: read
1253+
1254+
steps:
1255+
- name: Checkout code
1256+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
1257+
with:
1258+
fetch-depth: 0
1259+
1260+
# No talos-image download: the nodes boot from ghcr.io/siderolabs/talos
1261+
# directly (hack/e2e-compose.yaml), so this lane consumes no nocloud asset.
1262+
- name: Download PR patch
1263+
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
1264+
with:
1265+
name: pr-patch
1266+
path: _out/assets
1267+
1268+
- name: Apply patch
1269+
run: git apply _out/assets/pr.patch
1270+
1271+
- name: Set sandbox ID
1272+
run: echo "SANDBOX_NAME=cozy-e2e-sandbox-c$(echo "${GITHUB_REPOSITORY}:${GITHUB_WORKFLOW}:${GITHUB_REF}" | sha256sum | cut -c1-10)" >> $GITHUB_ENV
1273+
1274+
# The container lane takes its kernel from the host, and Talos cannot load
1275+
# modules in container mode -- machine.kernel.modules is a silent no-op
1276+
# there. hack/e2e-container-up.sh asserts both modules rather than
1277+
# installing anything, because a contributor running the lane locally
1278+
# should be told what is missing, not have their host modified. Supplying
1279+
# them is the caller's job, which on CI is this step.
1280+
- name: Prepare container-lane host
1281+
run: |
1282+
sudo modprobe openvswitch
1283+
if ! grep -q '^zfs ' /proc/modules; then
1284+
sudo apt-get update -qq
1285+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
1286+
sudo modprobe zfs
1287+
fi
1288+
grep -E '^(openvswitch|zfs) ' /proc/modules
1289+
1290+
- name: Prepare environment
1291+
run: |
1292+
attempt=0
1293+
until make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME apply \
1294+
&& make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME prepare-cluster-container; do
1295+
attempt=$((attempt + 1))
1296+
if [ $attempt -ge 3 ]; then
1297+
echo "Attempt $attempt failed, exiting..."
1298+
exit 1
1299+
fi
1300+
echo "Attempt $attempt failed, retrying..."
1301+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true
1302+
done
1303+
echo "Prepare environment completed after $((attempt + 1)) attempts"
1304+
1305+
# This lane has no VIP: .11 is a plain control-plane node standing in for
1306+
# the QEMU lane's 192.168.123.10 (hack/e2e-prepare-cluster-container.bats).
1307+
- name: Install Cozystack into sandbox
1308+
env:
1309+
COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
1310+
run: |
1311+
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
1312+
echo "❌ Install failed (no retry — see diagnostics below)"
1313+
echo "::group::Diagnostics: HelmRelease status"
1314+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -100' || true
1315+
echo "::endgroup::"
1316+
echo "::group::Diagnostics: recent events"
1317+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -50' || true
1318+
echo "::endgroup::"
1319+
exit 1
1320+
fi
1321+
1322+
- name: Select E2E tests
1323+
id: select
1324+
if: ${{ !contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
1325+
env:
1326+
BASE_REF: ${{ github.base_ref }}
1327+
run: |
1328+
git diff --name-only "origin/${BASE_REF}...HEAD" > /tmp/changed.txt
1329+
apps=$(./hack/select-e2e.sh /tmp/changed.txt)
1330+
echo "Selected apps: ${apps:-<none>}"
1331+
echo "apps=${apps}" >> $GITHUB_OUTPUT
1332+
if [ -z "${apps}" ]; then
1333+
echo "skip=true" >> $GITHUB_OUTPUT
1334+
else
1335+
echo "skip=false" >> $GITHUB_OUTPUT
1336+
fi
1337+
1338+
- name: Run OpenAPI tests
1339+
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
1340+
1341+
- name: Run E2E tests
1342+
id: e2e_tests
1343+
if: ${{ steps.select.outputs.skip != 'true' }}
1344+
env:
1345+
SELECTED_APPS: ${{ steps.select.outputs.apps }}
1346+
FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
1347+
run: |
1348+
if [ "$FULL_E2E" = "true" ]; then
1349+
suites=""
1350+
echo "Running full chainsaw suite"
1351+
else
1352+
suites="$SELECTED_APPS"
1353+
echo "Running selected chainsaw suites: $suites"
1354+
fi
1355+
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES="$suites"; then
1356+
echo "❌ Chainsaw E2E failed on the container lane"
1357+
echo "::group::Diagnostics: HelmRelease status"
1358+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
1359+
echo "::endgroup::"
1360+
echo "::group::Diagnostics: recent events"
1361+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
1362+
echo "::endgroup::"
1363+
exit 1
1364+
fi
1365+
echo "✅ All selected E2E tests passed"
1366+
1367+
# Artifact names carry the -container suffix: upload-artifact@v4 rejects a
1368+
# duplicate name within a run, and the `e2e` job claims the unsuffixed ones.
1369+
- name: Collect chainsaw report
1370+
if: always()
1371+
run: |
1372+
mkdir -p _out
1373+
docker cp "$SANDBOX_NAME":/workspace/hack/e2e-chainsaw/chainsaw-report.xml \
1374+
_out/chainsaw-report.xml || true
1375+
1376+
- name: Upload chainsaw report
1377+
if: always()
1378+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
1379+
with:
1380+
name: chainsaw-report-container
1381+
path: _out/chainsaw-report.xml
1382+
if-no-files-found: ignore
1383+
1384+
- name: Collect report
1385+
id: collect_report
1386+
if: always()
1387+
timeout-minutes: 30
1388+
continue-on-error: true
1389+
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME collect-report || true
1390+
1391+
- name: Upload cozyreport.tgz
1392+
if: always()
1393+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
1394+
with:
1395+
name: cozyreport-container
1396+
path: _out/cozyreport.tgz
1397+
if-no-files-found: ignore
1398+
1399+
# Both halves, and in this order: the compose project owns the nodes and
1400+
# the zpools outlive it. A leftover pool makes the next run on a reused
1401+
# host adopt storage that already carries a dead cluster's volumes.
1402+
- name: Tear down sandbox
1403+
if: always()
1404+
run: |
1405+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true
1406+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
1407+
for n in 1 2 3; do sudo zpool destroy "data-srv${n}" 2>/dev/null || true; done
1408+
sudo rm -rf /var/lib/cozy-e2e-zpools
1409+
12131410
# Report the required "E2E Tests" gate as a commit status rather than relying
12141411
# on a job named "E2E Tests". A required *job* that is skipped counts
12151412
# as PASSED by branch protection — desired for docs-only PRs, but the exact

‎docs/agents/e2e-testing.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,8 @@ Today the sandbox boots three QEMU guests and Cozystack runs KubeVirt inside the
200200

201201
Three properties of container mode drive the whole design, and each one fails quietly rather than loudly if ignored. `machine.kernel.modules` is a **silent no-op** — `kernel_module_spec.go` returns early on `ModeContainer` with no error and no event — so the host must load `openvswitch` and `zfs` before the nodes start, and both the host script and the BATS file assert it rather than trusting it. Docker gives a privileged container its **own tmpfs `/dev`, seeded once at creation**, so `/dev/kvm` works but ZFS zvols created later never appear and linstor-csi fails `ControllerPublishVolume`; the compose file binds the host devtmpfs instead, which is also why `talosctl cluster create docker` can never carry LINSTOR. And container mode has **no ARP VIP**, so `192.168.123.11` stands in for the QEMU lane's `192.168.123.10` and `hack/e2e-install-cozystack.bats` takes the endpoint from `COZY_APISERVER_ENDPOINT`, defaulting to the unchanged QEMU value.
202202

203+
In CI the lane runs as its own job, `E2E (container lane, informational)` in `.github/workflows/pull-requests.yaml`, on a second runner beside the QEMU `e2e` job and against the same commit. It is deliberately absent from `e2e-report`'s `needs`, so it cannot move the required `E2E Tests` status and a red there never blocks a merge -- correct until the lane has a run history, since every measurement behind it was taken on hand-built spike stands rather than a CI runner. It is a separate job rather than a `lane:` matrix leg on `e2e` because `e2e-report` reads `needs.e2e.outputs.soft_red`, and a matrixed job publishes one output map across all legs, which would make the node-join soft-red decision depend on whichever leg finished last. The two lanes take different runners, so they are not a controlled A/B on hardware; `make apply` records the host's `svm features:` on both, which is what lets results be compared within the `vgif`-present and `vgif`-absent strata separately, the way cozystack#3513 itself was established.
204+
203205
The nodes run as **siblings on the runner's docker daemon**, not nested inside the sandbox, because those host binds have to resolve against the real host devtmpfs. That is why provisioning is two steps: `hack/e2e-container-up.sh` does the host-side half and needs a docker socket, then the sandbox is attached to the compose network and `hack/e2e-prepare-cluster-container.bats` runs inside it exactly as the QEMU lane's file does.
204206

205207
What the lane gives up is **DRBD, and with it more than the `replicated` StorageClass**. DRBD's resource registry is global per kernel, so three containers sharing one kernel cannot represent three DRBD nodes — a second `new-resource` for the same name returns success while changing nothing, which is worse than an error because LINSTOR then records a replica it does not have. It applies to kind equally; it is a property of one kernel, not of Talos. `linstor` must therefore run with `drbd.enabled: false`, without which every satellite CrashLoops and no PVC binds at all. Because linstor-csi couples RWX to DRBD, tenant worker disks become RWO and **live migration is unavailable on this lane** — a larger loss than the missing StorageClass and the reason the QEMU lane must not be retired from nightly coverage. Everything else survives: kube-ovn runs the real OVS dataplane, and LINSTOR keeps real ZFS pools and the production `local` StorageClass.

‎hack/e2e-container-up.sh‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,20 @@ KUBERNETES_VERSION="${KUBERNETES_VERSION:-v1.33.12}"
2929
log() { echo "[e2e-container-up] $*"; }
3030
die() { echo "[e2e-container-up] FATAL: $*" >&2; exit 1; }
3131

32+
# modprobe and zpool need root; docker does not, and must NOT be run through
33+
# sudo, or the compose project ends up owned by a different user than the
34+
# teardown in packages/core/testing/Makefile runs as. So escalate per command
35+
# rather than re-exec'ing the whole script. Empty when already root, which is
36+
# how this runs on a dev box; `sudo` on a CI runner, which runs as an
37+
# unprivileged user with passwordless sudo.
38+
SUDO=""
39+
if [ "$(id -u)" -ne 0 ]; then
40+
command -v sudo >/dev/null 2>&1 \
41+
|| die "not root and no sudo available; this script needs to load kernel
42+
modules and create zpools on the host."
43+
SUDO="sudo"
44+
fi
45+
3246
# ---------------------------------------------------------------------------
3347
# 1. Host kernel modules.
3448
#
@@ -40,12 +54,14 @@ die() { echo "[e2e-container-up] FATAL: $*" >&2; exit 1; }
4054
# or CNI error with no obvious cause.
4155
# ---------------------------------------------------------------------------
4256
for mod in openvswitch zfs; do
43-
modprobe "$mod" 2>/dev/null || true
57+
$SUDO modprobe "$mod" 2>/dev/null || true
4458
if ! grep -q "^${mod} " /proc/modules; then
4559
die "kernel module '${mod}' is not loaded on the host.
4660
The container lane takes its kernel from the host and Talos cannot load
4761
modules in container mode, so this must be loaded before the nodes start.
48-
Both ship in the base linux-modules package: 'modprobe ${mod}'."
62+
openvswitch ships with the distro kernel, so 'modprobe openvswitch' is
63+
usually enough; zfs needs the OpenZFS packages first (on Ubuntu,
64+
'apt-get install zfsutils-linux', then 'modprobe zfs')."
4965
fi
5066
log "host module ${mod}: loaded"
5167
done
@@ -57,14 +73,14 @@ done
5773
# node-distinct and each satellite pinned to its own; nothing enforces the
5874
# pinning. Acceptable for CI, not a production pattern.
5975
# ---------------------------------------------------------------------------
60-
mkdir -p "$ZPOOL_BACKING_DIR"
76+
$SUDO mkdir -p "$ZPOOL_BACKING_DIR"
6177
for n in 1 2 3; do
62-
if zpool list "data-srv${n}" >/dev/null 2>&1; then
78+
if $SUDO zpool list "data-srv${n}" >/dev/null 2>&1; then
6379
log "zpool data-srv${n}: already present"
6480
continue
6581
fi
66-
truncate -s "$ZPOOL_SIZE" "${ZPOOL_BACKING_DIR}/srv${n}.img"
67-
zpool create -f "data-srv${n}" "${ZPOOL_BACKING_DIR}/srv${n}.img"
82+
$SUDO truncate -s "$ZPOOL_SIZE" "${ZPOOL_BACKING_DIR}/srv${n}.img"
83+
$SUDO zpool create -f "data-srv${n}" "${ZPOOL_BACKING_DIR}/srv${n}.img"
6884
log "zpool data-srv${n}: created (${ZPOOL_SIZE})"
6985
done
7086

‎packages/core/testing/Makefile‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,14 @@ prepare-cluster-container:
6161
delete-cluster-container:
6262
docker compose -p cozy-e2e -f $(ROOT_DIR)hack/e2e-compose.yaml down -v || true
6363

64+
# COZY_APISERVER_ENDPOINT crosses into the sandbox explicitly: docker exec
65+
# starts a fresh environment, so a value exported by the caller does not reach
66+
# the bats file that reads it. Unset expands to empty here, and the default in
67+
# hack/e2e-install-cozystack.bats uses `:-`, which treats empty as absent -- so
68+
# the QEMU lane keeps 192.168.123.10 without naming it. The container lane has
69+
# no VIP and passes .11 instead.
6470
install-cozystack:
65-
docker exec "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-install-cozystack.bats'
71+
docker exec -e COZY_APISERVER_ENDPOINT="${COZY_APISERVER_ENDPOINT}" "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-install-cozystack.bats'
6672

6773
test-openapi:
6874
docker exec "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-test-openapi.bats'

0 commit comments

Comments
 (0)