CI: self-hosted ARM64 runners can't start service containers — backend gate red since ~04:57 UTC 2026-08-28 #61

Summary

The test (backend) gate has failed on every commit since ~2026-08-28 04:57 UTC. Root cause is a degraded container runtime on the [self-hosted, Linux, ARM64] runner host(s) — not a code regression. It needs the runner host repaired/recycled; no repo change fixes it.

Symptom

The postgres:17 service container fails during the runner's Initialize containers step, before any job step runs:

docker pull postgres:17@sha256:e384... → all layers Pull complete   (pull is FINE)
exec /usr/local/bin/docker-entrypoint.sh: invalid argument
Failed to initialize container postgres:17@sha256:e384...
One or more containers failed to start.

The image pulls successfully (correct arch); the failure is at entrypoint exec (EINVAL).

Evidence it is NOT code

• The exact pinned digest postgres:17@sha256:e38411452a464af89e5adadb8d223bf53b898d47d6ef918b2d58c08707350449 boots cleanly on arm64 locally (database system is ready to accept connections).
• Gate was green at 04:31, red on every commit from 04:57 onward. The postgres digest hasn't changed since Aug 21 (e74413ff), and passed for days on the same runners.
• Runners are not provisioned from this repo (no ARC/runner config), and no commit in the last 2 days touched runner infra.
• Re-ran across multiple commits (04:58 → 06:51) — identical failure, no recovery.

Impact

Every lego/backend-touching push fails CI, and it also fails deploy.yml's backend workflow_call gate. Blocks merges/deploys that touch the backend.

Remediation

Repair/recycle the self-hosted ARM64 runner host(s): inspect Docker/containerd/runc/overlay2 state; most likely a broken container runtime or a bad node-image/kernel change around 04:57 UTC. Rolling the node(s) should restore the gate. Once healthy, backend goes green automatically on the next lego/backend push.

Related

scripts (test) was a separate, code-level failure (unpinned dev-env agent images from ee7ecebb) and is already fixed + green on main (15c78715). This issue is only the runner-host half.

Filed automatically while working the "get CI green" task — the backend gate is the one remaining red gate and it requires host access to fix.