Controller restore-path resilience: stop mass-pruning sandboxes on restart and restore failures #301

Problem

Intermittent CI failures across the stack trace back to a multi-layer reliability gap in the forkd controller's snapshot restore path. The failure chain:

1. restore_many_with fails — firecracker VMs fail to start during restore, logging socket /tmp/forkd-daemon-<tag>-<id>/child-N.sock never appeared within 10s (forkd-vmm wait_for_sock, lib.rs ~line 902). Observed to be systemic (affects multiple image tags, not one).
2. Controller restart prunes ALL sandboxes — on startup, Registry::reconcile() (state.rs:213) prunes every sandbox entry whose PID is no longer alive. After a watchdog-triggered restart, every running sandbox is pruned (observed: 22 pruned in one incident), invalidating all upstream lease references in a single stroke.
3. Orphaned firecracker processes survive backend restarts — when the upstream lease manager restarts, its Shutdown() path does not fully kill all sandbox VMs via the forkd API, leaving orphaned firecracker processes. These orphans cause resource contention (tap device conflicts, netns conflicts, fd limits) that prevents new restore_many_with operations from succeeding, producing a cascade.

Root cause analysis

• The 10-second wait_for_sock timeout (Duration::from_secs(10) at lib.rs:980 and :1486) is a fixed budget with no load-awareness. Under contention from orphaned VMs or parallel restore bursts, firecracker processes take longer than 10s to open their API socket, and the wait bails out.
• reconcile() is a blunt instrument: it cannot distinguish "VM died unexpectedly" from "VM is alive but the controller just restarted." On restart, all live VMs become unreconcilable (no live_vms handles can be reconstructed from registry data alone) and are pruned.
• There is no pre-restore orphan detection: restore_many_with does not check whether firecracker processes from a prior (crashed) run still hold tap devices or netns slots before attempting a new restore.

Resource exhaustion ruled out

The host has ample capacity during failures: 62 GB RAM (58 GB available), 22 GB disk free, 32 GB tmpfs. The 22 firecracker processes and 4m37s CPU / 2.7 GB peak of the controller are well within limits. The failures are not resource exhaustion — they are startup-path contention and blunt-restart side effects.

Proposed fix areas

1. Restore-path resilience

• Investigate why firecracker processes fail to open their API socket within 10s under load — tap/netns conflicts from orphaned VMs, fd limits, or controller-internal serialization.
• Make the wait_for_sock timeout configurable and/or load-aware (longer budget under concurrent restore bursts), rather than a hardcoded 10s.
• Add per-VM error reporting so partial restore_many_with failures surface which specific child failed, rather than a single bail that loses all children.

2. Graceful reconcile / restart recovery

• Distinguish "controller crashed with surviving VMs" from "VM died" during reconcile(). Surviving firecracker processes with valid sockets are manageable orphans, not stale entries to prune — either reattach (if feasible) or kill targeted PIDs before pruning the registry.
• (In progress on branch fix/controller-restart-recovery — kill_orphans() kills orphaned firecracker processes on startup after reconcile(), with PID-reuse guards and bounded death polling. Tracked separately; this issue covers the broader restore-path resilience.)

3. Orphan detection before restore

• Before restore_many_with, detect and reap orphaned firecracker processes (processes not tracked by a live live_vms entry) holding tap devices or netns slots, so a new restore is not doomed by a prior crash's leftovers.

4. Watchdog: targeted recovery instead of mass kill

• The operational watchdog (forkd-spawn-watchdog.sh, deployed on the server, not in this repo) currently pkill -9 firecracker — killing all running VMs, not just the failed ones. A single-VM restore failure becomes a full-pool outage.
• A graduated response: first attempt soft restart of just the failed restore, only escalate to controller restart on repeated failures. Kill firecracker processes associated with the failed daemon dir only, not all firecracker processes.
• (The watchdog script is operational tooling living outside this repo; this issue tracks the controller-side changes that would enable targeted recovery — per-VM error reporting, orphan detection, graceful reconcile.)

Acceptance criteria

1. Single VM restore failure does NOT prune all running sandboxes
2. restore_many_with reports per-VM failures so partial successes are not lost
3. Orphaned firecracker processes from a prior crash are detected and reaped before a new restore is attempted
4. wait_for_sock timeout is no longer a hardcoded 10s under load
5. A controller restart does not mass-prune VMs that are still alive

Related

• The upstream lease manager has a companion issue for its side of the cascade (runner retry on 410 Gone, graceful drain before restart, orphan detection on backend startup).