Runtime update rolls back the binary after the new daemon already migrated the global DB (20 s restart watchdog vs ~80 s boot), leaving an unbootable install #559

Summary

compozy update (runtime target, install_method: direct-binary) swapped the binary from 0.3.0-beta.22 to 0.3.0-beta.23 and restarted the daemon. The new daemon migrated the global database (goose stream global 99 → 107) within its first second, then took ~80 s to become ready. The restart watchdog only waits ~20 s, declared the replacement daemon failed, and the updater rolled the binary back to beta.22 — but the database migration is irreversible, so the restored beta.22 can no longer open ~/.compozy/compozy.db:

error: daemon: open global database "~/.compozy/compozy.db": store: initialize sqlite database "~/.compozy/compozy.db": globaldb: inspect operator home migration status: store: database schema is ahead of this binary: ~/.compozy/compozy.db stream global is at version 107, binary head is 99; install a newer CompozyOS binary or stop

Net effect: a slow boot was turned into a permanently unbootable install (CLI daemon and desktop app both dead) that required manually installing the runtime tarball and the desktop app bundle to recover. The same "update leaves the home unbootable" outcome was previously reported in #451 (different trigger). The structural defect here is that the rollback path runs after an irreversible schema migration has already been applied.

Environment

• OS: macOS (Darwin 25.6.0), Apple Silicon
• Runtime: 0.3.0-beta.22 → 0.3.0-beta.23, direct binary at ~/.local/bin/compozy, updated via compozy update
• Desktop app: /Applications/CompozyOS.app 0.3.0-beta.22, managed runtime copy at ~/.compozy/bin/compozy (.desktop-provenance.json → installed_by: desktop-app)
• Home size (relevant, this is what makes boot slow): ~/.compozy/compozy.db 145 MB; ~/.compozy/sessions 262 session dirs, 836 MB
• checksums.txt of the release matched both downloads (compozy_darwin_arm64.tar.gz 03f83fa5…, CompozyOS-0.3.0-beta.23-mac-arm64.zip 22f7bcc0…), so this is not a corrupt asset

Steps to reproduce

The only precondition is a home directory where the new daemon needs more than ~20 s to reach readiness. A large home does that naturally (see numbers above). If you don't have one, a deterministic way to simulate it (suggested, I have not run this variant myself) is to kill -STOP the replacement daemon right after it logs store.migrations.applied stream=global, wait 30 s, then kill -CONT it.

1. Have the daemon running on beta.22 (compozy status → running).
2. Run compozy update (both targets pending: runtime beta.23 + app beta.23).
3. Watch ~/.compozy/logs/compozy.log and ~/.compozy/restarts/<id>.json.

Actual result (timeline, UTC, taken from compozy.log, restarts/*.json, update-history.jsonl)

~/.compozy/restarts/f02bf261-….json after the fact:

{
  "operation_id": "f02bf261-48f1-44da-9717-62292ea3c36d",
  "status": "failed",
  "old_pid": 83857,
  "old_started_at": "2026-09-08T04:16:40Z",
  "active_session_count": 0,
  "failure_reason": "daemon: replacement daemon did not become ready before timeout",
  "started_at": "2026-09-08T21:33:50.237675Z",
  "completed_at": "2026-09-08T21:34:10.9292Z"
}

compozy update -o json afterwards (runtime block):

"runtime": {
  "status": "failed",
  "install_method": "direct-binary",
  "current_version": "0.3.0-beta.22",
  "latest_version": "v0.3.0-beta.23",
  "restored_version": "0.3.0-beta.22",
  "daemon_restarted": false,
  "message": "The runtime update failed and the previous version was restored.",
  "last_error": "cli: daemon restart did not complete before timeout: context deadline exceeded"
}

State after the update "completed": no daemon process running, compozy daemon start fails instantly with the schema-ahead error, desktop app stuck at state: error, code: runtime_start_failed with the same message. Nothing in the user-facing output mentions that the DB was migrated or that the restored binary cannot open it.

Expected result

Any one of these would have avoided the outage:

1. Never roll back the binary after an irreversible migration has run. If the new daemon has already applied schema migrations, rollback must either restore a pre-migration DB backup taken by the updater, or be skipped with a clear message ("new daemon migrated the DB; keeping new binary, retry compozy daemon start").
2. Order boot so migrations happen after readiness gating, or run them as a separate, explicit pre-flight step whose duration is not counted against the restart watchdog.
3. Make the restart watchdog progress-aware (extend while the daemon keeps logging progress) or at least much longer than 20 s. 20 s is below the boot time of a moderately used home; the daemon here was 60 s away from being healthy when it was declared dead.

Related defects observed while recovering (same root pattern: fixed short timeouts + kill-on-timeout)

• compozy daemon start (CLI) returned error: cli: daemon did not become ready before timeout: context deadline exceeded while the detached daemon kept booting and answered compozy status ~35 s later. The exit code says "failed" for a start that succeeded.
• Desktop app kills a foreign-version daemon. With a healthy beta.23 daemon running (started from ~/.local/bin), launching the beta.22 app went through phase: resolve for 28 s, then sent SIGTERM to that daemon (daemon: received shutdown signal signal=terminated at 21:44:52) and started its own bundled beta.22 runtime ("resolution":"start"), which of course died on the schema-ahead error. There is no attach-to-newer-runtime path, so the running daemon was the only working thing and the app removed it.
• Staged app update never applies once the app can't boot. compozy update reported CompozyOS app update staged; applies on next launch, but the app never reached the point of applying it (it errors in runtime_start_failed first). ~/Library/Caches/@compozydesktop-updater/ only contained the previous beta.22 zip, nothing for beta.23. So the app-side update cannot be the recovery path either.
• Desktop bootstrap: 15 s per attempt × 5, and it reaps the daemon it started. After manually installing app beta.23, desktop-bootstrap.jsonl shows resolution: provision → Starting the CompozyOS runtime → 15 s later The runtime did not become ready; retrying, 5 times, ending in cli: reap unready detached daemon: signal: killed. Each retry kills a daemon that was mid-boot and starts over, so it can never succeed on a home where boot > 15 s. The only thing that worked was starting the daemon from the CLI first, waiting for compozy status, then launching the app → Attached to the running CompozyOS runtime.
• Stale failed status. After the runtime was current again, compozy update --check kept reporting runtime.status: failed / "The runtime update failed and the previous version was restored." because the last runtime block in ~/.compozy/logs/update-history.jsonl has phase: rolled-back. It only cleared after deleting that line.

Impact

Complete outage of the local CompozyOS install (CLI daemon and desktop app) after a routine compozy update, with no automatic recovery path and an error message that only makes sense if you already know the updater rolled back the binary. This is the second time on this machine that a runtime update's rollback left the DB ahead of the binary (beta.21→beta.22 on 2026-09-01 had a different crash cause but the same rollback-after-migration outcome), and #451 describes a third unbootable-after-update case.

Workaround that recovered the install (manual, in this order)

1. gh release download v0.3.0-beta.23 --repo compozy/compozy -p compozy_darwin_arm64.tar.gz -p checksums.txt; verify sha256; install -m 0755 compozy ~/.local/bin/compozy.
2. compozy daemon start (ignore its timeout error), wait until compozy status answers.
3. Do not launch the beta.22 desktop app at this point (it will kill the daemon). Download CompozyOS-0.3.0-beta.23-mac-arm64.zip, verify sha256, ditto -x -k, codesign --verify --deep --strict, replace /Applications/CompozyOS.app.
4. Launch the app only while the daemon is already ready → it attaches ("resolution":"attach") and provisions ~/.compozy/bin/compozy with the beta.23 copy.
5. compozy update --cancel to clear the dormant waiting-for-app operation; remove the phase: rolled-back line from update-history.jsonl to clear the stale failed status.