Skip to content

fix(workspace): wait briefly for a held extension lock before refusing a load - #2657

Merged
davidfarah2003 merged 4 commits into
Cotal-AI:mainfrom
drakeo338:claude/2649-fix
Oct 10, 2026
Merged

davidfarah2003 merged 4 commits into
Cotal-AI:mainfrom
drakeo338:claude/2649-fix

Conversation

@drakeo338

@drakeo338 drakeo338 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2649.

Loading an installed extension claimed the extension lock with a zero wait, so a cotal command that started a moment before another process finished installing or removing extensions was refused with "extension install/remove is in progress", although a retry would have succeeded. The load now waits up to five seconds for the lock and fails with the same message if the other process is still running after that. I added a changeset and extended the materialize-concurrency smoke test, which fails without the change and passes with it (12 of 12 cells).

The maintainers completed this change on the contributor's branch. It is rebased on main, where the advisory lock code now lives in core, so the bounded wait moved there and the changeset lists core too, and the review findings are addressed. Two independent reviewers from different model families approved the exact head below, each with its own reproduction: a load that meets a held extension lock now waits and proceeds once the holder releases, and a lock still held after five seconds is refused with the same messages.

Approved-at: 6c7e8ba

@davidfarah2003

Copy link
Copy Markdown
Contributor

Thanks. The reader collision in #2649 reproduces at the merge-base and the bounded wait fixes it at this head. Three things before it can land:

  1. The wait is passed into the synchronous claimExtensionMutationLock / acquireLock, so the whole Node event loop blocks for the duration. With a live holder we measured materializeFromManifest blocking for about five seconds with a 50 ms interval timer firing zero times, where the merge-base refused in 2 ms. This is not only a short CLI path: the manager calls this API while it is serving requests and renewing leases, so timers, incoming RPCs and cancellation cannot run during the wait. Use a non-blocking bounded acquisition for the async materialization path while keeping the pass-to-writer exclusion.
  2. The five seconds are budgeted separately for the pass lock and the writer lock rather than for the whole acquisition. Holding the pass for about four seconds while a child borrows it and keeps the writer for about eight seconds, the load acquired the pass, then began a fresh writer wait and succeeded at around 7.7 s instead of refusing at the advertised five. Carry one deadline through both acquisitions, or change the promised bound.
  3. The changeset names "extension install/remove is in progress" as the timeout message, but a held mutation lock first fails on its pass guard with "another extension update or mutation is in progress (pid ) - retry once it finishes"; the quoted text only appears in the writer-only case. Correct the changeset, and add a short note on the affected operator page (docs/architecture.md or the cotal ext section of docs/cli.md) describing the bounded wait and the refusal.

We will re-review the next push.

@drakeo338

Copy link
Copy Markdown
Contributor Author

All three are in bb9c4d2, a plain commit on top. Materialization now waits through a new async lock acquisition, so the event loop keeps running; the smoke checks a 50 ms interval keeps firing while a load is queued. The pass and writer waits share one deadline, with a smoke cell that refuses at about 1.5 s. The changeset names the pass-guard message first.

@davidfarah2003

Copy link
Copy Markdown
Contributor

Thanks for the second push. The event-loop point from the first round is narrower now but still open, and two smaller things came up on this head:

  1. The new async entrypoint (advisory-lock.ts around line 298) still calls the whole synchronous acquireLock retry loop rather than one non-blocking attempt. With a stale canonical writer record and a live .reclaim sentinel, the call retries synchronously until the sentinel releases: a 700 ms reclaimer produced about 814 ms of event-loop blockage with zero 50 ms timer ticks, and a 6.2 s reclaimer made the load fulfil after 6.3 s with zero ticks, past the advertised five-second budget. One shared try-acquire/reclaim attempt that reports both holder and reclaimer contention, with the awaited retries and a single deadline on the async side, would close this. The synchronous loop itself is inherited and is tracked separately in fix(workspace): unify advisory-lock retry policies to bound stale reclaim contention #3249.
  2. The async mutation and update-pass functions in extension-mutation.ts (lines 80-121) copy the existing policy bodies almost verbatim: parent validation, the npm marker refusal, pass-before-writer ordering, the timeout wording, the paired releases and the error cleanup. Materialization is the only async caller and does not borrow a parent, yet the copy keeps that branch. Keep one policy and confine the sync-versus-async difference to the acquisition step.
  3. The operator note at docs/cli.md line 2143 attributes the pass-guard message ("another extension update or mutation is in progress (pid N) - retry once it finishes") to a held writer lock. A live writer refuses with the install/remove text from materialize.ts, including when only the pass is borrowed. The changeset describes the split correctly; the docs sentence does not.

Both reviews reproduced these through importInstalledExtension under isolated homes. We will re-review the next push.

@drakeo338

Copy link
Copy Markdown
Contributor Author

Pushed c16f716: async lock acquisition now does one non-blocking try-acquire/reclaim attempt (reporting holder and .reclaim-sentinel contention) with awaited retries under one deadline, extension-mutation keeps one pass-then-writer policy, and the docs/cli.md note matches the refusal messages.

@davidfarah2003

Copy link
Copy Markdown
Contributor

Two independent reviews approved this head (c16f716) with their own reproductions under isolated configuration roots, so the change itself is ready. It cannot be merged as it stands: the pull request now conflicts with main in packages/workspace/src/advisory-lock.ts (the lock reader changed in 15d10c9, "fix: contain competing same-host inbox consumers"), and GitHub reports it as not mergeable, which also keeps the attribution check from running on this head.

Please rebase onto current main and resolve that file. The new head will get a fresh review round; the approval paragraph above applies to c16f716 only.

drakeo338 and others added 4 commits October 10, 2026 01:25
…nt loop

The load's bounded wait ran inside the synchronous lock acquisition, which
blocked every timer and request for up to five seconds, and the five seconds
were granted separately to the pass lock and the writer lock.

Add an awaited variant of the acquisition and use it for materialization, with
one deadline shared by the pass and writer locks. Correct the changeset to name
the pass-lock refusal first and describe the bounded wait in docs/cli.md.
…ing attempt

The async lock acquisition called the whole synchronous acquire loop, so a
stale writer record behind a live reclaim sentinel still blocked the event
loop and ran past the advertised five-second bound.

Split one non-blocking try-acquire/reclaim attempt out of acquireLock. It
reports both a live holder and a live reclaimer; the sync loop sleeps between
attempts and the async one awaits timers under a single deadline.

In extension-mutation.ts, keep one pass -> writer policy (parent validation,
npm marker refusal, timeout wording, paired releases, error cleanup) and let
the sync and async claimers differ only in how they acquire. The async claim no
longer carries the parent-borrow branch, which materialization never uses.

Correct the docs/cli.md note: a live add, remove or update holds the pass lock
too, so the pass-guard message is what a load sees; the install/remove text
appears only when the pass is free but the writer is still held.

Signed-off-by: drakeo338 <[email protected]>
The lock primitive now lives in @cotal-ai/core, so the async acquirer this change adds ships from core and the changeset names it.
@davidfarah2003
davidfarah2003 merged commit f9a71ce into Cotal-AI:main Oct 10, 2026
16 of 25 checks passed
@davidfarah2003

Copy link
Copy Markdown
Contributor

Thanks @drakeo338, this landed in f9a71ce. Loading an installed extension now waits up to five seconds for a held extension lock without blocking the process, instead of refusing at once; we rebased it on main, where the lock code now lives in core. We checked it by reproducing the refusal at the old base and the successful wait at this head, running the extended materialize-concurrency smoke, and typechecking the whole repo merged with current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage:needs-info A specific missing input prevents a meaningful reproduction attempt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

workspace: loading an installed connector takes the exclusive extension mutation lock with no wait, so concurrent CLI commands refuse each other

2 participants