Repository navigation
fix: recover stalled and wedged multi-runs - #230
Conversation
Introduce a resolved StallPolicy on model.RuntimeConfig (enabled, idle timeout, child-backstop timeout, terminal-command cap, retries) plus the parked-alert sound, with on-by-default values applied in ApplyDefaults and the child > idle invariant enforced. Expose the knobs as [defaults.stall] TOML overrides and [sound] on_parked, wired through the daemon project and per-run apply paths with duration parsing consistent with existing overrides. StallPolicy() gives downstream tasks a single coherent read.
Add terminal parked status to the recovery and daemon run-status vocabularies, map it in recoveryStatusForRunStatus, and treat it as terminal in isTerminalRunStatus so a parked child unblocks the parent await loop. Add job.stalled and job.parked event kinds with a JobParkedPayload carrying triage detail, and classify job.parked as a terminal journal event to force flush+fsync.
Terminal and Bash tool commands ran on a detached context.Background(), so a hung command could never be cancelled and the agent blocked forever on wait-for-exit, surfacing as a tool call stuck in PENDING (ADR-004). - Parent each terminal command to the session run/attempt context so cancellation and shutdown propagate to the subprocess tree. - Apply StallPolicy.TerminalCap as an absolute wall-clock backstop per command, plumbed through runshared.Config.Stall and ClientConfig. - Enforce a process-group kill ladder: SIGTERM, grace, SIGKILL, then reap the tree so no orphaned subprocess survives; force-kill on ctx cancel. - Deadline the permission and filesystem handlers we own, returning a structured failure instead of a silent hang. Commands that complete within the cap are unaffected.
Turn the per-attempt activity watchdog into the fast stall-detection layer. The watchdog now keys off the resolved StallPolicy.IdleTimeout instead of the init/activity Timeout (which continues to drive ResolveACPInitTimeout only), and arms whenever StallPolicy.Enabled even when Timeout<=0 so a run is never left unguarded. Any recorded activity still resets the idle clock. On a full-silence timeout the watchdog reaps the session's terminals via the graceful process-group kill ladder (agent.clientImpl.CloseTerminals) before cancelling the attempt with a typed activity-timeout cause, and it emits one structured diagnostic when idle first crosses 50% and 80% of the window, reusing the task_03 diagnostics channel. Introduce a runshared.Clock/Ticker seam (RealClock in production, injectable fake in tests) threaded through ActivityMonitor and the watchdog so idle windows advance under test control without real sleeps.
…ackstop The parent await loop now watches each child's durable journal high-water sequence. A child whose sequence stops advancing for StallPolicy.ChildTimeout is cancelled through the existing cancel path, so it settles on a terminal status and the batch join proceeds in enqueued, parallel-multi and parallel-tasks modes. A child that keeps advancing is never touched, and reaping one child never blocks its siblings.
…covery summary Render the recovery states the run TUI could not show before. A job that stalls gets its own transient state and a plain-language reason on the timeline; a job that stalls again after its clean-state retry renders as terminal parked, with its preserved worktree, rather than being folded into the failure count. Close every run with a completed/recovered/parked line. Recovered is the subset of completions that needed a stall recovery to get there, derived from the sticky Job.Stalled flag in-process and from the presence of a job.stalled event in the child's journal on the daemon side. Classifying off the journal rather than the child run row sidesteps the gap where a self-parked child still surfaces as a failed run row. Parked is disjoint from completed and failed in every counter, so run-completion denominators now key off settled = completed + failed + parked. The new task.multi.summary event is emitted first on every finalization path, including cancellation and aggregate child failure, because a batch that parked a child is exactly the batch whose closing counts the user needs. It journals through a detached context, so it survives parent cancellation. Also fixes a pre-existing alignment bug in the summary box, where mixed 9- and 10-wide stat labels sat the SUCCEEDED value one column left of the others.
A park is now the only job transition that alerts: EventKindJobParked maps to a new OnParked sound, fired once from markParked. Stalls and their clean-state retries stay silent, so an alert always means a job needs a human. The parked job's JobParkedPayload detail - plain reason, last tool call, journal high-water sequence, and the preserved worktree and log - now renders in a TUI triage panel on the closing summary and in 'compozy runs watch' output, so the returning user can rerun or inspect without digging. SoundOnParked was orphaned on RuntimeConfig since task_01; wire it through core.Config, the CLI state, runshared.Config, and the extension hook mirror, and validate sound.on_parked alongside its siblings.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedReview was skipped due to path filters ⛔ Files ignored due to path filters (1)
CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThis PR adds stall/park recovery across configuration, execution, daemon, ACP, UI, CLI, sound, and event payloads. It introduces stall policy settings, new stalled/parked job states, parked run reporting, per-child liveness backstops, terminal command caps, and updated rendering and notification paths. ChangesStall/Park Feature
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant jobRunner
participant jobLifecycle
participant worktreeReset
participant childBackstop
participant RunManager
jobRunner->>jobLifecycle: markStalled()
jobRunner->>worktreeReset: resetWorktreeForStallRetry()
alt stall budget remains
jobRunner->>jobRunner: retry attempt from clean snapshot
else park required
jobRunner->>jobLifecycle: markParked()
end
childBackstop->>RunManager: reap() cancels wedged child
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| Filename | Overview |
|---|---|
| internal/core/run/executor/runner.go | Introduces the stall retry loop, attemptBudget tracking, handleStall/parkJob, and worktree reset logic. The two independent budgets (ordinary vs stall) are well-reasoned and the retry arithmetic looks correct. |
| internal/core/run/internal/acpshared/session_exec.go | Wires the stall watchdog with the new ActivityWatchdogConfig. The Clock: activityClock direct read bypasses activityClockMu that was added in this same PR to protect the variable. |
| internal/daemon/task_multi.go | Adds the child backstop ticker loop and recovery summary emission. A transient GetRun error inside the ticker path aborts the wait early, a new failure mode not present before the polling was added. |
| internal/daemon/child_backstop.go | New file: clean implementation of the per-child durable liveness guard. Budget invariant enforcement and nil-safe no-op design are correct. |
| internal/core/run/executor/lifecycle.go | Adds markStalled and markParked lifecycle transitions. Parked correctly records a failure for non-zero exit while keeping the job status distinct from failed. |
| internal/core/run/executor/execution.go | Replaces raw failed int32 with per-job countRunRecovery, adds parked-aware deriveRunStatus, and suppresses the duplicate terminal sound for parked runs. |
| internal/core/agent/terminal.go | Adds graceful SIGTERM→grace→SIGKILL kill ladder, wall-clock cap via context.WithTimeoutCause, and CloseTerminals capability used by the stall watchdog. killOnce ensures the ladder runs at most once. |
| internal/core/run/internal/acpshared/aliases.go | Adds activityClockMu RWMutex to protect activityClock in newActivityMonitor and SetActivityClockForTest, but a direct unprotected read remains in session_exec.go. |
| internal/daemon/task_multi_summary.go | New file: reads each child's journal for stalled/parked/completed events and folds them into the recovery summary. Best-effort design (errors return nil events, not failures) is correct. |
| internal/core/model/runtime_config.go | Adds StallPolicy struct, pointer-typed StallEnabled/StallRetries for override semantics, and applyStallDefaults with invariant enforcement. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Attempt starts] --> B[Activity watchdog armed]
B --> C{Session update?}
C -- yes --> D[Reset idle clock] --> C
C -- no, idle >= IdleTimeout --> E[KillTerminals + Cancel attempt]
E --> F[HandleSessionTimeout: Stalled=true]
F --> G{handleStall}
G -- stall budget = 0 --> H[parkJob]
G -- pre-retry hook fails --> H
G -- worktree reset fails --> H
G -- budget > 0 and reset OK --> I[Reset worktree to pre-snapshot]
I --> J[budget.stall--, budget.total++]
J --> A
H --> K[markParked: status=parked, triage alert]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[Attempt starts] --> B[Activity watchdog armed]
B --> C{Session update?}
C -- yes --> D[Reset idle clock] --> C
C -- no, idle >= IdleTimeout --> E[KillTerminals + Cancel attempt]
E --> F[HandleSessionTimeout: Stalled=true]
F --> G{handleStall}
G -- stall budget = 0 --> H[parkJob]
G -- pre-retry hook fails --> H
G -- worktree reset fails --> H
G -- budget > 0 and reset OK --> I[Reset worktree to pre-snapshot]
I --> J[budget.stall--, budget.total++]
J --> A
H --> K[markParked: status=parked, triage alert]
Reviews (6): Last reviewed commit: "fix: accept snake_case stall runtime ove..." | Re-trigger Greptile
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (10)
internal/core/run/executor/runner_stall_test.go (1)
213-213: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winWrap the flat top-level tests in
Should…subtests.Several new tests here (e.g.
TestJobRunnerStallRetryIsIndependentOfMaxRetries,...SecondStallParksWithPopulatedPayload,...StallRetryRunsFromACleanWorktree,...ZeroStallBudgetParksOnFirstStall,...StalledJobDoesNotAffectSiblings) run assertions directly at the top level rather than inside at.Run("Should…")subtest, unlike the ones that already do. Please align them with the convention. As per path instructions: "MUST use t.Run("Should...") pattern for ALL test cases".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/run/executor/runner_stall_test.go` at line 213, The new runner stall tests are asserting at the top level instead of following the required Should-prefixed subtest convention. Update the affected test functions in runner_stall_test.go, including TestJobRunnerStallRetryIsIndependentOfMaxRetries and the related stall retry cases, to wrap their assertions inside t.Run("Should...", ...) subtests so they match the existing pattern used by the other tests and the path requirement.Source: Path instructions
internal/core/run/executor/parked_alert_test.go (1)
38-110: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse the
t.Run("Should...")subtest pattern.The table cases here are named
"nil config is silent","sound disabled...", etc., andTestNotifyParkedAlertIsANoOpWhenSoundIsDisabled/TestNotifyParkedAlertOnNilExecutionContextDoesNotPanicare flat. Rename case names to theShould…form (and wrap the flat tests in aShould…subtest) for consistency with the repo test convention. As per path instructions: "MUST use t.Run("Should...") pattern for ALL test cases".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/run/executor/parked_alert_test.go` around lines 38 - 110, Update the parked alert tests to follow the repo’s required t.Run("Should...") convention. In TestSoundConfigForGatesOnTheSoundFeatureFlag, rename each table case to a Should... subtest name, and in TestNotifyParkedAlertIsANoOpWhenSoundIsDisabled and TestNotifyParkedAlertOnNilExecutionContextDoesNotPanic wrap the assertions in a single t.Run("Should...") subtest. Keep the existing behavior and assertions intact while making the names consistent with the test pattern used around soundConfigFor and notifyParkedAlert.Source: Path instructions
internal/core/workspace/config_test.go (1)
487-561: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winWrap standalone stall tests in
t.Run("Should...")subtests.
TestLoadConfigParsesStallSectionandTestLoadConfigMergesStallWorkspaceOverGlobalConfigdon't use thet.Run("Should...")pattern. As per path instructions, all test cases must use this pattern.♻️ Proposed refactor for `TestLoadConfigParsesStallSection`
func TestLoadConfigParsesStallSection(t *testing.T) { - isolateWorkspaceConfigHome(t) - root := t.TempDir() - writeWorkspaceConfig(t, root, ` + t.Run("Should parse stall section fields and sound.on_parked", func(t *testing.T) { + isolateWorkspaceConfigHome(t) + root := t.TempDir() + writeWorkspaceConfig(t, root, ` [defaults.stall] enabled = false timeout = "2m" child_timeout = "8m" terminal_command_timeout = "30m" retries = 2 [sound] on_parked = "ping" `) - - cfg, _, err := LoadConfig(context.Background(), root) - if err != nil { - t.Fatalf("load config: %v", err) - } - stall := cfg.Defaults.Stall - if stall.Enabled == nil || *stall.Enabled { - t.Fatalf("unexpected stall.enabled: %#v", stall.Enabled) - } - if stall.Timeout == nil || *stall.Timeout != "2m" { - t.Fatalf("unexpected stall.timeout: %#v", stall.Timeout) - } - if stall.ChildTimeout == nil || *stall.ChildTimeout != "8m" { - t.Fatalf("unexpected stall.child_timeout: %#v", stall.ChildTimeout) - } - if stall.TerminalCommandTimeout == nil || *stall.TerminalCommandTimeout != "30m" { - t.Fatalf("unexpected stall.terminal_command_timeout: %#v", stall.TerminalCommandTimeout) - } - if stall.Retries == nil || *stall.Retries != 2 { - t.Fatalf("unexpected stall.retries: %#v", stall.Retries) - } - if cfg.Sound.OnParked == nil || *cfg.Sound.OnParked != "ping" { - t.Fatalf("unexpected sound.on_parked: %#v", cfg.Sound.OnParked) - } + + cfg, _, err := LoadConfig(context.Background(), root) + if err != nil { + t.Fatalf("load config: %v", err) + } + stall := cfg.Defaults.Stall + if stall.Enabled == nil || *stall.Enabled { + t.Fatalf("unexpected stall.enabled: %#v", stall.Enabled) + } + if stall.Timeout == nil || *stall.Timeout != "2m" { + t.Fatalf("unexpected stall.timeout: %#v", stall.Timeout) + } + if stall.ChildTimeout == nil || *stall.ChildTimeout != "8m" { + t.Fatalf("unexpected stall.child_timeout: %#v", stall.ChildTimeout) + } + if stall.TerminalCommandTimeout == nil || *stall.TerminalCommandTimeout != "30m" { + t.Fatalf("unexpected stall.terminal_command_timeout: %#v", stall.TerminalCommandTimeout) + } + if stall.Retries == nil || *stall.Retries != 2 { + t.Fatalf("unexpected stall.retries: %#v", stall.Retries) + } + if cfg.Sound.OnParked == nil || *cfg.Sound.OnParked != "ping" { + t.Fatalf("unexpected sound.on_parked: %#v", cfg.Sound.OnParked) + } + }) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/workspace/config_test.go` around lines 487 - 561, Wrap each standalone stall test in a t.Run subtest using the required Should naming pattern. Update TestLoadConfigParsesStallSection and TestLoadConfigMergesStallWorkspaceOverGlobalConfig to place their existing assertions inside t.Run("Should...", ...) blocks, preserving the current test logic and helper usage like isolateWorkspaceConfigHome, writeWorkspaceConfig, writeGlobalConfig, and LoadConfig.Source: Path instructions
internal/core/run/ui/stall_states_test.go (1)
127-163: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winWrap single-scenario tests in
t.Run("Should...")subtests.
TestParkedJobRendersTerminalParkedStateandTestZeroStallRunLeavesPerJobLayoutUnchangeddon't use thet.Run("Should...")pattern, unlike the other tests in this file which correctly follow it.As per path instructions: "MUST use t.Run("Should...") pattern for ALL test cases".
♻️ Suggested wrapping
func TestParkedJobRendersTerminalParkedState(t *testing.T) { t.Parallel() + t.Run("Should render terminal parked state after stall and retry", func(t *testing.T) { + t.Parallel() - m, translator := newStallTestModel(1) + m, translator := newStallTestModel(1) - applyEventToModel(t, m, translator, jobStalledEvent(t, 0, 1)) - applyEventToModel(t, m, translator, jobRetryEvent(t, 0, 2)) - applyEventToModel(t, m, translator, jobParkedEvent(t, 0)) + applyEventToModel(t, m, translator, jobStalledEvent(t, 0, 1)) + applyEventToModel(t, m, translator, jobRetryEvent(t, 0, 2)) + applyEventToModel(t, m, translator, jobParkedEvent(t, 0)) - job := &m.jobs[0] + job := &m.jobs[0] - if job.state != jobParked { - t.Fatalf("expected jobParked, got %v", job.state) - } + if job.state != jobParked { + t.Fatalf("expected jobParked, got %v", job.state) + } - if got, want := m.jobStateIcon(job.state), jobIconParked; got != want { - t.Fatalf("expected icon %q, got %q", want, got) - } + if got, want := m.jobStateIcon(job.state), jobIconParked; got != want { + t.Fatalf("expected icon %q, got %q", want, got) + } - if got := m.sidebarTimeString(job); got != "parked" { - t.Fatalf("expected sidebar time string %q, got %q", "parked", got) - } + if got := m.sidebarTimeString(job); got != "parked" { + t.Fatalf("expected sidebar time string %q, got %q", "parked", got) + } - if got, want := job.worktreePath, "/tmp/wt/task_02"; got != want { - t.Fatalf("expected preserved worktree %q, got %q", want, got) - } + if got, want := job.worktreePath, "/tmp/wt/task_02"; got != want { + t.Fatalf("expected preserved worktree %q, got %q", want, got) + } - if m.parked != 1 { - t.Fatalf("expected parked=1, got %d", m.parked) - } + if m.parked != 1 { + t.Fatalf("expected parked=1, got %d", m.parked) + } - if m.completed != 0 || m.failed != 0 { - t.Fatalf("parked must not count as completed or failed, got completed=%d failed=%d", m.completed, m.failed) - } + if m.completed != 0 || m.failed != 0 { + t.Fatalf("parked must not count as completed or failed, got completed=%d failed=%d", m.completed, m.failed) + } - if !m.isRunComplete() { - t.Fatal("expected a parked job to settle the run") - } + if !m.isRunComplete() { + t.Fatal("expected a parked job to settle the run") + } - if m.hasActiveJobs() { - t.Fatal("expected no active jobs after a park") - } + if m.hasActiveJobs() { + t.Fatal("expected no active jobs after a park") + } - if got := m.composerDisabledLabel(job); got != "Task parked - needs attention" { - t.Fatalf("unexpected composer label %q", got) - } + if got := m.composerDisabledLabel(job); got != "Task parked - needs attention" { + t.Fatalf("unexpected composer label %q", got) + } + }) }Also applies to: 271-286
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/run/ui/stall_states_test.go` around lines 127 - 163, Wrap the single-scenario tests in this file, especially TestParkedJobRendersTerminalParkedState and TestZeroStallRunLeavesPerJobLayoutUnchanged, in t.Run("Should...") subtests to match the required test pattern used by the other stall state tests. Keep the existing assertions and setup inside the new subtest blocks, and ensure the top-level test functions only orchestrate the subtest structure.Source: Path instructions
internal/core/run/ui/parked_detail_test.go (1)
23-48: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAlign subtest names with the "Should..." pattern.
Several tests in this file don't follow the
t.Run("Should...")naming convention required by the project's test path instructions.TestParkedJobCarriesFullTriageDetailuses bare names like"reason"and"last tool call", andTestSummaryParkedBoxRendersActionableDetail,TestSummaryViewOmitsParkedBoxWithoutParkedJobs,TestParkedDetailLinesSkipEmptyFields, andTestParkedProgressValuelack subtests entirely.As per path instructions: "MUST use t.Run("Should...") pattern for ALL test cases".
♻️ Suggested naming adjustments
func TestParkedJobCarriesFullTriageDetail(t *testing.T) { t.Parallel() job := &parkedDetailModel(t).jobs[0] cases := []struct { name string got string want string }{ - {"reason", job.stallReason, "no output for 3m0s"}, - {"last tool call", job.stallLastToolCall, "Bash go test ./..."}, - {"worktree", job.worktreePath, "/tmp/wt/task_02"}, - {"log", job.parkLogPath, "/tmp/logs/task_02.out.log"}, + {"Should carry the stall reason", job.stallReason, "no output for 3m0s"}, + {"Should carry the last tool call", job.stallLastToolCall, "Bash go test ./..."}, + {"Should carry the preserved worktree path", job.worktreePath, "/tmp/wt/task_02"}, + {"Should carry the preserved log path", job.parkLogPath, "/tmp/logs/task_02.out.log"}, }For single-scenario tests, wrap the body in a
t.Run:func TestSummaryParkedBoxRendersActionableDetail(t *testing.T) { t.Parallel() + t.Run("Should render all parked triage fields in the summary box", func(t *testing.T) { + t.Parallel() - m := parkedDetailModel(t) + m := parkedDetailModel(t) - parked := m.parkedJobs() - if len(parked) != 1 { - t.Fatalf("expected exactly one parked job, got %d", len(parked)) - } + parked := m.parkedJobs() + if len(parked) != 1 { + t.Fatalf("expected exactly one parked job, got %d", len(parked)) + } - box := xansi.Strip(m.renderSummaryParkedBox(80, parked)) - for _, want := range []string{ - "RUN.PARKED", - statusLabelParked, - "task_02", - "no output for 3m0s", - "Bash go test ./...", - "seq 42", - "/tmp/wt/task_02", - "/tmp/logs/task_02.out.log", - } { - if !strings.Contains(box, want) { - t.Fatalf("parked box missing %q:\n%s", want, box) - } - } + box := xansi.Strip(m.renderSummaryParkedBox(80, parked)) + for _, want := range []string{ + "RUN.PARKED", + statusLabelParked, + "task_02", + "no output for 3m0s", + "Bash go test ./...", + "seq 42", + "/tmp/wt/task_02", + "/tmp/logs/task_02.out.log", + } { + if !strings.Contains(box, want) { + t.Fatalf("parked box missing %q:\n%s", want, box) + } + } + }) }Also applies to: 50-74, 78-90, 94-110, 112-121
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/run/ui/parked_detail_test.go` around lines 23 - 48, Update the parked detail tests to follow the required t.Run("Should...") convention. In TestParkedJobCarriesFullTriageDetail, rename each subtest from generic labels like "reason" and "last tool call" to descriptive "Should..." names, and wrap the other single-scenario tests (TestSummaryParkedBoxRendersActionableDetail, TestSummaryViewOmitsParkedBoxWithoutParkedJobs, TestParkedDetailLinesSkipEmptyFields, and TestParkedProgressValue) in t.Run calls as well. Keep the existing assertions and behavior in parked_detail_test.go unchanged, only adjust the test case structure and names.Source: Path instructions
internal/core/run/internal/acpshared/diagnostics_test.go (1)
43-104: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSubtest names should follow the required
Should...pattern.The
TestLogProgressSignalDiagnosticssubtests (e.g."logs non-zero drop and backpressure counters when drops occurred") do not use the mandatedt.Run("Should...")form. Please rename accordingly.As per path instructions: "MUST use t.Run("Should...") pattern for ALL test cases".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/run/internal/acpshared/diagnostics_test.go` around lines 43 - 104, The subtest names in TestLogProgressSignalDiagnostics do not follow the required Should... convention. Rename each t.Run case in logProgressSignalDiagnostics-related tests to start with "Should..." so they comply with the test naming rule, and update the descriptive strings for the scenarios covering drops, clean runs, non-journal submitters, and nil sessions.Source: Path instructions
internal/core/agent/session_test.go (1)
193-245: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest-case names should follow the required
Should...pattern.The new subtests use descriptive names (e.g.
"critical completed update is delivered under saturated buffer","tool call started is critical") rather than the mandatedt.Run("Should...")form. Please rename to theShould...convention.As per path instructions: "MUST use t.Run("Should...") pattern for ALL test cases".
Also applies to: 356-409
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/agent/session_test.go` around lines 193 - 245, The new subtests in session_test.go do not follow the required Should... naming convention. Rename each t.Run case in the affected tests, including the ones around the critical publish/backpressure scenarios and the related cases later in the file, so every test name starts with Should... and remains descriptive. Use the existing subtest blocks and identifiers like t.Run, session.publish, and the critical/non-critical update scenarios to update the names consistently across all affected test cases.Source: Path instructions
internal/core/run/internal/acpshared/watchdog_test.go (1)
122-161: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPrefix table-driven subtest names with "Should" to match the required pattern.
The subtest names in
TestWatchdogIdleTimeoutArmsIndependentlyOfTimeoutare descriptive but don't follow thet.Run("Should...")pattern required for all test cases. Renaming them is straightforward and improves test output consistency.♻️ Suggested subtest name updates
{ - name: "armed with Timeout=0 when stall enabled", + name: "Should arm with Timeout=0 when stall is enabled", cfg: &config{Timeout: 0, Stall: model.StallPolicy{Enabled: true, IdleTimeout: 3 * time.Minute}}, want: 3 * time.Minute, }, { - name: "disabled stall never arms even with a positive Timeout", + name: "Should not arm when stall is disabled even with a positive Timeout", cfg: &config{ Timeout: 10 * time.Minute, Stall: model.StallPolicy{Enabled: false, IdleTimeout: 3 * time.Minute}, }, want: 0, }, { - name: "zero idle window never arms", + name: "Should not arm when idle window is zero", cfg: &config{Stall: model.StallPolicy{Enabled: true, IdleTimeout: 0}}, want: 0, }, { - name: "nil config never arms", + name: "Should not arm when config is nil", cfg: nil, want: 0, },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/run/internal/acpshared/watchdog_test.go` around lines 122 - 161, The table-driven subtests in TestWatchdogIdleTimeoutArmsIndependentlyOfTimeout do not follow the required naming pattern; update each t.Run case name to start with “Should” while keeping the same scenarios. Make this change in the subtest name strings inside the loop so the test output is consistent with the expected convention.Source: Path instructions
internal/core/agent/terminal_lifecycle_test.go (1)
144-179: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winWrap standalone test cases in
t.Run("Should...")subtests.
TestResolveTerminalCapFallsBackToStallDefault,TestTerminalBaseContextPrefersSessionRunContext, andTestClientRequestPermissionReturnsWithinDeadlineexercise meaningful business logic but skip thet.Run("Should...")pattern required for all test cases. Wrapping them improves test reporting and complies with the path instruction.♻️ Suggested refactor for t.Run pattern compliance
func TestResolveTerminalCapFallsBackToStallDefault(t *testing.T) { - t.Parallel() - - client := &clientImpl{} - if got := client.resolveTerminalCap(); got != model.DefaultStallTerminalCap { - t.Fatalf("resolveTerminalCap() = %s, want default %s", got, model.DefaultStallTerminalCap) - } - client.terminalCommandTimeout = 5 * time.Minute - if got := client.resolveTerminalCap(); got != 5*time.Minute { - t.Fatalf("resolveTerminalCap() = %s, want 5m", got) - } + t.Parallel() + t.Run("Should fall back to the stall default when terminalCommandTimeout is unset", func(t *testing.T) { + t.Parallel() + client := &clientImpl{} + if got := client.resolveTerminalCap(); got != model.DefaultStallTerminalCap { + t.Fatalf("resolveTerminalCap() = %s, want default %s", got, model.DefaultStallTerminalCap) + } + }) + t.Run("Should return the configured timeout when terminalCommandTimeout is set", func(t *testing.T) { + t.Parallel() + client := &clientImpl{terminalCommandTimeout: 5 * time.Minute} + if got := client.resolveTerminalCap(); got != 5*time.Minute { + t.Fatalf("resolveTerminalCap() = %s, want 5m", got) + } + }) }Also applies to: 286-308
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/agent/terminal_lifecycle_test.go` around lines 144 - 179, Wrap the standalone tests in this file, including TestResolveTerminalCapFallsBackToStallDefault and TestTerminalBaseContextPrefersSessionRunContext, in t.Run("Should...") subtests to match the required test naming pattern. Keep the existing assertions and setup intact, but move each case into a descriptive Should-style subtest so reporting and compliance are consistent; apply the same refactor to the related TestClientRequestPermissionReturnsWithinDeadline case as indicated.Source: Path instructions
internal/core/run/internal/acpshared/session_exec.go (1)
228-278: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider adding a
sync.WaitGroupfor the watchdog goroutine.The
runActivityWatchdoggoroutine is started inStartACPActivityWatchdogand exits viastopChorcfg.Ctx.Done(), but the stop function returns without waiting for the goroutine to actually exit. This means the goroutine may briefly outlive the caller. While the practical risk is low (the goroutine only reads a ticker and calls idempotent operations), the coding guidelines require tracking all goroutines withsync.WaitGroupor equivalent.As per coding guidelines: "No fire-and-forget goroutines; track all goroutines with sync.WaitGroup or equivalent."
♻️ Proposed WaitGroup integration
func StartACPActivityWatchdog(cfg ActivityWatchdogConfig) func() { if cfg.Monitor == nil || cfg.IdleTimeout <= 0 || cfg.Cancel == nil { return func() {} } if cfg.Ctx == nil { cfg.Ctx = context.Background() } if cfg.Clock == nil { cfg.Clock = runshared.RealClock{} } if cfg.Logger == nil { cfg.Logger = silentLogger() } stopCh := make(chan struct{}) var stopOnce sync.Once + var wg sync.WaitGroup + wg.Add(1) - go runActivityWatchdog(cfg, watchdogInterval(cfg.IdleTimeout), stopCh) + go func() { + defer wg.Done() + runActivityWatchdog(cfg, watchdogInterval(cfg.IdleTimeout), stopCh) + }() return func() { stopOnce.Do(func() { close(stopCh) }) + wg.Wait() } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/core/run/internal/acpshared/session_exec.go` around lines 228 - 278, StartACPActivityWatchdog launches runActivityWatchdog as a fire-and-forget goroutine, but the returned stop function only closes stopCh and does not wait for shutdown. Add a sync.WaitGroup (or equivalent) in StartACPActivityWatchdog to track the watchdog goroutine, have the goroutine signal completion when runActivityWatchdog exits, and make the stop closure wait for that completion before returning. Keep the change localized around StartACPActivityWatchdog, runActivityWatchdog, and the stopCh/stopOnce flow.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/cli/run_observe_parked_test.go`:
- Around line 22-40: The test cases in renderObservedRunEvent should follow the
required t.Run("Should...") pattern instead of being standalone top-level tests.
Update TestRenderObservedJobParkedCarriesFullTriageDetail and the other related
cases in run_observe_parked_test.go to wrap each scenario in a descriptive
t.Run("Should...") subtest, matching the style used by run_observe_stall_test.go
and the path instructions.
In `@internal/daemon/task_multi_summary_test.go`:
- Around line 30-77: The `TestClassifyChildOutcome` table-driven subtest names
in the `classifyChildOutcome` coverage do not follow the required `Should...`
convention. Update each `t.Run` case name in `task_multi_summary_test.go` to
start with `Should...` and keep the descriptions consistent with the existing
`TestSummarizeChildOutcomesOverRecordedBatch` style so all test cases use the
same naming pattern.
In `@pkg/compozy/events/kinds/job_test.go`:
- Around line 58-65: The empty-field marshaling behavior for JobParkedPayload is
wrong because LogPath still serializes as an empty string; update the struct
definition in JobParkedPayload (the field tagged as LogPath in job.go) to add
omitempty so mustMarshalMap and the “Should omit empty optional fields” test in
job_test.go produce only the index field when LogPath is unset.
---
Nitpick comments:
In `@internal/core/agent/session_test.go`:
- Around line 193-245: The new subtests in session_test.go do not follow the
required Should... naming convention. Rename each t.Run case in the affected
tests, including the ones around the critical publish/backpressure scenarios and
the related cases later in the file, so every test name starts with Should...
and remains descriptive. Use the existing subtest blocks and identifiers like
t.Run, session.publish, and the critical/non-critical update scenarios to update
the names consistently across all affected test cases.
In `@internal/core/agent/terminal_lifecycle_test.go`:
- Around line 144-179: Wrap the standalone tests in this file, including
TestResolveTerminalCapFallsBackToStallDefault and
TestTerminalBaseContextPrefersSessionRunContext, in t.Run("Should...") subtests
to match the required test naming pattern. Keep the existing assertions and
setup intact, but move each case into a descriptive Should-style subtest so
reporting and compliance are consistent; apply the same refactor to the related
TestClientRequestPermissionReturnsWithinDeadline case as indicated.
In `@internal/core/run/executor/parked_alert_test.go`:
- Around line 38-110: Update the parked alert tests to follow the repo’s
required t.Run("Should...") convention. In
TestSoundConfigForGatesOnTheSoundFeatureFlag, rename each table case to a
Should... subtest name, and in TestNotifyParkedAlertIsANoOpWhenSoundIsDisabled
and TestNotifyParkedAlertOnNilExecutionContextDoesNotPanic wrap the assertions
in a single t.Run("Should...") subtest. Keep the existing behavior and
assertions intact while making the names consistent with the test pattern used
around soundConfigFor and notifyParkedAlert.
In `@internal/core/run/executor/runner_stall_test.go`:
- Line 213: The new runner stall tests are asserting at the top level instead of
following the required Should-prefixed subtest convention. Update the affected
test functions in runner_stall_test.go, including
TestJobRunnerStallRetryIsIndependentOfMaxRetries and the related stall retry
cases, to wrap their assertions inside t.Run("Should...", ...) subtests so they
match the existing pattern used by the other tests and the path requirement.
In `@internal/core/run/internal/acpshared/diagnostics_test.go`:
- Around line 43-104: The subtest names in TestLogProgressSignalDiagnostics do
not follow the required Should... convention. Rename each t.Run case in
logProgressSignalDiagnostics-related tests to start with "Should..." so they
comply with the test naming rule, and update the descriptive strings for the
scenarios covering drops, clean runs, non-journal submitters, and nil sessions.
In `@internal/core/run/internal/acpshared/session_exec.go`:
- Around line 228-278: StartACPActivityWatchdog launches runActivityWatchdog as
a fire-and-forget goroutine, but the returned stop function only closes stopCh
and does not wait for shutdown. Add a sync.WaitGroup (or equivalent) in
StartACPActivityWatchdog to track the watchdog goroutine, have the goroutine
signal completion when runActivityWatchdog exits, and make the stop closure wait
for that completion before returning. Keep the change localized around
StartACPActivityWatchdog, runActivityWatchdog, and the stopCh/stopOnce flow.
In `@internal/core/run/internal/acpshared/watchdog_test.go`:
- Around line 122-161: The table-driven subtests in
TestWatchdogIdleTimeoutArmsIndependentlyOfTimeout do not follow the required
naming pattern; update each t.Run case name to start with “Should” while keeping
the same scenarios. Make this change in the subtest name strings inside the loop
so the test output is consistent with the expected convention.
In `@internal/core/run/ui/parked_detail_test.go`:
- Around line 23-48: Update the parked detail tests to follow the required
t.Run("Should...") convention. In TestParkedJobCarriesFullTriageDetail, rename
each subtest from generic labels like "reason" and "last tool call" to
descriptive "Should..." names, and wrap the other single-scenario tests
(TestSummaryParkedBoxRendersActionableDetail,
TestSummaryViewOmitsParkedBoxWithoutParkedJobs,
TestParkedDetailLinesSkipEmptyFields, and TestParkedProgressValue) in t.Run
calls as well. Keep the existing assertions and behavior in
parked_detail_test.go unchanged, only adjust the test case structure and names.
In `@internal/core/run/ui/stall_states_test.go`:
- Around line 127-163: Wrap the single-scenario tests in this file, especially
TestParkedJobRendersTerminalParkedState and
TestZeroStallRunLeavesPerJobLayoutUnchanged, in t.Run("Should...") subtests to
match the required test pattern used by the other stall state tests. Keep the
existing assertions and setup inside the new subtest blocks, and ensure the
top-level test functions only orchestrate the subtest structure.
In `@internal/core/workspace/config_test.go`:
- Around line 487-561: Wrap each standalone stall test in a t.Run subtest using
the required Should naming pattern. Update TestLoadConfigParsesStallSection and
TestLoadConfigMergesStallWorkspaceOverGlobalConfig to place their existing
assertions inside t.Run("Should...", ...) blocks, preserving the current test
logic and helper usage like isolateWorkspaceConfigHome, writeWorkspaceConfig,
writeGlobalConfig, and LoadConfig.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: c13d52e1-59bd-4eae-80ea-927024db77e3
⛔ Files ignored due to path filters (1)
docs/events.mdis excluded by!**/*.md
📒 Files selected for processing (80)
internal/cli/reviews_exec_daemon.gointernal/cli/run_observe.gointernal/cli/run_observe_parked_test.gointernal/cli/run_observe_stall_test.gointernal/cli/state.gointernal/cli/workspace_config.gointernal/core/agent/client.gointernal/core/agent/client_test.gointernal/core/agent/session.gointernal/core/agent/session_test.gointernal/core/agent/terminal.gointernal/core/agent/terminal_lifecycle_test.gointernal/core/api.gointernal/core/model/constants.gointernal/core/model/model_test.gointernal/core/model/runtime_config.gointernal/core/model/task_runtime.gointernal/core/run/exec/aliases.gointernal/core/run/executor/aliases.gointernal/core/run/executor/event_stream.gointernal/core/run/executor/execution.gointernal/core/run/executor/execution_acp_test.gointernal/core/run/executor/execution_test.gointernal/core/run/executor/hooks.gointernal/core/run/executor/lifecycle.gointernal/core/run/executor/parked_alert_test.gointernal/core/run/executor/result.gointernal/core/run/executor/run_recovery_counts_test.gointernal/core/run/executor/runner.gointernal/core/run/executor/runner_stall_test.gointernal/core/run/internal/acpshared/aliases.gointernal/core/run/internal/acpshared/command_io.gointernal/core/run/internal/acpshared/diagnostics.gointernal/core/run/internal/acpshared/diagnostics_test.gointernal/core/run/internal/acpshared/session_exec.gointernal/core/run/internal/acpshared/session_handler.gointernal/core/run/internal/acpshared/stall_result_test.gointernal/core/run/internal/acpshared/watchdog_test.gointernal/core/run/internal/runshared/buffers.gointernal/core/run/internal/runshared/buffers_test.gointernal/core/run/internal/runshared/clock.gointernal/core/run/internal/runshared/config.gointernal/core/run/internal/runshared/types.gointernal/core/run/journal/journal.gointernal/core/run/journal/journal_test.gointernal/core/run/journal/terminal_event_test.gointernal/core/run/recovery/outcome.gointernal/core/run/recovery/parked_status_test.gointernal/core/run/ui/model.gointernal/core/run/ui/parked_detail_test.gointernal/core/run/ui/sidebar.gointernal/core/run/ui/stall_states_test.gointernal/core/run/ui/summary.gointernal/core/run/ui/timeline.gointernal/core/run/ui/types.gointernal/core/run/ui/update.gointernal/core/run/ui/view.gointernal/core/sound/subscriber.gointernal/core/sound/subscriber_test.gointernal/core/workspace/config_merge.gointernal/core/workspace/config_test.gointernal/core/workspace/config_types.gointernal/core/workspace/config_validate.gointernal/core/worktree/reset.gointernal/core/worktree/reset_test.gointernal/daemon/child_backstop.gointernal/daemon/child_backstop_test.gointernal/daemon/run_manager.gointernal/daemon/run_manager_test.gointernal/daemon/stall_status_test.gointernal/daemon/task_multi.gointernal/daemon/task_multi_summary.gointernal/daemon/task_multi_summary_test.gointernal/daemon/task_multi_test.gopkg/compozy/events/docs_test.gopkg/compozy/events/event.gopkg/compozy/events/kinds/job.gopkg/compozy/events/kinds/job_test.gopkg/compozy/events/kinds/task.gosdk/extension/types.go
Integrates upstream compozy#231 (parallel-tasks worktree lifecycle) and the v0.2.12 release with the stall/recovery work. Key conflict resolutions: - waitForTaskMultiChild: keep upstream's event-driven wait on the child done channel and layer the durable journal-sequence backstop (ADR-003) on top as a periodic safety net that re-reads run status, so a child wedged hard enough to never close its done channel still settles the join. - journal durability: adopt upstream's isDurabilityCriticalEvent / RequiresDurablePublish and OR in EventKindJobParked so parked triage detail still forces an immediate flush and fsync. - events: keep both new kinds (task.multi.summary and task.multi.queue_failed) and both payload field sets (completed/recovered/parked plus worktree_reason/result_branch). - finalizeTaskMultiParallel: adopt upstream's structure where the queue terminal is emitted once by emitTaskMultiQueueSettlement; this also resolves the review finding that a best-effort recovery-summary write failure must not fail an otherwise-successful batch.
pedronauck
left a comment
There was a problem hiding this comment.
Validated at head 2cb26b3524ee505ea97bb68d040f7c9ea8c24d4c with the full make verify gate (exit 0), but there is a blocking runtime-override contract bug.
runtimeOverrideInput.Stall is decoded from JSON with DisallowUnknownFields, while workspace.StallOverrides only defines toml tags. As a result, the API-style payload {"stall":{"enabled":true,"timeout":"200ms","child_timeout":"500ms","terminal_command_timeout":"1s","retries":1}} fails with json: unknown field "child_timeout" instead of applying the policy. The integration helper in internal/daemon/child_backstop_test.go currently hides this by sending Go/PascalCase keys (Enabled, Timeout, ChildTimeout).
Please add the corresponding json tags to every StallOverrides field, update the integration payload to use the public snake_case shape, and add a parser-level regression proving all stall fields decode and reach RuntimeConfig through parseRuntimeOverrides/applyRuntimeOverrideInput. Then rerun make verify.
|
@pedronauck Done! Can you review again? |
pedronauck
left a comment
There was a problem hiding this comment.
Approved at head e695fdc. The snake_case stall override contract is now covered through the public parser/application path, all active review threads are resolved, the focused worktree cleanup regression passed 5/5 with the race detector, and CI run 29432190781 executed the exact current merge ref 29f6303 with make verify fully green: lint reported 0 issues, 4,769 tests passed, build completed, and Playwright E2E passed.
Follow-up hardening (2026-07-22)PR #242, in commit
The tradeoff is explicit: shorter idle windows and more retries can increase model-token usage. Concurrent review isolation also requires a committed Git baseline and rejects source changes outside the workflow artifact scope; conflicting or parked worktrees are preserved for triage. |
Reverts six PRs that rode along on this branch but are independent of the Task Groups feature (verified: no task-group code depends on them): - compozy#234 cy-qa-workflow module + host.tasks.create v2 (extension SDK) - compozy#237 cy-capture-decisions extension - compozy#226 Kiro CLI ACP model handling - compozy#248 sync-test isolation (touched cy-capture-decisions) - compozy#252 release v0.2.15 prep - compozy#253 test-daemon reaping Kept the task-group-relevant infra: compozy#230 (multi-run recovery), compozy#236 (reviews-fix progress), compozy#241 (parallel task status), compozy#250 (task artifacts).
The problem
A multi-run could freeze partway through and never come back. When an agent stopped emitting output, the attempt sat idle until some external timeout, and in a parallel run a wedged child would hold the parent's join open with no way out. Nothing told you a job had stalled or that one needed a look.
What changed
The branch adds stall detection and recovery to the run lifecycle.
A watchdog looks for progress on a dedicated idle window. Any ACP session update resets its clock, so it only fires on an attempt that has genuinely gone quiet. When it fires, the job gets one retry from a clean worktree. If it stalls again, the job parks instead of failing: its worktree and journal are kept so you can triage what happened.
Parallel runs get a second layer. Each child's durable journal high-water sequence is tracked, and if it stops advancing for the whole budget, the parent reaps the child through the normal cancel path so the join proceeds. The backstop budget sits strictly above the idle window, so the in-attempt watchdog always gets first chance to heal.
Two supporting fixes make recovery reliable. Terminal commands are now cancellable, wall-clock capped, and parented to the attempt, so a stuck command can't orphan a process. Critical ACP session updates are no longer dropped under backpressure, and stall diagnostics are logged when the watchdog trips.
Visibility
There's a new parked run status plus job stall and park events. The run UI shows stalled, retrying, and parked states, and prints an end-of-run recovery summary with completed, recovered, and parked counts. When a job parks, an alert fires with the triage detail: why it parked, the last tool call, and the worktree path.
Config
New stall-detection knobs (idle timeout, child timeout, terminal cap, retries), on by default. An explicit disable turns the whole thing off.
Testing
go buildandgo test -racepass oninternal/core/run/executorandinternal/daemon, and golangci-lint is clean on both. The last commit folds in three review fixes: the stall-retry attempt budget (it was reporting an impossible3/2when a stall retry preceded ordinary retries), a duplicate alert on parked runs, and a child-policy defaulting gap that could silently disarm the backstop.Summary by CodeRabbit
New Features
Bug Fixes
Tests