Repository navigation
fix: enable runtime evidence profiles - #242
Conversation
|
@richardfogaca is attempting to deploy a commit to the Compozy Team on Vercel. A member of the Team first needs to authorize it. |
|
Warning Review limit reached
More reviews will be available in 2 minutes and 9 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (8)
WalkthroughThis PR adds runtime evidence mode to task execution profiles, enabling conditional session permission auto-approval and selective worker targeting. A new ChangesTask Execution Profile Runtime Evidence
🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Introduces a new task execution-profile runtime policy (default vs evidence) that is persisted in global DB, surfaced via OpenAPI/CLI, and used to relax session permissions + add guidance overlays when evidence mode is enabled.
Changes:
- Add
RuntimePolicy/RuntimeModeto task execution profiles, including normalization + validation. - Persist
runtime_modein global DB with migration + store read/write support. - Propagate runtime evidence mode into session creation (permissions + prompt overlays) and update API/types/fixtures/docs accordingly.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| web/src/systems/tasks/mocks/fixtures.ts | Adds runtime to task execution profile fixtures and updates override precedence. |
| web/src/generated/agh-openapi.d.ts | Updates generated TS types to include runtime. |
| packages/site/content/runtime/core/configuration/config-toml.mdx | Adjusts coordinator configuration table formatting. |
| openapi/agh.json | Adds runtime object to execution profile schema and marks it required. |
| internal/tools/builtin/testdata/native-tool-catalog.json | Updates schema digest to reflect profile schema changes. |
| internal/tools/builtin/tasks.go | Extends execution profile JSON schema to include runtime. |
| internal/task/profile.go | Adds runtime mode/policy types, normalization, and validation. |
| internal/task/manager_profile.go | Sets default runtime mode in default execution profile. |
| internal/store/globaldb/schema_task_orchestration_profile.go | Adds runtime_mode column to schema. |
| internal/store/globaldb/migrate_task_orchestration_profile.go | Adds migration to backfill/add runtime_mode column. |
| internal/store/globaldb/global_db_test.go | Registers new migration in expected migration list. |
| internal/store/globaldb/global_db_task_profile_test.go | Extends store tests to cover runtime mode persistence. |
| internal/store/globaldb/global_db_task_profile.go | Reads/writes runtime_mode in execution profile store. |
| internal/store/globaldb/global_db_task_orchestration_schema_test.go | Includes runtime_mode in schema assertions. |
| internal/store/globaldb/global_db.go | Adds migration v41 for runtime_mode. |
| internal/session/manager_start.go | Allows session start to override permissions via CreateOpts. |
| internal/session/manager.go | Adds Permissions to session CreateOpts. |
| internal/daemon/task_runtime.go | Applies runtime evidence mode to session opts (permissions + overlay). |
| internal/daemon/task_role_runtime_test.go | Adds coverage for profile-selected worker + runtime evidence behavior. |
| internal/daemon/task_role_runtime.go | Loads execution profiles to decide worker target; adds capability-aware claim command text. |
| internal/cli/task.go | Displays runtime mode in CLI execution profile output. |
Comments suppressed due to low confidence (1)
openapi/agh.json:1
- The OpenAPI schema allows any string for
runtime.mode, but server-side validation only acceptsdefaultorevidence(and treats empty as default after normalization). This mismatch will generate overly-permissive TS types (mode: string) and can lead clients to send values that pass schema validation but are rejected by the API; consider adding anenum(and adefault) forruntime.modematching server behavior. Also consider whetherruntime(and/ormode) should truly be required in the OpenAPI contract if the server can safely default it when omitted.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
b3d14e1 to
b3df09a
Compare
pedronauck
left a comment
There was a problem hiding this comment.
Automated review — self + Codex (gpt-5.5, xhigh)
Two reviews were attempted on PR #242 (fix: enable runtime evidence profiles): an AGH self-review and an external Codex peer review (gpt-5.5, reasoning xhigh).
- Self verdict: FIX_BEFORE_SHIP (1 blocker, 3 risks, 2 nits)
- Codex verdict: unavailable — the Codex run technically completed but ignored the scoped-write contract and produced no findings artifact, so it returned no usable verdict. The findings below are from the self-review only.
The core design is solid: runtime.mode is opt-in, validated, persisted via an appended numbered migration (v41, prior checksums unchanged), contract regen co-ships (OpenAPI + web agh-openapi.d.ts + native-tool-catalog digest + fixtures), and unit coverage is good. go build ./internal/... is clean.
Blockers
- Dropping
--waitfrom the worker claim prompt can strand sessions.internal/daemon/task_role_runtime.gotaskRoleClaimCommand(~L558) now emitsagh task next -o json(was--wait -o json), andtaskRolePromptOverlaytells the agent to run it "once". Without--wait, the command does a single non-blocking claim and returns empty if nothing is immediately claimable. A freshly spawned profile-selected worker can reach the prompt before its run is claimable (boot/lease/visibility timing) and never claim its assigned run. Fix: keep--waitas the base arg and append--capabilityflags after it.
Risks
- Modifying the body of already-shipped migration v38.
internal/store/globaldb/migrate_task_orchestration_profile.go: the v38 migration body now also addsruntime_mode. Harmless in practice (idempotent + v41 covers it) but violates append-only migration discipline. Fix: revert the v38 body change and rely solely on v41. - Evidence mode grants
approve-allwithout requiring sandbox isolation.internal/daemon/task_runtime.goapplyTaskSessionRuntimeProfile(L222-235) setsPermissions=approve-allwhenever mode isevidence, independent of sandbox isolation — full auto-approve on the operator host. Fix: gate evidence-mode approve-all behind an isolated sandbox, or make the trust boundary explicit with a warning. - No docs for the new
runtime.modefield. The onlypackages/sitechange (config-toml.mdx) is whitespace cleanup. Fix: add execution-profile docs forruntime.mode(+sandbox.mode), or file a tracked follow-up.
Nits
- Implicit worker-target precedence in
workerTargetForRun(AgentName→PreferredAgentNames[0]→AllowedAgentNames) — add a one-line comment documenting the order. - Redundant idempotent column add across v38 and v41 — disappears once v38 is reverted to its original body.
Note: the external Codex round will be re-run in a healthy environment to add its independent verdict.
b3df09a to
eb32cba
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
internal/session/manager_start.go (1)
614-627:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy liftPersist the permission override across resume paths.
This override only survives the initial create.
prepareCreateStartfillssessionStartSpec.permissions, but resumed sessions still reachstartSpecPermissions(..., resolved.Permissions)with an empty override, so evidence-mode sessions come back with agent-default permissions after restart/resume. Persist the effective permission mode in session metadata and hydrate it inprepareResumeStart.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/session/manager_start.go` around lines 614 - 627, The override in sessionStartSpec.permissions isn't persisted across resumes; update the create path (prepareCreateStart) to store the effective permission mode (the result of startSpecPermissions or resolved.Permissions) into the session's metadata (e.g., a well-known key like "effective_permissions") when the session is first created, and update prepareResumeStart to hydrate sessionStartSpec.permissions from that metadata so startSpecPermissions(s, resolved.Permissions) will return the persisted override on resume; touch the functions startSpecPermissions, prepareCreateStart, and prepareResumeStart to read/write that metadata and ensure resume and new-session flows use the same effective permission value.
🧹 Nitpick comments (3)
internal/store/globaldb/global_db_task_orchestration_schema_test.go (1)
80-98: ⚡ Quick winCover upgraded legacy profile rows, not just the DDL text.
This only proves the statements mention
runtime_mode. It doesn’t prove that an existingtask_execution_profilesrow from the pre-runtime_modeschema upgrades and reopens withruntime_mode = "default", which is the behavior this PR relies on.A small observed-history case here would catch migration drift much earlier: seed a legacy profile row, run
OpenGlobalDB, reopen once more, and assert the migrated row loads withRuntimeModeDefault. As per coding guidelines, "Migration drift fixes require observed-history tests. Cover fresh DB, upgrade/reopen, and the real recorded migration prefix that failed."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/store/globaldb/global_db_task_orchestration_schema_test.go` around lines 80 - 98, Add an observed-history test that seeds a legacy task_execution_profiles row (one that lacks the taskExecutionProfileRuntimeModeColumn), then calls OpenGlobalDB to run migrations, reopens the DB, and asserts the migrated row now has RuntimeModeDefault; specifically, in the same test file augment the existing case by inserting a pre-migration profile row into the DB (using the legacy schema), call OpenGlobalDB (the function that applies the migrations), reopen the database connection, query the task_execution_profiles row and assert its RuntimeMode equals RuntimeModeDefault, and keep using the existing helpers like taskOrchestrationProfileMigrationSchemaStatements and schemaStatementsContainColumn to locate the migration context.internal/store/globaldb/global_db_task_profile_test.go (1)
90-98: ⚡ Quick winAssert that partial updates reset
runtime.modeback todefault.This update intentionally omits
Runtime, but the test never checks that the previously storedevidencemode is cleared. A regression that accidentally preserves opt-in evidence mode would still pass here.Suggested assertion
if !updated.UpdatedAt.Equal(updatedAt) { t.Fatalf("updated.UpdatedAt = %v, want %v", updated.UpdatedAt, updatedAt) } + if got, want := updated.Runtime.Mode, taskpkg.RuntimeModeDefault; got != want { + t.Fatalf("updated.Runtime.Mode = %q, want %q", got, want) + } assertStringSliceGlobal(t, updated.Worker.AllowedAgentNames, []string{"runner"}) assertStringSliceGlobal(t, updated.Review.AllowedAgentNames, nil)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/store/globaldb/global_db_task_profile_test.go` around lines 90 - 98, The test updates an ExecutionProfile omitting Runtime but never asserts that the previously set runtime.mode (and any opt-in evidence mode) was cleared; after calling UpsertExecutionProfile (the updated variable), add an assertion that updated.Runtime.Mode equals the default runtime mode (e.g., taskpkg.RuntimeModeDefault or the zero value) and that any Runtime.Evidence/opt-in evidence field is nil or reset, so partial updates do not preserve an opt-in evidence mode.internal/daemon/task_role_runtime_test.go (1)
24-42: ⚡ Quick winUse
Should ...subtest names in the table.The new cases use
empty,simple, and similar labels, which breaks the repo’s required test naming pattern. Rename them toShould quote empty input,Should quote metacharacters, and so on.As per coding guidelines, "
**/*_test.go: MUST use t.Run("Should...") pattern for ALL test cases".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/daemon/task_role_runtime_test.go` around lines 24 - 42, Rename the test case names in the tests table to follow the "Should ..." pattern so subtests use descriptive names; update each entry in the tests slice (the struct with name, value, want) to e.g. "Should quote empty input", "Should quote simple input", "Should quote metacharacters", "Should quote single-quote input" (keeping the same value and want expectations) so that t.Run(tt.name, ...) still exercises shellQuoteSimple correctly and matches the repository test naming guideline.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/daemon/task_role_runtime.go`:
- Around line 341-352: The active role session reuse logic is ignoring
profile-specific startup settings; modify the reuse key logic around
activeRoleSession to include the execution profile (or disable reuse when the
incoming activation.Profile differs) so provider/model selection and
sandbox/evidence-mode setup from applyTaskSessionSandboxProfile and
applyTaskSessionRuntimeProfile are not skipped; update any functions that
compute the session identity (e.g., taskRoleSessionName or the lookup that finds
activeRoleSession) to incorporate activation.Profile (or a stable fingerprint of
it) so taskRoleCreateOpts is only bypassed for truly matching-profile sessions.
---
Outside diff comments:
In `@internal/session/manager_start.go`:
- Around line 614-627: The override in sessionStartSpec.permissions isn't
persisted across resumes; update the create path (prepareCreateStart) to store
the effective permission mode (the result of startSpecPermissions or
resolved.Permissions) into the session's metadata (e.g., a well-known key like
"effective_permissions") when the session is first created, and update
prepareResumeStart to hydrate sessionStartSpec.permissions from that metadata so
startSpecPermissions(s, resolved.Permissions) will return the persisted override
on resume; touch the functions startSpecPermissions, prepareCreateStart, and
prepareResumeStart to read/write that metadata and ensure resume and new-session
flows use the same effective permission value.
---
Nitpick comments:
In `@internal/daemon/task_role_runtime_test.go`:
- Around line 24-42: Rename the test case names in the tests table to follow the
"Should ..." pattern so subtests use descriptive names; update each entry in the
tests slice (the struct with name, value, want) to e.g. "Should quote empty
input", "Should quote simple input", "Should quote metacharacters", "Should
quote single-quote input" (keeping the same value and want expectations) so that
t.Run(tt.name, ...) still exercises shellQuoteSimple correctly and matches the
repository test naming guideline.
In `@internal/store/globaldb/global_db_task_orchestration_schema_test.go`:
- Around line 80-98: Add an observed-history test that seeds a legacy
task_execution_profiles row (one that lacks the
taskExecutionProfileRuntimeModeColumn), then calls OpenGlobalDB to run
migrations, reopens the DB, and asserts the migrated row now has
RuntimeModeDefault; specifically, in the same test file augment the existing
case by inserting a pre-migration profile row into the DB (using the legacy
schema), call OpenGlobalDB (the function that applies the migrations), reopen
the database connection, query the task_execution_profiles row and assert its
RuntimeMode equals RuntimeModeDefault, and keep using the existing helpers like
taskOrchestrationProfileMigrationSchemaStatements and
schemaStatementsContainColumn to locate the migration context.
In `@internal/store/globaldb/global_db_task_profile_test.go`:
- Around line 90-98: The test updates an ExecutionProfile omitting Runtime but
never asserts that the previously set runtime.mode (and any opt-in evidence
mode) was cleared; after calling UpsertExecutionProfile (the updated variable),
add an assertion that updated.Runtime.Mode equals the default runtime mode
(e.g., taskpkg.RuntimeModeDefault or the zero value) and that any
Runtime.Evidence/opt-in evidence field is nil or reset, so partial updates do
not preserve an opt-in evidence mode.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 63bafd32-f305-4e26-83a2-6e06f241fb40
⛔ Files ignored due to path filters (5)
internal/tools/builtin/testdata/native-tool-catalog.jsonis excluded by!**/*.jsonopenapi/agh.jsonis excluded by!**/*.jsonpackages/site/content/runtime/core/autonomy/execution-profiles.mdxis excluded by!**/*.mdxpackages/site/content/runtime/core/configuration/config-toml.mdxis excluded by!**/*.mdxweb/src/generated/agh-openapi.d.tsis excluded by!**/generated/**
📒 Files selected for processing (20)
internal/api/spec/spec.gointernal/api/spec/spec_test.gointernal/cli/task.gointernal/daemon/task_role_runtime.gointernal/daemon/task_role_runtime_test.gointernal/daemon/task_runtime.gointernal/daemon/task_runtime_test.gointernal/session/manager.gointernal/session/manager_start.gointernal/store/globaldb/global_db.gointernal/store/globaldb/global_db_task_orchestration_schema_test.gointernal/store/globaldb/global_db_task_profile.gointernal/store/globaldb/global_db_task_profile_test.gointernal/store/globaldb/global_db_test.gointernal/store/globaldb/migrate_task_orchestration_profile.gointernal/store/globaldb/schema_task_orchestration_profile.gointernal/task/manager_profile.gointernal/task/profile.gointernal/tools/builtin/tasks.goweb/src/systems/tasks/mocks/fixtures.ts
What this changes
This PR introduces an explicit runtime evidence mode for task execution profiles. Profiles can now ask workers to run with broader local runtime permissions and prompt guidance when the task needs browser/simulator/app evidence.
Why
AGH already lets agents execute code, but frontend/mobile work needs a deliberate mode for installing dependencies, booting apps, driving browsers or simulators, and collecting evidence. Keeping this behind an execution-profile policy gives us that capability without changing the default conservative task path.
Implementation notes
runtime.modeto task execution profiles withdefaultandevidencemodes.runtime_modein GlobalDB with migration and schema coverage.Reviewer focus
Validation
PATH="$HOME/.bun/bin:$PATH" make verifySummary by CodeRabbit
Release Notes
New Features
Improvements
Database
runtime_modecolumn to task execution profiles table.