Repository navigation
fix(surveyor): require selection evidence before survey completion - #193
Conversation
The baseline failed the new operative-selection contract check; the fix passes all seven independently removed-obligation checks, the existing surveyor tests, ShellCheck, manifest and digest validation, and current-head hosted CI. An independent correctness and simplification review found no actionable issues. I also exercised the definition with a separate agent against six synthetic evidence scenarios: complete censuses without ranking, a failed older-candidate control read, a live claim before a ready candidate, an evidenced-empty queue, an expired claim with an unsupported label, and an elapsed measurement date. All produced the expected selection and freshness decisions while retaining Operate evidence. The reusable fixture records those cases separately from the automated definition-drift checks. Consumer deployment still requires its normal reviewed plugin rollout. @coderabbitai full review |
The correctly shaped CodeRabbit request at 09:34:55 UTC has no reaction, acknowledgement, review, or changed head status after the five-minute no-response window. The current head still only reports that automatic review is disabled. Advancing to the next configured reviewer; CI remains green. @codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
|
|
Warning Review limit reachedNext included review available in 23 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: 📒 Files selected for processing (8)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7455dd0360
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
All three findings are fixed, replied to, and resolved at 5f0e5db, which includes current main 495d152. Consumers retain oldest-actionable-first when no order is declared; nominated candidates need their own current actionability evidence; and the self-test runs directly. The new requirements failed before the fixes. All nine independent obligation-removal checks now pass, as do the existing surveyor test, ShellCheck, manifests, digest validation, and current-head hosted CI. An independent agent evaluated all eight evidence scenarios, including both newly reported gaps, with the expected results. Independent correctness review found no remaining issue. @coderabbitai full review |
The current-head CodeRabbit request from 10:00:23 UTC has no reaction, acknowledgement, review, or service response after more than eight minutes. Its response window has expired; advancing to Codex. All CI checks pass and all three prior findings are resolved. @codex review |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '== Current HEAD and working tree =='
git rev-parse HEAD
git status --short
printf '%s\n' '== PR merge base and changed files =='
base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main)"
printf 'base=%s\n' "$base"
git diff --stat "$base"...HEAD
git diff --name-status "$base"...HEAD
printf '%s\n' '== Changed-file outlines =='
git diff --name-only "$base"...HEAD | while IFS= read -r file; do
printf '\n--- %s ---\n' "$file"
ast-grep outline "$file" --items all 2>/dev/null || sed -n '1,80p' "$file"
done
printf '%s\n' '== Diff =='
git diff --find-renames --find-copies "$base"...HEADLength of output: 21977 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '== Surveyor definition map and focused sections =='
ast-grep outline plugins/agentic-engineering/agents/portfolio-surveyor.agent.md --items all 2>/dev/null || true
nl -ba plugins/agentic-engineering/agents/portfolio-surveyor.agent.md | sed -n '90,160p;470,720p'
printf '%s\n' '== References to surveyor selection and validation =='
rg -n -i -C 3 \
'(portfolio-surveyor|surveyor-selection|selection evidence|ready_work|last_full_survey|freshness.cursor|freshness cursor|oldest-actionable)' \
plugins .github README.md 2>/dev/null || true
printf '%s\n' '== Shell test inventory and invocation references =='
fd -t f -e sh . plugins/agentic-engineering | sort
rg -n -C 3 '(shellcheck|digest|contract\.test|\.test\.sh|scripts/)' \
.github plugins README.md 2>/dev/null || true
printf '%s\n' '== Changed JSON validation inputs =='
for f in \
.claude-plugin/marketplace.json \
.github/plugin/marketplace.json \
plugins/agentic-engineering/.claude-plugin/plugin.json \
plugins/agentic-engineering/plugin.json \
plugins/agentic-engineering/resources/provider-neutral.desired-state.json \
plugins/agentic-engineering/scripts/fixtures/surveyor-selection.json; do
printf '\n--- %s ---\n' "$f"
jq . "$f" >/dev/null
nl -ba "$f" | sed -n '1,120p'
doneLength of output: 1784 🏁 Script executed: #!/bin/bash
set -euo pipefail
numbered_slice() {
local file=$1 ranges=$2
awk -v ranges="$ranges" '
BEGIN {
n=split(ranges, a, ";")
for (i=1; i<=n; i++) {
split(a[i], b, "-")
start[i]=b[1]; end[i]=b[2]
}
}
{
for (i=1; i<=n; i++) {
if (FNR >= start[i] && FNR <= end[i]) {
printf "%6d %s\n", FNR, $0
break
}
}
}' "$file"
}
printf '%s\n' '== Surveyor definition sections =='
numbered_slice plugins/agentic-engineering/agents/portfolio-surveyor.agent.md '90-160;470-720'
printf '%s\n' '== References to surveyor selection and validation =='
rg -n -i -C 3 \
'(portfolio-surveyor|surveyor-selection|selection evidence|ready_work|last_full_survey|freshness.cursor|freshness cursor|oldest-actionable)' \
plugins .github README.md 2>/dev/null || true
printf '%s\n' '== Shell test inventory =='
fd -t f -e sh . plugins/agentic-engineering | sort
printf '%s\n' '== Shell test and validation invocations =='
rg -n -C 3 '(shellcheck|digest|contract\.test|\.test\.sh|scripts/)' \
.github plugins README.md 2>/dev/null || true
printf '%s\n' '== Changed JSON parses and contents =='
for f in \
.claude-plugin/marketplace.json \
.github/plugin/marketplace.json \
plugins/agentic-engineering/.claude-plugin/plugin.json \
plugins/agentic-engineering/plugin.json \
plugins/agentic-engineering/resources/provider-neutral.desired-state.json \
plugins/agentic-engineering/scripts/fixtures/surveyor-selection.json; do
printf '\n--- %s ---\n' "$f"
jq . "$f" >/dev/null
awk '{printf "%6d %s\n", FNR, $0}' "$f" | sed -n '1,120p'
doneLength of output: 201991 🏁 Script executed: #!/bin/bash
set -euo pipefail
line_range() {
local file=$1 start=$2 end=$3
awk -v start="$start" -v end="$end" \
'FNR >= start && FNR <= end {printf "%6d %s\n", FNR, $0}' "$file"
}
printf '%s\n' '== Agentic engineer: survey digest consumption and selection =='
rg -n -i -C 8 \
'(survey digest|QUERY-UNKNOWN|ready_work|last_full_survey|freshness|oldest-actionable|Advance)' \
plugins/agentic-engineering/agents/agentic-engineer.agent.md \
plugins/agentic-engineering/skills/portfolio-maintenance/SKILL.md
printf '%s\n' '== Portfolio maintenance selection workflow =='
line_range plugins/agentic-engineering/skills/portfolio-maintenance/SKILL.md 175 390
printf '%s\n' '== GitHub workflow files and test invocation =='
fd -t f -e yml -e yaml .github | sort
rg -n -i -C 5 \
'(agentic-engineering|shellcheck|\.test\.sh|scripts/|digest|plugin)' \
.github/workflows .github 2>/dev/null || true
printf '%s\n' '== Static integrity checks =='
for f in \
plugins/agentic-engineering/scripts/surveyor-selection-contract.test.sh \
plugins/agentic-engineering/scripts/fixtures/surveyor-selection.json; do
printf '%s: ' "$f"
if [ -x "$f" ] || [ "${f##*.}" = json ]; then
printf 'present\n'
fi
done
bash -n plugins/agentic-engineering/scripts/surveyor-selection-contract.test.sh
jq empty plugins/agentic-engineering/scripts/fixtures/surveyor-selection.json
actual="$(sha256sum plugins/agentic-engineering/agents/portfolio-surveyor.agent.md | awk '{print $1}')"
declared="$(jq -r '.spec.roles["portfolio-surveyor"].definitionSha256' plugins/agentic-engineering/resources/provider-neutral.desired-state.json)"
printf 'definition_sha actual=%s declared=%s match=%s\n' "$actual" "$declared" "$([ "$actual" = "$declared" ] && printf yes || printf no)"
printf '%s\n' '== Test naming conventions near surveyor tests =='
fd -t f -e sh . plugins/agentic-engineering/scripts | grep -E '(surveyor|contract).*test\.sh$' | sortLength of output: 97419
The selection-evidence rules are consistent with the consumer’s You are interacting with an AI system. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f0e5db83e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Both findings are fixed at current head 0be5454: consumer-bound claim expiry and all-author open-PR association evidence. Validation: 11 independent definition-removal controls; 3 valid association counts and 13 invalid/partial-error responses; 12 blind model-pressure scenarios; 374 guard, 19 adapter, and 25 digest cases. Manifest validation and the existing dependency-query test pass. GitHub CI is green at this head. @coderabbitai full review |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '== HEAD, merge base, and changed files =='
git rev-parse HEAD
base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main)"
printf 'base=%s\n' "$base"
git diff --stat "$base"...HEAD
git diff --name-status "$base"...HEAD
printf '%s\n' '== Changed-file outlines =='
git diff --name-only "$base"...HEAD | while IFS= read -r file; do
printf '\n--- %s ---\n' "$file"
ast-grep outline "$file" --items all 2>/dev/null || true
done
printf '%s\n' '== Full PR diff =='
git diff --find-renames --find-copies --unified=80 "$base"...HEAD
printf '%s\n' '== Current surveyor and consumer selection references =='
rg -n -i -C 5 \
'(selection evidence|selection_evidence|ready_work|empty queue|survey complete|claim expiry|claim.*expire|open.?PR|pull request|association|oldest-actionable|QUERY-UNKNOWN|freshness)' \
plugins/agentic-engineering .github 2>/dev/null || true
printf '%s\n' '== Relevant file slices =='
for file in \
plugins/agentic-engineering/agents/portfolio-surveyor.agent.md \
plugins/agentic-engineering/agents/agentic-engineer.agent.md \
plugins/agentic-engineering/skills/portfolio-maintenance/SKILL.md \
plugins/agentic-engineering/scripts/surveyor-selection-contract.test.sh \
plugins/agentic-engineering/scripts/fixtures/surveyor-selection.json \
plugins/agentic-engineering/resources/provider-neutral.desired-state.json; do
if [ -f "$file" ]; then
printf '\n--- %s ---\n' "$file"
nl -ba "$file" | sed -n '1,900p'
fi
done
printf '%s\n' '== Static structural checks =='
for file in \
plugins/agentic-engineering/scripts/surveyor-selection-contract.test.sh \
plugins/agentic-engineering/scripts/fixtures/surveyor-selection.json \
plugins/agentic-engineering/resources/provider-neutral.desired-state.json; do
if [ -f "$file" ]; then
case "$file" in
*.sh) bash -n "$file" ;;
*.json) jq empty "$file" ;;
esac
printf 'validated %s\n' "$file"
fi
doneLength of output: 202224 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '== Changed surveyor sections =='
sed -n '120,165p;525,755p' \
plugins/agentic-engineering/agents/portfolio-surveyor.agent.md
printf '%s\n' '== Consumer workflow and related selection behavior =='
sed -n '1,380p' plugins/agentic-engineering/skills/portfolio-maintenance/SKILL.md
sed -n '1,150p' plugins/agentic-engineering/agents/agentic-engineer.agent.md
printf '%s\n' '== CI test discovery and shell conventions =='
sed -n '1,280p' .github/workflows/ci.yaml
fd -t f -e sh . plugins/agentic-engineering/scripts | sort
rg -n -C 4 --glob '*.sh' \
'(find.*\.test\.sh|fd .*test|\.test\.sh|shellcheck)' \
.github plugins/agentic-engineering README.md 2>/dev/null || true
printf '%s\n' '== Forge guard grammar relevant to the added GraphQL command =='
sed -n '1,360p' plugins/agentic-engineering/scripts/forge-readonly-guard.sh
printf '%s\n' '== Static syntax and JSON/hash checks =='
bash -n plugins/agentic-engineering/scripts/surveyor-open-pr-links.test.sh
bash -n plugins/agentic-engineering/scripts/surveyor-selection-contract.test.sh
jq empty plugins/agentic-engineering/scripts/fixtures/surveyor-selection.json
actual="$(sha256sum plugins/agentic-engineering/agents/portfolio-surveyor.agent.md | awk '{print $1}')"
declared="$(jq -r '.spec.roles["portfolio-surveyor"].definitionSha256' plugins/agentic-engineering/resources/provider-neutral.desired-state.json)"
printf 'definition-sha match=%s actual=%s declared=%s\n' \
"$([ "$actual" = "$declared" ] && printf yes || printf no)" "$actual" "$declared"
printf '%s\n' '== Version consistency =='
jq -r '.version' plugins/agentic-engineering/plugin.json
jq -r '.version' plugins/agentic-engineering/.claude-plugin/plugin.json
jq -r '.plugins[] | select(.name=="agentic-engineering") | .version' .claude-plugin/marketplace.json
jq -r '.plugins[] | select(.name=="agentic-engineering") | .version' .github/plugin/marketplace.jsonLength of output: 108148 🤖 web_search executed:
Length of output: 3972
The new selection rules prevent unsupported ranked, empty, and freshness-complete outcomes. The association query uses supported GitHub GraphQL arguments. CI discovers the new contract tests. Static syntax, JSON, manifest digest, and version consistency checks passed. You are interacting with an AI system. |
CodeRabbit completed its review without findings in comment 5551226588, but its response does not identify a reviewed commit and it emitted no current-head review object. That completed response therefore does not satisfy the commit-bound review gate. Advancing to the next provider for unchanged head 0be5454; CI is terminal green and all finding threads are resolved. @codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Ready after current-head review and terminal CI at 0be5454, based on main 495d152. All five prior findings are fixed and their threads resolved. Selection now proves the nominee and preceding skips, preserves the default oldest-first order, requires declared claim timing, and collects all-author open-PR associations without retrieving linked repository nodes. The exact-head clean Codex review is comment 5551249455, posted by the trusted connector after this round's authenticated request. CodeRabbit completed without findings but supplied no reviewed-commit identity, so its reply was recorded as a completed no-gate result before advancing. Validation includes 11 independent definition-removal controls, 12 blind model-pressure cases, 3 valid association counts and 13 invalid/partial-error responses, read-only guard admission, and the existing guard/adapter/dependency/digest suites. Hosted CI covers both new executable tests. The plugin version is 4.4.23 with matching desired-state digests and all four manifests in parity. This delivers the portable plugin definition. Consumer pin/compatibility-overlay rollout and runtime cache adoption remain separate deployment work; no runtime cache was changed. |
Post-merge verification passed at d5ee81a. The complete merged tree equals reviewed head 0be5454. All 11 selection removal controls and the open-PR projection test (3 valid counts, 13 invalid/partial-error responses, guard admission) passed again against those bytes. Default-branch CI 33961622030 and CodeQL 33961621716 succeeded at that exact merge commit. Issue #165 is closed; its claim and this task's merged remote branch are absent. Consumer rollout remains separate from this portable plugin delivery. |
1 participant
Why
A complete inventory could be reported as an empty work queue before the surveyor had ranked issues or established why they could not be started. That false completion could delay actionable work.
What
Require current selection evidence before reporting the highest-ranked issue, an empty queue, or a completed survey. Missing evidence stays explicit while useful health findings remain available.
Fixes #165