Skip to content

feat: analyze bundled hook execution surfaces - #404

Closed
chrisknvidia wants to merge 22 commits into
NVIDIA:mainfrom
chrisknvidia:feat/christopherk/issue-399-hook-surface
Closed

chrisknvidia wants to merge 22 commits into
NVIDIA:mainfrom
chrisknvidia:feat/christopherk/issue-399-hook-surface

Conversation

@chrisknvidia

@chrisknvidia chrisknvidia commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #399

Scope

This draft is the hooks-only first slice of #399:

  • BH1 reports bundled lifecycle hooks from the exact supported hook/settings surfaces.
  • BH2 reports only closed, direct exfiltration proofs from those hook declarations.
  • Hook parsing, evidence, ledger events, scoring, report output, baseline behavior, and recursive single-child CLI routing are included.

BH3 permission analysis is intentionally excluded. The stacked draft follow-up is #429.

Implementation

  • one bounded analyzer module; no script interpreter, runtime engine, marketplace simulator, or duplicate archive traversal
  • exact hooks/hooks.json, .claude/settings.json, and .claude/settings.local.json routing after existing input normalization
  • aggregate, deterministic, sanitized BH1/BH2 evidence with per-document failure isolation
  • an effective, well-formed project/local disableAllHooks: true suppresses ordinary bundled hooks from all three surfaces; no managed-settings exemption is asserted without managed provenance
  • malformed modeled settings remain PARTIAL, preserve valid hook findings, and cannot make disableAllHooks trustworthy
  • closed direct BH2 forms for external HTTP handlers and literal curl/wget/scp/rsync uploads, including sensitive hook stdin
  • CRITICAL BH2 findings retain the existing blocking score/recommendation behavior

Verification

  • focused analyzer, acceptance, registry, report, and CLI suite: 304 passed
  • full suite on exact head 9759c6f: 3,055 passed, 13 skipped, 38 deselected, 4 expected xfails
  • lint, formatting, C901 complexity, targeted mypy, lockfile, and diff checks: clean
  • real directory and ZIP CLI scans: an ordinary plugin hook beside effective disableAllHooks emitted no BH findings, remained complete, scored 0/SAFE, and exited 0
  • malformed same-file modeled settings retained BH1, reported incomplete analysis, exited 0 by default, and exited 1 with --fail-on-incomplete
  • Docker smoke: image build, version/git checks, local fixture scan, and public GitHub URL scan passed
  • integration regression comparison on the final stacked head and clean origin/main: both produced 30 passed, 2 skipped, 3 failed; the same two live-Codex model-resolution failures and graph/LLM telemetry expectation reproduce on main

Deliberate limits

The analyzer does not interpret referenced scripts, execute permission globs, simulate full settings precedence, discover new archive roots, or claim that absence of BH2 makes a hook safe. Live authenticated Claude/IDE/Desktop hook dispatch was not run because the local Claude environment is not authenticated.

This PR remains draft for maintainer review.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes because the new hook-flow model has several deterministic false negatives for credential/event-data exfiltration and several blocking false positives. The focused analyzer suite passes, so these behaviors need explicit regression cases before the model can safely drive BH2 install blocking.

Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_flow.py Outdated
Comment thread src/skillspector/nodes/analyzers/bundled_hook_runtime.py Outdated
@mohgupta-ship-it

mohgupta-ship-it commented Aug 25, 2026 •

Copy link
Copy Markdown
Member

Powered by Codex: PR council review result.

This is a triage signal, not a maintainer approval.

  • Rating: new feature
  • Confidence: high
  • Status read: Draft, changes requested, merge blocked, green checks
  • Review method: fresh GitHub metadata/body/files/reviews/checks plus selected diffs; council lenses were spec fit, dead-code/reachability, YAGNI/scope, design/coupling, and code standards/tests.
  • Council assessment: Hook execution-surface analyzer still has live BH2 blockers per current review trail: false negatives for shell/Python exfiltration paths and some blocking false positives.
  • Recommended action: Do not merge or describe as ready; fix BH2 blockers and add explicit regression coverage.

@rng1995
rng1995 marked this pull request as ready for review August 26, 2026 19:03
@chrisknvidia
chrisknvidia marked this pull request as draft August 26, 2026 21:21
chrisknvidia added a commit to chrisknvidia/SkillSpector that referenced this pull request Aug 27, 2026
Signed-off-by: Christopher Kevin <[email protected]>
@chrisknvidia
chrisknvidia marked this pull request as ready for review August 27, 2026 01:22

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

The current-main conflict is resolved at 4c06d7d by retaining #429's already-merged BH3 additions; the resolved tree is byte-for-byte identical to current main. I resolved the four threads whose requested behavior is now satisfied (|&/|| are explicitly unmodeled, the dig .env and tuple-unpack false positives are gone, and loopback classification uses one normalizer).

Still requesting changes for the five remaining data-flow cases: shell parameter expansion, event stdin through command substitution, aws s3 cp - stdin semantics, sys.stdin.buffer.read(), and local Python-import traversal. Exact current-tree reproductions emit BH1 only with payload_analysis_level="unmodeled", but the ledger remains COMPLETED; the acceptance path can therefore report analysis complete and recommend SAFE without analyzing the payload. Please either model these flows or make unmodeled payload analysis incomplete/fail closed, with exact regressions.

Because merged #429 already subsumes #404 plus BH3, closing #404 as superseded and fixing this behavior on main in a focused follow-up may be cleaner.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-review at dbfb033: merging current main leaves zero file changes; the head tree is byte-for-byte identical to d3a849a. The five previously requested flows remain unaddressed: shell parameter expansion, event stdin through command substitution, aws s3 cp - stdin, sys.stdin.buffer.read(), and local Python-import traversal each emit BH1 only with payload_analysis_level="unmodeled" while the ledger reports COMPLETED. The focused 113-test suite and hosted checks pass, but this fail-open behavior remains. Still requesting changes. Since #429 already landed this tree, close #404 as superseded and address the behavior in a focused follow-up instead of merging a no-op PR.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-review of 8146c18: the current head only merges main and the PR now has an empty diff. It therefore does not provide a PR-owned change that resolves the five previously reported unmodeled hook payload flows (shell parameter expansion, event stdin through command substitution, aws s3 cp -, sys.stdin.buffer.read(), and local Python-import traversal). The existing action items remain open even though hosted checks are green. This PR has been superseded by the already-landed hook work; please close it and address any remaining fail-open completeness behavior in a focused follow-up.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed current head 39995d5af6d90b6b1276d1830d9d9e24c80598a2. The new head only synchronizes main; GitHub still reports an empty PR diff, so there is no PR-owned change to reassess and none of the five outstanding hook payload flows is fixed here: shell parameter expansion, event stdin through command substitution, aws s3 cp - stdin, sys.stdin.buffer.read(), and local Python-import traversal remain unmodeled while completeness can still be reported as completed.

All hosted checks pass, but this no-op PR is superseded by the hook work already merged through #429. Please close it and address any remaining fail-open completeness behavior in a focused follow-up with regressions. The existing threads contain the exact cases, so I have not duplicated them. The unresolved action items and BEHIND merge state prevent approval and merge.

@chrisknvidia

Copy link
Copy Markdown
Collaborator Author

Closing as superseded, per review. PR #573 contains the focused fail-closed correction and exact regressions for all five remaining hook-payload cases; the original inline threads now link to that follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants