Skip to content

fix(cli): print recursive SARIF to stdout and keep status text off it - #565

Merged
rng1995 merged 1 commit into
NVIDIA:mainfrom
kevin9327:fix/recursive-sarif-stdout
Sep 16, 2026
Merged

rng1995 merged 1 commit into
NVIDIA:mainfrom
kevin9327:fix/recursive-sarif-stdout

Conversation

@kevin9327

@kevin9327 kevin9327 commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

skillspector scan <collection> --recursive --format sarif > results.sarif does not produce a SARIF file. stdout contains only the human-readable progress and summary, and the merged SARIF log is never printed:

$ skillspector scan ./multi --recursive --no-llm --format sarif
Multi-skill directory detected: 2 skills found

  [1/2] Scanning a (a/)
         Score: 0/100 (LOW)
...
═══ Multi-Skill Summary ═══
...
$ echo $?
0

The --help text for --output says "If not specified, prints to stdout." #467 fixed this for --format json (combined document to stdout, status text to stderr). SARIF is the other format tools parse from stdout, and it was left on the old routing, so several single-skill paths also print status text ahead of the log:

command (--no-llm --format sarif) first line of stdout on main
scan ./multi --recursive Multi-skill directory detected: 2 skills found (and no SARIF follows)
scan ./multi (two sub-skills, no --recursive) Warning: Found 2 skills in this directory. Use --recursive to scan each
scan ./no-skill-md --recursive Warning: --recursive specified but no sub-skills detected. Scanning as single
scan ./skill --verbose Running scan...
scan ./skill --transitive when a child scan fails (reproduced through the unit test below) Warning: Transitive scan failed for https://github.com/...

In each case json.loads(stdout) fails at character 0.

Fix

  • Add _MACHINE_READABLE_FORMATS = {json, sarif} and use it at the five places that currently check format == FormatChoice.json to send status text to stderr.
  • In _scan_multi_skill, build the merged SARIF log whether or not --output is set. With --output it is written to the file as before; without it, it is printed. This mirrors what fix(cli): emit recursive JSON reports to stdout #467 did for the JSON branch.

--output behaviour, the SARIF content, the output bound and exit codes are unchanged. Markdown and terminal routing are untouched.

Tests (tests/unit/test_cli.py)

  • test_recursive_single_skill_advisory_does_not_pollute_stdout and test_multi_skill_advisory_does_not_pollute_stdout: the existing JSON tests from fix(cli): emit recursive JSON reports to stdout #467, now parametrized over json and sarif (the json in their names was dropped).
  • test_recursive_sarif_without_output_writes_only_the_log_to_stdout: recursive, verbose, no --output. stdout must pass validate_sarif_report and contain the aggregate run; progress, "Running scan" and the summary must be on stderr.
  • test_transitive_failure_warning_stays_off_sarif_stdout: SARIF variant of test_transitive_resolver_failure_preserves_direct_report.

Against unmodified cli.py (Windows 11, Python 3.12.10, main at 4148ab3):

E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
E       assert 'Transitive scan failed' not in 'Warning: Tr... }\n  ]\n}\n'
FAILED tests/unit/test_cli.py::test_recursive_single_skill_advisory_does_not_pollute_stdout[sarif]
FAILED tests/unit/test_cli.py::test_multi_skill_advisory_does_not_pollute_stdout[sarif]
FAILED tests/unit/test_cli.py::test_recursive_sarif_without_output_writes_only_the_log_to_stdout
FAILED tests/unit/test_cli.py::test_transitive_failure_warning_stays_off_sarif_stdout
4 failed, 2 passed

With the fix: 6 passed. Every changed line runs in these tests (checked with branch coverage).

I also re-ran the first four rows against the real CLI. After the fix, stdout passes validate_sarif_report in every row (for ./multi --recursive it holds 3 runs: two skills plus the aggregate), and the status text shows up on stderr.

Suite

pytest -m "not integration and not provider" -p no:randomly, same machine:

main this branch
tests/unit, tests/opencode, tests/test_*.py 10 failed, 1721 passed 10 failed, 1725 passed
tests/nodes 3 failed, 3588 passed, 4 errors 3 failed, 3588 passed, 4 errors

The +4 are the new SARIF cases. The failures and errors are the same on both sides and Windows-only (release and compare_scan_accuracy harnesses, CRLF fixtures, a backslash inside a POSIX file name, and parametrized test IDs over the Windows environment-variable length limit).

ruff check src/ tests/: All checks passed. ruff format --check src/ tests/: 246 files already formatted.

🤖 Generated with Claude Code

A recursive scan with --format sarif and no --output printed only the
human-readable progress and summary to stdout; the merged SARIF log was
written only when --output was set. Other status lines (the multi-skill
and no-sub-skill advisories, --verbose "Running scan...", and transitive
failure warnings) were also printed to stdout ahead of SARIF, because
NVIDIA#467 routed them to stderr for JSON only.

Treat SARIF like JSON for stdout routing, and print the merged recursive
SARIF log when no --output is given.

Co-Authored-By: Claude Opus 5 <[email protected]>
Signed-off-by: kevin9327 <[email protected]>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Approved at commit 8e1882b.

I traced the affected single-skill, recursive, verbose, and transitive paths. JSON and SARIF are now treated consistently as machine-readable output: status text is routed away from stdout, recursive SARIF without --output emits one bounded and validated log, and the existing file-output, exit-code, cleanup, and completeness behavior is preserved.

The added tests cover the reported regressions, including aggregate SARIF output and transitive failure warnings. All current CI jobs pass.

@rng1995
rng1995 merged commit 20fdba7 into NVIDIA:main Sep 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants