Skip to content

Fix indexed path traversal and prepare v1.1.0 for agent runtime integration - #179

Merged
Shengyu Fu (shengyfu) merged 5 commits into
mainfrom
shengyfu-bump-tgrep-to-1-1-0
Oct 7, 2026
Merged

Shengyu Fu (shengyfu) merged 5 commits into
mainfrom
shengyfu-bump-tgrep-to-1-1-0

Conversation

@shengyfu

@shengyfu Shengyu Fu (shengyfu) commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Summary

Prepare tgrep v1.1.0 for Copilot CLI agent runtime integration, harden ordinary indexed file reads, and remove the flaky live-memory comparison. This PR includes the requested path-containment and macOS test fixes in addition to version metadata; it is no longer metadata-only.

Version preparation

  • Set workspace.package.version from 1.0.11 to 1.1.0.
  • Synchronize tgrep-cli / tgrep-core in the root Cargo.lock and tgrep-core in fuzz/Cargo.lock.
  • Both crates inherit the workspace version; CLI version output and Windows VERSIONINFO continue to derive from it.

Indexed-read containment and review follow-up

  • Validate stored content-index and filename-sidecar paths before use, sharing validation with shared-base/checkpoint readers. Reject invalid UTF-8, non-normalized or escaping paths, and Windows path prefixes/alternate streams where applicable. Preserve valid Unix colon-bearing names, with a builder/reader/rooted-open/sidecar round-trip regression.
  • Use existing pinned-root file handles for ordinary local and server content reads, indexed metadata sorting, and file-stamp collection. Descendant symlinks/reparse points are not followed.
  • Require an absolute, resolvable metadata root covering the requested scope; otherwise scan the requested tree instead of guessing the index root. The requested root remains the authority for local content reads; metadata hashes are not treated as authentication.
  • Preserve the public collect_filestamps -> HashMap<String, FileStamp> signature. This best-effort compatibility API diagnoses and omits unsafe/unreadable paths. Add try_collect_filestamps -> Result<HashMap<String, FileStamp>> for fallible callers, retaining omission of missing files. Both APIs use the same validated rooted-open helper.
  • Enforce source-byte limits during local, scan, and server reads, not only in metadata prefilters. A shared bounded reader consumes at most limit + 1 bytes and rejects oversized input before decoding/caching. Local mmap uses an explicitly approved extent and rechecks the same handle before accepting the mapping. The explicit-file exemption from the inherited default cap is unchanged.
  • Retain raw source-byte counts in cached decoded entries. A stricter query cannot accept oversized cached content based on its smaller decoded length; a bounded fresh read allows a subsequently smaller replacement to become eligible. Server size-prefilter optimizations do not bypass read-time bounds.
  • Document indexes as local generated output that must not be committed or distributed with repositories. Explicit-file and scan-follow semantics are unchanged.
  • Add deterministic coverage for stored-path rejection, whole/subtree containment, pinned-handle mmap/decoding, ordinary server cold/cache/encoding paths before reconciliation, metadata-root fallback, and Windows alternate-stream rejection. Additional review regressions cover growth, capped mappings, raw-versus-decoded sizes, cache replacements, exact/zero/unlimited caps, and bounded byte consumption. Fixtures use invented data only.

Deterministic memory-budget tests

  • Replace comparisons between separate live memory samples with fixed-input tests of the selection policy used by production.
  • Cover private-byte preference, lazy RSS fallback, zero/maximum values, and unavailable counters on every platform. Verify RSS is not queried when private bytes are available and is queried exactly once otherwise.
  • Retain a single-sample supported-platform smoke test. Runtime selection remains private bytes first, RSS only as fallback; no retries, sleeps, or widened tolerances are introduced.

Review follow-up commit: 195ecb1e781f6b7fd6dd63bf9b43b1b8125381f9. It addresses all three inline threads plus the related local-read size-limit finding in the review summary. The branch started from main at 677a4baa1a8d5e5c886feaff7140a5a12cacd550.

Validation

Completed locally on Windows for the review follow-up:

  • Core library: 315 passed. corrupt_index_reader, shared_worktrees, and snapshot_consistency: 41 passed.
  • Latest CLI unit suite: 282 passed, 1 ignored, including the new bounded-read/cache regressions and the deterministic memory-budget tests.
  • indexed_containment, indexed_files, indexed_hidden, large_file_search, default_max_filesize, and concurrent_search: 34 passed.
  • cargo check --locked --offline --quiet --workspace --tests: passed.
  • cargo clippy --locked --offline --quiet --workspace --all-targets -- -D warnings: passed.
  • cargo fmt --all -- --check and CRLF-aware git diff --check: passed. The decoder's committed CRLF line endings are preserved.
  • Full cargo metadata --locked --offline --format-version 1 graphs: root 194 packages, separate fuzz workspace 111 packages; tgrep package versions remain 1.1.0.
  • cargo build --locked --offline --quiet -p tgrep-cli --bin tgrep: passed. Built CLI returns exit 0, empty stderr, and exact stdout tgrep 1.1.0\n.
  • All Cargo test commands used --locked --offline. Final follow-up diff contains seven Rust files only; no dependency churn, manifest/lockfile edits, or vendor changes.

Native Unix-specific regressions, including the new colon-filename round trip, are included for CI but were not executed locally. The previously observed macOS memory-sampling flake is addressed by the deterministic tests above.

Release boundary

No runtime-repository edits or integration implementation, tags, release publication, workflow dispatches, or pipeline changes. Checksum generation and Windows signing/release-build work remain in the compliant internal Azure DevOps pipeline and are unchanged here. Vendored code, vendor/ignore/Cargo.lock, and historical benchmark/provenance data are untouched. This description does not include the coordinated-disclosure report, reporter details, or a standalone exploit generator.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 22:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

All version metadata is consistent, scoped, and matches the stated release boundary.

Review effort: Balanced
Findings: None

What changed in this PR

Updates release metadata to prepare tgrep v1.1.0.

Changes:

  • Bumps the workspace version to 1.1.0.
  • Synchronizes root and fuzz lockfiles.
File Description
Cargo.toml Updates workspace version.
Cargo.lock Updates CLI and core package versions.
fuzz/​Cargo.lock Updates the core package version.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Validate stored paths and reuse pinned-root opens for ordinary indexed search, metadata, and sorting. Cover containment and decoding with regression tests.

Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 02:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The current implementation introduces a public API break, rejects valid Unix filenames, and retains a server size-check race.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity

Open (3)

Comment thread tgrep-core/src/rooted.rs Outdated
Comment thread tgrep-cli/src/serve.rs
Comment thread tgrep-core/src/meta.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 7, 2026 02:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Server reads can bypass size limits, and the minor release introduces a breaking public API change.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity

Open (3)

Replace comparisons between live memory samples with fixed-input coverage of private-byte preference, lazy RSS fallback, and unavailable counters. Keep a single-sample platform smoke test and preserve runtime selection behavior.

Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 02:56
@shengyfu Shengyu Fu (shengyfu) changed the title Bump version to 1.1.0 for agent runtime integration Fix indexed path traversal and prepare v1.1.0 for agent runtime integration Oct 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Reader/writer path incompatibility, a public API break, and size-limit race conditions remain.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity

Open (3)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Enforce max file size during reading

tgrep-cli/​src/​search.rs:1486

The size check is only a metadata snapshot; read_open_text_lossy then maps or reads the handle without a byte bound. If the file grows after this check, --max-filesize can still search and allocate beyond the requested limit. Pass the limit into the read path and reject data beyond limit + 1, including before accepting an mmap.

Preserve Unix colon filenames and the legacy file-stamp API. Enforce source-byte limits during local and server reads, including mmap and cached content, with deterministic regressions.

Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 03:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-platform filesystem containment and concurrent read behavior warrant final human security review.

Review effort: Balanced
Findings: None

Resolved since last review (3)

@shengyfu
Shengyu Fu (shengyfu) merged commit 8ae6090 into main Oct 7, 2026
12 checks passed
@shengyfu
Shengyu Fu (shengyfu) deleted the shengyfu-bump-tgrep-to-1-1-0 branch October 7, 2026 04:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants