You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add opt-in shared repository daemon and worktree lifecycle - #171
Layer 3 above #170 (synchronization), following #169 (immutable generations) and merged #168. Head: 7afdc11444bc8454783286a58092a410ed2af5f3, targeting shengyfu-shared-worktree-sync at 2af22f08bfb74d61d11da75d5dc9d96976464f78. Fifteen own commits, zero behind the exact parent. Native stack #172: 169 → 170 → 171, base main, is unchanged; all PRs remain open/unmerged. No ancestor branch mutations, extra agents/sessions or merges.
Implements an opt-in repository-scoped loopback daemon with protocol-v1 negotiation, exact immutable generation pins, private synchronized worktree overlays, independent recoverable leases and external delta-only checkpoints. Attachment is the opt-in: ordinary search, --files and status discover separate worktree registration. Unattached roots retain legacy behavior. Stale/incompatible/not-ready registrations scan directly, never a bare shared base or stale ordinary index; status reports the error. Explicit --index-path preserves legacy intent and --no-index always scans.
Native registration precedes reconciliation. Bounded queues, overflow/unknown-event full repair, completion-based polling and explicit no-watch behavior feed each view. One indexing worker and two query workers avoid a repository-global search lock. Existing decoding/matching/rendering helpers are reused. There is no cross-worktree content cache or ordinary full-index overlay merge.
Candidate opens use the parent's retained-root WorktreeSnapshot::open_file seam. Core guards atomically close readiness on open failures, including swallowed errors. Shared matching bounded-reads owned handles outside the guard, invalidates later read failures, and verifies root/epoch again before publishing content, filenames or empty results. Readiness additionally requires a successfully checkpointed epoch and no active reconciliation.
Shared authority checks and cheap/core discovery share lossless Unix gitfile/commondir decoding, retaining canonical root/Git-directory/common-directory comparisons and zero daemon Git subprocesses on hot queries. UTF-8 canonical worktree roots remain required.
Separate module-level Unix regressions cover native gitfiles and independently native commondir targets through full raw/CLI lifecycle, ignore-source precedence, nested repositories, malformed-ignore failure/repair, scan parity and three client/zero server Git starts. Existing pointer-tamper checks remain enabled. Only the precise EILSEQ fixture-creation impossibility is skipped on filesystems that cannot create native-byte names; decoder unit coverage still executes.
The locked ignore 0.4.25 also rejected native metadata, preventing complete walk readiness. It is vendored without upgrading; only upstream src/dir.rs is patched. All original licenses/provenance/checksum are preserved. Errors and ignore semantics are not suppressed. Direct path-only dependency wiring preserves the fix for checkout installs, release/cross builds, Git/path consumers and the separate fuzz workspace; packaging cannot silently substitute unpatched registry code. See the auditable patch/distribution notes.
Valid JSON-RPC errors may carry an explicit null ID when rejected before request parsing, preserving shared connection queue full. Success still requires the exact request ID. Mismatched/missing IDs, invalid errors, simultaneous result/error and malformed envelopes remain errors. Production TCP and real CLI regressions fail before/pass after the fixes.
Accepted mock sockets inherit listener nonblocking mode on macOS. The protocol/CLI mocks explicitly restore blocking mode, retaining bounded read/write timeouts. An existing automatic-watch fixture also retries the specific reconciliation race within its original deadline. No production/runtime/workflow changes were required. The actual 3e89778 macOS job and entire matrix passed; a subsequent review misattributed the old-head read-line EAGAIN failure to new-head timeout line numbers. That thread was resolved with exact passing job logs, without removing timeouts.
Regenerated tracked fuzz/Cargo.lock for local ignore and the complete current core dependency graph, adding 60 previously absent package/version pairs plus the ignore source switch. Every previously locked package version is preserved. Locked metadata reproducibly failed before and now passes on Windows/Linux without rewriting the lock; all four native-Linux fuzz targets pass locked all-target checking. The review thread is resolved. This commit changes only the fuzz lockfile.
The user separately requested a macOS cleanup workaround, explicitly expanding scope to this repository's Python adapter and CI configuration. No external agent-runtime repository was changed; the final-review followups do not edit runtime/workflows.
After os.killpg raises PermissionError, scripts/agent/runtime.py waits at most 250 ms for confirmed child exit. If still alive, it re-raises the original error instead of suppressing it or waiting indefinitely.
Deterministic delayed-exit/live-child denial tests verify exception identity, bounded waiting and pipe cleanup; they fail before/pass after.
CI matrix fail-fast: false; no tests disabled or marked continue-on-error.
8b767f2 CI passed all three platforms, including all 33 macOS Python tests and the original max-results failure. 3e89778 CI also passed Windows, Ubuntu, macOS, Format and Clippy after the socket fixture correction. Current-head checks are independently green below.
Wait for ready: true. Persist a unique 1–128-character ASCII alphanumeric/hyphen/underscore token before transmission. Repeating a live token/root/literal revision/profile recovers a lost response without another lease. Tokens detach independently. Restart requires reattachment and full checkpoint revalidation; views never automatically repin. Release leases and finish in-flight operations before removing a worktree; Windows retains root handles. The runtime supervises the daemon. Escape the literal subcommand with tgrep -- shared ..
Versioned methods: hello, attach, lookup, refresh, detach, status, search, files. Queries bind root/view/generation/epoch; successful responses identify backend: "shared-v1". Separate daemon/common-dir and view/worktree-git-dir registration never uses legacy serve.json. Raw passthru: true is rejected; CLI filesystem-scan parity, including nonmatching files, remains unchanged. Malformed/incompatible requests never return empty successes. README, SHARED_WORKTREE_INDEXES.md and AGENTS.md document the complete contract.
Validation and final review
Full Windows workspace 1,035 passed, one pre-existing ignored; full exact-source native-Linux workspace 1,080 unique top-level tests passed, one pre-existing ignored. Three child FIFO probes are not double-counted.
Final CLI unit suites 270 Windows / 292 Linux and shared integrations 24/29 pass; formatting and all-target Clippy pass on both platforms.
Complete vendored upstream suite 148 Windows / 153 Linux, including doctests, passes. All 21 upstream files are byte-identical except the documented parser file; archive SHA256 matches pre-fix Cargo.lock.
Independent immutable Windows/Linux binaries pass both native metadata lifecycle cases, all 12 real-CLI envelope cases, ignore/scan parity and Git-start counts. Full Windows shared smoke passes sharing/isolation, leases, hints, fallback, checkpoints, passthru and actual detached-worktree removal while a sibling stays served.
Independent private-root checkout installation succeeds; installed binary passes protocol/full shared smoke. Main and separate fuzz metadata select exactly one local ignore0.4.25, excluded from main workspace membership. Final Windows/Linux --locked checks preserve both lockfile hashes. Tested production/install inputs match published inputs; final lock-only head binaries are byte-identical to independently tested 3e89778 binaries.
Exact-head CI 37231855937 passed Windows, Ubuntu, macOS, Format and Clippy. CodeQL and CLA also pass. All runs were automatic; no manual reruns requested.
Completed review 37231890496: Findings None. Its overview and all review/comment pages were read; all 12 inline threads are resolved, pagination complete. The overview recommends final human review because of the daemon/filesystem/concurrency scope; this is not represented as human approval. No merge or approval was performed.
Sharing and deliberate limits
Three identical tracked files require three reads/decodes and zero private extractions; a linked worktree adds its ordinary .git pointer as one private extraction. Four CRLF-transformed paths plus the pointer need five initial reads/decodes/extractions, zero new extractions on no-op verification. Views reuse the same immutable base; newer generations reuse unchanged content without repinning existing views. Sharing does not eliminate reads or transformation costs.
V1 supports the default raw-auto tracked-regular-file 64 MiB profile. Storage is trusted/external, with separate generation/overlay children. Defaults: 32 views, 256 leases, 8192 native subscriptions, 16384 hint slots; work queue=max views, query queue=16, incoming queue=32. Requests cap at 1 MiB, encoded responses at 64 MiB; this is not a total-process memory cap.
Native subscription failure/budget exhaustion falls back to polling. Automatic full verification defaults to 120 seconds after completion. Failed repairs back off 1/2/4/8/16/30 seconds; explicit refresh bypasses/resets this. No-watch needs runtime refresh after initial verification. Readiness/hint acknowledgments are not filesystem-wide atomic freshness guarantees. No content cache, online GC, checkpoint eviction, automatic repinning or lease expiry.
Ensure detach completes despite malformed or removed view markers
tgrep-cli/src/serve/shared/server.rs:571
A malformed or concurrently removed view marker makes final detach return here before the lease and in-memory view are released. Because stale/corrupt markers are already treated as safe scan fallbacks, this can unnecessarily consume a lease forever and prevent reattachment until daemon restart. Make marker cleanup best-effort and always complete the server-side detach.
Watch nested .tgrep directories outside the root storage directory
tgrep-cli/src/serve/shared/server.rs:942
This basename check also skips nested directories such as src/.tgrep, but WorktreeView excludes only the root's .tgrep storage directory. A nested .tgrep directory can therefore be part of the hidden searchable corpus while receiving no native subscription, even though the view reports native watch coverage; changes remain stale until periodic reconciliation. Skip only the root storage directory while retaining the repository-wide .git exclusion.
Validate UTF-8 root ID before creating a retained generation
tgrep-cli/src/serve/shared/server.rs:678
Compute/validate the UTF-8 root ID before manager.ensure. A raw RPC can name a Unicode symlink that canonicalizes to a non-UTF-8 worktree; in that case ensure may build and publish a new retain-all generation, and only this later conversion rejects the attach, leaving permanent storage from a request that can never create a view.
Require RPC scope to be an existing directory
tgrep-cli/src/serve/shared/server.rs:1104
The public RPC contract says scope is a directory, but this accepts an existing regular-file scope: it remains inside the worktree and then returns a successful empty result because the generated prefix ends in /. Validate scoped_root.is_dir() so malformed direct RPC requests fail explicitly instead of silently looking like no matches.
These shared-only limits are accepted when serve runs without --shared, but the ordinary-server branch ignores both values. Add the same requires = "shared" constraint used by shared_storage so a mistyped invocation fails instead of silently discarding configuration.
Detect dangling marker symlinks to preserve fail-closed fallback
tgrep-cli/src/serve/shared/protocol.rs:83
try_exists() follows symlinks, so a dangling/corrupt view-marker symlink is reported as absent. The query then becomes eligible for the legacy index even though a marker directory entry still exists, contradicting the fail-closed “mere presence selects shared fallback” contract. Detect the marker entry with symlink_metadata so any unreadable/corrupt marker routes to scanning.
Include required lease parameter in operation summary
SHARED_WORKTREE_INDEXES.md:153
The operation summary omits the required caller-owned lease parameter, while the v1 wire table and implementation require it on every attach. Include lease here so protocol implementers do not construct an invalid attach request.
Add backoff for persistently failing view reconciles
tgrep-cli/src/serve/shared/server.rs:271
A persistently broken view is retried every 200 ms. Fast permanent failures such as a deleted worktree or unreadable metadata therefore create an endless full-reconcile/log loop, and with only one reconcile worker they can repeatedly delay healthy views and lifecycle requests. Add capped failure backoff (reset on success or an explicit new refresh) instead of using the fixed initial-readiness delay for every failure.
Charge encoded rows, filenames, statistics and envelope data during accumulation; stop oversized per-file output before allocating JSON rows and bound final serialization. Back off failed repairs with explicit-refresh recovery and expose retry status. Cover canonical subtree hints after synchronizing the parent.
Co-authored-by: Copilot App <[email protected]>
Document releasing all leases and finishing in-flight operations before removing a worktree. Verify the last independent lease releases the linked root while sibling queries remain served.
Co-authored-by: Copilot App <[email protected]>
Use retained snapshot file handles and verify final epochs for content, filenames and empty results. Keep status and queries unavailable until the matching checkpoint epoch is published, including failed reconciliation retries. Cover Unix replacement-root isolation and preserve strict backoff assertions.
Co-authored-by: Copilot App <[email protected]>
Allow a 250ms exit grace after group-signal permission errors, preserving the original error for a still-live child. Cover delayed exit and bounded denial, and disable matrix fail-fast so one platform cannot cancel the others.
Co-authored-by: Copilot App <[email protected]>
Repository::discover preserves native Git-directory paths, but this revalidation calls git_index::git_dir, which reads the .git pointer with read_to_string. On Unix, a UTF-8 linked-worktree root can point into a repository path containing non-UTF-8 bytes: attach succeeds through Repository::discover, then every lookup/search/detach fails here as “missing worktree Git directory,” leaving the lease until restart. Parse the gitfile and commondir losslessly (or validate via a retained native identity handle) before comparing them with the stored Repository paths.
When the incoming connection queue is full, the server has not read the request yet and intentionally returns an error with id: null. This validation rejects that envelope before the following error branch can surface “shared connection queue full,” so clients only report “invalid shared RPC response” and lose the actionable saturation diagnostic. Allow a null ID specifically for error envelopes, while retaining id == 1 for successful responses.
Keep native gitfile and commondir paths lossless across shared validation and ignore discovery. Vendor only the locked ignore release with its small native parser patch, preserving ignore semantics and direct checkout distribution. Validate null-ID JSON-RPC errors without relaxing successful or mismatched response IDs. Add full lifecycle, precedence, repair and envelope regressions.
Co-authored-by: Copilot App <[email protected]>
macOS inherits the nonblocking listener flag on accepted mock sockets. Restore blocking mode explicitly and retain bounded read/write timeouts in the protocol and CLI fixtures. No production or workflow changes.
Co-authored-by: Copilot App <[email protected]>
Record local ignore and the complete current tgrep-core dependency graph in the separate fuzz workspace. Preserve every previously locked package version. Locked metadata now resolves without modifying the lock on Windows and Linux; all four native Linux fuzz targets check successfully.
Co-authored-by: Copilot App <[email protected]>
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
The cross-platform daemon, filesystem authority checks, watchers, concurrency, and lifecycle state warrant final human review despite successful CI and extensive coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Layer 3 above #170 (synchronization), following #169 (immutable generations) and merged #168. Head:
7afdc11444bc8454783286a58092a410ed2af5f3, targetingshengyfu-shared-worktree-syncat2af22f08bfb74d61d11da75d5dc9d96976464f78. Fifteen own commits, zero behind the exact parent. Native stack #172: 169 → 170 → 171, basemain, is unchanged; all PRs remain open/unmerged. No ancestor branch mutations, extra agents/sessions or merges.Implements an opt-in repository-scoped loopback daemon with protocol-v1 negotiation, exact immutable generation pins, private synchronized worktree overlays, independent recoverable leases and external delta-only checkpoints. Attachment is the opt-in: ordinary search,
--filesandstatusdiscover separate worktree registration. Unattached roots retain legacy behavior. Stale/incompatible/not-ready registrations scan directly, never a bare shared base or stale ordinary index; status reports the error. Explicit--index-pathpreserves legacy intent and--no-indexalways scans.Native registration precedes reconciliation. Bounded queues, overflow/unknown-event full repair, completion-based polling and explicit no-watch behavior feed each view. One indexing worker and two query workers avoid a repository-global search lock. Existing decoding/matching/rendering helpers are reused. There is no cross-worktree content cache or ordinary full-index overlay merge.
Candidate opens use the parent's retained-root
WorktreeSnapshot::open_fileseam. Core guards atomically close readiness on open failures, including swallowed errors. Shared matching bounded-reads owned handles outside the guard, invalidates later read failures, and verifies root/epoch again before publishing content, filenames or empty results. Readiness additionally requires a successfully checkpointed epoch and no active reconciliation.Final-review findings addressed
bc39fe8 — native metadata and RPC diagnostics
ignore 0.4.25also rejected native metadata, preventing complete walk readiness. It is vendored without upgrading; only upstreamsrc/dir.rsis patched. All original licenses/provenance/checksum are preserved. Errors and ignore semantics are not suppressed. Direct path-only dependency wiring preserves the fix for checkout installs, release/cross builds, Git/path consumers and the separate fuzz workspace; packaging cannot silently substitute unpatched registry code. See the auditable patch/distribution notes.shared connection queue full. Success still requires the exact request ID. Mismatched/missing IDs, invalid errors, simultaneous result/error and malformed envelopes remain errors. Production TCP and real CLI regressions fail before/pass after the fixes.3e89778 — test portability
Accepted mock sockets inherit listener nonblocking mode on macOS. The protocol/CLI mocks explicitly restore blocking mode, retaining bounded read/write timeouts. An existing automatic-watch fixture also retries the specific reconciliation race within its original deadline. No production/runtime/workflow changes were required. The actual 3e89778 macOS job and entire matrix passed; a subsequent review misattributed the old-head read-line EAGAIN failure to new-head timeout line numbers. That thread was resolved with exact passing job logs, without removing timeouts.
7afdc11 — standalone fuzz lock
Regenerated tracked
fuzz/Cargo.lockfor local ignore and the complete current core dependency graph, adding 60 previously absent package/version pairs plus the ignore source switch. Every previously locked package version is preserved. Locked metadata reproducibly failed before and now passes on Windows/Linux without rewriting the lock; all four native-Linux fuzz targets pass locked all-target checking. The review thread is resolved. This commit changes only the fuzz lockfile.User-approved CI workaround at 8b767f2
The user separately requested a macOS cleanup workaround, explicitly expanding scope to this repository's Python adapter and CI configuration. No external agent-runtime repository was changed; the final-review followups do not edit runtime/workflows.
os.killpgraisesPermissionError,scripts/agent/runtime.pywaits at most 250 ms for confirmed child exit. If still alive, it re-raises the original error instead of suppressing it or waiting indefinitely.fail-fast: false; no tests disabled or marked continue-on-error.8b767f2 CI passed all three platforms, including all 33 macOS Python tests and the original max-results failure. 3e89778 CI also passed Windows, Ubuntu, macOS, Format and Clippy after the socket fixture correction. Current-head checks are independently green below.
Public CLI/RPC contract
Wait for
ready: true. Persist a unique 1–128-character ASCII alphanumeric/hyphen/underscore token before transmission. Repeating a live token/root/literal revision/profile recovers a lost response without another lease. Tokens detach independently. Restart requires reattachment and full checkpoint revalidation; views never automatically repin. Release leases and finish in-flight operations before removing a worktree; Windows retains root handles. The runtime supervises the daemon. Escape the literal subcommand withtgrep -- shared ..Versioned methods: hello, attach, lookup, refresh, detach, status, search, files. Queries bind root/view/generation/epoch; successful responses identify
backend: "shared-v1". Separate daemon/common-dir and view/worktree-git-dir registration never uses legacy serve.json. Rawpassthru: trueis rejected; CLI filesystem-scan parity, including nonmatching files, remains unchanged. Malformed/incompatible requests never return empty successes. README, SHARED_WORKTREE_INDEXES.md and AGENTS.md document the complete contract.Validation and final review
--lockedchecks preserve both lockfile hashes. Tested production/install inputs match published inputs; final lock-only head binaries are byte-identical to independently tested 3e89778 binaries.Sharing and deliberate limits
Three identical tracked files require three reads/decodes and zero private extractions; a linked worktree adds its ordinary
.gitpointer as one private extraction. Four CRLF-transformed paths plus the pointer need five initial reads/decodes/extractions, zero new extractions on no-op verification. Views reuse the same immutable base; newer generations reuse unchanged content without repinning existing views. Sharing does not eliminate reads or transformation costs.V1 supports the default raw-auto tracked-regular-file 64 MiB profile. Storage is trusted/external, with separate generation/overlay children. Defaults: 32 views, 256 leases, 8192 native subscriptions, 16384 hint slots; work queue=max views, query queue=16, incoming queue=32. Requests cap at 1 MiB, encoded responses at 64 MiB; this is not a total-process memory cap.
Native subscription failure/budget exhaustion falls back to polling. Automatic full verification defaults to 120 seconds after completion. Failed repairs back off 1/2/4/8/16/30 seconds; explicit refresh bypasses/resets this. No-watch needs runtime refresh after initial verification. Readiness/hint acknowledgments are not filesystem-wide atomic freshness guarantees. No content cache, online GC, checkpoint eviction, automatic repinning or lease expiry.