Skip to content

Avoid watcher reindex churn from virtualized file hydration #131

Description

@shengyfu

Problem

On a warm tgrep serve index over a Windows virtualized repository, a read-only search can trigger dozens of watcher reindexes for distinct candidate files. In the reported 286,937-file Substrate enlistment, searching for pattern resolved 13,426 candidates and was followed by many reindex: modified ... messages even though the user made no edits.

Windows virtualized files can hydrate their primary data stream when read. notify 7 reports the resulting FILE_ACTION_MODIFIED notifications only as generic Modify(Any), so the watcher cannot distinguish hydration from a real same-size, timestamp-preserving write. Dropping attribute/modify notifications or suppressing recently searched paths would therefore risk stale results.

PR #130 suppresses exact duplicate events once a path has a live-overlay entry, but a reader-only path has no in-memory byte evidence before its first hydration event. Its first read-induced event still creates an unnecessary overlay mutation, log line, and possible save.

Proposed solution

Persist a bounded content identity with each indexed file's existing filestamps.json record:

  • Compute BLAKE3-128 from the exact decoded text used to build trigram postings.
  • Preserve the existing FileStamp Rust API and metadata comparison semantics; the content ID is separate optional evidence in the serialized record.
  • Populate evidence in full, delta, resumed/batched, and live-reindex paths.
  • On a forced event for a reader-only path, suppress the overlay commit only when the newly decoded content identity exactly matches the persisted identity and the current reader still owns the path.
  • Continue invalidating decoded-content cache state and refreshing metadata on a suppressed event.
  • Missing, legacy, corrupt, or stale evidence must fail open and reindex.
  • Invalidate old persisted evidence before publishing new core index files, publish new evidence only after the complete core set, and restore it on transaction rollback.

Correctness requirements

  • Preserve detection of real same-size/same-timestamp content changes.
  • Never use a time-based cooldown or generic metadata equality as content proof.
  • Never suppress when a live overlay or tombstone supersedes the reader.
  • Withhold evidence for files reported during the initial build's race window, just like current stamps.
  • Cover full builds, memory-cap checkpoints/resume, stale delta merges, reloads, empty-index recovery, and interrupted publication.

Validation

  • Reader-backed identical forced event causes no overlay mutation or reindex: modified log.
  • Same metadata with different bytes still commits.
  • Legacy/missing/corrupt evidence reindexes.
  • Unchanged IDs survive stale merges; changed IDs are replaced; deleted/binary/unreadable IDs are removed or withheld.
  • Interrupted publication cannot pair old evidence with a new reader generation.
  • Re-run the reported virtualized-repository search and a real same-size rewrite using a release build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions