Support indexed hidden-file searches while preserving ignore rules - #158

Summary

Make index and serve include hidden, non-ignored files by default, while ordinary searches continue to apply ripgrep-compatible hidden-file filtering. serve --hidden is accepted as a redundant flag; no new server startup flag is required.

This fixes the indexed-search performance regression for Copilot's existing arguments, including --index-path <index> --files-with-matches --hidden --glob !.git --with-filename -- needle <root>. Content, filename-only, count, JSON, and --files queries can use compatible local indexes or the server without dropping hidden matches or disabling ignore rules.

• Apply shared visibility before candidate limits/counts/output selection, respecting explicitly requested hidden roots and Windows hidden attributes without per-candidate filesystem probes.
• Carry the broader corpus through builds, resumed checkpoints, reloads, incremental publication, native watching, polling, and startup reconciliation. Discover and update ignore rules inside hidden directories; preserve .gitignore/.ignore and other existing ignore semantics.
• Exclude the actual configured index directory and its staging/retired generations, including custom storage inside the source tree. Preserve existing Windows generation retirement and rollback handling.
• Reconcile descendants when a native directory rename/removal reports only the directory, so old content/cache and filename entries do not survive.

Compatibility and migration

Legacy, incomplete, or mismatched coverage metadata falls back to a filesystem scan. A current server reconciles older indexes and publishes hidden coverage only when it has established the eligible corpus; older servers must explicitly confirm support before a new client renders indexed results.

The new file-table format prevents older local readers from silently exposing hidden results by ignoring visibility metadata. Filename-only membership is published atomically with its visibility and file-table identity, including coherent startup hydration after interrupted publication. New readers retain legacy-format migration support. Downgrading to an older binary requires rebuilding its index, preferably at a separate --index-path.

--hidden never implies --no-ignore. Positive glob overrides intentionally scan because they can reinclude ignored files absent from the default index; negative-only globs such as !.git remain indexed. Other unsupported corpus-widening flags retain their scan fallback. README and agent guidance document these boundaries.

No package version bump, runtime-repository change, tag, or release is included.

Validation

• Workspace unit tests: 261 CLI passed, 1 existing ignored; 257 core passed.
• Related integration tests: 362 passed — ripgrep_compat 230, ripgrep_parity 102, indexed_hidden 8, indexed_files 4, concurrent_search 7, memory_cap 1, watcher_dot_ignore 2, watcher_watch_registration 6, warm_start_gitignore 1, and serve_max_filesize 1. The final committed eight-test indexed_hidden target was rerun successfully.
• Focused regressions cover real Copilot arguments and backend diagnostics, ordinary visibility, ignored hidden files, legacy/incomplete coverage and migration, custom storage, Windows attributes, hidden updates/ignore transitions/restarts, interrupted filename publication/startup hydration, native directory rename, and retained publication rollback assertions.
• cargo clippy --workspace --benches --locked --quiet -- -D warnings, cargo build --workspace --benches --locked --quiet, cargo fmt --all -- --check, and Git whitespace checks passed.
• Independent acceptance against the frozen binary: 82 parity/lifecycle cases (including native directory creation/rename/deletion with an active watcher) and 10 exact v1.0.8 reverse-client local/server cases, all passed. Parent acceptance checked content/-l/-c routing and --files parity; Rust regressions separately assert filename backend diagnostics.