Second file-backed Fluree on a live root can block forever in recover_wal (grp_index test hang) #2001

What happens

it_indexing_workflow::file_based_indexing_then_new_connection_loads_and_queries (in grp_index) hung for more than 18 minutes at 0% CPU during a full cargo test -p fluree-db-api --no-fail-fast run, on a branch based on origin/main at bf523e24e. Run alone, it passes 3 out of 3 times, but each run takes 28–37 s, and most of that is spent waiting.

Where it was stuck

The test thread's stack, from sample:

FlureeBuilder::build
  FileStorage::recover_wal
    futures_executor::block_on(RwLock::write_owned())
      std::thread::park

Other live threads: fluree-index-rebuild ×2, fluree-cas-helper ×2, fluree-wal-flush.

Suspected cause

Not confirmed. I didn't pin down which reader holds the gate.

• wal::operation_gate is one process-wide tokio::sync::RwLock per canonical storage root (fluree-db-core/src/storage/wal.rs).
• Every file operation holds the read side for its whole duration (begin_operation, fluree-db-core/src/storage/file.rs).
• recover_wal takes the write side with futures::executor::block_on, which parks the calling thread.
• The test opens its second instance, FlureeBuilder::file(path).build(), inside local.run_until(...) on a #[tokio::test] current-thread runtime. The first instance's background indexer is also running on that runtime.
• If a read guard belongs to a task that can only run on that same thread, the writer never gets the lock. tokio's RwLock is fair, so once the writer is queued, every later reader queues behind it as well.

The passing runs taking ~30 s suggest the writer waits a long time for readers to drain even when it does get through.

This has the same shape as the CI hang fixed in #1357: a blocking wait parked a thread that another task needed in order to make progress.

Reproduce

• Intermittent, under parallel load: cargo test -p fluree-db-api --test grp_index
• Alone (passes, but slowly): cargo test -p fluree-db-api --test grp_index -- --exact it_indexing_workflow::file_based_indexing_then_new_connection_loads_and_queries

Possible directions

• Make recovery await the write lock, or run it under spawn_blocking, so it never parks a runtime thread a reader needs.
• Find out what still holds the read side for about 30 s after an index build has completed.
• In the test, stop the first instance's indexer, or drop the instance, before opening the second.