wasm32 groundwork: the engine stack compiles and runs on wasm32-unknown-unknown, native untouched - #1714

scripts/wasm-clang.sh / wasm-ar.sh pick a wasm32-capable clang per platform
(Homebrew LLVM on macOS where Apple clang has no WebAssembly backend, system
clang on Linux/CI); .cargo/config.toml routes zstd-sys through them instead
of a hardcoded macOS path. New ci.yml wasm32 job checks the full engine
stack and the no-default-features parse layer for the wasm target.

Three shared foundations for making the wasm32-unknown-unknown build
survive at runtime (std::time clock reads and moka cache construction
abort there):

- fluree-db-core::clock — Instant/SystemTime that work on every target.
  Native neutrality: on native these are plain re-exports of std::time
  (zero-cost type aliases; call sites compile to exactly what they had),
  wasm32 gets web-time (performance.now/Date.now via wasm-bindgen).
  web-time is already in the wasm dependency graph via ulid, so no new
  transitive deps.

- fluree-db-core::wasm_cache — clock-free moka::sync::Cache stand-in
  (byte-weighted exact LRU over the lru crate, uncontended parking_lot
  Mutex — cheap on a single-threaded target). moka 0.12 constructs its
  eviction clock from std::time::Instant::now() inside Cache::builder(),
  which aborts on wasm32. None of the wasm-reachable caches use TTL/TTI,
  so LRU-with-weigher is a faithful replacement. Compiled on native only
  under cfg(test) so its unit tests run there; native builds never link
  it (lru is a wasm-target + dev dep only).

- chrono 'wasmbind' at the workspace dep: activates only chrono's own
  wasm32 cfg branch (js-sys/wasm-bindgen are target-gated optional deps
  inside chrono) so Utc::now() reads Date.now() instead of aborting in
  SystemTime::now(). Native cargo tree for chrono verified byte-identical
  before/after.

…efault

std::time::Instant::now()/SystemTime::now() abort on wasm32-unknown-
unknown. Every reachable call site moves to fluree_db_core::clock —
which on native re-exports std::time, so these are import-line swaps
with zero native codegen change; the timing instrumentation itself is
kept (performance.now works on wasm via web-time):

- query operator driver (execute/runner.rs — fired on EVERY query) and
  the sort/groupby/aggregate/group_aggregate/optional/join drain timers,
  datalog fixpoint timing, COUNT fast-path timers
- binary-index dict reader slow-lookup timing
- novelty bulk_apply_commits timing (ledger-load path)
- core tracking (opt-in track_time)
- transact generate_txn_id: SystemTime via the shim — the previous
  unwrap_or_default() could not help because the panic is inside now()
  itself; fires on every staged transaction
- api ledger_manager idle-TTL stamps (monotonic_secs, entry touch) and
  compaction timing

ledger_manager also gets a cfg'd default_cache_dir(): the previous
std::env::temp_dir() default PANICS on wasm32 ("no filesystem on this
platform") and runs inside FlureeBuilder::memory(), before any build
method. The wasm synthetic path is never touched at runtime — the disk
cache opens with a zero write budget there and reads map Unsupported to
a miss.

moka 0.12 aborts at Cache::builder()/new() on wasm32 (std Instant clock
origin), which made every Fluree init path panic: the leaflet cache,
GovernanceCache, and R2rmlCache are all constructed inside
finalize_with_backend, and the ontology-imports + fulltext caches on
first use.

Each seam shadow-imports fluree_db_core::wasm_cache::Cache under
cfg(target_arch = "wasm32") — one cfg'd use line (or a SyncCache alias
where the moka path was spelled inline). Native neutrality: the native
arm of every cfg pair is the exact moka type as before; no wrapper, no
dyn. The seams' moka API surface (get/get_with/try_get_with/insert/
contains_key/invalidate_all/run_pending_tasks/weighted_size/entry_count/
policy().max_capacity()) is mirrored by the stand-in, including
try_get_with's Arc<E> error type.

moka itself moves to cfg(not(wasm32)) target deps in binary-index,
query, and api — native resolves identically; the wasm artifact drops
moka entirely (smaller binary, and no accidental future construction).

wasm-bindgen-futures lands in api's wasm target deps here as well; its
consumer (the spawn_detached seam) arrives in the next commit.

…beat cfg

wasm32 has no ambient tokio runtime (tokio::spawn panics "must be
called from within a runtime") and cannot spawn OS threads
(std::thread::spawn panics). Per-site treatments, chosen so the native
path keeps its exact concurrency shape:

- api::wasm_compat::spawn_detached — #[inline] tokio::spawn on native
  (same discarded JoinHandle as before), wasm_bindgen_futures::
  spawn_local on wasm32. Applied at the init-path cache event listener,
  which fires on every build with ledger caching on (the default).

- cypher_typed prefetch fallback: the native chunked tokio::spawn
  fan-out is unchanged inside cfg(not(wasm32)); wasm32 runs the same
  point reads inline (single thread — the fan-out buys nothing, and
  available_parallelism there reported a fictitious 4).

- transact PendingRawTxnUpload (opt-in store_raw_txn): wasm32 twin
  defers the upload as data and runs it inside finish(); nothing is
  stored before that, so abort/Drop have nothing to release and the
  native Drop-guard/abort machinery isn't needed. Native struct and
  impls are byte-identical inside their cfg.

- multi-query dispatcher: native parallel JoinSet + semaphore + deadline
  path unchanged inside cfg(not(wasm32)); wasm32 twin runs sub-queries
  sequentially with no deadline enforcement (JoinSet::spawn would panic
  and tokio::time aborts at its first clock read). The per-sub-query
  size check is factored into success_or_size_error, shared by both —
  the one native-code refactor in this series (once-per-subquery call,
  not hot-loop).

- batched_lookup progress heartbeat (telemetry only): native-only cfg
  block; its file-level Instant import moves to core::clock (the scan
  timer at the top of the lookup was another every-call abort).

dict_io.rs / binary-index lib.rs / tx.rs hunks are rustfmt import-order
corrections to spike-branch files (this branch now passes cargo fmt
--check as a whole).

On wasm32-unknown-unknown every std::fs call fails with
ErrorKind::Unsupported. try_read_cached_bytes mapped only NotFound to a
miss, and fetch_cached_bytes/_cid apply ? to its result BEFORE
attempting the CAS fetch — so on wasm the pre-checks propagated
Unsupported as the read's error and the authoritative fetch was never
reached: every binary-index artifact load failed outright. (The
coalesced_fetch leader path already had the correct tolerant shape;
only the outer pre-checks were wrong.)

Fix at the single choke point: map Unsupported to Ok(None) alongside
NotFound, extracted as read_result_as_cache_outcome so the mapping is
unit-testable with synthetic errors on native. Real I/O failures
(EIO, EACCES) still propagate. Correct on all targets — native gains
nothing but loses nothing (Unsupported from a local fs read means the
cache dir can't serve reads; falling through to the remote fetch is
exactly what the coalesced leader already did in that case).

Writes were already safe: budget 0 on wasm (create_dir_all fails →
writes disabled) and best_effort_write early-returns at budget 0, so
eviction/scan paths never run there.

On wasm32 there is no sync->async CAS bridge (single thread, no block_on), so
the Tier-1 spike stubbed `run_sync_on_runtime` to a flat error: a cache miss
failed the query with no way to recover. Give the sync read path a way to
report *what* it needs instead.

`ContentStore::resolve_cached_bytes(&ContentId) -> Option<Arc<[u8]>>` is the
new sync residency tier: default `None`, forwarded through the `Arc<dyn>`
blanket impl and `BranchedContentStore`. On a hit it is one hash lookup
returning a shared `Arc` clone — no copy, no I/O, never blocks. On a miss the
read path raises `NeedFetch { cid, kind }`, carried inside `io::Error` as a
custom payload so no accessor signature changes: the same channel the fuel
tracker already tunnels through the sync cursor. An async caller above the
operator boundary catches it (`from_io_error`, or `find_in_chain` for wrapped
errors), fetches the named object, and re-runs.

All five CAS-backed read families get a wasm arm that consults residency and
bypasses the filesystem tiers — those probe `std::fs`, which returns
`Unsupported` on wasm and would propagate as a hard error rather than falling
through to a fetch: leaf blobs and history sidecars (via a zero-copy
`SharedBlobLeafHandle`), leaf byte ranges, dict-tree leaves, forward packs,
and vector shards. Reaching the bridge stub now means a read path gained a
CAS call without a residency arm, so it stays an error rather than becoming
unreachable.

Native is untouched by construction: each accessor cfg-selects between its
existing body (renamed `*_native`, unchanged) and the wasm arm, so there is no
added branch, no dynamic dispatch, and no layout change on the hot leaf-open
path. `NeedFetch` is constructed on zero native paths.

Tests drive the miss/fetch/retry round trip natively against a miss-injecting
content store, asserting the miss actually fires, names its CID, walks a
wrapped `source()` chain, and returns the same allocation on a repeat hit.

`next_batch` advanced its cursor state before the fallible reads, so an error
lost data instead of being retryable:

- The leaf open ran after `current_leaf_idx += 1` and overlay slicing, so a
  failed open skipped that leaf entirely on any later call.
- Worse at leaflet granularity: the open leaf is `take()`n for the duration of
  the loop, so a `?` on the column load or the history-sidecar read dropped
  the leaf handle along with every remaining leaflet in it.

Neither was reachable as data loss on native today (a failed CAS read fails
the query), but both block retry-after-fetch — the recovery model wasm needs,
where a miss is normal and the caller fetches the named object and re-enters.

Open the leaf before advancing, and isolate the leaflet's fallible reads so a
failure rewinds the leaflet index and restores the leaf before propagating.

The test measures a remote scan's CAS call count, then replays the scan once
per call position against a store that fails exactly that call, requiring
exactly one surfaced error and zero lost rows each time. Both fixes were
mutation-checked against it: restoring the advance-before-open loses rows at
call 1, dropping the leaflet rewind loses them at call 4.

Found by RUNNING the wasm probe under node (wasm-pack --target nodejs +
manual poll loop), not by the static audit: with every other fix in
place, build_memory/create_ledger/insert all survived and the trap
moved to Fluree::query_with_options — six std::time::Instant::now()
sites in view/query.rs (parse/plan/exec timers on both the
QueryInput and the reasoning-override arms).

These are the api-layer counterparts of the fluree-db-query operator
timers already routed through the shim; the audit's api sweep missed
them because it filtered by native-gated MODULE and these live in an
un-gated one. Same treatment, same native neutrality: on native
core::clock::Instant IS std::time::Instant.

With this the probe completes the full path in a real wasm runtime —
FlureeBuilder::memory().build_memory() -> create_ledger -> JSON-LD
insert -> SPARQL query returns Ok on the first poll, in both the dev
and release builds.

Each item is attribution, not behavior: Durability's env/sync helpers cfg-allow
dead when the native FileStorage that consumes them is compiled out; the
connection File-storage arm becomes cfg-if instead of early-return (unreachable
statement); DT_ID's re-export follows its only consumers behind the import
feature; two api helpers whose callers are native-gated get matching gates.
wasm32 api check and native workspace check both clean.

run-node.mjs builds the probe with wasm-pack (--target nodejs), appends a
raw-exports handle to the generated glue (the nodejs wrapper re-exports only
wasm_bindgen items and the probe's entry points are deliberately raw
extern-C), instantiates under Node, and drives probe_start + probe_poll to
completion.

Exit 0 requires the positive ran-marker — poll code 1, the probe future's
own Ok — never "didn't crash": Pending exhaustion, Err, not-started, and
traps all exit non-zero. The failure lane is exercised (poll-before-start
returns -2 and would exit 1).

wasm-pack's bundled wasm-opt pass is disabled for the release profile via
package metadata: installed wasm-opt binaries can reject bulk-memory output,
and correctness is this harness's job — size measurement keeps its own
explicit wasm-opt invocation (README).

Run at the wasm/groundwork merge (shims + stage0): dev and release both
PASS — build_memory -> create_ledger -> JSON-LD insert -> SPARQL query
returns Ok on the first poll under node v22.

…ut dirs

The probe is becoming the CI smoke test — a fresh clone must resolve the
same dep graph the harness was verified against. pkg-node-{dev,release}
are generated by run-node.mjs and stay untracked.

… rewrite

The tier-1 spike replaced .cargo/config.toml wholesale with the wasm
clang wiring, deleting the workspace [env] RUST_MIN_STACK = 8MB setting
whose comment names proxy_integration as a suite whose debug-build async
chains overflow the 2MB default test-thread stack. Result: on the spike
base (before any later commit) test_remote_mount_queries_and_write_rejection
aborts deterministically with a stack overflow, while untouched main
passes — verified three-point (main pass 3/3, spike base fail 3/3,
this branch fail 3/3, all isolated runs).

Merge both concerns: the original [env] block is restored verbatim with
the spike's wasm CC/AR entries appended and the wasm32 target rustflags
kept unchanged.

…ning)

Review finding: the wasm32 job was compile-only, so the entire runtime-hazard
class this branch fixes (clock panics, thread spawns, moka construction) could
regress unseen. The job now installs wasm-pack and runs wasm-probe/run-node.mjs,
which exits 0 only on the probe future's own Ok.

Two gaps kept the sync residency tier from working end to end:

The typed `NeedFetch` error was the only miss-reporting channel, and it is
severed before any retry frame can see it — `QueryError` has no io-carrying
variant, `from_io` stringifies, and ~200 query-crate sites format errors
into strings. The new `storage::residency` module makes the STORE the
carrier instead: a `MissRegister` on the content store records every
`(ContentId, FetchKind)` want at miss time, deduplicated, and a retry frame
reacts to ANY execution error by draining it — non-empty means fetch the
wants and re-run, empty means the error was real. Immune to every present
and future error conversion. Callers that know their unit's whole routed
want set record all of it, so one retry round learns N objects, not one.

The hook was also unreachable through the assembly path a browser peer
uses: `build_with(storage)` wraps `StorageRead` in `StorageContentStore`,
which never overrode `resolve_cached_bytes`, and `StorageRead` had no
residency method to forward to. `StorageRead` now carries the same
CID-keyed `resolve_cached_bytes` (no address formatting on the lookup) and
`miss_register`, forwarded through `Arc<dyn Storage>`,
`StorageContentStore` (mirroring `resolve_local_path`), `Arc<dyn
ContentStore>`, and `BranchedContentStore`.

The participation contract lives on `ContentStore::miss_register`:
returning `Some` switches the sync read path to residency-only serving, and
obliges the store to pin bytes returned by `get` (fetch-pins), which is
what makes retry progress monotone and progress-based termination sound.

…ing, retry primitives

Reworks the read tier's wasm arms around the store-level miss register and
makes the residency machinery natively testable:

- The residency arms are no longer wasm-only: every dispatch site compiles
  under `cfg(any(wasm32, feature = "residency"))` and engages at runtime
  only for a store that exposes a miss register (`residency_mode()`), so a
  native test with a residency-tier store drives the exact read path wasm
  gets. Production native builds keep the feature off and compile the
  pre-residency path unchanged; wasm without a residency store falls
  through to the native body rather than a dead end.

- Multi-want reporting (one round learns N objects, not one): a cursor
  leaf-open miss registers the scan's whole remaining routed leaf set (and
  sidecars when replaying); the three `batched_lookup_*` probes register
  every non-resident leaf their routed chunk ranges will touch before
  opening anything; `resident_or_need_fetch` records each individual miss.

- Async retry primitives in `need_fetch`: `fetch_wants` (concurrent
  fetch-and-verify against the fetch-pins contract) and `RetryBudget`
  (drain → fetch → re-run with progress-based termination — a round must
  pin at least one new object — plus a sanity cap). These are the drain
  API the api-level loop and the browser driver build on.

- `leaflet_cache` loader errors preserve a typed `NeedFetch` through the
  `Arc<io::Error>` re-wrap instead of stringifying it (the `open_leaf_dir`
  dir-load closure is the path that cannot resolve residency before
  entering the loader; everything else should, and the dict-leaf path
  does).

- The module docs no longer claim the typed error is catchable above the
  operator boundary — it is not (real wrappers stringify); the register is
  the contract and the typed payload is best-effort for io-preserving
  frames and log messages.

Tests: register round trip (miss recorded, drained, fetched, re-run hits,
zero-copy pin), want-set dedupe and one-round multi-fetch, no-progress
stall surfacing as an error, and — feature-gated — a two-leaf cursor scan
whose first miss must register BOTH routed leaves and complete after one
fetch round (mutation-checked: dropping the whole-set registration fails
it on the single-CID register).

The scan operator's cursor loops (`next_batch` and `drain_count`) gain an
in-frame drain/fetch/retry arm under `cfg(any(wasm32, feature =
"residency"))`: on a cursor error with a register-bearing store, the async
frame drains the miss register, awaits the fetches (concurrently, the
unit's whole want set), and re-enters the SAME cursor — which the earlier
re-enterability fix made safe — so scans recover in place with no
whole-query re-run and no wasted decode. One-shot paths (fast paths,
policy sub-queries) still surface the error and recover through the outer
drain loop.

Belt-and-braces typing alongside the register: `QueryError::NeedFetch`
carries the typed payload where the `io::Error` survives, downcast in
`from_io` exactly like the `FuelExceededError` precedent, and the 30
`Internal(format!("leaf open: {e}"))` sites across the fast paths now
route through `from_io` (same message when the payload is absent). The
register remains the load-bearing channel — these conversions only improve
what io-preserving frames and logs see.

Also: the wasm `Pattern::S2Search` arm returns `UnsupportedFeature`
instead of `Internal` (it is a capability gap, not a bug), and the
`wasm_compat` module docs stop calling themselves spike artifacts — they
are the canonical native import path for the re-exported types.

…embly

Drives REAL queries through a residency-mode storage backend assembled the
way a browser peer is built — `FlureeBuilder::memory().build_with(storage,
NameServiceMode::ReadOnly(..))`, storage behind `StorageContentStore` — so
the hook forwarding, the register, the operator-frame retry, and the outer
drain/fetch/re-run loop are all exercised against a plain native instance
as ground truth (identical results required):

- operator path: completes with leaf misses consumed IN-FRAME by the scan
  operator (the outer loop must see no index-leaf wants — mutation-checked
  by disabling the frame arm);
- fast path (predicate COUNT): the first attempt must surface the miss
  (one-shot, no in-frame retry) with the want already in the register
  (mutation-checked by disabling register recording), then recovers
  through the outer loop;
- dir-only walk (COUNT(DISTINCT ?s), `open_leaf_dir` inside the
  leaflet-cache dir-load closure — the stringification point): registers
  and recovers; a whole-graph COUNT(*) cannot exercise this path because
  it is answered from index stats without touching leaves;
- policy-filtered query (f:query sub-query): the policy layer stringifies
  errors, so recovery works only because the register, not the error
  chain, carries the wants.

Every test asserts a positive miss marker; recovery is never inferred from
the absence of an error. The `residency` feature is enabled through
dev-dependencies only — production native builds keep the pre-residency
read path.

Internal pre-review of #1714 (approve-with-notes; findings applied on merit):
- MissRegister documents its one-draining-query-per-store scope — a future
  multi-query driver must serialize drains or scope wants (M-1)
- the wasm32 CI job now gates warnings via -D warnings in its restated
  RUSTFLAGS; cargo check alone exits 0 on warnings (M-2)
- the node probe asserts the inserted names round-trip instead of
  accepting any Ok (L-1)
- spawn_maintenance and the bm25_worker module are native-gated: they can
  only spawn onto an ambient tokio runtime (L-2)
- the core uuid-unification comment names the real wasm uuid consumers
  (query UUID()/STRUUID(), policy) instead of native-only moka; wasm_cache
  documents its TTL-free boundary

The L-1 assertion checked Debug output, which renders encoded bindings —
the literal names never appear there, so the previous commit's probe
failed. Format to SPARQL JSON (via the view's snapshot) and assert both
inserted names, verified passing with an unmasked exit code this time.

CI's clippy (--all --all-features --all-targets -D warnings) failed on
five workspace-denied manual_assert sites in it_residency_retry.rs.
They were invisible to the local per-crate runs that reported clean: a
clippy invocation over an already-checked tree can be a fingerprint
no-op that prints nothing and exits 0. Verified this time by grepping
the pattern to zero, forcing a re-check, and running the CI-shaped
workspace invocation with -D warnings (minus the vector feature, whose
usearch/cxx build is broken on this box).

Suite still 4/4.

…e call per use

`residency_mode()` asked `dyn ContentStore::miss_register()` every time — a
vtable call, at six sites, up to ~3 times per leaf open. The answer is a fixed
property of the store: `cas` is assigned once in each constructor and never
reassigned, and the field is private, so it cannot change under us. Compute it
at construction and read the bool.

Second reason, beyond the call itself: it takes dyn dispatch out of the
`residency`-enabled build. Per the dev-dependency feature unification noted in
fluree-db-api/Cargo.toml, that is the build every api bench compiles, so this
narrows the gap between what the benches measure and what production ships.

The field carries the same cfg as the method, so a production native build has
neither. Verified in all three configurations: default native (field absent),
`--features residency`, and wasm32.

…e missed

The wasm32 CI job runs `cargo check`, and the local clippy gate only ever
targeted the two browser crates — so no gate ever ran `clippy --target
wasm32` over the engine crates, and two lints sat red under it:

- binary_index_store.rs run_sync_on_runtime: `let _ = fut` trips
  clippy::let_underscore_future (a dropped, un-awaited future). The drop is
  intentional — there is no runtime to drive it on wasm and reaching the stub
  is itself the bug — so make it an explicit `drop(fut)`.
- connection/src/lib.rs (two File-storage arms): `return Err(...)` inside the
  non-native cfg block is needless once the native sibling is cfg-stripped on
  wasm, so it becomes the arm tail. Baseline code; only a wasm-target clippy
  pass sees it.

A follow-up commit adds the wasm-clippy gate that would have caught both.

…here is no filesystem

The wasm read path already treats a cache read as a miss and suppresses
every cache write, but both index loaders still opened with an
unconditional create_dir_all. On wasm32 that returns Unsupported, so a
browser peer's first ledger open died with 'failed to load binary index:
operation not supported on this platform' before a single CAS fetch was
attempted. Found by the browser-peer smoke against a real server.

…progress on the blocking object

Two coupled defects in the wasm/residency read path, both surfaced by an
adversarial review of the retry machinery:

- register_remaining_wants recorded the WHOLE remaining routed leaf set on one
  leaf-open miss. A LIMIT-shaped scan that reads one leaf and stops therefore
  pulled the entire predicate run over the network first, and a byte-budgeted
  resident tier turned it into a livelock: each retry round re-fetched a tail
  the tier had just evicted, so newly_resident>0 held every round while the
  cursor never advanced — only the sanity cap stopped it, after thousands of
  wasted fetches. Now records a bounded read-ahead window (32); the consumer
  stops pulling at its LIMIT, so only the first window is ever fetched.

- RetryBudget::after_error treated 'some want became resident' as progress.
  Under a tier too small to hold one round that is satisfiable forever without
  the cursor advancing. Now gates Ok(true) on the specific blocking object
  (the first-recorded want; MissRegister is insertion-ordered) being resident
  after the fetch — a finite, monotonically-advancing guarantee. A tier that
  cannot hold even the blocking object fails fast and names it, instead of
  spinning to the cap.

Tests: a 40-leaf scan registers exactly the 32-leaf window (mutation-checked
against an inflated window); the non-pinning store fails on round 1 at the
blocking-object gate, not by spinning to the cap.

std::time::Instant::now() TRAPS (unreachable) on wasm32 rather than returning
an error, and the OWL2-RL fixpoint runs on a query path reachable ungated from
wasm (reasoning_prep -> reason_owl2rl -> run_fixpoint). clock::Instant aliases
web_time on wasm and std::time on native, so this is a no-op off-wasm.

…it surfaces

The wasm32 job ran 'cargo check', which cannot see clippy-only lints, and no
other job runs clippy against the wasm target — so a class of wasm-only lints
could rot undetected (this stack shipped three: a dropped future in the CAS
bridge stub, a needless return in a wasm-cfg storage arm, and an explicit
into_iter loop in the sequential multi-query twin). Run clippy instead; it
lints every workspace member in the -p closure at lib level. No --all-targets:
it drags in fluree-db-api's dev-deps, whose tokio net/mio does not build for
wasm32. Fixes the last of the three lints (dispatch.rs) here.