fix(query): compare terms by what they are, not how they're encoded - #2010

…triple

A single-triple OPTIONAL substituted a correlated value into its pattern
only when the value was decoded, plus an encoded subject in the subject
slot. Any other encoded value (late-materialized: an indexed ledger with
no live novelty, or a batched join lane at any time) was left a variable.
The OPTIONAL then scanned the whole predicate and left the correlation to
`unify_check`, which compares with `==`:

- an IRI reached as a predicate (`EncodedPid`) and met as a subject or
  object (`EncodedSid`), or the reverse, compares unequal, so the match
  was silently dropped (`ex:s1 ?p ?o . OPTIONAL { ?p rdfs:label ?l }`
  never found a label);
- an encoded object from a batched join against the scan's decoded
  object (novelty pending) compares unequal too, dropping every match;
- where the forms agreed the answer was right but cost rows x the
  predicate's size, and the scan was cached under the one key every row
  shared (#1973: a 13 s delete).

The nested-loop join already decoded every encoded form into every slot.
Its per-slot substitution is now a shared function both operators call,
so the two copies cannot drift again; the join's behaviour is unchanged.
OPTIONAL uses it, answers no match without a scan when a value cannot
fill its slot (a literal as a subject), and refuses an encoded value it
cannot decode rather than leave the slot free.

The scan cache key follows: every value substitution binds keys by its
identity in every position (encoded values by their encoded id), so two
rows share a cached scan only when they would run the same lookup. The
unit test that pinned an encoded object keying as free now pins the
opposite.

The scan encodes one IRI twice: as `EncodedPid` where a pattern reaches
it as a predicate, as `EncodedSid` where it is a subject or object.
Decoded producers carry it as `Sid`/`Iri`. `normalize_for_key` mapped
decoded IRIs to `EncodedSid` but left `EncodedPid` alone, and the hash
join's `join_key` kept its own copy of the rule, so on the encoded lane
one IRI met as a predicate and as a subject was two terms to DISTINCT,
GROUP BY, MINUS, EXISTS / NOT EXISTS, a trailing VALUES and hash joins:
duplicate rows, split groups, MINUS removing nothing, EXISTS finding
nothing. (`Iri` against `EncodedSid` also keyed apart in the hash join.)

One rule now, `canonical_iri_id`: an IRI keys by its persisted subject
id when it has one, else by its predicate id. `encoded_equivalent` and
`join_key` both go through it. The store memoizes each predicate's
subject id on first use, so a key over `EncodedPid` values pays a load
per row; an all-`EncodedSid` key and a decoded key with a subject id
take the same path as before.

OPTIONAL's `unify_check` compares with `same_term`: equal or same-form
values answer as before, and only a pair of different forms is
canonicalized, through the same normalization.

…ST probe

`?s ... OPTIONAL { ?c <p> ?s }` correlates on the OPTIONAL triple's
object. With the substitution fix each required row ran its own bound
lookup; now the whole required batch is answered by one sorted pass over
OPST, binding the optional-only subject, like the subject-keyed probe
does for `OPTIONAL { ?s <p> ?o }`. The required side is coalesced into
that one probe when the lane is admitted.

The OPST walk is the nested-loop join's bound-object lane, extracted
into `for_each_object_probe_match` and shared rather than copied: one
leaf read per touched leaf, a binary search per object inside each
leaflet, and the same novelty merge (`ObjectProbeOps`). The join's lane
calls it with its existing emit path; its rows and fuel are unchanged.

The OPTIONAL lane is admitted under the same gates as the join's
(single ledger, encoded output allowed, a mergeable overlay, no history
range or policy-filtered predicate) and declines to the per-row lookups
when a row's object is not a ref with an id (unbound, a literal, an IRI
the dictionaries do not hold). It stamps `optional_object_probe` so a
test pins that it answered.

An EXISTS inside an expression (`FILTER(EXISTS {...} || ...)`) is
answered from a cached subject set of its predicate when the index has
no live novelty. The probe took only a decoded `Sid`, but in exactly
that state the scan emits subjects encoded, so every row declined and
ran the EXISTS body as a seeded subplan. An `EncodedSid` carries the
persisted subject id the cache holds, so it now probes directly.

The test counts the seeded subplans a query runs (a nested-loop join
opened per declined row): three before, none after, on the fresh and
the adopted-index states.

The per-row path returns a batch as soon as one required row's matches
are drained, so each holds a row or a few, but every column was
allocated with `batch_size` (1,000) slots. Any consumer that buffers the
output kept all of them: #1973 measured 160-171 KB per required row, a
43 GB OOM on a 61k-row vocabulary page. Columns of a batch under half
full are now shrunk to their rows before the batch is built.

The batched lanes still fill whole batches; this only changes the
small ones.

Adds COUNT(DISTINCT), JSON-LD not-exists, groupBy and values, and a
Cypher count over OPTIONAL MATCH to the mixed-form cases, all with
hand-written expectations in the three index states. Also pins the
surfaces that already treated the forms as one term (inner joins on a
predicate binding or on an object used as a predicate, FILTER = and
sameTerm, a leading VALUES, subquery and group joins, a multi-pattern
OPTIONAL), so the canonical form cannot regress them.

Measured on a 1M-triple ledger (warm server, dev-fast), the canonical
form first cost +9% on GROUP BY / DISTINCT over predicate bindings on
the indexed lane, and +31% with novelty pending, where predicate keys
are decoded `Sid`s: each missed the subject dictionary and then built
its IRI string to ask the predicate dictionary.

- `predicate_subject_id` resolves every predicate on first use (up to
  4,096 of them; lazily per predicate above that) and records whether
  any predicate IRI is a subject at all. Usually none is, so the
  per-row call is an inlined load and a branch.
- A decoded IRI asks the predicate table first: a namespace bit test,
  then a hash probe on the `Sid`, no string building. A decoded
  predicate key no longer pays a subject-dictionary lookup per row at
  all, so GROUP BY / DISTINCT over predicates with novelty pending run
  about 5x faster than on main (300 ms -> 60 ms on the 1M ledger). An
  IRI in no predicate's namespace pays one bit test before its subject
  lookup.

… bound

`?s <p> ?t . OPTIONAL { ?s <q> ?t }` correlates on both subject and
object. The substitution fix decoded the row's encoded object into the
scan as well, a dictionary lookup per row that only narrowed a lookup
already narrowed to the subject; the `query_hot_optional`
object_correlated bench measured it at +3.5%. When the row binds the
subject, an encoded object now stays free in the scan and the
correlation is enforced by `unify_check`, whose `same_term` treats the
encoded and decoded forms of one term as one. The cache key follows the
same per-row decision, so a row keys its object as free exactly when its
scan leaves it free.

Tests add the shape, plus one where the unify must match an object
across forms: with novelty pending the batched joins emit the subject
and the correlated object encoded while the OPTIONAL's scan decodes.

… forms

…arms

`normalize_for_key` and `binding_to_group_key_normalized` now ask
`encoded_iri_canonical` first: an `EncodedSid` returns at once (it is
canonical already), and an `EncodedPid` costs the store's per-predicate
load, before the NUM_BIG check and the decoded arms of
`encoded_equivalent`.

Paired measurement on the 1M-triple ledger (main and fix servers warm
side by side, 15 alternating repetitions per query): GROUP BY / DISTINCT
over subjects and objects within +/-1% of main on both lanes; GROUP BY
over predicates +0.7%, DISTINCT over predicates +2.5-2.9% on the indexed
lane (about 1 ns per row); GROUP BY / DISTINCT over predicates with
novelty pending -79%.

… nodes

Equality surfaces resolved a decoded term to its id through the persisted
dictionaries alone, and `encoded_equivalent` refused blank nodes while the
hash join resolved them. With novelty pending a plain scan decodes every
term, but the batched lanes bind a subject minted since the last index by
its novelty id and a persisted blank node by its subject id. The two forms
of one term then keyed apart: MINUS kept rows it must remove, DISTINCT and
GROUP BY split one term in two, COUNT(DISTINCT) overcounted, and sameTerm
and a trailing VALUES dropped matches. The OPTIONAL bound-object lane made
this reachable from the plainest reverse OPTIONAL.

`TermDicts` is now the one place a query resolves a decoded term to its
id: the persisted dictionary first, then the novelty dictionary, the order
every lane assigns ids in. The canonical IRI id is the subject id when the
IRI has one (persisted or novelty), else the persisted predicate id.
Blank nodes follow the same rule on every surface. A decoded IRI asks the
subject dictionary first again, so a decoded subject key costs the one
lookup it cost before predicates had ids here; the novelty dictionary is
probed only after a persisted miss, only when it holds entries, and the
probe no longer allocates.

Every surface goes through it: `normalize_for_key` (DISTINCT, GROUP BY,
MINUS, EXISTS), the group-aggregate keys and COUNT(DISTINCT), the hash
join's `join_key`, OPTIONAL's unify (`same_term`, its normalization now
built once per operator), VALUES compatibility, BIND onto a bound
variable, and the sameTerm fast path, which compares canonical ids and
hands a miss to the decoding comparison instead of answering false. The
four copies of the persisted-then-novelty subject lookup (OPTIONAL, the
nested-loop join's batched key, the property join, the annotation
enumeration lane) now call it.

…nd the join

A dataset of several graphs of one ledger (`FROM <g1> FROM <g2>`) has no
graph view, while each graph's scan still binds encoded ids. OPTIONAL then
refused an encoded correlation value with an internal error, and the
nested-loop join left the slot free with nothing to correlate it: its
unify instructions were built from right-side variables that exclude
every left variable, so they were always empty, and a value left free
matched every row of the slot (each match repeated once per left row).
The join also ignored a value that can never fill its slot (a grouped,
list or map value as an object) and matched the free slot instead.

`CorrelationView` is what a correlated scan decodes through: the active
graph's view, or, with several graphs of one ledger active, a view over
the ledger's store, through which the ids a ledger shares across its
graphs (subject, predicate, string dictionary) and inline literals
decode. An arena-backed literal handle (NUM_BIG, vector) names a value
only within its own graph and stays free.

The nested-loop join now acts on the substitution outcome: a value that
cannot fill its slot yields no right rows, and every variable the right
batch still binds that the left row carries (a slot substitution left
free) must be the same term on both sides (`same_term`). OPTIONAL leaves
such a slot free for its unify check instead of raising an error. The
dead unify-instruction construction in the join is removed.

…ints

A correlated property-path endpoint resolved `EncodedSid` through the
persisted dictionary only and had no `EncodedPid` arm. A predicate bound
in predicate position (`ex:s1 ?p ?o . ?p ex:parentProp+ ?super`), or a
subject a batched join bound by its novelty id, resolved to nothing, and
the row fell into the full-closure branch, pairing it with every closure
pair. The endpoint now resolves an `EncodedSid` through the novelty-aware
graph view and an `EncodedPid` through the predicate dictionary.

`GRAPH ?g`'s binding extraction gets the same `EncodedPid` arm so it stays
total across binding kinds. No plan is known to reach it encoded: the
`ex:s2 ?g ?o . GRAPH ?g { ... }` case the test pins was already right.

…t must not

Every OPTIONAL case now states what the lane must do, so a lane that stops
firing, or fires where it must not, fails on the same ledger as its twin:
it fires for ref-valued object correlations, declines every window of
literal objects, declines the windows it cannot probe in a mixed stream,
and is never consulted when the OPTIONAL also correlates its subject. The
JSON-LD, UPDATE and Cypher twins assert the lane's stamp rather than any
bound lookup. Inside an UPDATE's WHERE with novelty pending the lane
declines at the shared probe-lane admission (a transaction's WHERE runs
over a one-member dataset), and the test pins that too.

New cases: the lane's novelty merge (a pending retract of a base link and
pending asserts), an UPDATE template and a JSON-LD transaction filled from
a `?c` minted in novelty, and a non-root policy that leaves the lane on
when it cannot touch the predicate and keeps it out when it can.

…ect is bound

With the subject bound, OPTIONAL left an encoded object to its unify check
instead of decoding it into the scan. For a numeric that changed the
match: a substituted numeric matches by value across numeric datatypes
(`1` and `"1"^^xsd:long`), the inner join's rule, and the unify check
compares terms, so the answer depended on whether the subject happened to
be bound. Numerics are substituted, so OPTIONAL matches a numeric the way
the join does in every index state. Main matched it by value only on the
novelty lane, where the value arrives decoded, and as a term on the
indexed lane. Strings, dates and IRIs, for which the two matches agree,
stay with the unify check and keep its saving.

The OPTIONAL subject lane also checks the eager-materialization gate the
bound-object lane checks. No caller reaches it eagerly today.

…lazy memo

Keyed surfaces over decoded predicates, and the first query after an index
publish, paid for the canonical form:

- A decoded IRI asked the subject dictionary first, so a decoded predicate
  key (novelty pending: `GROUP BY ?p`, `DISTINCT ?p`) paid a subject lookup
  that misses, then a predicate probe, then a novelty probe per row. The
  predicate table answers first again, now behind a per-namespace mask of
  its predicates' name lengths, so an IRI outside every predicate's
  namespace or of a length no predicate there has costs a bit test, not a
  hash. Which persisted predicates are novelty subjects is found once per
  operator rather than probed per key.
- The predicate-to-subject memo resolved every predicate on first use, a
  dictionary lookup per predicate the index holds inside the first query
  after a publish. Each predicate now resolves on its own first use, and
  the per-key read is one relaxed atomic load.
- `EqualityNorm` reads which novelty layers hold entries once, not on every
  per-row `dicts()`.
- OPTIONAL's bound-object lane coalesced up to the 512K-row cap of the
  hash-join lanes before probing. It now coalesces at most the window the
  join's own bound-object lane flushes at (100,000 rows); main answered
  these OPTIONALs row by row and buffered nothing.
- A per-row output batch is shrunk to its rows only when it is under an
  eighth full, not under half.

The BIND operator's clobber check kept a row only when the bound value and
the computed one were structurally equal. A pattern binds a subject encoded
on an indexed ledger while an expression computes it decoded, so the check
dropped a row whose computed IRI is the bound term. It now uses
`same_term`, as the inline BIND check does since the canonicalization
commit. The JSON-LD shapes the tests pin all reach the inline check; no
query shape found reaches the operator's check with a bound target, so this
keeps the two paths on one rule.

… does not

The performance notes claimed every equality surface keyed an IRI by one
canonical form. That was not true for novelty-minted ids, blank nodes,
the sameTerm fast path, property-path endpoints or BIND. It is now, and
the notes say so, along with the two cases still outside the rule (an IRI
no dictionary holds, an arena-backed literal with no graph view) and the
OPTIONAL lanes' correlation rules.

…join

…BY and DISTINCT

The group-aggregate key builders and DISTINCT built the per-row view of the operator's normalization on every row, twice per row in GROUP BY. They now read it once per batch.

…g it

same_term canonicalized a mixed pair by encoding the decoded side, a
reverse dictionary lookup per pair. The commonest mixed pair, a batched
lane's encoded subject against a scan's decoded one with novelty pending,
now decodes the encoded side through the graph view, which memoizes it per
subject.

On the 1M-triple bench ledger with novelty pending, an OPTIONAL correlated
on subject and object over 192,000 rows (`?s schema:about ?t . ?s
prov:wasDerivedFrom ?d . OPTIONAL { ?s ?x ?d }`) went from 1,448 ms to
1,245 ms (paired runs on a loaded box). Main answers it in 1,093 ms but
binds `?x` on none of the rows; this branch binds it on all of them.

The numeric-object case runs with the index fresh and with novelty pending: main matched the decoded value of the novelty lane by value and the encoded one of the indexed lane as a term.

A big number's or a vector's encoded key is a handle into an arena scoped
to one graph and predicate, counted from 0. A dataset of several graphs of
one ledger (`FROM <g1> FROM <g2>`) bound those handles in each member scan
and compared, joined, grouped and projected them in a context with no
graph to decode them in, so g1's 1.5 and g2's 9.5 under one predicate,
both handle 0, were one value: a join paired the wrong rows, MINUS removed
the wrong ones, GROUP BY merged them, and projection decoded g2's handle
through g1's arena and printed 1.5 for 9.5.

A scan whose context is one member of a union of graphs (`with_graph_ref`
under a parent with several active graphs) now binds those literals
decoded, so a graph-scoped handle never leaves its graph. Scans of a single
graph are unchanged; the scan's open event records which ones decode
(`arena_literals_decoded`).

`same_term` refuses, as an internal error, to compare an arena-backed
literal with no graph view to decode it, rather than certify equality or
inequality by handle; the nested-loop join's check no longer compares such
handles structurally; and the keyed normalizers assert, in debug builds,
that no arena handle reaches them unviewed. Member decoding makes all of
these unreachable.

Amounts in both graphs of a two-graph dataset share arena handles. Projection, a join, an OPTIONAL, MINUS and GROUP BY over them, with the member scans required to decode, and a single-graph scan required not to.

…e's first key

With novelty pending, the first predicate key an operator canonicalized probed the novelty dictionary for every persisted predicate, a per-operator cost that grows with the ledger's predicate count. Each predicate is now probed on its own first key, into a per-operator slot table read with one relaxed load after that.

…ctions

A GRAPH scope's scans bind big numbers and vectors as handles into its own
graph's arenas. Its exit decoded big numbers only, and only when the
scope's graph id differed from the outer context's. So:

- a vector left the scope as a handle. Outside, it was printed or matched
  as another graph's vector, or it reached a union of graphs, where
  comparing it is an internal error;
- in a union whose first graph is the scope's own, the ids match although
  the union decodes in no graph, so big numbers left undecoded too;
- a row seeding the scope carried the outer graph's handles in, and the
  scope decoded them through its own arenas.

ArenaCrossing decodes a handle crossing in either direction through the
graph that bound it, unless both sides read one single graph. It replaces
the two copies of the big-number exit.

One vector and one big number per graph, each handle 0 of its own arena,
so every crossing that decodes through the wrong graph changes the answer:

- a vector projected out of GRAPH, alone and inside a union, and a
  default-graph vector joined against a GRAPH scope (with a JSON-LD twin);
- a GRAPH-scope vector joined with a union, and the vector and big-number
  joins with a union whose first graph is the scope's own;
- a default-graph vector and big number carried into a GRAPH scope by an
  OPTIONAL (with a JSON-LD twin).

A same-ledger SERVICE reads the dataset's first graph of that ledger, which
is another graph when the SERVICE sits inside a GRAPH scope. Its body then
crosses graphs as a GRAPH scope does. A seeding row's vector or big number
was decoded through the target's arenas, and the body's own handles were
decoded downstream through the scope's. It now uses the same ArenaCrossing.

… a no-op

The batched R2RML GRAPH path seeds the whole parent batch without decoding
its arena handles. That is sound only because it runs without a dataset,
where the source scope inherits the outer graph id. A debug assertion now
pins the crossing as a no-op there.