Skip to content

T3 #1300: type/null metadata propagation contract for BoundIR/LogicalPlan seams - #1304

Merged
lmeyerov merged 8 commits into
masterfrom
1300-type-null-metadata
May 5, 2026
Merged

lmeyerov merged 8 commits into
masterfrom
1300-type-null-metadata

Conversation

@lmeyerov

@lmeyerov lmeyerov commented May 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a stable metadata contract over the type/nullability data BoundIR/LogicalPlan already carry (BoundVariable.nullable, RowSchema.columns, ScalarType.nullable), plus a propagation-continuity invariant in the LogicalPlan verifier. Strictly additive — no behavior change in existing GFQL/Cypher execution paths.

Issue: #1300 (T3 under #1262, umbrella #1046).

What's in the diff

New — graphistry/compute/gfql/ir/metadata.py (179 LOC). Eight helpers, exported via __all__ and re-exported from graphistry.compute.gfql.ir:

  • is_nullable(typ) — True only for ScalarType(nullable=True); structural types pass through as non-nullable
  • with_nullable(typ, nullable) — return ScalarType copy with the requested flag; structural types pass-through
  • widen_to_nullable(typ) — force scalar nullability to True
  • column_logical_type(schema, name) / column_is_nullable(schema, name) — schema accessors with tri-state None for "absent or non-scalar"
  • merge_types(left, right) — least-upper-bound for column merges; per-kind rules for ScalarType (with unknown wildcard), NodeRef (label union), EdgeRef (field-equal-or-None widening), ListType (recurse), PathType (loosest bounds); cross-kind returns None
  • bound_variable_type(bv) / bound_variable_is_nullable(bv) — reconcile BoundVariable.nullable with the LogicalType; structural pass-through

Modified — graphistry/compute/gfql/ir/verifier.py (+117 LOC). Adds invariant 6: type propagation continuity across a unary op's input slot. For columns appearing in both input.output_schema and op.output_schema, kinds must agree (NodeRef/EdgeRef/ScalarType/PathType/ListType) and ScalarType nullability is monotone-widening. Carved out as row-droppers that may legitimately narrow: Filter, PatternMatch, SemiApply, AntiSemiApply. Skipped when either schema is empty so existing planner-emitted plans that initialise output_schema=RowSchema() remain valid until the schema-population slice lands.

Three pre-existing inline patterns also consolidated onto helpers within ir/: verifier.py:188 (invariant 5 optional-arm) and verifier.py:295 (invariant 6 monotonicity) onto is_nullable; query_graph.py:224 (is_required = not var.nullable) onto bound_variable_is_nullable.

Modified — graphistry/compute/gfql/ir/__init__.py (+18 LOC). Re-exports all 8 helpers via the package surface.

Modified — graphistry/compute/gfql/ir/query_graph.py (+1/-1 LOC). One inline .nullable consolidated.

New — graphistry/tests/compute/gfql/test_ir_type_propagation.py (603 LOC, 61 tests across 7 classes):

  • TestNullableHelpers — 10 tests
  • TestSchemaAccessors — 5 tests
  • TestMergeTypes — 15 tests (per-kind rules, cross-kind rejection, wildcard semantics)
  • TestBoundVariableType — 6 tests
  • TestBoundVariableIsNullable — 10 tests (parametrized across all 5 LogicalType families × both nullability values)
  • TestPropagationContinuity — 12 tests (kind mismatch, narrowing on Project/Distinct/OrderBy, Filter/PatternMatch carve-outs, dropped columns)
  • TestSeamWith1303LoweringSplit — 4 tests pinning co-import behavior with the post-GFQL monolith shrinkdown S3: split lowering.py projection + reentry concerns #1303 projection_planning.py / cypher/reentry/runtime.py modules

Modified — CHANGELOG.md (+1 LOC). Internal entry under #1262.

Acceptance (per #1300)

  1. Bound/logical plan nodes expose stable type/nullability metadata where required — eight-helper contract surface in graphistry.compute.gfql.ir.
  2. No behavior regressions in existing GFQL/Cypher execution paths — full GFQL suite: 1777 passed, 113 skipped, 15 xfailed, 0 failed.
  3. Tests prove metadata survives representative transforms/stages — 61 tests covering helper contract + verifier invariant 6.

Caveats

  • ScalarType.kind="unknown" is the dominant binder output today (11 of 13 ScalarType(...) constructions in binder.py). The helpers operate correctly on this hollow data; richer per-column kinds land in T4 (Arrow type bridge), not T3.
  • Verifier invariant 6 is exercised by passes/manager.py (after every tier1/tier2 pass; fatal on diagnostics) and cypher/lowering.py (_verify_selected_logical_plan gates LogicalPlan routing). The check is short-circuited when output_schema is empty, which is how most planner-emitted Project nodes initialise today — invariant 6 mainly guards hand-built fixtures and pass-emitted plans that explicitly populate schemas, until the schema-population slice lands.

Deferred (T3.b, tracked in #1309)

Three further inline-nullable consolidation candidates were inspected but not applied in this PR to avoid sibling-PR collision; each is a one-helper-call swap:

  • binder.py:154 (nullable=existing.nullable or branch_var.nullable — BoundVariable union merge)
  • binder.py:1062 (nullable = any(scope[ref].nullable for ref in refs) — BoundVariable nullability OR-fold)
  • lowering.py:529 (variable.nullable or bool(variable.null_extended_from) — nullability flatten)

Test plan

  • pytest -q graphistry/tests/compute/gfql/test_ir_type_propagation.py — 61 passed
  • pytest -q graphistry/tests/compute/gfql/test_ir_verifier.py — no regressions on invariants 1-5
  • pytest -q graphistry/tests/compute/gfql/ — 1777 passed, 0 failed
  • ./bin/lint.sh (ruff) clean
  • ./bin/typecheck.sh (mypy 1.20.2, 239 source files) clean
  • CI green

@lmeyerov
lmeyerov force-pushed the 1300-type-null-metadata branch from d415e0e to 7f71ce3 Compare May 5, 2026 06:38
lmeyerov added a commit that referenced this pull request May 5, 2026
User pressure-tested PR #1304 against #1260-style code-reduction goals
and repo-quality / adoption goals. Honest critiques landed; applying
the cheap mitigations:

- Item 1: re-export the eight metadata helpers from
  `graphistry.compute.gfql.ir` so external adopters can discover them
  via the package surface, not the deep `ir.metadata` path.

- Item 2: drop `assert hasattr(...)` private-symbol coupling from the
  TestSeamWith1303LoweringSplit co-import test. #1303's projection_planning
  and reentry/runtime modules `globals().update(vars(lowering))` to
  inherit the shared symbol table, which clobbers `__name__` and
  `__file__`. Use sys.modules registration as the right load-success
  witness — purely behavioral, no internal-symbol coupling.

- Item 5: verifier.py docstring now disclosures honestly that
  `verify(...)` is currently called only by the test suite — invariants
  document intended semantics, not always-on runtime safety. Callers
  integrating strict-mode validation (T2 #1302) or arrow coercion (T4)
  should plan on calling verify explicitly.

- Item 3: opened **#1309** as the T3.b tracking issue for the deferred
  binder.py:151 + lowering.py:571 consolidations onto helpers. Linked
  from CHANGELOG so the punch list lives in GitHub, not just in
  plan.md.

Items 4 (PR description framing on `ScalarType.kind="unknown"`) and
6 (coordinator handoff messaging on indirect GraphistryGPT relevance)
are non-code edits handled separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@lmeyerov
lmeyerov force-pushed the 1300-type-null-metadata branch from 0524245 to cd9649d Compare May 5, 2026 08:29
lmeyerov added a commit that referenced this pull request May 5, 2026
User pressure-tested PR #1304 against #1260-style code-reduction goals
and repo-quality / adoption goals. Honest critiques landed; applying
the cheap mitigations:

- Item 1: re-export the eight metadata helpers from
  `graphistry.compute.gfql.ir` so external adopters can discover them
  via the package surface, not the deep `ir.metadata` path.

- Item 2: drop `assert hasattr(...)` private-symbol coupling from the
  TestSeamWith1303LoweringSplit co-import test. #1303's projection_planning
  and reentry/runtime modules `globals().update(vars(lowering))` to
  inherit the shared symbol table, which clobbers `__name__` and
  `__file__`. Use sys.modules registration as the right load-success
  witness — purely behavioral, no internal-symbol coupling.

- Item 5: verifier.py docstring now disclosures honestly that
  `verify(...)` is currently called only by the test suite — invariants
  document intended semantics, not always-on runtime safety. Callers
  integrating strict-mode validation (T2 #1302) or arrow coercion (T4)
  should plan on calling verify explicitly.

- Item 3: opened **#1309** as the T3.b tracking issue for the deferred
  binder.py:151 + lowering.py:571 consolidations onto helpers. Linked
  from CHANGELOG so the punch list lives in GitHub, not just in
  plan.md.

Items 4 (PR description framing on `ScalarType.kind="unknown"`) and
6 (coordinator handoff messaging on indirect GraphistryGPT relevance)
are non-code edits handled separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@lmeyerov
lmeyerov marked this pull request as ready for review May 5, 2026 08:29
lmeyerov and others added 8 commits May 5, 2026 01:31
Add a stable helper surface over the type/nullability metadata BoundIR /
LogicalPlan already carry, plus a propagation-continuity invariant in the
LogicalPlan verifier. T3 of #1262 / umbrella #1046.

graphistry/compute/gfql/ir/metadata.py exposes seven helpers:
is_nullable, with_nullable, widen_to_nullable, column_logical_type,
column_is_nullable, merge_types, bound_variable_type. ScalarType is the
only nullable carrier today; structural types (NodeRef, EdgeRef,
PathType, ListType) pass through unchanged.

verifier.py gains invariant 6: for unary ops, columns shared between
input.output_schema and op.output_schema must agree on kind family;
ScalarType nullability is monotone-widening, with Filter as the
sole carve-out (it may drop NULL rows). The check is skipped when
either schema has no columns so existing planner-emitted plans
that initialise output_schema=RowSchema() remain valid.

Adds graphistry/tests/compute/gfql/test_ir_type_propagation.py
(48 tests) covering helper contract surface and verifier
pass/fail cases (kind mismatch, nullability narrowing on
Project/Distinct/OrderBy, Filter carve-out, dropped columns,
structural pass-through, list-element recursion).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wave 1 review found:
- I1 IMPORTANT: PatternMatch with predicates can narrow nullability the
  same way Filter does (predicates may drop rows where pattern matches
  fail). Add PatternMatch to _NULLABILITY_NARROWING_OPS. Optional arms
  remain locked nullable=True via invariant 5; this carve-out only
  changes invariant 6 narrowing-detection on the *non-optional* lane.
- I3 SUGGESTION→IMPORTANT: bound_variable_type structural pass-through
  was only tested for NodeRef. Parametrize across NodeRef / EdgeRef /
  PathType / ListType to pin the documented contract.

Adds positive PatternMatch carve-out test in TestPropagationContinuity.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wave 2 amplification on wave-1 fixes was clean. New IMPORTANT findings:

- B-I1: SemiApply / AntiSemiApply (Cypher EXISTS / NOT EXISTS subquery
  filters) are filter-shaped row-droppers. Same forward-correctness risk
  as wave-1 PatternMatch carve-out. Add both to
  _NULLABILITY_NARROWING_OPS, lowercase the type alias to match the rest
  of verifier.py, and update the file-header invariant 6 doc.

- B-I2: bound_variable_type silently dropped bv.nullable for structural
  variables (the binder records nullable=True on optional-arm
  whole-row aliases even though NodeRef/EdgeRef have no nullable
  dimension). Adding bound_variable_is_nullable as the canonical
  "is this variable nullable" helper that returns bv.nullable directly,
  with the docstring in bound_variable_type now pointing callers there.

- B-S2: CHANGELOG test count refresh after wave 1 added 4 tests.

Adds parametrized TestBoundVariableIsNullable across all 5 LogicalType
families so the contract is pinned for both nullable=True and =False
on every kind.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wave 3 converged the review (waves 2 + 3 both non-significant advance,
0 BLOCKER + 0 IMPORTANT new findings in either). Applying the two most
actionable SUGGESTIONs from wave-3:

- B-S1: invariant-6 error message previously read "only Filter may drop
  NULL rows" — wave-2 extended the carve-out to PatternMatch / SemiApply
  / AntiSemiApply, so the user-visible diagnostic now lists the full set
  derived from `_NULLABILITY_NARROWING_OPS`. Existing test substring
  assertions (`"narrowed nullability"`) still match.

- A3: CHANGELOG test count refreshed from "≈60" to the actual 57 (wave-2
  B-S2 specifically asked for an accurate count).

Two additional fixes that fell out of wave-3 review naturally are kept
as deferred-cosmetic SUGGESTIONs (e.g. parametrizing the carve-out tests
across all four narrowing ops); they would be additive only and do not
gate convergence.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…5-05)

Master moved ahead with #1303 (GFQL monolith shrinkdown S3 — split
lowering.py projection + reentry concerns into cypher/projection_planning.py
and cypher/reentry/runtime.py). Diff overlap with T3 was zero (T3 lives in
ir/, #1303 in cypher/; CHANGELOG was the only candidate-conflict file and

Both lanes share the conceptual surface "lowering produces LogicalPlan-shaped
output that the IR verifies", so adding TestSeamWith1303LoweringSplit (4
tests) to pin the seam:
- Co-import of #1303 split modules + T3 metadata module without circular
  surprise; lowering still re-exposes the split delegators.
- Realistic NodeScan→PatternMatch→Filter→Project chain — invariant 6
  passes through carve-outs and column drops.
- OPTIONAL MATCH plan with non-nullable scalar — pins that invariant 5 fires
  exactly once; invariant 6 stays silent (no false-positive double signal).
- Helper contract on a RowSchema shape mirroring projection_planning.py's
  whole-row + scalar-projection emits (NodeRef + ScalarType columns).

Refresh CHANGELOG count from 57 to 61.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…lable

User raised that this PR was net additive (+869/-1) — surprising next to
the #1260 monolith-shrinkdown lane. Audit found two safe in-house
consolidations: invariants 5 and 6 in verifier.py both contained
inline `isinstance(typ, ScalarType) and ... typ.nullable` checks that
T3's new is_nullable helper now expresses directly. Apply both.

Two further candidates were inspected and intentionally deferred:
- binder.py:151 nullable-OR union merge — T2 #1302 hot zone
- lowering.py:571 BoundVariable nullable+null_extended flatten — #1295/#1303
  hot zone

Both deferrals are documented in CHANGELOG and plan.md so the T3.b
follow-on slice has a concrete punch list.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
User pressure-tested PR #1304 against #1260-style code-reduction goals
and repo-quality / adoption goals. Honest critiques landed; applying
the cheap mitigations:

- Item 1: re-export the eight metadata helpers from
  `graphistry.compute.gfql.ir` so external adopters can discover them
  via the package surface, not the deep `ir.metadata` path.

- Item 2: drop `assert hasattr(...)` private-symbol coupling from the
  TestSeamWith1303LoweringSplit co-import test. #1303's projection_planning
  and reentry/runtime modules `globals().update(vars(lowering))` to
  inherit the shared symbol table, which clobbers `__name__` and
  `__file__`. Use sys.modules registration as the right load-success
  witness — purely behavioral, no internal-symbol coupling.

- Item 5: verifier.py docstring now disclosures honestly that
  `verify(...)` is currently called only by the test suite — invariants
  document intended semantics, not always-on runtime safety. Callers
  integrating strict-mode validation (T2 #1302) or arrow coercion (T4)
  should plan on calling verify explicitly.

- Item 3: opened **#1309** as the T3.b tracking issue for the deferred
  binder.py:151 + lowering.py:571 consolidations onto helpers. Linked
  from CHANGELOG so the punch list lives in GitHub, not just in
  plan.md.

Items 4 (PR description framing on `ScalarType.kind="unknown"`) and
6 (coordinator handoff messaging on indirect GraphistryGPT relevance)
are non-code edits handled separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wave 6 found 1 BLOCKER + 1 IMPORTANT:

BLOCKER (Track A): verifier.py docstring "honesty" wording from the
pressure-test commit overshot. Claimed `verify(...)` is "currently
invoked only by the test suite; no production caller in the repo runs
it on planner-emitted plans" — wrong. `passes/manager.py:71,86` runs
verify after every tier-1 / tier-2 pass and treats any diagnostic as
fatal; `cypher/lowering.py:429` runs it via `_verify_selected_logical_plan`
to gate LogicalPlan routing for covered Cypher shapes. So the invariants
ARE real safety nets on those paths. Replaced the docstring section with
an accurate "Production callers" list and a narrower "Scope caveat for
invariant 6" note: the kind/nullability check short-circuits when
output_schema columns are empty, which is most planner-emitted Project
nodes today — that part of the framing is still correct.

IMPORTANT (Track B): missed consolidation candidate at
`ir/query_graph.py:224` (`is_required = not var.nullable` on a
BoundVariable). Same package as T3, no sibling-PR hot-zone risk.
Apply `bound_variable_is_nullable(var)` directly. CHANGELOG bumped
from "two consolidations" to "three consolidations within `ir/`".

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant