GFQL type system follow-on C: public schema-Arrow APIs + plottable boundary enforcement #1339

Activity

1. lmeyerov

   added

   enhancement

   gfql

   on May 7, 2026
2. lmeyerov commented on May 7, 2026

   

   lmeyerov

   on May 7, 2026

   ContributorAuthor

   More actions

   Scoping alignment note: lane D #1345 tracks cross-lane API contract alignment in parallel.\n\nThis lane C issue remains independently executable and non-blocked by D; please include compatibility notes needed for D drift checks.
3. lmeyerov commented on May 16, 2026

   

   lmeyerov

   on May 16, 2026

   ContributorAuthor

   More actions

   Coordinator schedule note: #1464 now owns the primary community-facing schema tutorial and is intentionally placed after #1338 inference.

   For #1339, assume the tutorial baseline will already cover infer/refine/bind/validate. Keep this issue focused on schema↔Arrow APIs and plottable boundary enforcement/coercion; any docs here should extend or link from #1464 rather than becoming the first schema tutorial.
4. lmeyerov commented on May 16, 2026

   

   lmeyerov

   on May 16, 2026

   ContributorAuthor

   More actions

   Coordinator schedule update: filed #1465 for sending bound typed GraphSchema through gfql_remote().

   For this #1339 lane, please keep an eye on serialization boundaries: #1465 can either consume a stable schema serialization defined here, or define a narrow remote JSON contract itself. Avoid making #1339 implement the whole remote request path unless explicitly retargeted.
5. lmeyerov commented on May 17, 2026

   

   lmeyerov

   on May 17, 2026

   ContributorAuthor

   More actions

   Coordinator research follow-up note for #1339:

   The Arrow/plottable boundary should own the strict/coercion/conflict details that #1457 should not absorb.

   Recommended constraints:

   • Arrow remains the property type substrate.
   • Same property + same Arrow type: ok.
   • Nullability differences should merge through explicit presence semantics, not a raw boolean.
   • Incompatible Arrow types should reject in strict declared schemas unless an explicit widening/resolver mode is chosen.
   • Preserve Arrow metadata and extension types where compatible; conflict on incompatible extension metadata unless a resolver is supplied.
   • JSON-compatible schema serialization should remain debuggable for GFQL remote: send bound typed GraphSchema with gfql_remote requests #1465, while optional Arrow IPC can preserve exact Arrow schemas when both sides support it.

   This keeps #1339 as the place to settle exact conversion/coercion before remote schema transport and tutorials overpromise behavior.
6. lmeyerov commented on May 20, 2026

   

   lmeyerov

   on May 20, 2026

   ContributorAuthor

   More actions

   Coordinator audit cross-link: typed-schema cluster dispatch audit posted on #1058:

   #1058 (comment)

   This issue/PR was included in the 2026-05-20 audit covering #1457/#1337/#1338/#1339/#1485/#1465 and downstream implications for #1567/#1580.
7. lmeyerov commented on May 25, 2026

   

   lmeyerov

   on May 25, 2026

   ContributorAuthor

   More actions

   #1339 implementation prep after #1457 landing

   Context: #1567 closed receipt-only for this worker because #1580 protects the
   post-#1457 IR seams. This is the fallback implementation-prep pass for #1339.

   Dependency state checked 2026-05-25:

   • GFQL type system follow-on A: public declarative schema model + stable exports #1337 / feat(gfql): add public declarative schema model #1457 is merged at 3d85344d083295a05c7619baa1a5e415a8d69b8f.
   • GFQL type system follow-on B: schema inference API + typed topology extraction #1338 remains open. Its latest prep scopes inference as returning the public
     feat(gfql): add public declarative schema model #1457 GraphSchema and explicitly leaves public Arrow/plottable boundary
     enforcement to GFQL type system follow-on C: public schema-Arrow APIs + plottable boundary enforcement #1339.
   • GFQL type system follow-on C: public schema-Arrow APIs + plottable boundary enforcement #1339 remains open and should be treated as Arrow/boundary work, not inference
     or remote transport.

   Current #1457 surface #1339 should build on

   #1457 already provides:

   • graphistry.schema.NodeType, EdgeType, GraphSchema, EdgeTopology
   • NodeType(..., properties=pyarrow.Schema | pyarrow.Field | pyarrow.DataType | RowSchema | mapping)
   • EdgeType(..., properties=...)
   • NodeType.to_arrow(...) and EdgeType.to_arrow(...)
   • Internal graphistry.compute.gfql.ir.arrow_bridge.to_arrow()/from_arrow()
   • GraphSchema.to_catalog() carrying node_row_schemas, edge_row_schemas,
     strict, columns-by-label/type, and edge topology metadata

   Important narrowing: #1457 partially covers basic Arrow declaration/export.
   #1339 should not re-add those basics. It should focus on stable public import
   helpers and plottable boundary enforcement/coercion.

   Recommended #1339 implementation scope

   Public schema conversion:

   • Add public import helpers, likely:
     • NodeType.from_arrow(name, schema, *, labels=None, include_labels=True, coercion="widen")
     • EdgeType.from_arrow(name, source, destination, schema, *, include_type_label=True, coercion="widen")
     • Optional GraphSchema.to_arrow(...)/GraphSchema.from_arrow(...) only if the API can represent nodes/edges without inventing inference semantics. If not, defer graph-level import until GFQL type system follow-on B: schema inference API + typed topology extraction #1338.
   • Keep exact logical metadata via the existing ir.arrow_bridge keys:
     gfql.logical_type, gfql.schema_confidence, and bridge version.
   • Document that strict mode rejects unsupported Arrow types / incompatible declared logical types; widen mode string-bridges structural values and records confidence.

   Plottable boundary enforcement/coercion:

   • Add an explicit boundary helper rather than overloading existing upload behavior silently, e.g. one of:
     • g.validate_arrow_schema(table="edges"|"nodes", schema=None, coercion="strict"|"widen")
     • g.coerce_to_schema(table="edges"|"nodes", schema=None, coercion="strict"|"widen")
     • or g.to_arrow(..., schema=..., schema_validate="strict"|"widen") if the API owner prefers enhancing the existing debug/export method.
   • Default schema=None should use g._gfql_schema when bound, otherwise only current Arrow conversion behavior.
   • Preserve current to_arrow(validate='autofix'|'strict'|'strict-fast') mixed-type behavior unless the user opts into schema enforcement. Existing upload paths should not start rejecting data by default.
   • Do not implement gfql_remote() schema transport here; GFQL remote: send bound typed GraphSchema with gfql_remote requests #1465 owns remote request serialization.

   Semantics / conflict rules to settle in #1339:

   • Same property + same Arrow/logical type: accept.
   • Nullability/presence should not collapse into a raw boolean when GFQL type system follow-on B: schema inference API + typed topology extraction #1338 data is available; use explicit declared/inferred confidence and leave presence semantics to GFQL type system follow-on B: schema inference API + typed topology extraction #1338 if needed.
   • Incompatible Arrow logical types: reject in strict mode; widen only through documented resolver/coercion behavior.
   • Preserve Arrow metadata and extension metadata when compatible; conflict on incompatible extension metadata unless a resolver is provided.
   • JSON-compatible serialization should be debuggable for GFQL remote: send bound typed GraphSchema with gfql_remote requests #1465, but exact Arrow IPC preservation can stay optional.

   Test plan

   • Public NodeType.from_arrow() / EdgeType.from_arrow() round-trip with to_arrow().
   • Strict reject for unsupported Arrow types and structural NodeRef/EdgeRef export where strict cannot represent them.
   • Widen mode preserves logical metadata and confidence round-trip.
   • Bound graph schema enforcement against pandas edges/nodes:
     • matching declared schema passes
     • missing declared column fails in strict mode
     • incompatible dtype fails in strict mode
     • compatible nullable/widen case behaves as documented
   • Existing g.to_arrow(validate=...) behavior remains backward compatible when no schema enforcement is requested.
   • cuDF representative boundary smoke on DGX RAPIDS 25.02 + 26.02 if implementation touches dataframe conversion/coercion.

   Guardrails

   • Do not duplicate ir.arrow_bridge mapping tables; route through the existing bridge.
   • Do not touch ir/metadata.py or IR seam inventory: surfaces protected from #1567 (IR/verifier) future reassignment #1580 protected IR seams.
   • Do not implement GFQL type system follow-on B: schema inference API + typed topology extraction #1338 inference, GFQL type system follow-on: schema effects for graph-growing calls #1485 schema effects, GFQL remote: send bound typed GraphSchema with gfql_remote requests #1465 remote schema transport, or GFQL schema tutorial: infer, refine, bind, and validate Cypher #1464 tutorial scope here.
   • Keep docs as focused recipes extending docs/source/gfql/schema.rst; do not turn GFQL type system follow-on C: public schema-Arrow APIs + plottable boundary enforcement #1339 into the primary inference tutorial.

   Recommended next action: assign #1339 only after deciding whether to wait for
   #1338 inference. If started before #1338, keep it to explicit declared-schema
   Arrow import/export and opt-in plottable boundary validation, with no inferred
   presence/topology behavior.
8. lmeyerov

   closed this as completedin #1635on May 25, 2026
9. lmeyerov commented on May 25, 2026

   

   lmeyerov

   on May 25, 2026

   ContributorAuthor

   More actions

   #1339 coordination note from #1338 implementation PR:

   #1636

   Current inferred schema shape for Arrow-boundary work:

   • Public return object remains GraphSchema.
   • Property logical types are inferred through the existing GFQL Arrow bridge (from_arrow()), then scalar nullability is corrected from observed dataframe nulls.
   • Source/destination/id binding columns are stored on GraphSchema and adapted through the existing GraphSchema.to_catalog() path.
   • Presence detail is separate from the public schema contract through SchemaInferenceReport, with required, optional, maybe_absent, and unknown.

   Suggested #1339 consumption:

   • Treat GraphSchema as the stable schema object to enforce against.
   • Use SchemaInferenceReport only when boundary policy needs presence/nullability provenance that GraphSchema does not represent directly.
   • Keep Arrow coercion/boundary failure policy in GFQL type system follow-on C: public schema-Arrow APIs + plottable boundary enforcement #1339; GFQL type system follow-on B: schema inference API + typed topology extraction #1338 does not coerce dataframes or enforce plottable boundaries.
10. lmeyerov commented on May 25, 2026

    

    lmeyerov

    on May 25, 2026

    ContributorAuthor

    More actions

    Merged PR #1635 as dcdf93b.\n\nDelivered #1339 scope:\n- Experimental public schema↔Arrow import/export: NodeType.from_arrow(), EdgeType.from_arrow(), EdgeTopology.from_metadata(), GraphSchema.node_arrow()/edge_arrow()/to_arrow()/from_arrow().\n- Opt-in bound-schema Arrow boundary validation/coercion on to_arrow(), plot(), upload(), and validate_arrow_schema(). Default behavior remains schema_validate=False.\n- Boundary behavior covers strict missing/type/null rejection, autofix Arrow casts, active label/type fragments, type-local non-nullability, and logical string/binary compatibility.\n- Docs + CHANGELOG updated. No #1338 inference, no #1465 remote transport, no IR/compiler-plan surface touched.\n\nValidation:\n- Review skill converged: waves 1-2 fixed findings; waves 3-4 clean; credentials gate clean.\n- Local focused pytest/docs/ruff/typecheck + old mypy pass.\n- DGX RAPIDS 26.02 and 25.02 focused cuDF smoke pass.\n- Full PR CI green, including RTD, changed-line coverage, TCK GFQL, docs, GFQL core, and compat/full matrices.\n\nNext typed-schema step pre-scoped on #1485.
11. lmeyerov commented on May 25, 2026

    

    lmeyerov

    on May 25, 2026

    Last edited by lmeyerov

    ContributorAuthor

    More actions

    Coordination note: PR #1637 adds public read-only g.schema / g.has_schema() over the same _gfql_schema storage path consumed by the Arrow-boundary work. It does not change bind(schema=...), Arrow validation/coercion, upload, or plotting behavior.

    PR: #1637