docs(gfql): publish the traceable q1-q9 board and rewrite the prose around it by lmeyerov · Pull Request #1815 · graphistry/pygraphistry

lmeyerov · 2026-07-28T02:19:19Z

Review-only. Do not self-merge. Editorial half of the #1806 split. Stacked on #1813 (docs/bench-numbers-gate-and-deletions) — review that one first; it is mechanical and urgent, this one is not. Base it back to master after #1813 lands.

What this PR is for

#1813 makes the public docs not-wrong by deleting. This one makes them good: it publishes the one board that traces cell-by-cell to committed artifacts, and rewrites the prose the deletions left behind.

What is now published, and with what scope

Two lanes from the matched, row-validated prrao87/graph-benchmark q1–q9 Cypher suite, measured on a quiet, perf-locked dgx-spark with position-balanced slots. Every value is a :bench: reference resolved from docs/source/_data/gfql_benchmarks.json; no figure on the page is typed by hand, and the gate from #1813 fails the build if that stops being true.

GFQL-Polars beats GFQL-pandas on all nine queries, at both graph sizes. Every per-query speedup is larger on the 100k graph than on the 20k one — so the "gap grows with size" claim is shown rather than asserted. Both absolute medians and both ratios are published.
On the 100,000-person lane, GFQL with engine='polars' answers five of the nine queries faster than embedded Kuzu 0.11.3 — q1 3.63×, q2 5.06×, q3 1.95×, q8 1.93×, q9 1.26×, with both sides' medians shown.

The page states, in the body text next to the table, that this is the q1–q9 suite at 100,000 persons, on engine='polars', against Kuzu 0.11.3, warm medians, rows validated per cell, and that it is five of the nine queries in that suite at that size — not a claim about every query, every engine, or every size.

Full provenance for both runs (date, host, perf lock, quiet host, rep protocol, pygraphistry commit, pyg-bench commit, runtime image, dataset, competitor version, raw artifact path) renders on the page from the source of truth, and the build fails if a page publishes a number without it.

Prose changes

engines.rst's vendor table is reorganized from "where each tool wins / where it complements" into "coming from X → what GFQL gives you when the query runs here". It keeps the vendor names (they are how people find this page) and drops the per-vendor commentary.
engines.rst's engine bullets now describe each engine's architecture and the shape it is built for, instead of asserting head-to-head wins that no surviving measurement supports. [F1] cites the published board.
Pages with no publishable figures — indexing.rst, index_adjacency.rst, benchmark_filter_pagerank.rst — now say plainly that the lane's numbers are not published until it runs under the provenance-carrying harness. They no longer narrate what used to be on the page.
performance.rst (top level) replaces "can become dramatically faster" with the actual mechanism and a link to the measured numbers. An intensifier with no evidence behind it is the slop guide's DON'T Report server-side ETL errors to client #9.
Structure follows reader questions — one keyword, against an embedded graph database, provenance, how it is fast, how a number gets published — rather than a generic outline.

Where the stance and the provenance rule pulled against each other

Three places, and how each was resolved. Flagging them because the resolution is a judgement call, not a mechanical one.

We lose six of nine at 20k and win five of nine at 100k. Selecting which lane to publish is allowed; implying a win in a lane we lose is not. Resolution: publish the 100k lane, name it in the sentence that makes the claim, say "five of the nine", and say explicitly that the four queries not listed are not published here. The 20k lane still appears — but only in the pandas-vs-Polars table, where GFQL wins every cell, and it is labelled as the 20k column. There is no sentence a reader can take as "GFQL beats Kuzu", full stop.
The stance says do not concede Neo4j's concurrency and durability, because they are achievable with GFQL. I removed the concessions ("Neo4j remains the transactional system-of-record", "keep Neo4j or Kuzu as the system-of-record"). I did not replace them with a positive GFQL durability/concurrency claim, because I have no documentation or measurement to point at for one. The page is now silent on that axis. If there is a supportable claim to make there, it needs a source before it goes in; silence was the only option that satisfied both rules.
The stance says no history, but the withdrawals are provenance information. The per-page "these figures have been withdrawn" tombstones from docs(gfql): publish benchmark numbers from a provenance-carrying source of truth #1806 are gone; each page instead states forward-looking policy — this lane's numbers are not published until it runs under the harness. The withdrawal itself is recorded permanently in CHANGELOG.md (in docs(gfql): delete refuted and unverifiable benchmark numbers, and gate the rest #1813) rather than repeated on six pages. Nothing is being hidden: no claim is made, and the reason no claim is made is on the page.

One thing I could not find a truthful framing for, so it is simply absent: there is no publishable "GFQL vs Neo4j" latency comparison at all right now. engines.rst says what GFQL gives you structurally on that row (one in-process call, no projection, no write-back) and points at the pipeline description. It does not imply a latency win.

Deliberately out of scope

benchmark_graphframes.rst is untouched. Its numbers are verifiable — raw results are committed and every headline cell round-trips — but it contains statements the stance would change (e.g. where CPU PageRank via igraph trails GraphFrames), and editing a verified benchmark page's conclusions deserves its own review. Follow-up: bring it under the gate first, then revisit the prose.
demos/gfql/benchmark_filter_pagerank_cpu_gpu.ipynb renders the same withdrawn pipeline figures from the same missing results directory. Noted in docs(gfql): delete refuted and unverifiable benchmark numbers, and gate the rest #1813; a notebook rewrite is separate work.

Verification

bin/lint.sh clean.
rstcheck clean on every edited .rst.
bin/check_bench_numbers.py — reference integrity, freshness, hand-typed literals, and commit drift all pass.
Dockerised docs/ci.sh build (the same one test-docs runs) exits 0, with 0 gfql-bench warnings; rendered HTML spot-checked for every published cell and both provenance blocks.

🤖 Generated with Claude Code

https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

…te the rest Two published OLAP ratios are not stale, they are wrong. `gfql/performance.rst` published q8 at "5.0 ms vs 1,004 ms for embedded Kuzu (200x)"; a matched, row-validated re-run measures 5.06 ms vs 9.76 ms = 1.93x. It published q9 at 14.2x; measured, 66.61 vs 84.06 = 1.26x. An audit of every other published performance figure found most of them can be neither confirmed nor refuted: the reproducers wrote to /tmp or only printed, recorded no commit/host/timestamp, and their artifacts were never committed. Two are worse than unverifiable - the LadybugDB competitor column was a hardcoded `LADYBUG = {...}` literal with no citation, and the filter->PageRank SVG charts are generated from a results directory that has never existed in any commit. All of it is removed rather than restated. The mechanism that stops it recurring ships with the deletion: a single generated source of truth (docs/source/_data/gfql_benchmarks.json), a Sphinx extension that resolves `:bench:` against it and fails the build on a missing, stale, non-quotable or undisclosed number, and a standalone gate that adds commit-drift and hand-typed-literal checks. Deletion and mechanism only - no editorial rewriting. The prose rewrite is a separate PR. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

…round it The editorial half of the #1806 split. It publishes the one benchmark board that traces cell-by-cell to committed artifacts, and rewrites the prose that the deletions left behind. Two lanes, each named where it is stated: GFQL-Polars beats GFQL-pandas on all nine queries of the graph-benchmark q1-q9 Cypher suite at both sizes measured, with every per-query speedup larger on the bigger graph; and on the 100,000-person lane GFQL with engine='polars' answers five of the nine queries faster than embedded Kuzu 0.11.3. The page says which engine, which size, which suite, and that it is five of nine. engines.rst's vendor table is reorganized from "where each tool wins" into "what GFQL gives you when the query runs here". Pages with no publishable figures now state that the lane is unpublished until it runs under the provenance-carrying harness, rather than narrating what used to be there. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

lmeyerov · 2026-07-28T02:22:42Z

CI note — test-docs is SKIPPED on this PR, and that is a pre-existing gating hole, not a failure.

test-docs declares needs: [changes, python-lint-types, docs-latex-unicode-guard]. This PR changes no .py file, so python-lint-types is skipped by its path filter — and a job whose needs includes a skipped job is itself skipped, regardless of its own if:. The result is that a docs-only PR never runs the docs build in CI. That holds after a rebase onto master too, since the diff would still be docs-only.

#1813 does not hit this: it adds bin/check_bench_numbers.py, docs/source/_ext/*.py and docs/test_bench_numbers.py, so python-lint-types runs, test-docs runs, and the Dockerised -W build passes there with the gate live.

For this PR the same build was run locally via the identical docs/ci.sh script (DOCS_FORMAT=html): exit 0, 0 gfql-bench warnings, and the rendered HTML was spot-checked for every published cell and both provenance blocks.

Worth fixing in ci.yml (adding !cancelled() to test-docs's condition, or dropping python-lint-types from its needs), but that needs the workflow OAuth scope this token does not have.

… lane Merge origin/master (8764eb8) and re-point the board at receipts: - Withdraw every citation of results/graphbench-matched-q1q9-20260726 (both 20k and 100k runs): the lane carries no receipts and is nonreproducible. Its runs and cells leave gfql_benchmarks.json, and the restored q9 66.6/84.1 ms cell leaves gfql/performance.rst. - Publish the receipted 20k lane results/graphbench-board-20k-20260802 (pygraphistry 8764eb8, 4 position-balanced slots x 51 runs after 5 warmups, host-load/spike receipts, row-set-validated cells) as a full nine-row board: 5 wins, 2 ties, 2 losses for GFQL-Polars vs embedded Kuzu, q4/q5 flagged WEAK (overlapping slot ranges) in data and prose, plus the comparator's compare.txt table transcribed verbatim. - Leave every 100k cell as an explicit pending marker for the lane running 2026-08-02 (results/graphbench-board-100k-v2-20260802). - engines.rst drops "both sizes measured" / "gap grows with size" until the 100k lane republishes them; vendor rows stay qualitative. - Adopt master's landed bench-data infra (_ext, contract, pytest gate); drop the branch's superseded bin/check_bench_numbers.py. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

lmeyerov · 2026-08-02T18:40:06Z

Refreshed onto today's receipted lanes (3e01a1a4, merges origin/master @ 8764eb82). Review-only as before; no merge.

Cut (nonreproducible)

Every number from results/graphbench-matched-q1q9-20260726 (pyg-bench) is gone — that lane has no receipts. Concretely:

Both graphbench-q1q9-{20k,100k}-20260726 runs and all their cells removed from docs/source/_data/gfql_benchmarks.json.
The "five of the nine at 100k" table and claim this PR previously published: cut.
The pandas-vs-Polars two-size speedup table (its ratios came from the same lane): cut. Its ratios are not re-derived; the receipted board shows the pandas and polars medians side by side instead.
Master's restored q9 "66.6 ms vs 84.1 ms (1.26x)" OLAP cell (same lane): cut.
engines.rst no longer says "both sizes measured" or "gap grows with size"; those return only if the 100k lane republishes them.

Replaced by: the receipted 20k board

results/graphbench-board-20k-20260802 (pyg-bench), pygraphistry 8764eb82 (= today's master), 4 position-balanced K/G/I slots x 51 timed runs after 5 warmups, 10% tie band, host-load + spike receipts, runner sha256s, row-set-validated cells. Published in gfql/performance.rst as a full nine-row board with :bench: references, one provenance block, one disclosures block, and the comparator's table transcribed verbatim:

q     rows   kuzu ms   pandas ms   polars ms  best GFQL   ratio  verdict
------------------------------------------------------------------------
q1       3     15.30       35.18        8.95     polars    1.71  WIN 1.71x
q2       1     35.67       37.88       13.06     polars    2.73  WIN 2.73x
q3       5      5.68       11.67        5.80     polars    0.98  TIE
q4       2      3.35        9.49        3.97     polars    0.84  LOSE 1.19x (WEAK: slot ranges overlap)
q5       1      5.39       80.73        4.50     polars    1.20  WIN 1.20x (WEAK: slot ranges overlap)
q6       5      8.90       80.44        5.73     polars    1.55  WIN 1.55x
q7       1      5.19       19.82        5.68     polars    0.91  TIE
q8       1      2.80       19.60        8.24     polars    0.34  LOSE 2.94x
q9       1     10.76       28.26        8.32     polars    1.29  WIN 1.29x

5W / 2T / 2L. The two losses (q4 weak, q8 clear at ~3x) are printed with the same weight as the wins; q4/q5 WEAK flags travel as disclosures in the published data, so the page cannot render those ratios without the caveat.

One provenance note: the lane's SOURCE_COMMITS receipt records pyg-bench checkout 7dca9392; the run record carries that alongside the ad24d789 publish ref rather than silently picking one.

Pending (not filled in)

The 100k lane is running today. Every 100k cell is an explicit "pending: lane running 2026-08-02" marker pointing at results/graphbench-board-100k-v2-20260802; no carried-over or invented 100k number anywhere.

Merge notes

Master's landed bench-data infra won every infra conflict (_ext/*, contract, docs/test_bench_numbers.py, conf.py); the branch's superseded bin/check_bench_numbers.py (incompatible with master's module API) is dropped.
benchmark_filter_pagerank.rst resolves to master's restored, receipted version.
CHANGELOG: union, newest entry prepended.
Verified: docs/test_bench_numbers.py 18/18 pass, contract re-verification clean, static point-of-use audit (provenance + disclosures) clean, rstcheck clean on every touched page.

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

results/graphbench-board-100k-v2-20260802 sealed VALID (pygraphistry 8764eb8, pyg-bench 7dca9392 per SOURCE_COMMITS; 4 position-balanced slots x 51 runs after 5 warmups; 263 receipted spike seconds): - 100k board published: 6 wins, 2 ties, 1 loss for GFQL-Polars vs embedded Kuzu, no weak verdicts; comparator table transcribed verbatim; cells added to gfql_benchmarks.json as :bench: references. - Pending markers removed; cross-scale reading added: q8 loses at both scales (comparator verdicts LOSE 2.94x / LOSE 2.44x), q4 flips from weak 20k loss to clean 100k win, Polars beats pandas on all nine queries at both scales with a wider gap on the larger graph. - engines.rst restores "both sizes measured" / "gap grows with size", now citing the receipted boards. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

lmeyerov · 2026-08-02T19:02:28Z

100k lane sealed VALID — pending markers replaced with the receipted board (1f8ebb2e).

results/graphbench-board-100k-v2-20260802 (pyg-bench PR #165), SOURCE_COMMITS pygraphistry 8764eb82 / pyg-bench 7dca9392, same protocol as the 20k lane (4 position-balanced slots x 51 timed runs after 5 warmups, 10% tie band, host-load + spike receipts — 263 receipted spike seconds, VALIDITY: no external process in any over-limit capture). Comparator table transcribed verbatim:

q     rows   kuzu ms   pandas ms   polars ms  best GFQL   ratio  verdict
------------------------------------------------------------------------
q1       3    153.62      199.30       31.29     polars    4.91  WIN 4.91x
q2       1    278.64      209.32       44.33     polars    6.29  WIN 6.29x
q3       5     34.23       73.48       13.66     polars    2.51  WIN 2.51x
q4       3     13.50       64.41       10.51     polars    1.28  WIN 1.28x
q5       1     13.17      412.25       13.68     polars    0.96  TIE
q6       5     24.11      410.76       14.84     polars    1.62  WIN 1.62x
q7       1      9.59      120.21        9.88     polars    0.97  TIE
q8       1     13.62      152.75       33.25     polars    0.41  LOSE 2.44x
q9       1     83.71      218.58       36.41     polars    2.30  WIN 2.30x

100k board: 6W / 2T / 1L, no weak verdicts. Stated plainly on the page:

q8 loses at both scales — the comparator's verdicts read LOSE 2.94x (20k) and LOSE 2.44x (100k). It is the one query where embedded Kuzu beats GFQL outright, and it stays a loss as the graph grows.
q4 flips: weak loss inside overlapping slot ranges at 20k, clean 1.28x win at 100k.
GFQL-Polars beats GFQL-pandas on all nine queries at both scales, by more on the larger graph — so engines.rst gets its "both sizes measured" / "gap grows with size" lines back, now citing the receipted boards.

Both boards render from :bench: references; the Provenance subsection carries both run records; the 100k cells and run are in gfql_benchmarks.json.

Verified again: docs/test_bench_numbers.py 18/18 pass, static provenance/disclosures audit clean, rstcheck clean, and no pending/re-running marker remains on the page.

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

Owner review on #1815: - The board section now leads with the receipts-grounded topline: GFQL-Polars beats embedded Kuzu on 11 of 18 cells across the two boards (4 ties, 3 losses; wins reaching 6.29x), exceptions are ties and losses on small point-lookup-shaped work (q8 loses at both scales; q4's loss is weak and 20k-only, flipping to a win at 100k), and the ratio moves further toward GFQL from 20k to 100k on eight of nine queries (q5 slips from narrow win to tie). Neo4j is folded in only through evidence the docs carry: the on-page LDBC SNB pairs (four of five to GFQL) and the receipted filter->PageRank->filter pipeline page. - All previous-version/withdrawal prose is cut from the pages: the July-lane withdrawal paragraphs and the "until rerun" clause. Docs state current facts; the history stays in the changelog and PR. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

lmeyerov · 2026-08-02T19:37:25Z

Review revisions applied (babc7465).

1. Archaeology cut. All previous-version/withdrawal prose is gone from the pages: the "Earlier q8/q9 figures from the July lane are withdrawn..." paragraph, the "No number from any earlier 100k run appears on this page..." paragraph, and the bulk-sweep "until rerun on a current tag" clause. A sweep for withdrawn / earlier figures / July lane / supersedes / the old results-dir name across every touched page comes back clean. The history lives in CHANGELOG.md and this PR, not on the pages.

2. Topline up front. The board section now opens with:

GFQL with engine='polars' wins the large majority of these head-to-head cells at both scales: across the two boards it beats embedded Kuzu on 11 of the 18 cells, ties 4, and loses 3, with wins reaching 6.29x. The exceptions are a few ties and losses on small, point-lookup-shaped work: q8, a single-row two-hop count that Kuzu answers very fast, loses at both scales, and q4's loss is weak-flagged and only at 20k — it flips to a win at 100k. The gap otherwise runs in GFQL's favor and grows with scale: on eight of the nine queries the Kuzu ÷ Polars ratio moves further toward GFQL from 20k to 100k, the exception being q5, which slips from a narrow win into the tie band. Against Neo4j, this page carries the seeded LDBC SNB pairs above — GFQL takes four of the five — and the receipted filter → PageRank → filter pipeline against Neo4j + GDS is in benchmark_filter_pagerank.

Notes on grounding: the 6.29x renders from the :bench: q2-100k ratio cell; the scale-trend sentence is stated as eight-of-nine with q5 named, because the receipts show q5's ratio slipping 1.20 → 0.96 (and q8's actually improving 0.34 → 0.41) — "everything except q8/ties" would not have been accurate. Neo4j is claimed only through evidence the docs carry (the on-page LDBC table, 4 of 5 to GFQL, and the receipted pipeline page); no "faster than Neo4j" claim rests on cells this page lacks.

Verified again: docs/test_bench_numbers.py 18/18, point-of-use audit clean, rstcheck clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

lmeyerov and others added 2 commits July 27, 2026 19:06

This was referenced Jul 28, 2026

docs(gfql): publish benchmark numbers from a provenance-carrying source of truth #1806

Closed

docs(gfql): delete refuted and unverifiable benchmark numbers, and gate the rest #1813

Merged

lmeyerov changed the base branch from docs/bench-numbers-gate-and-deletions to master August 2, 2026 18:41

lmeyerov merged commit f875724 into master Aug 2, 2026
35 checks passed

lmeyerov mentioned this pull request Aug 3, 2026

docs(gfql): q1-q9 board update to the candidate 20260803 lanes (17W/0T/1L) #1848

Merged

Navigation Menu

Repository navigation

docs(gfql): publish the traceable q1-q9 board and rewrite the prose around it - #1815

docs(gfql): publish the traceable q1-q9 board and rewrite the prose around it#1815
lmeyerov merged 5 commits into
masterfrom
docs/bench-numbers-editorial

lmeyerov commented Jul 28, 2026

Uh oh!

lmeyerov commented Jul 28, 2026

Uh oh!

lmeyerov commented Aug 2, 2026

Uh oh!

lmeyerov commented Aug 2, 2026

Uh oh!

lmeyerov commented Aug 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lmeyerov commented Jul 28, 2026

What this PR is for

What is now published, and with what scope

Prose changes

Where the stance and the provenance rule pulled against each other

Deliberately out of scope

Verification

Uh oh!

lmeyerov commented Jul 28, 2026

Uh oh!

lmeyerov commented Aug 2, 2026

Cut (nonreproducible)

Replaced by: the receipted 20k board

Pending (not filled in)

Merge notes

Uh oh!

lmeyerov commented Aug 2, 2026

Uh oh!

lmeyerov commented Aug 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant