Repository navigation
bench: baseline host/corpus identity, phase-share gating, and noise floors - #1550
Conversation
…loor Four extensions to the baseline gate #1477 introduced, all of them aimed at the same failure: a compare run that produces a number which looks like a result and is not one. host block. `runner_class` becomes `host.class` inside a block that also records os, arch, cpu_model, and physical/logical cores. The class names a comparability SET, not a machine — two hosts share one when a human has decided their absolute numbers may be compared. A mismatch is now a refusal (exit 2) rather than a silent downgrade to advisory; `--allow-host-mismatch` opts back into the advisory posture, which is what CI runs today. The derived default is `{os}-{arch}`, deliberately coarse: an M1 and an M4 both derive `macos-aarch64` and are not interchangeable, so any host whose numbers are meant to gate should set FLUREE_BENCH_HOST_CLASS explicitly. corpus identity. A new `meta` sidecar (target/fluree-bench-meta/, same shape as the mem sidecar) carries sha256 + byte_len + element_count + input/output syntax + thread count per scenario, declarable per entry or once per file. Comparing a run over one input against a baseline captured over another is the most expensive silent failure available here, because the output is indistinguishable from a real regression — so it is a refusal with no override, unlike the host check. SHA-256 rather than BLAKE3: blake3 is not in the tree, sha2 already is, and a corpus is hashed once per capture. phase shares. Per-phase {ns, share_pct}, with the share derived from the ns at record time so the two can never disagree. Absolute phase ns gates only within a host class; share drift gates regardless, because a share is a ratio within one run and the machine cancels. Only share GROWTH gates — shares sum to 100, so gating both directions would fail a clean improvement twice, once for the phase that got better and once for its complement. noise floor. `capture --accumulate` folds repeat runs into a median + MAD over >= 5 samples, and the gate flags only when the median move exceeds max(budget_pct, 3*MAD/median). MAD over stddev because a single outlier run is the common case here and barely moves MAD. The floor only ever widens an envelope, never tightens one. Schema bumps to 2; schema-1 files still load (`runner_class` maps to `host.class`, a bare `host` string to `host.name`), so the committed guardrails-pre.json stays usable untouched. A baseline claiming a NEWER version is a refusal — silently ignoring fields we don't understand is how a gate stops gating.
…mple capture The per-PR and nightly compares were relying on a silent behaviour — a host-class mismatch quietly turning both metrics advisory. That is now a refusal by default, so both jobs pass --allow-host-mismatch and say why. The flag is the phase-1 posture and the comment names the condition for dropping it: a committed ci-ubuntu-latest baseline. Nothing about what these jobs report changes; what changes is that the downgrade is written down. bench-capture gains a `capture_samples` dispatch input. At 1 (default) it behaves as before. At >= 5 it reruns the subset and folds each pass in with --accumulate, producing the median + MAD the compare gate needs to widen budgets past runner flap — which is the thing standing between the current advisory annotations and a baseline that can actually gate on ubuntu-latest. Both compare/capture steps also clear target/fluree-bench-meta alongside the mem sidecars. A stale corpus block is worse than a stale timing: it asserts an input identity for a run that never read that input.
…ly do Every claim in the file checked against the merged main + #1477 tree. Five were wrong or stale: - "Where benches live" pointed at .github/workflows/bench-nightly.yml, which has never existed — the file is bench.yml. Both entries also carried "lands in bench-5" / "lands in bench-nightly PR" process notes for work that landed. Replaced with the real layout, including bench-baselines/ and the three target/ directories capture reads. - The bench table was missing six of the eighteen registered [[bench]] entries: query_overlay_matrix, query_hot_bsbm_bi, query_hot_property_path, query_hot_whole_graph_agg, annotation_hydration, annotation_planner. (Five of those are missing on main; #1477 adds query_overlay_matrix.) Rows written from each bench's own module docs. Also says outright that nothing enforces this table, unlike the [[bench]] <-> budget reconcile, which is why it rots. - The env-var table's defaults are right for a laptop and were being read as though they described CI. Both jobs pin quick/tiny — the nightly included, despite `Full`'s doc calling itself the nightly profile. Noted in the table and corrected at the source in runtime.rs, where BenchProfile::Full claimed "Nightly CI" and referenced the unbuilt `bench-nightly`. - bench-baselines/README.md still described the pre-5d9d69513 asymmetric gate ("enforces memory across any runner class, time advisory"). That has been false on this branch since the gate was made symmetric. The rest of the section is rewritten for the extensions in the previous commit: the metric table with its per-metric portability column, the refusal conditions and their exit codes, the noise-floor accumulation loop, and why {os}-{arch} is a coarse default rather than a promise.
fluree-bench-support/README.md documented every module except the three the baseline gate runs on. Adds `mem`/`meta` (what a sidecar is for, which dir, what it carries) and `baseline`, and corrects the test count, which said 44 against an actual 102.
Review findings from an adversarial pass over the baseline gate. Each of these produced a clean-looking report for a change that was genuinely worse. New-phase invisibility (the expensive one). The per-phase loop walked only the BASELINE's phases, so moving work into a phase the baseline never had made the old phase's share FALL — reported as "improved" — while the new phase carrying the cost produced no row at all. A 2x regression exited 0. Phases are now compared over the union of both sides: a phase new in the current run breaches when it claims more than the drift threshold, and a phase that vanished breaches when it used to. Neither emits a phase_time row, because there is no ratio to report against a missing side and a row reading "ok" against a zero baseline is worse than no row. Rows are marked (NEW)/(VANISHED). Timing noise widened the MEMORY budget. One budget was computed per scenario and handed to every metric, so a baseline with a noisy clock (MAD 100 on a median of 1000 = a 30% floor) let a +25% peak_bytes regression through a 5% memory budget. An allocator's peak is near-deterministic for a given workload; a noisy clock is no reason to forgive it. The floor now reaches time and phase_time only. The noise floor is baseline-only. It previously took max(baseline, current), which lets an erratic run widen its own budget — the worse a run behaves, the more it is forgiven. The baseline is the controlled measurement and alone gets to say how much a scenario moves. The host-mismatch remedy suggested relabelling the host (--host-class / FLUREE_BENCH_HOST_CLASS "to the class the baseline was captured on"). Renaming a machine does not make its numbers comparable: following that advice converts the refusal into a silent, confident ENFORCEMENT against hardware the baseline never ran on — the exact failure the refusal exists to prevent. The remedy now leads with --allow-host-mismatch, and the usage text says what --host-class is and is not for. Also here, from the same pass: accumulate now enforces host-class AND corpus identity, and both guards live in the library rather than in the CLI that happened to call it. Accumulation is the one operation that DESTROYS the evidence of a mismatch — two populations blended into a single median/MAD, with the provenance overwritten in the same breath — so a bench harness calling accumulate() directly must not be able to skip the check. Returns Result; the doc comment no longer describes protection that wasn't there. with_phase_ns MERGES instead of replacing, recomputing shares over the union. Replacing silently discarded an earlier call's phases while leaving the remaining shares looking perfectly well-formed — invisible in the output. And PhaseTiming's docs no longer claim ns and share_pct "can never disagree": that is a property of the constructor, not an invariant, since both fields are public and a baseline file is untrusted input. compare now reports a scenario whose shares don't sum to 100% instead of assuming they do — reported, not gated, since the drift comparison is still internally consistent. Regression tests come from the reviewer's three repro fixtures and all three are verified against the real binary, not just the library.
…new rules The phase and corpus machinery reads in BENCHMARKING.md as though it were running today. It isn't: no bench in the workspace calls meta::record_scenario, so no phase or corpus row appears in any report this branch produces. That is deliberate — the gate has to exist before the benches that depend on it — but a reader had no way to tell scaffolding from live coverage. Stated up front. Also documents the rules the fix commit introduced: one-sided phase gating and why comparing only same-named phases is itself a hiding place; the three bounds on the noise floor (widening-only, wall-clock-only, baseline-only) each with the inversion it prevents; and accumulate's refusal to pool across host classes or corpora. The compare quickstart now carries --allow-host-mismatch inline. Without it the copy-pasteable command refuses against the committed baseline, which is exactly the flag the reader needs to understand — leaving it in the options list taught the rule only to people who already knew it.
Cargo.lock recorded fluree-bench-virtual at 4.1.3 while the crate inherits `version.workspace = true` and [workspace.package] declares 4.1.4. The entry was stale on main itself, not introduced here: any plain `cargo build` or `cargo metadata` on a main-derived branch rewrites this same line and leaves a dirty tree. Nothing enforces it today — no workflow passes --locked or --frozen — so this is hygiene rather than a fix for a live failure. It rides along with this PR only because building this branch is what surfaced it; it is one line of lockfile churn with no dependency resolution change, and it is deliberately a separate commit so it can be dropped or cherry-picked independently of the bench work.
bplatz
left a comment
There was a problem hiding this comment.
@aaj3f I did a defect-focused pass over the gate and workflow changes. I found two fail-open paths that look merge-blocking (missing corpus metadata and invalid phase shares), plus several accumulation/provenance and input-hardening issues. Details are inline; I left this as a comment review rather than changing the approval state.
| // Corpus identity gates everything else about this scenario: if the two | ||
| // runs read different inputs, none of their numbers are comparable and | ||
| // emitting rows for them would be worse than emitting nothing. | ||
| if let (Some(base_corpus), Some(cur_corpus)) = |
There was a problem hiding this comment.
This treats (Some(base_corpus), None) as comparable. If a producer stops emitting its sidecar, or the sidecar write/read fails, the current run bypasses the corpus-identity gate and can exit 0. Once the baseline declares a corpus, missing current metadata should be a refusal. The same asymmetry exists in accumulate around line 626, where fresh.clone() can also erase the prior identity while pooling the sample. Could you add compare and accumulate regression tests for the Some/None case?
There was a problem hiding this comment.
Fixed in ecba51446. You're right that absence of evidence was being read as evidence of the same input. (Some, None) is now a refusal; the reverse (None, Some) stays permitted, since a schema-1 baseline predates corpus blocks and refusing there would make every old baseline unusable — that asymmetry is now explicit in a corpus_refusal helper that both compare and accumulate call, so the two can't drift apart again. accumulate had the same hole plus the fresh.clone() erasure you spotted. Regression tests both places: baseline_corpus_with_no_current_corpus_refuses and accumulate_refuses_when_fresh_drops_the_corpus.
| return; | ||
| } | ||
| let sum: f64 = phases.values().map(|p| p.share_pct).sum(); | ||
| if (sum - 100.0).abs() > SHARE_SUM_TOLERANCE_PP { |
There was a problem hiding this comment.
I don't think invalid shares are cosmetic here. On a host mismatch, phase_share is the only enforcing signal; a malformed current sidecar with all shares at 0 makes every phase look improved while this only emits an anomaly and the command exits 0. Since the file is explicitly treated as untrusted input, please refuse non-finite/negative values and a materially invalid sum (or recompute shares from ns) before gating.
There was a problem hiding this comment.
Agreed, and this was the worse of the two — thank you. Treating shares as cosmetic is exactly wrong in the case where they're the only thing enforcing. Fixed in ecba51446: shares are validated before anything gates on them, and non-finite, negative, or a sum more than 1pp off 100 is now a refusal (MismatchKind::PhaseShares) rather than an annotation. I went with refusing over recomputing from ns to match the rest of the gate — a file that disagrees with itself gets "I won't guess" — and the remedy string names ScenarioMeta::with_phase_ns as the way to regenerate. The anomalies channel is gone since this was its only user. Your exact scenario is pinned by all_zero_current_shares_refuse_instead_of_gating_as_improved, which asserts no bogus Improved row is emitted.
| /// - **host class** must match between `prev` and `fresh`. | ||
| /// - **corpus identity** must match for every scenario present in both, | ||
| /// resolved entry-then-file exactly as [`compare`] resolves it. | ||
| pub fn accumulate(prev: &BaselineFile, fresh: &BaselineFile) -> Result<BaselineFile, String> { |
There was a problem hiding this comment.
compare refuses a schema newer than this binary understands, but accumulate has no equivalent guard. An older binary can deserialize an additive schema-3 file, clone a fresh schema-2 capture, and save it back while silently dropping fields it did not understand. Please reject prev.schema_version > BASELINE_SCHEMA_VERSION before merging.
There was a problem hiding this comment.
Fixed in ecba51446 — accumulate now rejects prev.schema_version > BASELINE_SCHEMA_VERSION before merging, matching compare. The silent-field-drop path you describe is the real hazard: it writes the file back, so the loss is permanent rather than just a bad comparison. Test: accumulate_refuses_a_newer_schema.
| } | ||
| } | ||
|
|
||
| let mut out = fresh.clone(); |
There was a problem hiding this comment.
This replaces git_sha with the fresh SHA while pooling the previous samples without checking revision identity. Running --accumulate after a code change mixes pre/post-change populations, relabels all samples as the new revision, and can widen MAD enough to hide the change being measured. I think differing non-empty SHAs should refuse accumulation unless there is an explicit override.
There was a problem hiding this comment.
Fixed in ecba51446. Differing non-empty git_shas now refuse, with --allow-revision-drift (and AccumulateOptions::allow_revision_drift for library callers) as the explicit override. The MAD-widening consequence you name is the one that worried me most — it doesn't just mislabel the samples, it actively raises the threshold that would have caught the change. Test: accumulate_refuses_across_revisions_unless_overridden, plus accumulate_allows_a_matching_or_absent_revision for the normal loop and the no-git case.
| } | ||
| } | ||
|
|
||
| fn is_usable(&self) -> bool { |
There was a problem hiding this comment.
n, median_ns, and mad_ns are trusted independently of samples_ns. A stale or hand-edited baseline can claim n >= 5 with an arbitrary median/MAD and thereby move the comparison baseline or widen the budget. Since the raw samples are stored, load/compare should recompute these fields or reject an inconsistent/non-positive set.
There was a problem hiding this comment.
Fixed in ecba51446. Since the raw samples are stored, as you say, they're now the authority: n/median_ns/mad_ns are recomputed from samples_ns at load. A set that can't be recomputed — fewer than MIN_NOISE_SAMPLES, or any non-finite/non-positive sample — is returned unusable rather than trusted, which drops the floor for that entry. That's the safe direction, since the floor only ever widens a budget. Tests: noise_stats_are_recomputed_from_their_own_samples_at_load, fabricated_noise_without_samples_is_unusable, non_positive_samples_are_rejected_as_a_floor.
| #[serde(default)] | ||
| corpus: Option<CorpusInfo>, | ||
| #[serde(default)] | ||
| entries: BTreeMap<String, BaselineEntry>, |
There was a problem hiding this comment.
Because all top-level fields default, {} deserializes as a valid schema-1 baseline. With --allow-host-mismatch, every current scenario is merely reported as new, no metric breaches, and compare exits 0. Please validate that a loaded baseline has the required structure and at least one scenario rather than accepting an empty document.
There was a problem hiding this comment.
Fixed in ecba51446. load_baseline now requires schema_version >= 1 and at least one scenario. The failure mode you describe is the one I'd least want to ship — a gate that passes because it had no reference points looks identical to a gate that passed on merit. Test: empty_document_is_rejected_at_load covers both {} and an explicit empty entries.
| /// First 12 hex chars, for report lines. Never used for comparison. | ||
| pub(crate) fn short_hash(hash: &str) -> &str { | ||
| let end = hash.len().min(12); | ||
| &hash[..end] |
There was a problem hiding this comment.
This byte slice can panic when an untrusted sha256 contains Unicode and byte 12 is not a UTF-8 boundary, turning a documented refusal into exit 101. Validating the field as 64 lowercase ASCII hex characters would also protect identity semantics; otherwise truncate by characters here.
There was a problem hiding this comment.
Fixed in 55cc517e7, both ways you suggested. short_hash now slices on a character boundary so no input can panic it, and CorpusInfo::validate rejects anything that isn't exactly 64 lowercase ASCII hex digits. Doing only the truncation would have left the identity semantics resting on an unvalidated field, and doing only the validation would have left the panic reachable from the very error path that reports it — the crash landed precisely when the tool was trying to explain its refusal. Tests: short_hash_never_panics_on_multibyte_input and validate_rejects_malformed_hashes.
| for scale in tiny small; do | ||
| FLUREE_BENCH_SCALE=$scale cargo bench -p fluree-db-api --bench query_overlay_matrix | ||
| FLUREE_BENCH_SCALE=$scale cargo bench -p fluree-db-api --bench query_hot_bsbm | ||
| samples="${{ github.event.inputs.capture_samples || '1' }}" |
There was a problem hiding this comment.
capture_samples=0 makes seq 1 0 run no passes. The baseline is never created, and upload-artifact only warns on a missing file by default, so the capture job can succeed without an artifact. Please validate a positive integer and set if-no-files-found: error on the upload step.
There was a problem hiding this comment.
Fixed in fb2f60b41. Validates a positive integer before the loop and sets if-no-files-found: error on the upload. A capture job that goes green having captured nothing is the worst outcome available here, because the next person reasonably assumes the artifact exists.
|
|
||
| Two ways to reach phase 2: | ||
|
|
||
| 1. **Committed CI-class baseline (implemented).** Run `bench.yml`'s |
There was a problem hiding this comment.
The capture job writes guardrails-pre-ci.json, while both CI compare jobs read guardrails-pre.json. Following 'download the artifact, commit it, and remove --allow-host-mismatch' literally leaves CI reading the old local baseline and refusing every run. Please explicitly say to replace/rename guardrails-pre.json, or update the compare path as part of phase 2.
There was a problem hiding this comment.
Fixed in 517a03c5a. You're right that following it literally lands phase 2 as a red gate — the flag comes off and the compare then refuses against the stale local-class file. The doc now spells out the mv guardrails-pre-ci.json bench-baselines/guardrails-pre.json replace step explicitly, and names the alternative (keep the -ci filename and repoint every --baseline path) so the choice is visible rather than implied.
…iew 7) `short_hash` sliced `&hash[..12]` by BYTE offset. A baseline file is untrusted input, so `sha256` can hold anything — and when byte 12 lands mid-codepoint the slice panics. It is called from the refusal path, so the failure mode was a documented exit-2 refusal turning into an exit-101 crash: the gate crashes exactly when it is trying to tell you why it won't compare. Two fixes, because either alone leaves a gap. `short_hash` now slices on a character boundary, so no input can panic it. And `CorpusInfo::validate` rejects any hash that is not exactly 64 lowercase ASCII hex digits — a hash nothing validated cannot establish identity, so accepting one would assert "same input" on the strength of an unparsed field.
Six review items, one theme: a baseline file is untrusted input, and in each
case the gate accepted something it could not verify and exited 0. Grouped in
one commit because they are interleaved edits to the same load/compare path in
baseline.rs; splitting them would mean commits that don't build.
[BLOCKING] Corpus, `(Some, None)` (item 1). `compare` only checked identity
when BOTH sides declared a corpus, so a producer that stopped emitting its
sidecar — or a sidecar write that failed — bypassed the identity gate entirely.
Absence of evidence was read as evidence of the same input. Now: baseline
declares a corpus and the current run declares none is a REFUSAL. The reverse
stays permitted, and that asymmetry is deliberate — a schema-1 baseline
predates corpus blocks, so refusing there would make every old baseline
unusable. Both directions are pinned. `accumulate` had the same hole plus a
worse consequence, since `fresh.clone()` also erased the prior identity while
pooling the sample; it now shares one `corpus_refusal` helper with `compare` so
the two cannot drift apart.
[BLOCKING] Shares (item 2). These were treated as cosmetic — a report line,
never a gate. That is wrong in the one case that matters: on a host mismatch
`phase_share` is the ONLY enforcing metric, so a malformed current sidecar with
every share at 0.0 marked every phase "improved" while the absolute metrics
were merely advisory, and the command exited 0. Shares are now validated before
anything gates on them: non-finite, negative, or a sum more than 1pp off 100 is
a refusal (`MismatchKind::PhaseShares`). Refusing rather than silently
recomputing from `ns` matches the rest of the gate — a file that disagrees with
itself gets "I won't guess", and the remedy names the constructor that derives
shares correctly. The `anomalies` channel is removed; it existed only for this.
accumulate schema guard (item 3). `compare` refused a schema newer than the
binary understands; `accumulate` did not, so an older binary could deserialize
an additive future schema, clone a fresh capture over it, and write it back
having silently dropped every field it didn't recognise.
accumulate revision guard (item 4). Pooling samples across a code change mixes
pre- and post-change populations, relabels every sample as the new revision,
and can widen the MAD enough to hide the change being measured. Differing
non-empty `git_sha`s now refuse unless `--allow-revision-drift` says otherwise.
NoiseStats trust (item 5). `n`, `median_ns`, and `mad_ns` were trusted
independently of the `samples_ns` they claim to summarise, so a hand-edited
file could claim n=5 beside an arbitrary median and MAD and thereby move the
comparison point and widen the budget. They are now recomputed from the samples
at load. A set with too few samples, or any non-finite/non-positive sample,
cannot be recomputed and is returned unusable rather than trusted — losing a
floor is the safe direction, since a floor only ever widens.
Empty-document guard (item 6). Every top-level field carries a serde default so
schema-1 files keep loading, which meant `{}` also deserialized "successfully".
An empty baseline compares against nothing, breaches no metric, and exits 0 —
a gate that passes because it had no reference points. `load_baseline` now
requires schema_version >= 1 and at least one scenario.
`capture_samples: 0` made `seq 1 0` run zero passes, so no baseline was ever written — and `upload-artifact` only warns on a missing file by default, so the job went green having produced nothing. A capture job that silently captures nothing is the worst outcome available here, because the next person assumes an artifact exists. Validates a positive integer before the loop, and sets `if-no-files-found: error` on the upload so the belt has a brace.
The capture job writes `guardrails-pre-ci.json`; both compare jobs read `guardrails-pre.json`. Following "download the artifact, commit it, and remove --allow-host-mismatch" literally leaves CI reading the old host_class=local file with the flag gone — which refuses every run, so phase 2 would land as a red gate rather than a working one. Spells out the replace step with the actual command, and notes the alternative (keep the -ci name, repoint every --baseline path) so the choice is visible rather than implied.
|
@bplatz -- ready for re-review when you have time |
bplatz
left a comment
There was a problem hiding this comment.
Re-reviewed. All nine items are addressed, and I checked each against the code rather than the replies — the two fail-open paths I called merge-blocking are genuinely closed:
corpus_refusalis a single helper called from bothcompareandaccumulate, so the asymmetry can't drift apart again, and the(Some, None)direction refuses.- Share validation runs ahead of any gating decision, and
MismatchKind::PhaseSharesrefuses rather than annotates. The all-zero-shares case emits no bogusImprovedrow. - Noise stats are recomputed from
samples_nsat the serde boundary, so every reader sees derived numbers; an unrecomputable set drops the floor, which is the widening-only direction. {}and an explicit emptyentriesare both rejected at load.short_hashis char-boundary safe and the hash is validated — agreed that doing only one would have left either the panic or the identity semantics exposed.- The capture job fails a non-positive
capture_samplesand errors on an empty upload. - The doc names the
mvexplicitly, so phase 2 doesn't land as a red gate.
Verified locally: 139/139 tests (17 new, one per item), clippy --all-targets clean, fmt --check clean, and the committed schema-1 guardrails-pre.json still refuses at exit 2 and compares advisory under --allow-host-mismatch.
Approving. Two residuals inline — both non-blocking and both latent today. They're the same shape as items 1 and 4 on the sibling code path, so worth a look before merge rather than a reason to hold it.
| } | ||
| // Not rerun this pass: carry the previous entry forward whole. | ||
| _ => { | ||
| out.entries.insert(id.clone(), prev_entry.clone()); |
There was a problem hiding this comment.
The corpus guard above skips ids absent from fresh (the continue at the top of that loop), but this arm re-inserts exactly those entries into out — whose file-level corpus is fresh's. An entry that declared no entry-level corpus and was inheriting prev.corpus therefore resolves against a different corpus after the merge, with its samples unchanged.
Reproduced with prev = {q1, q2} under a file-level corpus A, and a pass that reran only q3 under corpus B:
prev q2 corpus = ("A", "61be55a8")
merged q2 corpus = ("B", "81cc5b17") # same samples, relabelled
The overlapping-id case is correctly refused, so this only bites when a pass doesn't rerun every scenario — --only subsetting, or a bench that emits no criterion output that pass. The CI loop reruns the same subset each pass, and nothing produces corpus blocks yet, so it isn't reachable today. It is the same silent identity rewrite as the (Some, None) case though, just on the carry-forward path instead of the pooling path, so I'd rather it not sit latent: either stamp the resolved prev.corpus_for(id) onto the carried-forward entry, or run the identity check over the union of prev.entries and fresh.entries rather than the intersection.
There was a problem hiding this comment.
Fixed in e06187439, and I did both — they turn out to be complementary rather than alternatives, which I didn't expect going in.
The union walk alone still lets the case where prev resolved to NO corpus and fresh introduces a file-level one slip through: the carried entry inherits an identity its samples never had. The stamp alone has nothing to stamp in exactly that case. So: carried-forward entries get prev.corpus_for(id) stamped when it resolves to something, and where it resolves to nothing while fresh declares a corpus, the merge refuses — "explicitly no corpus" isn't expressible through inheritance, so there's no honest value to write.
Resolution stays single-sourced as you asked: every decision on both sides goes through corpus_for, and the intersection case still calls the same corpus_refusal helper compare uses.
Your repro is the regression test verbatim (carried_forward_entries_keep_their_own_corpus): prev {q1,q2} under file-level A, a pass rerunning only q3 under B, asserting q2 still resolves to A with its samples unchanged and that q3 keeps B. Plus carrying_forward_into_a_newly_declared_corpus_refuses for the unstampable case. Both confirmed failing before the fix.
| )); | ||
| } | ||
| if !opts.allow_revision_drift { | ||
| if let (Some(prev_sha), Some(fresh_sha)) = (&prev.git_sha, &fresh.git_sha) { |
There was a problem hiding this comment.
This requires both sides Some, and the merge below takes fresh's git_sha — so a single pass with no revision resets the accumulator and reopens the gate for every pass after it:
after a no-git pass, git_sha = None
next pass at a different revision -> ACCEPTED
pooled samples = [1000.0, 1010.0, 5000.0]
labelled git_sha = Some("ccccccc")
That's the MAD-widening case this guard exists for, reached in two steps rather than one. The trigger is git_short_sha() returning None, so it's a local/container concern rather than a CI one — CI always has git. Carrying prev.git_sha forward when fresh has none, or treating None vs Some as drift, would close it, and either way --allow-revision-drift stays the explicit path.
There was a problem hiding this comment.
Fixed in e06187439. I went with treating None vs Some as drift rather than carrying prev.git_sha forward.
Carrying it forward would assert that the fresh samples were taken at a revision nothing actually observed — fabricating provenance in order to preserve a label, which is the same shape as the identity rewrites in R1 and item 1. Treating absence as part of revision identity says the honest thing instead: a sample whose revision is unknowable is precisely the one that can't be pooled safely. Both-absent still accumulates fine, so the no-git tarball/container workflow is unaffected, and --allow-revision-drift remains the explicit path through.
Your two-step repro is the test (a_no_git_pass_cannot_launder_a_revision_change), written to accept either outcome — the no-git pass itself refusing, or it being accepted and the next pass at a different revision refusing — so it pins the hole rather than my particular choice of where to close it. Confirmed failing before the fix. One earlier test flipped with the contract: one-sided None was asserted as allowed and is now asserted as drift in both directions.
Both are the same defect class as the (Some, None) corpus hole, on sibling paths in the same function, so they land together. Both repros reproduced first: three tests written against the unfixed code, confirmed failing, then fixed. R1 — carry-forward relabelling. The corpus guard walked the INTERSECTION of prev and fresh entries, but the carry-forward arm re-inserts precisely the entries it skipped, into a file whose file-level corpus is fresh's. An entry that declared no entry-level corpus and was inheriting prev's therefore resolved against a different corpus after the merge, samples unchanged. Fixed by walking the UNION and, for carried-forward entries, stamping the corpus they actually resolved to. Both of the suggested shapes, and they turn out to be complementary rather than alternatives: the union walk alone still lets prev-had-none inherit a newly declared file-level corpus, and the stamp alone has nothing to stamp in that case. So the stamp preserves identity where identity exists, and where it does not — prev resolved to no corpus, fresh declares one — the merge refuses, because "explicitly no corpus" cannot be expressed through inheritance and the entry would otherwise claim an identity its samples never had. Resolution stays single-sourced: every decision goes through corpus_for on both sides. R2 — revision laundering. The guard required both sides Some and the merge took fresh's git_sha, so ONE pass with no git reset the accumulator to None and every later pass sailed through a guard that needed two Somes to fire. The MAD-widening case the guard exists for, reached in two steps instead of one. Fixed by comparing revision identity as a whole, absence included: None vs Some is drift like any other. Chosen over carrying prev's sha forward because that would assert the fresh samples were taken at a revision nothing observed — fabricating provenance to preserve a label. A sample whose revision is unknowable is exactly the one that cannot be pooled safely. Both-absent still accumulates fine (tarball or container builds have nothing to disagree about), and --allow-revision-drift remains the explicit path through. One earlier test changes with the contract: one-sided None was asserted as allowed and is now asserted as drift, in both directions.
…conflict-free review Single conflict: fluree-db-api/src/lib.rs — convergent cfg-orphan fix. Both #1566 and the audit side moved build()'s #[cfg(native)] doc block below with_secret_resolver (kept iceberg-gated). The cfg fix is byte-identical on both sides; resolved the lone doc-wording hunk to main's #1566 text, so the merged file equals origin/main's api/lib.rs (zero diff, no audit feature lost). hash_join.rs, optional.rs, planner.rs auto-merged (disjoint hunks: audit #1477 fast-path vs main cypher #1553).
PR-bench: baseline extensions (host class, corpus identity, phase shares, noise floor)
Why
#1477 gave the workspace a real baseline capture/compare tool. This PR closes the
four gaps between that tool and what the
fluree rdfconversion effort needs fromit (
riot-analog-bench-strategy.md§6b, F5/F6), and corrects a doc that haddrifted from the workflows it describes.
The gaps share a theme: a compare run that emits a number which looks like a
result and is not one. Today the tool will happily compare a laptop against a CI
runner, a Turtle corpus against an N-Triples one, or a single flaky sample against
another, and print a tidy Δ% table either way.
What changed
1. Host block, and a refusal instead of a silent downgrade
runner_classbecomeshost.classinside a block that also records os, arch,cpu model, and physical/logical cores (probed via
sysctlon Darwin,/proc/cpuinfoon Linux, falling back toavailable_parallelism).A class names a comparability set, not a machine — two hosts share one when a
human has decided their absolute numbers may be compared. A mismatch is now a
refusal (exit 2), not a quiet flip to advisory.
--allow-host-mismatchoptsback into the advisory posture.
The derived default is
{os}-{arch}, and the docs say plainly that it is coarseand not a promise: an M1 and an M4 both derive
macos-aarch64. Any host whosenumbers are meant to gate should set
FLUREE_BENCH_HOST_CLASSexplicitly.FLUREE_BENCH_RUNNER_CLASSis still read as the pre-rename spelling, so noexisting workflow breaks.
2. Corpus identity — the expensive silent failure
A new
metasidecar (target/fluree-bench-meta/, same shape as #1477'smemsidecar) carries per scenario:
sha256,byte_len,element_count,input_syntax,output_syntax,thread_count. Declarable per entry, or once atfile level and inherited.
A mismatch in hash, syntax, or thread count is a refusal with no override.
Unlike the host check there is no posture in which comparing different inputs is
meaningful, and the resulting table is indistinguishable from a real regression.
nameandelement_countare deliberately not part of the identity — two pathsto identical bytes are the same corpus.
SHA-256, not BLAKE3. blake3 is not in the workspace;
sha2already is. Acorpus is hashed once per capture (streamed in 1 MiB chunks), so the throughput
difference does not pay for a new dependency. No new workspace dependencies.
3. Per-phase timings and share drift
Per-entry
{phase: {ns, share_pct}}, with the share derived from the ns at recordtime (
ScenarioMeta::with_phase_ns) so the two can never disagree.phase_timegates only within a host class, liketimeandpeak_mem.phase_sharegates regardless of host class. A share is a ratio within onerun, so the machine cancels. This is the only signal a cross-host comparison
keeps, and it is the one that catches a regression an aggregate number hides.
Only share growth gates. Shares sum to 100, so gating both directions would fail
a clean improvement twice — once for the phase that got better, once for its
complement. Shrink is reported as
improvedso the whole picture stays visible.One-sided phases are gated on share alone. Comparing only same-named phases
is itself a hiding place: move work into a phase the baseline never had and the
old phase's share falls — reading as an improvement — while the new phase
carrying the cost produces no row. So a phase new in the current run breaches
when it claims more than the drift threshold, and a phase that has vanished
breaches when it used to. Rows are marked
(NEW)/(VANISHED).Here is the case that motivates the whole feature, from the end-to-end smoke run —
whole-scenario time is inside its noise floor and clean, while the phase breakdown
localizes a 28% parse regression masked by a write improvement:
4. Noise floor
capture --accumulatefolds repeat runs into one file: each pass appends the freshcriterion mean to a per-scenario sample vector and recomputes median + MAD. At
n ≥ 5 the gate compares medians instead of single-run means, and widens the
wall-clock budget to
max(budget_pct, 3 × MAD / median).Criterion's own confidence interval describes iterations within one run and says
nothing about run-to-run spread — which is exactly what shared-runner flap is. MAD
rather than standard deviation because one outlier run is the common failure here
and barely moves MAD.
Three bounds, each of which inverts the gate if dropped: the floor only widens
(a tighter-than-noise budget can only false-alarm); it is wall-clock only,
never touching
peak_mem(an allocator peak is near-deterministic, so a noisyclock is no reason to forgive an allocation regression); and it is baseline
only (otherwise the worse a run behaves, the more it is forgiven).
accumulateitself refuses to pool samples across host classes or corpora, andboth guards live in the library rather than the CLI. Accumulation is the one
operation that destroys the evidence of a mismatch — two populations blended
into one median/MAD, with the provenance overwritten in the same breath — so a
direct caller must not be able to skip the check.
5. BENCHMARKING.md corrections
Every claim checked against the merged main + #1477 tree. Five were wrong:
.github/workflows/bench-nightly.yml(in "Where benches live")bench.yml[[bench]]entries are registered — six missingBenchProfile::Full= "Nightly CI" (runtime.rs)quick/tiny, nightly includedbench-baselines/README.md: "enforces memory across any runner class, time advisory"Note on the "runs on every PR" and "dedicated 4-core runners" claims from the
original brief: #1477 already removed both in its rewrite of the CI-gate
section. They are gone from the merged result; nothing to do.
The six missing bench rows are
query_overlay_matrix,query_hot_bsbm_bi,query_hot_property_path,query_hot_whole_graph_agg,annotation_hydration,annotation_planner(five are missing on main; #1477 adds the first). Each row iswritten from the bench's own module docs. The table now also states outright that
nothing enforces it — unlike the
[[bench]]↔ budget reconcile — which is why itrots.
Sample baseline (schema 2)
{ "schema_version": 2, "label": "convert-pre", "git_sha": "c7d82614d", "captured_at": "2026-07-28T23:54:15Z", "profile": "quick", "scale": "small", "host": { "class": "bench-m8gd", "name": "ip-10-0-3-7", "os": "linux", "arch": "x86_64", "cpu_model": "AMD EPYC 9R14", "physical_cores": 16, "logical_cores": 32 }, "entries": { "convert/ttl_to_nq/small": { "mean_ns": 990000.0, "median_ns": 990000.0, "mem": { "peak_bytes": 24117248, "total_allocated_bytes": 310378496 }, "noise": { "n": 5, "median_ns": 1000000.0, "mad_ns": 10000.0, "samples_ns": [980000.0, 990000.0, 1000000.0, 1010000.0, 1020000.0] }, "corpus": { "name": "dblp.ttl", "sha256": "aaaa1111bbbb2222cccc3333dddd4444eeee5555ffff6666aaaa7777bbbb8888", "byte_len": 1048576, "element_count": 40000, "input_syntax": "turtle", "output_syntax": "nquads", "thread_count": 8 }, "phases": { "read": { "ns": 100000.0, "share_pct": 10.0 }, "parse": { "ns": 600000.0, "share_pct": 60.0 }, "write": { "ns": 300000.0, "share_pct": 30.0 } } } } }Every added block is optional. A schema-1 file still loads:
runner_classmaps tohost.class, a barehoststring maps tohost.name, and missingnoise/corpus/phasessimply switch off their respective rules. The committedbench-baselines/guardrails-pre.jsonis unmodified and still compares. Abaseline claiming a newer schema version is a refusal — silently ignoring fields
we don't understand is how a gate stops gating.
Gating semantics, complete
timepeak_memphase_timephase_shareThe noise floor is wall-clock-only and baseline-only. It never touches
peak_mem— an allocator peak is near-deterministic, so a noisy clock is noreason to forgive an allocation regression — and the current run's own spread
never widens its own budget, or the worse a run behaves the more it is forgiven.
Phase rows also carry a presence marker. A phase present on both sides compares
normally; one new in the current run breaches when its share exceeds the drift
threshold (
(NEW)); one that has vanished breaches when it used to(
(VANISHED)). Neither one-sided case emits aphase_timerow.--allow-host-mismatchschema_versionnewer than the binaryCI impact: none, by construction
The per-PR
bench-compareand nightlybench-gatecompare ahost_class=localbaseline on
ci-ubuntu-latest, which the new default would refuse. Both steps nowpass
--allow-host-mismatchexplicitly, reproducing #1477's exact behaviour(annotations, exit 0). The change is that the downgrade is written down, with a
comment naming the condition for removing the flag.
bench-capturegains acapture_samplesdispatch input (default1, unchangedbehaviour). At
≥ 5it reruns the subset and accumulates, producing the median +MAD a baseline needs before it can gate through runner flap.
Files
fluree-bench-support/src/meta.rsCorpusInfo,PhaseTiming,ScenarioMeta, sidecar IO, SHA-256fluree-bench-support/src/baseline.rsHostBlock,NoiseStats,CompareOptions,FatalMismatch,Metric, refusal + share + noise gating, schema-1 read pathfluree-bench-support/src/bin/bench_baseline.rs--host-class,--allow-host-mismatch,--share-drift-pp,--accumulate,--clean-sidecars; exit code 2; per-metric renderingfluree-bench-support/src/runtime.rsFullis not the CI profile)fluree-bench-support/src/lib.rs,Cargo.tomlmetamodule;sha2(existing workspace dep).github/workflows/{ci,bench}.yml--allow-host-mismatch;capture_samples; clear the meta sidecar dirBENCHMARKING.md,bench-baselines/README.md,fluree-bench-support/README.mdVerification
57 of the 121 lib tests are new (measured against this branch's base at
06f1c10cf, which has 64). Every gating rule has one: host mismatch fatal /allowed / matched / forced-advisory, corpus mismatch by hash, by syntax, by thread
count, file-level vs entry-level precedence, one-sided declaration, share growth /
shrink / within-threshold / configurable-threshold / cross-host enforcement, MAD
arithmetic, floor widening, floor not tightening, floor not hiding a real move,
sub-threshold sample counts, accumulate arithmetic, and the schema-1 read path
including the exact shape of the committed
guardrails-pre.json.Also exercised end-to-end against the real binary: the committed schema-1 baseline
(refusal, then advisory under the flag), a captured schema-2 baseline accumulated
to n=5, a share-drift breach under a passing aggregate, a corpus swap, and
cross-host share enforcement.
Deviations from the §6b spec
sha2is. The specpermits this ("prefer no new deps").
|Δ|woulddouble-flag every real improvement, since shares sum to 100. Both directions are
still reported.
--allow-host-mismatchexists at all. The spec says compare hard-errors onhost mismatch; making that unconditional would red every PR today, since the
only committed baseline is cross-class. The refusal is the default and the
escape hatch is explicit, per-invocation, and documented — strictly louder than
bench+test: query correctness guardrails — differential harness, baseline gating, and four fast-path fixes #1477's silent downgrade.
thread_countis part of corpus identity, though the spec listed it as arecorded field. A K=8 run compared against a K=16 baseline is the same class of
silent failure as a corpus swap. Its refusal message is distinct so a reader can
tell a config mismatch from a data mismatch.
Open questions
guardrails-pre.jsonbe re-stamped to a specific host class? It wascaptured on Apple silicon but records
host_class=local. Re-stamping itmacos-aarch64would make a Mac dev's local compare enforce — but would alsoassert that any two Macs are comparable, which is false. Left as
localdeliberately; it is the honest value.
host_classstring does the official publication locus use? H-10approved
m8gd.4xlarge; the strategy's F7 wants publication tooling to refusenon-official classes. This PR provides the field; the allowlist is not
implemented and needs the canonical string.
bench-baselines/convert/per §6b, with rolling per-change baselines gitignored? Not created here (no
convert benches exist yet).