Skip to content

bench: baseline host/corpus identity, phase-share gating, and noise floors - #1550

Merged
aaj3f merged 12 commits into
mainfrom
bench/baseline-extensions
Jul 30, 2026
Merged

aaj3f merged 12 commits into
mainfrom
bench/baseline-extensions

Conversation

@aaj3f

@aaj3f aaj3f commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

PR-bench: baseline extensions (host class, corpus identity, phase shares, noise floor)

Builds on the BaselineFile introduced by #1477, which has now merged —
this branch is rebased onto 06f1c10cf (#1477's merge-reconcile head), which is
in main's history, so it merges cleanly with no rebase outstanding.

The last commit is a disclosed rider, deliberately separable: a one-line
Cargo.lock refresh (fluree-bench-virtual 4.1.3 → 4.1.4). That entry was
stale on main itself — any plain cargo build on a main-derived branch
rewrites it — and building this branch is what surfaced it. Drop or cherry-pick
it independently; it carries no dependency-resolution change.

Why

#1477 gave the workspace a real baseline capture/compare tool. This PR closes the
four gaps between that tool and what the fluree rdf conversion effort needs from
it (riot-analog-bench-strategy.md §6b, F5/F6), and corrects a doc that had
drifted from the workflows it describes.

The gaps share a theme: a compare run that emits a number which looks like a
result and is not one.
Today the tool will happily compare a laptop against a CI
runner, a Turtle corpus against an N-Triples one, or a single flaky sample against
another, and print a tidy Δ% table either way.

What changed

1. Host block, and a refusal instead of a silent downgrade

runner_class becomes host.class inside a block that also records os, arch,
cpu model, and physical/logical cores (probed via sysctl on Darwin,
/proc/cpuinfo on Linux, falling back to available_parallelism).

A class names a comparability set, not a machine — two hosts share one when a
human has decided their absolute numbers may be compared. A mismatch is now a
refusal (exit 2), not a quiet flip to advisory. --allow-host-mismatch opts
back into the advisory posture.

The derived default is {os}-{arch}, and the docs say plainly that it is coarse
and not a promise: an M1 and an M4 both derive macos-aarch64. Any host whose
numbers are meant to gate should set FLUREE_BENCH_HOST_CLASS explicitly.
FLUREE_BENCH_RUNNER_CLASS is still read as the pre-rename spelling, so no
existing workflow breaks.

2. Corpus identity — the expensive silent failure

A new meta sidecar (target/fluree-bench-meta/, same shape as #1477's mem
sidecar) carries per scenario: sha256, byte_len, element_count,
input_syntax, output_syntax, thread_count. Declarable per entry, or once at
file level and inherited.

A mismatch in hash, syntax, or thread count is a refusal with no override.
Unlike the host check there is no posture in which comparing different inputs is
meaningful, and the resulting table is indistinguishable from a real regression.
name and element_count are deliberately not part of the identity — two paths
to identical bytes are the same corpus.

SHA-256, not BLAKE3. blake3 is not in the workspace; sha2 already is. A
corpus is hashed once per capture (streamed in 1 MiB chunks), so the throughput
difference does not pay for a new dependency. No new workspace dependencies.

Nothing produces phases or corpus blocks yet. meta.rs is Tier-1
scaffolding for the conversion lane: schema, recorder, and gate are in place,
but no bench in the workspace calls meta::record_scenario, so no phase or
corpus row appears in any report this PR produces. The share gate and the
corpus refusal go live with the first producer. That is deliberate — the gate
has to exist before the benches that depend on it, and it ships with full test
coverage rather than waiting.

3. Per-phase timings and share drift

Per-entry {phase: {ns, share_pct}}, with the share derived from the ns at record
time (ScenarioMeta::with_phase_ns) so the two can never disagree.

  • Absolute phase_time gates only within a host class, like time and
    peak_mem.
  • phase_share gates regardless of host class. A share is a ratio within one
    run, so the machine cancels. This is the only signal a cross-host comparison
    keeps, and it is the one that catches a regression an aggregate number hides.

Only share growth gates. Shares sum to 100, so gating both directions would fail
a clean improvement twice — once for the phase that got better, once for its
complement. Shrink is reported as improved so the whole picture stays visible.

One-sided phases are gated on share alone. Comparing only same-named phases
is itself a hiding place: move work into a phase the baseline never had and the
old phase's share falls — reading as an improvement — while the new phase
carrying the cost produces no row. So a phase new in the current run breaches
when it claims more than the drift threshold, and a phase that has vanished
breaches when it used to. Rows are marked (NEW) / (VANISHED).

Here is the case that motivates the whole feature, from the end-to-end smoke run —
whole-scenario time is inside its noise floor and clean, while the phase breakdown
localizes a 28% parse regression masked by a write improvement:

scenario [metric]                                    status     baseline     observed        Δ   budget
convert/ttl_to_nq/small [time]                           ok     1.000 ms     1.025 ms   +2.50%     3.0%
convert/ttl_to_nq/small::parse [phase_time]        EXCEEDED   600.000 µs   768.750 µs  +28.12%     3.0%
convert/ttl_to_nq/small::parse [phase_share]       EXCEEDED        60.0%        75.0%  +15.0pp    5.0pp
convert/ttl_to_nq/small::read  [phase_time]              ok   100.000 µs   102.500 µs   +2.50%     3.0%
convert/ttl_to_nq/small::read  [phase_share]             ok        10.0%        10.0%   +0.0pp    5.0pp
convert/ttl_to_nq/small::write [phase_time]        improved   300.000 µs   153.750 µs  -48.75%     3.0%
convert/ttl_to_nq/small::write [phase_share]       improved        30.0%        15.0%  -15.0pp    5.0pp

4. Noise floor

capture --accumulate folds repeat runs into one file: each pass appends the fresh
criterion mean to a per-scenario sample vector and recomputes median + MAD. At
n ≥ 5 the gate compares medians instead of single-run means, and widens the
wall-clock budget to max(budget_pct, 3 × MAD / median).

Criterion's own confidence interval describes iterations within one run and says
nothing about run-to-run spread — which is exactly what shared-runner flap is. MAD
rather than standard deviation because one outlier run is the common failure here
and barely moves MAD.

Three bounds, each of which inverts the gate if dropped: the floor only widens
(a tighter-than-noise budget can only false-alarm); it is wall-clock only,
never touching peak_mem (an allocator peak is near-deterministic, so a noisy
clock is no reason to forgive an allocation regression); and it is baseline
only
(otherwise the worse a run behaves, the more it is forgiven).

accumulate itself refuses to pool samples across host classes or corpora, and
both guards live in the library rather than the CLI. Accumulation is the one
operation that destroys the evidence of a mismatch — two populations blended
into one median/MAD, with the provenance overwritten in the same breath — so a
direct caller must not be able to skip the check.

5. BENCHMARKING.md corrections

Every claim checked against the merged main + #1477 tree. Five were wrong:

Claim Reality
.github/workflows/bench-nightly.yml (in "Where benches live") Never existed; the file is bench.yml
"lands in bench-5" / "lands in bench-nightly PR" Both landed
Bench table lists 12 benches 18 [[bench]] entries are registered — six missing
BenchProfile::Full = "Nightly CI" (runtime.rs) Every CI job pins quick/tiny, nightly included
bench-baselines/README.md: "enforces memory across any runner class, time advisory" False since 5d9d695 made the gate symmetric — stale on #1477's own branch

Note on the "runs on every PR" and "dedicated 4-core runners" claims from the
original brief: #1477 already removed both in its rewrite of the CI-gate
section. They are gone from the merged result; nothing to do.

The six missing bench rows are query_overlay_matrix, query_hot_bsbm_bi,
query_hot_property_path, query_hot_whole_graph_agg, annotation_hydration,
annotation_planner (five are missing on main; #1477 adds the first). Each row is
written from the bench's own module docs. The table now also states outright that
nothing enforces it — unlike the [[bench]] ↔ budget reconcile — which is why it
rots.

Sample baseline (schema 2)

{
  "schema_version": 2,
  "label": "convert-pre",
  "git_sha": "c7d82614d",
  "captured_at": "2026-07-28T23:54:15Z",
  "profile": "quick",
  "scale": "small",
  "host": {
    "class": "bench-m8gd",
    "name": "ip-10-0-3-7",
    "os": "linux",
    "arch": "x86_64",
    "cpu_model": "AMD EPYC 9R14",
    "physical_cores": 16,
    "logical_cores": 32
  },
  "entries": {
    "convert/ttl_to_nq/small": {
      "mean_ns": 990000.0,
      "median_ns": 990000.0,
      "mem": { "peak_bytes": 24117248, "total_allocated_bytes": 310378496 },
      "noise": {
        "n": 5,
        "median_ns": 1000000.0,
        "mad_ns": 10000.0,
        "samples_ns": [980000.0, 990000.0, 1000000.0, 1010000.0, 1020000.0]
      },
      "corpus": {
        "name": "dblp.ttl",
        "sha256": "aaaa1111bbbb2222cccc3333dddd4444eeee5555ffff6666aaaa7777bbbb8888",
        "byte_len": 1048576,
        "element_count": 40000,
        "input_syntax": "turtle",
        "output_syntax": "nquads",
        "thread_count": 8
      },
      "phases": {
        "read":  { "ns": 100000.0, "share_pct": 10.0 },
        "parse": { "ns": 600000.0, "share_pct": 60.0 },
        "write": { "ns": 300000.0, "share_pct": 30.0 }
      }
    }
  }
}

Every added block is optional. A schema-1 file still loads: runner_class maps to
host.class, a bare host string maps to host.name, and missing
noise/corpus/phases simply switch off their respective rules. The committed
bench-baselines/guardrails-pre.json is unmodified and still compares. A
baseline claiming a newer schema version is a refusal — silently ignoring fields
we don't understand is how a gate stops gating.

Gating semantics, complete

metric units noise floor applies enforced within host class enforced across host classes
time ns yes yes no (advisory)
peak_mem bytes no yes no (advisory)
phase_time ns yes yes no (advisory)
phase_share percentage points no yes yes

The noise floor is wall-clock-only and baseline-only. It never touches
peak_mem — an allocator peak is near-deterministic, so a noisy clock is no
reason to forgive an allocation regression — and the current run's own spread
never widens its own budget, or the worse a run behaves the more it is forgiven.

Phase rows also carry a presence marker. A phase present on both sides compares
normally; one new in the current run breaches when its share exceeds the drift
threshold ((NEW)); one that has vanished breaches when it used to
((VANISHED)). Neither one-sided case emits a phase_time row.

condition outcome exit
within budget pass 0
enforced breach fail 1
host class differs, no --allow-host-mismatch refuse 2
corpus sha256 / syntax / thread count differs refuse 2
baseline schema_version newer than the binary refuse 2

CI impact: none, by construction

The per-PR bench-compare and nightly bench-gate compare a host_class=local
baseline on ci-ubuntu-latest, which the new default would refuse. Both steps now
pass --allow-host-mismatch explicitly, reproducing #1477's exact behaviour
(annotations, exit 0). The change is that the downgrade is written down, with a
comment naming the condition for removing the flag.

bench-capture gains a capture_samples dispatch input (default 1, unchanged
behaviour). At ≥ 5 it reruns the subset and accumulates, producing the median +
MAD a baseline needs before it can gate through runner flap.

Files

File What
fluree-bench-support/src/meta.rs new — CorpusInfo, PhaseTiming, ScenarioMeta, sidecar IO, SHA-256
fluree-bench-support/src/baseline.rs HostBlock, NoiseStats, CompareOptions, FatalMismatch, Metric, refusal + share + noise gating, schema-1 read path
fluree-bench-support/src/bin/bench_baseline.rs --host-class, --allow-host-mismatch, --share-drift-pp, --accumulate, --clean-sidecars; exit code 2; per-metric rendering
fluree-bench-support/src/runtime.rs corrected profile docs (Full is not the CI profile)
fluree-bench-support/src/lib.rs, Cargo.toml meta module; sha2 (existing workspace dep)
.github/workflows/{ci,bench}.yml explicit --allow-host-mismatch; capture_samples; clear the meta sidecar dir
BENCHMARKING.md, bench-baselines/README.md, fluree-bench-support/README.md corrections + new semantics

Verification

cargo nextest run -p fluree-bench-support                    122 passed, 0 skipped
cargo test -p fluree-bench-support --test workspace_reconcile  1 passed
cargo clippy -p fluree-bench-support --all-targets --no-deps   clean, 0 warnings
cargo fmt --check                                              clean

57 of the 121 lib tests are new (measured against this branch's base at
06f1c10cf, which has 64). Every gating rule has one: host mismatch fatal /
allowed / matched / forced-advisory, corpus mismatch by hash, by syntax, by thread
count, file-level vs entry-level precedence, one-sided declaration, share growth /
shrink / within-threshold / configurable-threshold / cross-host enforcement, MAD
arithmetic, floor widening, floor not tightening, floor not hiding a real move,
sub-threshold sample counts, accumulate arithmetic, and the schema-1 read path
including the exact shape of the committed guardrails-pre.json.

Also exercised end-to-end against the real binary: the committed schema-1 baseline
(refusal, then advisory under the flag), a captured schema-2 baseline accumulated
to n=5, a share-drift breach under a passing aggregate, a corpus swap, and
cross-host share enforcement.

Deviations from the §6b spec

  1. SHA-256 instead of BLAKE3 — blake3 is not in the tree; sha2 is. The spec
    permits this ("prefer no new deps").
  2. Share drift gates growth only, not absolute movement. Gating |Δ| would
    double-flag every real improvement, since shares sum to 100. Both directions are
    still reported.
  3. --allow-host-mismatch exists at all. The spec says compare hard-errors on
    host mismatch; making that unconditional would red every PR today, since the
    only committed baseline is cross-class. The refusal is the default and the
    escape hatch is explicit, per-invocation, and documented — strictly louder than
    bench+test: query correctness guardrails — differential harness, baseline gating, and four fast-path fixes #1477's silent downgrade.
  4. thread_count is part of corpus identity, though the spec listed it as a
    recorded field. A K=8 run compared against a K=16 baseline is the same class of
    silent failure as a corpus swap. Its refusal message is distinct so a reader can
    tell a config mismatch from a data mismatch.

Open questions

  1. Should guardrails-pre.json be re-stamped to a specific host class? It was
    captured on Apple silicon but records host_class=local. Re-stamping it
    macos-aarch64 would make a Mac dev's local compare enforce — but would also
    assert that any two Macs are comparable, which is false. Left as local
    deliberately; it is the honest value.
  2. What host_class string does the official publication locus use? H-10
    approved m8gd.4xlarge; the strategy's F7 wants publication tooling to refuse
    non-official classes. This PR provides the field; the allowlist is not
    implemented and needs the canonical string.
  3. Where should milestone convert baselines live — bench-baselines/convert/
    per §6b, with rolling per-change baselines gitignored? Not created here (no
    convert benches exist yet).

aaj3f added 7 commits July 28, 2026 21:11
…loor

Four extensions to the baseline gate #1477 introduced, all of them aimed at
the same failure: a compare run that produces a number which looks like a
result and is not one.

host block. `runner_class` becomes `host.class` inside a block that also
records os, arch, cpu_model, and physical/logical cores. The class names a
comparability SET, not a machine — two hosts share one when a human has
decided their absolute numbers may be compared. A mismatch is now a refusal
(exit 2) rather than a silent downgrade to advisory; `--allow-host-mismatch`
opts back into the advisory posture, which is what CI runs today. The derived
default is `{os}-{arch}`, deliberately coarse: an M1 and an M4 both derive
`macos-aarch64` and are not interchangeable, so any host whose numbers are
meant to gate should set FLUREE_BENCH_HOST_CLASS explicitly.

corpus identity. A new `meta` sidecar (target/fluree-bench-meta/, same shape
as the mem sidecar) carries sha256 + byte_len + element_count + input/output
syntax + thread count per scenario, declarable per entry or once per file.
Comparing a run over one input against a baseline captured over another is
the most expensive silent failure available here, because the output is
indistinguishable from a real regression — so it is a refusal with no
override, unlike the host check. SHA-256 rather than BLAKE3: blake3 is not in
the tree, sha2 already is, and a corpus is hashed once per capture.

phase shares. Per-phase {ns, share_pct}, with the share derived from the ns
at record time so the two can never disagree. Absolute phase ns gates only
within a host class; share drift gates regardless, because a share is a ratio
within one run and the machine cancels. Only share GROWTH gates — shares sum
to 100, so gating both directions would fail a clean improvement twice, once
for the phase that got better and once for its complement.

noise floor. `capture --accumulate` folds repeat runs into a median + MAD over
>= 5 samples, and the gate flags only when the median move exceeds
max(budget_pct, 3*MAD/median). MAD over stddev because a single outlier run
is the common case here and barely moves MAD. The floor only ever widens an
envelope, never tightens one.

Schema bumps to 2; schema-1 files still load (`runner_class` maps to
`host.class`, a bare `host` string to `host.name`), so the committed
guardrails-pre.json stays usable untouched. A baseline claiming a NEWER
version is a refusal — silently ignoring fields we don't understand is how a
gate stops gating.
…mple capture

The per-PR and nightly compares were relying on a silent behaviour — a
host-class mismatch quietly turning both metrics advisory. That is now a
refusal by default, so both jobs pass --allow-host-mismatch and say why. The
flag is the phase-1 posture and the comment names the condition for dropping
it: a committed ci-ubuntu-latest baseline. Nothing about what these jobs
report changes; what changes is that the downgrade is written down.

bench-capture gains a `capture_samples` dispatch input. At 1 (default) it
behaves as before. At >= 5 it reruns the subset and folds each pass in with
--accumulate, producing the median + MAD the compare gate needs to widen
budgets past runner flap — which is the thing standing between the current
advisory annotations and a baseline that can actually gate on ubuntu-latest.

Both compare/capture steps also clear target/fluree-bench-meta alongside the
mem sidecars. A stale corpus block is worse than a stale timing: it asserts an
input identity for a run that never read that input.
…ly do

Every claim in the file checked against the merged main + #1477 tree. Five
were wrong or stale:

- "Where benches live" pointed at .github/workflows/bench-nightly.yml, which
  has never existed — the file is bench.yml. Both entries also carried
  "lands in bench-5" / "lands in bench-nightly PR" process notes for work that
  landed. Replaced with the real layout, including bench-baselines/ and the
  three target/ directories capture reads.
- The bench table was missing six of the eighteen registered [[bench]]
  entries: query_overlay_matrix, query_hot_bsbm_bi, query_hot_property_path,
  query_hot_whole_graph_agg, annotation_hydration, annotation_planner. (Five
  of those are missing on main; #1477 adds query_overlay_matrix.) Rows written
  from each bench's own module docs. Also says outright that nothing enforces
  this table, unlike the [[bench]] <-> budget reconcile, which is why it rots.
- The env-var table's defaults are right for a laptop and were being read as
  though they described CI. Both jobs pin quick/tiny — the nightly included,
  despite `Full`'s doc calling itself the nightly profile. Noted in the table
  and corrected at the source in runtime.rs, where BenchProfile::Full claimed
  "Nightly CI" and referenced the unbuilt `bench-nightly`.
- bench-baselines/README.md still described the pre-5d9d69513 asymmetric gate
  ("enforces memory across any runner class, time advisory"). That has been
  false on this branch since the gate was made symmetric.

The rest of the section is rewritten for the extensions in the previous
commit: the metric table with its per-metric portability column, the refusal
conditions and their exit codes, the noise-floor accumulation loop, and why
{os}-{arch} is a coarse default rather than a promise.
fluree-bench-support/README.md documented every module except the three the
baseline gate runs on. Adds `mem`/`meta` (what a sidecar is for, which dir,
what it carries) and `baseline`, and corrects the test count, which said 44
against an actual 102.
Review findings from an adversarial pass over the baseline gate. Each of these
produced a clean-looking report for a change that was genuinely worse.

New-phase invisibility (the expensive one). The per-phase loop walked only the
BASELINE's phases, so moving work into a phase the baseline never had made the
old phase's share FALL — reported as "improved" — while the new phase carrying
the cost produced no row at all. A 2x regression exited 0. Phases are now
compared over the union of both sides: a phase new in the current run breaches
when it claims more than the drift threshold, and a phase that vanished
breaches when it used to. Neither emits a phase_time row, because there is no
ratio to report against a missing side and a row reading "ok" against a zero
baseline is worse than no row. Rows are marked (NEW)/(VANISHED).

Timing noise widened the MEMORY budget. One budget was computed per scenario
and handed to every metric, so a baseline with a noisy clock (MAD 100 on a
median of 1000 = a 30% floor) let a +25% peak_bytes regression through a 5%
memory budget. An allocator's peak is near-deterministic for a given workload;
a noisy clock is no reason to forgive it. The floor now reaches time and
phase_time only.

The noise floor is baseline-only. It previously took max(baseline, current),
which lets an erratic run widen its own budget — the worse a run behaves, the
more it is forgiven. The baseline is the controlled measurement and alone gets
to say how much a scenario moves.

The host-mismatch remedy suggested relabelling the host (--host-class /
FLUREE_BENCH_HOST_CLASS "to the class the baseline was captured on"). Renaming
a machine does not make its numbers comparable: following that advice converts
the refusal into a silent, confident ENFORCEMENT against hardware the baseline
never ran on — the exact failure the refusal exists to prevent. The remedy now
leads with --allow-host-mismatch, and the usage text says what --host-class is
and is not for.

Also here, from the same pass:

accumulate now enforces host-class AND corpus identity, and both guards live
in the library rather than in the CLI that happened to call it. Accumulation is
the one operation that DESTROYS the evidence of a mismatch — two populations
blended into a single median/MAD, with the provenance overwritten in the same
breath — so a bench harness calling accumulate() directly must not be able to
skip the check. Returns Result; the doc comment no longer describes protection
that wasn't there.

with_phase_ns MERGES instead of replacing, recomputing shares over the union.
Replacing silently discarded an earlier call's phases while leaving the
remaining shares looking perfectly well-formed — invisible in the output. And
PhaseTiming's docs no longer claim ns and share_pct "can never disagree": that
is a property of the constructor, not an invariant, since both fields are
public and a baseline file is untrusted input. compare now reports a scenario
whose shares don't sum to 100% instead of assuming they do — reported, not
gated, since the drift comparison is still internally consistent.

Regression tests come from the reviewer's three repro fixtures and all three
are verified against the real binary, not just the library.
…new rules

The phase and corpus machinery reads in BENCHMARKING.md as though it were
running today. It isn't: no bench in the workspace calls meta::record_scenario,
so no phase or corpus row appears in any report this branch produces. That is
deliberate — the gate has to exist before the benches that depend on it — but a
reader had no way to tell scaffolding from live coverage. Stated up front.

Also documents the rules the fix commit introduced: one-sided phase gating and
why comparing only same-named phases is itself a hiding place; the three bounds
on the noise floor (widening-only, wall-clock-only, baseline-only) each with the
inversion it prevents; and accumulate's refusal to pool across host classes or
corpora.

The compare quickstart now carries --allow-host-mismatch inline. Without it the
copy-pasteable command refuses against the committed baseline, which is exactly
the flag the reader needs to understand — leaving it in the options list taught
the rule only to people who already knew it.
Cargo.lock recorded fluree-bench-virtual at 4.1.3 while the crate inherits
`version.workspace = true` and [workspace.package] declares 4.1.4. The entry
was stale on main itself, not introduced here: any plain `cargo build` or
`cargo metadata` on a main-derived branch rewrites this same line and leaves a
dirty tree.

Nothing enforces it today — no workflow passes --locked or --frozen — so this
is hygiene rather than a fix for a live failure. It rides along with this PR
only because building this branch is what surfaced it; it is one line of
lockfile churn with no dependency resolution change, and it is deliberately a
separate commit so it can be dropped or cherry-picked independently of the
bench work.

@bplatz bplatz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaj3f I did a defect-focused pass over the gate and workflow changes. I found two fail-open paths that look merge-blocking (missing corpus metadata and invalid phase shares), plus several accumulation/provenance and input-hardening issues. Details are inline; I left this as a comment review rather than changing the approval state.

Comment thread fluree-bench-support/src/baseline.rs Outdated
// Corpus identity gates everything else about this scenario: if the two
// runs read different inputs, none of their numbers are comparable and
// emitting rows for them would be worse than emitting nothing.
if let (Some(base_corpus), Some(cur_corpus)) =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This treats (Some(base_corpus), None) as comparable. If a producer stops emitting its sidecar, or the sidecar write/read fails, the current run bypasses the corpus-identity gate and can exit 0. Once the baseline declares a corpus, missing current metadata should be a refusal. The same asymmetry exists in accumulate around line 626, where fresh.clone() can also erase the prior identity while pooling the sample. Could you add compare and accumulate regression tests for the Some/None case?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecba51446. You're right that absence of evidence was being read as evidence of the same input. (Some, None) is now a refusal; the reverse (None, Some) stays permitted, since a schema-1 baseline predates corpus blocks and refusing there would make every old baseline unusable — that asymmetry is now explicit in a corpus_refusal helper that both compare and accumulate call, so the two can't drift apart again. accumulate had the same hole plus the fresh.clone() erasure you spotted. Regression tests both places: baseline_corpus_with_no_current_corpus_refuses and accumulate_refuses_when_fresh_drops_the_corpus.

Comment thread fluree-bench-support/src/baseline.rs Outdated
return;
}
let sum: f64 = phases.values().map(|p| p.share_pct).sum();
if (sum - 100.0).abs() > SHARE_SUM_TOLERANCE_PP {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think invalid shares are cosmetic here. On a host mismatch, phase_share is the only enforcing signal; a malformed current sidecar with all shares at 0 makes every phase look improved while this only emits an anomaly and the command exits 0. Since the file is explicitly treated as untrusted input, please refuse non-finite/negative values and a materially invalid sum (or recompute shares from ns) before gating.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and this was the worse of the two — thank you. Treating shares as cosmetic is exactly wrong in the case where they're the only thing enforcing. Fixed in ecba51446: shares are validated before anything gates on them, and non-finite, negative, or a sum more than 1pp off 100 is now a refusal (MismatchKind::PhaseShares) rather than an annotation. I went with refusing over recomputing from ns to match the rest of the gate — a file that disagrees with itself gets "I won't guess" — and the remedy string names ScenarioMeta::with_phase_ns as the way to regenerate. The anomalies channel is gone since this was its only user. Your exact scenario is pinned by all_zero_current_shares_refuse_instead_of_gating_as_improved, which asserts no bogus Improved row is emitted.

Comment thread fluree-bench-support/src/baseline.rs Outdated
/// - **host class** must match between `prev` and `fresh`.
/// - **corpus identity** must match for every scenario present in both,
/// resolved entry-then-file exactly as [`compare`] resolves it.
pub fn accumulate(prev: &BaselineFile, fresh: &BaselineFile) -> Result<BaselineFile, String> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compare refuses a schema newer than this binary understands, but accumulate has no equivalent guard. An older binary can deserialize an additive schema-3 file, clone a fresh schema-2 capture, and save it back while silently dropping fields it did not understand. Please reject prev.schema_version > BASELINE_SCHEMA_VERSION before merging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecba51446 — accumulate now rejects prev.schema_version > BASELINE_SCHEMA_VERSION before merging, matching compare. The silent-field-drop path you describe is the real hazard: it writes the file back, so the loss is permanent rather than just a bad comparison. Test: accumulate_refuses_a_newer_schema.

}
}

let mut out = fresh.clone();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This replaces git_sha with the fresh SHA while pooling the previous samples without checking revision identity. Running --accumulate after a code change mixes pre/post-change populations, relabels all samples as the new revision, and can widen MAD enough to hide the change being measured. I think differing non-empty SHAs should refuse accumulation unless there is an explicit override.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecba51446. Differing non-empty git_shas now refuse, with --allow-revision-drift (and AccumulateOptions::allow_revision_drift for library callers) as the explicit override. The MAD-widening consequence you name is the one that worried me most — it doesn't just mislabel the samples, it actively raises the threshold that would have caught the change. Test: accumulate_refuses_across_revisions_unless_overridden, plus accumulate_allows_a_matching_or_absent_revision for the normal loop and the no-git case.

}
}

fn is_usable(&self) -> bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

n, median_ns, and mad_ns are trusted independently of samples_ns. A stale or hand-edited baseline can claim n >= 5 with an arbitrary median/MAD and thereby move the comparison baseline or widen the budget. Since the raw samples are stored, load/compare should recompute these fields or reject an inconsistent/non-positive set.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecba51446. Since the raw samples are stored, as you say, they're now the authority: n/median_ns/mad_ns are recomputed from samples_ns at load. A set that can't be recomputed — fewer than MIN_NOISE_SAMPLES, or any non-finite/non-positive sample — is returned unusable rather than trusted, which drops the floor for that entry. That's the safe direction, since the floor only ever widens a budget. Tests: noise_stats_are_recomputed_from_their_own_samples_at_load, fabricated_noise_without_samples_is_unusable, non_positive_samples_are_rejected_as_a_floor.

#[serde(default)]
corpus: Option<CorpusInfo>,
#[serde(default)]
entries: BTreeMap<String, BaselineEntry>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because all top-level fields default, {} deserializes as a valid schema-1 baseline. With --allow-host-mismatch, every current scenario is merely reported as new, no metric breaches, and compare exits 0. Please validate that a loaded baseline has the required structure and at least one scenario rather than accepting an empty document.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecba51446. load_baseline now requires schema_version >= 1 and at least one scenario. The failure mode you describe is the one I'd least want to ship — a gate that passes because it had no reference points looks identical to a gate that passed on merit. Test: empty_document_is_rejected_at_load covers both {} and an explicit empty entries.

Comment thread fluree-bench-support/src/meta.rs Outdated
/// First 12 hex chars, for report lines. Never used for comparison.
pub(crate) fn short_hash(hash: &str) -> &str {
let end = hash.len().min(12);
&hash[..end]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This byte slice can panic when an untrusted sha256 contains Unicode and byte 12 is not a UTF-8 boundary, turning a documented refusal into exit 101. Validating the field as 64 lowercase ASCII hex characters would also protect identity semantics; otherwise truncate by characters here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 55cc517e7, both ways you suggested. short_hash now slices on a character boundary so no input can panic it, and CorpusInfo::validate rejects anything that isn't exactly 64 lowercase ASCII hex digits. Doing only the truncation would have left the identity semantics resting on an unvalidated field, and doing only the validation would have left the panic reachable from the very error path that reports it — the crash landed precisely when the tool was trying to explain its refusal. Tests: short_hash_never_panics_on_multibyte_input and validate_rejects_malformed_hashes.

for scale in tiny small; do
FLUREE_BENCH_SCALE=$scale cargo bench -p fluree-db-api --bench query_overlay_matrix
FLUREE_BENCH_SCALE=$scale cargo bench -p fluree-db-api --bench query_hot_bsbm
samples="${{ github.event.inputs.capture_samples || '1' }}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

capture_samples=0 makes seq 1 0 run no passes. The baseline is never created, and upload-artifact only warns on a missing file by default, so the capture job can succeed without an artifact. Please validate a positive integer and set if-no-files-found: error on the upload step.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fb2f60b41. Validates a positive integer before the loop and sets if-no-files-found: error on the upload. A capture job that goes green having captured nothing is the worst outcome available here, because the next person reasonably assumes the artifact exists.

Comment thread BENCHMARKING.md Outdated

Two ways to reach phase 2:

1. **Committed CI-class baseline (implemented).** Run `bench.yml`'s

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The capture job writes guardrails-pre-ci.json, while both CI compare jobs read guardrails-pre.json. Following 'download the artifact, commit it, and remove --allow-host-mismatch' literally leaves CI reading the old local baseline and refusing every run. Please explicitly say to replace/rename guardrails-pre.json, or update the compare path as part of phase 2.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 517a03c5a. You're right that following it literally lands phase 2 as a red gate — the flag comes off and the compare then refuses against the stale local-class file. The doc now spells out the mv guardrails-pre-ci.json bench-baselines/guardrails-pre.json replace step explicitly, and names the alternative (keep the -ci filename and repoint every --baseline path) so the choice is visible rather than implied.

aaj3f added 4 commits July 29, 2026 11:00
…iew 7)

`short_hash` sliced `&hash[..12]` by BYTE offset. A baseline file is untrusted
input, so `sha256` can hold anything — and when byte 12 lands mid-codepoint the
slice panics. It is called from the refusal path, so the failure mode was a
documented exit-2 refusal turning into an exit-101 crash: the gate crashes
exactly when it is trying to tell you why it won't compare.

Two fixes, because either alone leaves a gap. `short_hash` now slices on a
character boundary, so no input can panic it. And `CorpusInfo::validate`
rejects any hash that is not exactly 64 lowercase ASCII hex digits — a hash
nothing validated cannot establish identity, so accepting one would assert
"same input" on the strength of an unparsed field.
Six review items, one theme: a baseline file is untrusted input, and in each
case the gate accepted something it could not verify and exited 0. Grouped in
one commit because they are interleaved edits to the same load/compare path in
baseline.rs; splitting them would mean commits that don't build.

[BLOCKING] Corpus, `(Some, None)` (item 1). `compare` only checked identity
when BOTH sides declared a corpus, so a producer that stopped emitting its
sidecar — or a sidecar write that failed — bypassed the identity gate entirely.
Absence of evidence was read as evidence of the same input. Now: baseline
declares a corpus and the current run declares none is a REFUSAL. The reverse
stays permitted, and that asymmetry is deliberate — a schema-1 baseline
predates corpus blocks, so refusing there would make every old baseline
unusable. Both directions are pinned. `accumulate` had the same hole plus a
worse consequence, since `fresh.clone()` also erased the prior identity while
pooling the sample; it now shares one `corpus_refusal` helper with `compare` so
the two cannot drift apart.

[BLOCKING] Shares (item 2). These were treated as cosmetic — a report line,
never a gate. That is wrong in the one case that matters: on a host mismatch
`phase_share` is the ONLY enforcing metric, so a malformed current sidecar with
every share at 0.0 marked every phase "improved" while the absolute metrics
were merely advisory, and the command exited 0. Shares are now validated before
anything gates on them: non-finite, negative, or a sum more than 1pp off 100 is
a refusal (`MismatchKind::PhaseShares`). Refusing rather than silently
recomputing from `ns` matches the rest of the gate — a file that disagrees with
itself gets "I won't guess", and the remedy names the constructor that derives
shares correctly. The `anomalies` channel is removed; it existed only for this.

accumulate schema guard (item 3). `compare` refused a schema newer than the
binary understands; `accumulate` did not, so an older binary could deserialize
an additive future schema, clone a fresh capture over it, and write it back
having silently dropped every field it didn't recognise.

accumulate revision guard (item 4). Pooling samples across a code change mixes
pre- and post-change populations, relabels every sample as the new revision,
and can widen the MAD enough to hide the change being measured. Differing
non-empty `git_sha`s now refuse unless `--allow-revision-drift` says otherwise.

NoiseStats trust (item 5). `n`, `median_ns`, and `mad_ns` were trusted
independently of the `samples_ns` they claim to summarise, so a hand-edited
file could claim n=5 beside an arbitrary median and MAD and thereby move the
comparison point and widen the budget. They are now recomputed from the samples
at load. A set with too few samples, or any non-finite/non-positive sample,
cannot be recomputed and is returned unusable rather than trusted — losing a
floor is the safe direction, since a floor only ever widens.

Empty-document guard (item 6). Every top-level field carries a serde default so
schema-1 files keep loading, which meant `{}` also deserialized "successfully".
An empty baseline compares against nothing, breaches no metric, and exits 0 —
a gate that passes because it had no reference points. `load_baseline` now
requires schema_version >= 1 and at least one scenario.
`capture_samples: 0` made `seq 1 0` run zero passes, so no baseline was ever
written — and `upload-artifact` only warns on a missing file by default, so the
job went green having produced nothing. A capture job that silently captures
nothing is the worst outcome available here, because the next person assumes an
artifact exists.

Validates a positive integer before the loop, and sets `if-no-files-found:
error` on the upload so the belt has a brace.
The capture job writes `guardrails-pre-ci.json`; both compare jobs read
`guardrails-pre.json`. Following "download the artifact, commit it, and remove
--allow-host-mismatch" literally leaves CI reading the old host_class=local
file with the flag gone — which refuses every run, so phase 2 would land as a
red gate rather than a working one.

Spells out the replace step with the actual command, and notes the alternative
(keep the -ci name, repoint every --baseline path) so the choice is visible
rather than implied.
@aaj3f

aaj3f commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

@bplatz -- ready for re-review when you have time

@bplatz bplatz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed. All nine items are addressed, and I checked each against the code rather than the replies — the two fail-open paths I called merge-blocking are genuinely closed:

  • corpus_refusal is a single helper called from both compare and accumulate, so the asymmetry can't drift apart again, and the (Some, None) direction refuses.
  • Share validation runs ahead of any gating decision, and MismatchKind::PhaseShares refuses rather than annotates. The all-zero-shares case emits no bogus Improved row.
  • Noise stats are recomputed from samples_ns at the serde boundary, so every reader sees derived numbers; an unrecomputable set drops the floor, which is the widening-only direction.
  • {} and an explicit empty entries are both rejected at load.
  • short_hash is char-boundary safe and the hash is validated — agreed that doing only one would have left either the panic or the identity semantics exposed.
  • The capture job fails a non-positive capture_samples and errors on an empty upload.
  • The doc names the mv explicitly, so phase 2 doesn't land as a red gate.

Verified locally: 139/139 tests (17 new, one per item), clippy --all-targets clean, fmt --check clean, and the committed schema-1 guardrails-pre.json still refuses at exit 2 and compares advisory under --allow-host-mismatch.

Approving. Two residuals inline — both non-blocking and both latent today. They're the same shape as items 1 and 4 on the sibling code path, so worth a look before merge rather than a reason to hold it.

Comment thread fluree-bench-support/src/baseline.rs Outdated
}
// Not rerun this pass: carry the previous entry forward whole.
_ => {
out.entries.insert(id.clone(), prev_entry.clone());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The corpus guard above skips ids absent from fresh (the continue at the top of that loop), but this arm re-inserts exactly those entries into out — whose file-level corpus is fresh's. An entry that declared no entry-level corpus and was inheriting prev.corpus therefore resolves against a different corpus after the merge, with its samples unchanged.

Reproduced with prev = {q1, q2} under a file-level corpus A, and a pass that reran only q3 under corpus B:

prev   q2 corpus = ("A", "61be55a8")
merged q2 corpus = ("B", "81cc5b17")   # same samples, relabelled

The overlapping-id case is correctly refused, so this only bites when a pass doesn't rerun every scenario — --only subsetting, or a bench that emits no criterion output that pass. The CI loop reruns the same subset each pass, and nothing produces corpus blocks yet, so it isn't reachable today. It is the same silent identity rewrite as the (Some, None) case though, just on the carry-forward path instead of the pooling path, so I'd rather it not sit latent: either stamp the resolved prev.corpus_for(id) onto the carried-forward entry, or run the identity check over the union of prev.entries and fresh.entries rather than the intersection.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e06187439, and I did both — they turn out to be complementary rather than alternatives, which I didn't expect going in.

The union walk alone still lets the case where prev resolved to NO corpus and fresh introduces a file-level one slip through: the carried entry inherits an identity its samples never had. The stamp alone has nothing to stamp in exactly that case. So: carried-forward entries get prev.corpus_for(id) stamped when it resolves to something, and where it resolves to nothing while fresh declares a corpus, the merge refuses — "explicitly no corpus" isn't expressible through inheritance, so there's no honest value to write.

Resolution stays single-sourced as you asked: every decision on both sides goes through corpus_for, and the intersection case still calls the same corpus_refusal helper compare uses.

Your repro is the regression test verbatim (carried_forward_entries_keep_their_own_corpus): prev {q1,q2} under file-level A, a pass rerunning only q3 under B, asserting q2 still resolves to A with its samples unchanged and that q3 keeps B. Plus carrying_forward_into_a_newly_declared_corpus_refuses for the unstampable case. Both confirmed failing before the fix.

Comment thread fluree-bench-support/src/baseline.rs Outdated
));
}
if !opts.allow_revision_drift {
if let (Some(prev_sha), Some(fresh_sha)) = (&prev.git_sha, &fresh.git_sha) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires both sides Some, and the merge below takes fresh's git_sha — so a single pass with no revision resets the accumulator and reopens the gate for every pass after it:

after a no-git pass, git_sha = None
next pass at a different revision -> ACCEPTED
  pooled samples   = [1000.0, 1010.0, 5000.0]
  labelled git_sha = Some("ccccccc")

That's the MAD-widening case this guard exists for, reached in two steps rather than one. The trigger is git_short_sha() returning None, so it's a local/container concern rather than a CI one — CI always has git. Carrying prev.git_sha forward when fresh has none, or treating None vs Some as drift, would close it, and either way --allow-revision-drift stays the explicit path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e06187439. I went with treating None vs Some as drift rather than carrying prev.git_sha forward.

Carrying it forward would assert that the fresh samples were taken at a revision nothing actually observed — fabricating provenance in order to preserve a label, which is the same shape as the identity rewrites in R1 and item 1. Treating absence as part of revision identity says the honest thing instead: a sample whose revision is unknowable is precisely the one that can't be pooled safely. Both-absent still accumulates fine, so the no-git tarball/container workflow is unaffected, and --allow-revision-drift remains the explicit path through.

Your two-step repro is the test (a_no_git_pass_cannot_launder_a_revision_change), written to accept either outcome — the no-git pass itself refusing, or it being accepted and the next pass at a different revision refusing — so it pins the hole rather than my particular choice of where to close it. Confirmed failing before the fix. One earlier test flipped with the contract: one-sided None was asserted as allowed and is now asserted as drift in both directions.

Both are the same defect class as the (Some, None) corpus hole, on sibling
paths in the same function, so they land together. Both repros reproduced
first: three tests written against the unfixed code, confirmed failing, then
fixed.

R1 — carry-forward relabelling. The corpus guard walked the INTERSECTION of
prev and fresh entries, but the carry-forward arm re-inserts precisely the
entries it skipped, into a file whose file-level corpus is fresh's. An entry
that declared no entry-level corpus and was inheriting prev's therefore
resolved against a different corpus after the merge, samples unchanged.

Fixed by walking the UNION and, for carried-forward entries, stamping the
corpus they actually resolved to. Both of the suggested shapes, and they turn
out to be complementary rather than alternatives: the union walk alone still
lets prev-had-none inherit a newly declared file-level corpus, and the stamp
alone has nothing to stamp in that case. So the stamp preserves identity where
identity exists, and where it does not — prev resolved to no corpus, fresh
declares one — the merge refuses, because "explicitly no corpus" cannot be
expressed through inheritance and the entry would otherwise claim an identity
its samples never had. Resolution stays single-sourced: every decision goes
through corpus_for on both sides.

R2 — revision laundering. The guard required both sides Some and the merge took
fresh's git_sha, so ONE pass with no git reset the accumulator to None and
every later pass sailed through a guard that needed two Somes to fire. The
MAD-widening case the guard exists for, reached in two steps instead of one.

Fixed by comparing revision identity as a whole, absence included: None vs Some
is drift like any other. Chosen over carrying prev's sha forward because that
would assert the fresh samples were taken at a revision nothing observed —
fabricating provenance to preserve a label. A sample whose revision is
unknowable is exactly the one that cannot be pooled safely. Both-absent still
accumulates fine (tarball or container builds have nothing to disagree about),
and --allow-revision-drift remains the explicit path through.

One earlier test changes with the contract: one-sided None was asserted as
allowed and is now asserted as drift, in both directions.
@aaj3f
aaj3f merged commit 271643b into main Jul 30, 2026
15 checks passed
@aaj3f
aaj3f deleted the bench/baseline-extensions branch July 30, 2026 20:16
aaj3f added a commit that referenced this pull request Jul 31, 2026
…conflict-free review

Single conflict: fluree-db-api/src/lib.rs — convergent cfg-orphan fix.
Both #1566 and the audit side moved build()'s #[cfg(native)] doc block
below with_secret_resolver (kept iceberg-gated). The cfg fix is
byte-identical on both sides; resolved the lone doc-wording hunk to
main's #1566 text, so the merged file equals origin/main's api/lib.rs
(zero diff, no audit feature lost). hash_join.rs, optional.rs, planner.rs
auto-merged (disjoint hunks: audit #1477 fast-path vs main cypher #1553).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants