ci: speed up tests, benchmarks, and Apple releases by bplatz · Pull Request #1802 · fluree/db

bplatz · 2026-09-07T13:52:06Z

Normal CI now completes in 7m34s, down from 14m48s in the sampled pre-change run (about 49% less elapsed time). Correctness checks remain on every PR/main push; benchmark execution moves to nightly/on-demand runs. The nightly benchmark workflow now completes its smoke tests instead of timing out, and Apple releases select a larger Apple silicon runner.

Measured results:

Workload	Before	Validated result
Normal CI, end to end	14m48s	7m34s
Workspace test job	14m44s	7m29s, with 47 additional doctests
Subset benchmark comparison	12m13s in normal CI	9m07s, now nightly/on demand
Full nightly benchmark run	~55m, timed out before smoke tests	29m43s, all 23 smoke targets passed

Sources: pre-change CI, final CI, on-demand comparison, previous timed-out nightly, and successful full nightly validation. These are observed runs, not guaranteed durations.

The 29m43s nightly measurement predates the final parallel split: its smoke build took 21m20s, smoke execution 17s, and comparison added 7m39s sequentially. In the final layout, smoke and comparison run independently, so comparison no longer extends the smoke job. The comparison-only path passed on the final commit; the final combined nightly duration has not been measured. The Apple runner mapping is validated, but its release-build speedup is also not yet measured.

Changes:

Run workspace tests on ci-cd-large (8 cores / 32 GB). Run default-feature compilation alongside Clippy, in parallel with nextest; retain all 12,361 tests, SPARQL compliance, SQL database integration tests, WASM/browser checks, and formatting. Add the previously missing workspace doctests.
Remove benchmark execution and its path-filter job from normal CI. Nightly runs full workspace benchmark smoke and subset performance comparison as independent jobs. gh workflow run bench.yml --ref <branch> -f compare_only=true runs just the comparison before merge when needed. Benchmark runtime/performance regressions are now detected nightly or on demand; Clippy still checks benchmark compilation on every PR.
Use ci-cd-large for benchmarks, align Rust 1.97.0 and host identity, build the selected binaries in one command, and clear stale Criterion output/sidecars. Preserve release optimization, benchmark scale, and sampling settings.
Select all 23 explicit nightly benchmark targets with --bench '*' and the GraphQL feature. Avoid extra libtest benchmark harnesses under fat LTO. Bound cold builds and prevent cancellation from starting more work.
Make baseline capture opt-in, validate 1–10 sequential samples, and allow 90 minutes for repeat capture. The committed local-machine baseline remains advisory for time/memory; phase-share drift still enforces.
Save Rust caches only on main; PRs restore shared dependencies. Run disk cleanup only below 15 GiB free, cancel superseded main CI, add workflow linting and job timeouts, pin Ubuntu 24.04/wasm-pack, and update handwritten checkout/setup-node/upload-artifact actions to v7.
Select macos-14-xlarge (5 cores / 14 GB) for Apple release builds through cargo-dist. Other release runner mappings and organization budgets remain unchanged.

Validation:

Final normal CI passed all eight jobs on 99bfce245 in 7m34s end to end. The test job took 7m29s (previous layout: 8m50s); Clippy plus relocated default-feature compilation finished in 4m18s, in parallel. On-demand comparison passed in 9m07s independently of normal CI; full smoke and capture were correctly skipped with compare_only=true. All 12,361 tests and 47 doctests passed on the final commit.
actionlint 1.7.12 and git diff --check pass. Structural checks verify that all correctness jobs/commands are retained, default-feature compilation moved intact, comparison moved intact, and full benchmark smoke/capture commands are preserved. Main branch protection/rulesets do not require either removed job.
Previous hosted validation passed all jobs: 12,361 tests (30 skipped), 47 doctests (163 explicitly ignored). Doctests added 18s. Conditional cleanup took under one second versus 35–68s; Clippy finished in 2m07s and SPARQL compliance in 2m30s.
Full benchmark smoke validation passed all 23 targets; compilation took 21m20s and smoke 17s. Comparison previously added 7m39s sequentially; it now runs independently. Earlier nightlies timed out before smoke and consumed about 55m.
cargo-dist 0.31.0 planning verified the Apple runner mapping. No release build was triggered. The previous Apple release build took 2h53m; the new machine's release duration is not yet measured.

Remaining coverage limitation: the standalone testsuite-shacl workspace has documented conformance failures and reports them without failing by default. A useful CI gate requires an expected-failure register; see docs/contributing/shacl-compliance.md.

aaj3f

@bplatz this is a good fix for both a real, blocking problem and one that all of us felt even when non-blocking. No arguments from me. I'll put Claude's review verbatim below:

This is a lot of CI time back, and the part I care most about is intact: I enumerated the job list at the head and every correctness gate is still there — fmt, clippy (with the default-feature check moved beside it rather than dropped), nextest with --all-features --no-fail-fast, the full W3C SPARQL suite, the SQL bridge, both WASM jobs — plus two things that are new and good, the doctest step (nextest never ran them) and an sha256-verified actionlint. The nightly story checks out too: every scheduled bench.yml run since at least 08-29 ends cancelled, so the smoke and compare it exists for have not run in weeks, and the dispatch runs on this branch show it completing. Two things I'd like decided on purpose rather than by default, both inline: ci-cd-large is billed while ubuntu-latest is free on a public repo, and this moves the every-push test job (and the nightlies, and the Apple release via macos-14-xlarge) onto paid minutes — small numbers, but worth choosing; and the per-PR perf signal now depends on a manual bench.yml dispatch before merging perf PRs, which is a habit we haven't started yet (#1803 and #1815 are open right now with none). Three nits beyond that: --locked on the nextest step too, a note on the main cache under cancel-in-progress, and the untested-until-tonight schedule path for bench-gate's compare_only guard.

Adherence to repo commitments:

Patterns/abstractions: ✔ n/a — workflow and docs only; the bench chassis, budgets, and reconcile test are unchanged.
Performance (speed first, memory second): ✔ no engine change; ⚠️ the per-PR perf-regression signal moves to nightly/on-demand — a deliberate trade, flagged so it is practiced.
Testing: ✔ every gate retained and green on the head; doctests added; the removed bench jobs were never required checks (ruleset carries only deletion / non-fast-forward).
Conventions: ✔ self-describing subjects, BENCHMARKING.md and bench-baselines/README.md updated to match; ⚠️ the seven commit bodies are one-liners with the reasoning in the PR body — fine here since the body is thorough, but worth a sentence each next time.

Verified locally at branch HEAD: job enumeration of ci.yml at head vs base; gh run list --workflow bench.yml (nightly cancellations 08-29→09-09, the three dispatches on this branch, none elsewhere); ruleset and branch-protection queries; repo visibility; BENCHMARKING.md still carries the whole_graph_agg void-numbers caveat at :375; .config/nextest.toml slow-timeout read.

Approving so you can merge when ready, but I'd like the runner-cost question answered on purpose and the bench.yml dispatch run on #1803 and #1815 before they land.

aaj3f · 2026-09-09T14:14:41Z

  test:
-    runs-on: ubuntu-latest
+    # 8 cores / 32 GB: the workspace compile and nextest run are the CI critical path.
+    runs-on: ci-cd-large


.github/workflows/ci.yml:89 — optional, a decision to make knowingly. fluree/db is public, so ubuntu-latest minutes are free; ci-cd-large is the org's billed larger runner, and this puts the job that runs on every push to every PR on it (plus the three nightly bench jobs, plus macos-14-xlarge for each Apple release in dist-workspace.toml:58). We already use asset-publisher-8-core for the Linux ARM release so this extends an existing posture rather than starting one, and the numbers are small — I make it roughly a quarter-dollar per CI run for test, a few dollars per nightly, and something like ten to fifteen dollars per Apple release — but "free → paid on the most frequent job" is the kind of thing I'd rather see chosen than inherited. If that's already been weighed, ignore me; if it hasn't, this is the line to weigh it on.

Weighed and accepted. The test-job speedup is almost entirely the larger runner (the removed bench-compare ran in parallel and was never the critical path), so going back to ubuntu-24.04 for test gives back most of the 14m→7m. At roughly a quarter dollar per CI run, a few dollars per nightly, and ten to fifteen per Apple release, the paid minutes are the price of the halved wall clock, and it extends the asset-publisher-8-core posture we already have rather than starting a new one.

aaj3f · 2026-09-09T14:14:41Z

    branches: [main]

+# Superseded pushes to main should cancel stale CI just like PR updates do.
 concurrency:


.github/workflows/ci.yml:10 — optional, process. With bench-compare gone from PR CI the per-PR perf signal — the phase-share drift check did exit 1, even though time/memory were annotation-only under --allow-host-mismatch — now lives in nightly (attributed to whoever merged last) and in the manual gh workflow run bench.yml --ref <branch> -f compare_only=true the docs describe. I think that's the right trade given the compile cost, but it only works if it becomes a habit for perf PRs, and right now the run list shows dispatches only on this branch: neither #1803 nor #1815, both open perf changes, has one. Worth starting the convention with those two before this lands. Commenting here because the removed bench-compare hunk has no line to anchor to.

Agreed on the habit, with one ordering constraint: gh workflow run bench.yml --ref <branch> runs the bench.yml from that ref, and neither #1803 nor #1815 carries this file yet. main's version has no compare_only input, so the flag is rejected, and #1815 is stacked on fix/incremental-stats-flat-ndv rather than main. Plan is: land this PR, update each perf branch from main, then dispatch with -f compare_only=true before they merge.

aaj3f · 2026-09-09T14:14:41Z

          tool: cargo-nextest

      - name: Test
        run: cargo nextest run --workspace --all-features --no-fail-fast


.github/workflows/ci.yml:113 — nit. --locked is on the doctest step (:119) but not on cargo nextest run above it, so a stale Cargo.lock surfaces after the seven-minute test run rather than at minute zero. One flag; fold-in-now material.

Done in 746725e. Added --locked to nextest, and to the clippy and default-feature check commands as well so the clippy job surfaces a drifted lockfile at the same time instead of building it.

aaj3f · 2026-09-09T14:14:41Z

 concurrency:
-  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
  cancel-in-progress: true


.github/workflows/ci.yml:12 — nit, informational. Cancelling superseded main runs plus save-if: main means a burst of merges can leave the Rust cache unsaved (rust-cache's post step skips a cancelled job) until one main run completes uninterrupted. Bounded and self-healing; just the explanation if a main run ever looks cold for no reason.

One correction to the premise: with cache-on-failure: true rust-cache's post-if is success() || CACHE_ON_FAILURE, so the save does run on a cancelled job. The real effect is a partially built target dir saved under the exact key; later main runs hit that key and skip saving until Cargo.lock, the toolchain, or another keyed input changes. Degraded rather than cold, and PRs still restore it. Wrote that down above concurrency: in 746725e.

aaj3f · 2026-09-09T14:14:42Z


 jobs:
  bench-gate:
+    if: ${{ !inputs.compare_only }}


.github/workflows/bench.yml:42 — nit, informational. if: ${{ !inputs.compare_only }} relies on inputs being empty on the schedule event, which it is, but no real nightly has run through this file yet (the schedule fires from main). The first night after merge is the test; if bench-gate unexpectedly skips, this is the line.

Made explicit in 746725e: github.event_name != 'workflow_dispatch' || !inputs.compare_only, mirroring the bench-capture guard. Will check the first 07:00 UTC run after merge shows both bench-gate and bench-compare.

aaj3f · 2026-09-09T14:14:42Z

-        run: cargo check --workspace --all-targets
+      # Nextest does not execute rustdoc examples. Reuse the all-feature build
+      # above so public API examples cannot silently stop compiling or running.
+      - name: Documentation tests


.github/workflows/ci.yml:117-119 — praise. The doctest step is a quiet but real coverage gain — nextest never ran rustdoc examples, so 47 public-API examples could have stopped compiling with every gate green.

aaj3f · 2026-09-09T14:14:42Z

+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v7
+      - name: Install actionlint


.github/workflows/ci.yml:31-38 — praise. Fetching actionlint by release URL and checking the sha256 before untarring is the right supply-chain posture for a binary we run on every PR.

…only guard Add --locked to clippy, the default-feature check, and nextest so a stale Cargo.lock fails at the start of each job instead of after the seven-minute test run, matching the doctest step that already had it. Guard bench-gate on the event name explicitly rather than relying on the `inputs` context being empty on the schedule event, mirroring bench-capture. Document that a cancelled main run can still save a partial rust-cache entry under the exact key: cache-on-failure makes the post step run on any status, so the effect of cancel-in-progress is a degraded cache, not a skipped save.

ci: use larger Apple silicon runner for releases

24966a6

bplatz requested review from aaj3f and zonotope September 7, 2026 13:52

ci: accelerate test and benchmark jobs and fix nightly cancellation

7da7f5a

bplatz changed the title ~~ci: use larger Apple silicon runner for releases~~ ci: speed up tests, benchmarks, and Apple releases Sep 7, 2026

bplatz added 5 commits September 7, 2026 10:10

ci: restrict nightly compilation to explicit benchmark targets

2fc9b4c

docs(ci): clarify nightly benchmark scope

fb2472b

ci: fix benchmark filtering and close workflow coverage gaps

a413004

ci: install verified actionlint release binary

77780d0

ci: move benchmark execution to nightly and parallelize checks

99bfce2

aaj3f approved these changes Sep 9, 2026

View reviewed changes

bplatz merged commit 9228e81 into main Sep 9, 2026
30 of 32 checks passed

bplatz deleted the ci/larger-apple-release-runner branch September 9, 2026 18:13

bplatz mentioned this pull request Sep 15, 2026

fix(raft): pin openraft fork that resamples election timeout per campaign #1845

Merged

Conversation

bplatz commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaj3f left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bplatz commented Sep 7, 2026 •

edited

Loading