Repository navigation
bench: compare AnalyticsEvent owned types - #327
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
iainmcgin
left a comment
There was a problem hiding this comment.
[claude code]
Thanks — the harness work here is careful, and I want to be clear up front that the measurement methodology is sound. The benchmark reuses the existing common.rs unchanged, so every variant runs an identical timed loop over identical input; data generation and message construction sit outside the timed region; outputs go through black_box; and the JSON denominators match across variants. I also confirmed it compiles and links under the bench profile (LTO, codegen-units=1), which matters more than usual because benchmarks/ is outside the cargo workspace and CI never builds it. The round-trip parity test over all 50 payloads is a genuine one, and each variant's build.rs config really does match its label — no copy-paste mismatch.
Four things to fix before I merge it.
1. build.rs:61 has a codename in it. The comment reads:
// ponytail: keep recursive children on Vec; add a heap-backed custom
// list axis only if a recursive-collection benchmark is requested.ponytail appears exactly once in the entire repository and means nothing to a reader. This is a public repo, so it needs to go. The stated reason is also wrong in a way worth correcting: keeping Nested.children on Vec isn't a scoping decision, it's a hard compile constraint — an inline SmallVec<[Nested; 4]> inside Nested makes Nested infinitely sized. Suggest:
// Nested.children stays Vec: an inline SmallVec<[Nested; 4]> inside Nested
// would be an infinitely sized type.2. The variants break per-message isolation, which desynchronises the charted history series. This is the one that actually matters. benches/analytics_event.rs opens with "only its own decoder is compiled" and carries a compile_error! guard enforcing it; benchmarks/history/README.md:213 explains why — "only the target shape's decoder is compiled, so no other shape can perturb it via the compiler's inlining." The guard only catches other messages, so it doesn't fire here, but linking three more full AnalyticsEvent codecs into that LTO/cgu=1 binary defeats the property it protects. Since build layout is the dominant noise source in this suite, the existing buffa/analytics_event/decode|encode|json_* rows would stop being comparable to v0.1.0–v0.9.0.
Please move the variants into their own bench target — benches/analytics_owned_types.rs with its own required-features — so the history binary is untouched. That also gives the comparison a natural home.
3. smolstr-smallvec4 is silently dropped by our own parser. It's the only benchmark id in the repo with a hyphen, and benchmarks/history/parse_criterion.py:62 is:
ID_RE = re.compile(r"^[A-Za-z0-9_]+(?:/[A-Za-z0-9_]+)+$")On a non-match it does current = None; continue, so two of the six new rows vanish from any capture with no error. Rename to smolstr_smallvec4.
4. The inline capacity doesn't match the data, so the headline would misanswer the issue. Across the 50 payloads, properties has a median length of 6.5 and only 28% of instances fit within the inline-4 capacity — so on the widest repeated field, SmallVec<[_; 4]> spills to the heap about 72% of the time, where it is a Vec plus a dead inline buffer. SmallList<[Nested; 4]> on sections also inline-stores four large Nested structs, each carrying its own inline SmallList<[Property; 4]>, which inflates size_of::<AnalyticsEvent>() considerably.
The benchmark measures all of that faithfully — but #215 asks for the optimal owned-type choices, and as built the smallvec4 axis measures SmallVec near its worst case. A reader would conclude "SmallVec doesn't help AnalyticsEvent" when the finding is really "inline-4 is the wrong capacity for this data." Either test a second capacity chosen from the distribution, or document the distribution alongside the results so the number can't be read as a general verdict. Your call which — the first is more useful, the second is cheaper.
None of this is structural; (1) and (3) are one-liners, (2) is a file move plus a Cargo.toml stanza, and (4) is a judgement call I'd rather you make than I. Happy to look again as soon as it's pushed.
One thing that surprised me in a good way: the borsh/cfg_aliases entries in the lockfile look alarming but are provably inert — they're smol_str's optional-dependency closure and never get compiled under the enabled feature set. Worth knowing so nobody rips them out later.
|
Thanks for the detailed review. I pushed f69bbef with all four items addressed:
The focused variant test, both isolated bench builds, workspace tests, clippy, rustdoc, fmt, and tracked Markdown lint all pass locally. |
The owned-type benches add buffa-smolstr as an optional path dependency. Cargo reads a path dep's manifest during resolution even when the feature is off, so Dockerfile.bench-buffa, which copies named directories and never examples/, fails at `cargo bench --no-run` with "failed to load manifest for dependency buffa-smolstr" before any benchmark runs. That breaks bench-cross and bench-docker outright. Copy the one directory it needs. Guard every isolated target against analytics_owned_types, not just the analytics one. Enabling the feature pulls the AnalyticsEvent codec and three variant codecs into the shared lib, so any target linking it loses the byte-identical layout the history series depends on. Selecting a single target with --bench, which is what the documented procedure and bench-iso both do, never compiles the one target whose guard already covered this. Record what the smallvec4 axis actually measures. It is dominated by inlining four 376-byte Nested values into `sections` (1520 bytes, ~79% of the 128 -> 1904 byte struct growth), not by list-length spill, so a uniformly larger inline capacity would make it worse rather than better. No-Verification-Needed: benchmark crate outside the workspace and CI; end-to-end path is the wire-parity test, which passes
|
[claude code] Re-reviewed. All three structural fixes landed cleanly, and I verified each by building rather than reading:
I pushed a commit rather than sending it back a second time, because one of the things I found is a hard build break and the rest are small. The build break. The isolation guard needed to go on all nine targets, not one. I'd added On the inline capacity — I got the mechanism wrong the first time, and the corrected version changes the recommendation. I'd written that
The 14.9x growth is dominated by That means inline-8 would make things worse, not better — Two things I deliberately left, both worth doing but not at merge time: the nine hand-maintained Nice detail I want to flag as good: the Merging. Thanks for the quick turnaround on the restructure — the separate target is the right shape, and the parity test over all 50 payloads is what made it safe to be confident about the variants. |
iainmcgin
left a comment
There was a problem hiding this comment.
[claude code] Approving on Iain's behalf — the three requested fixes landed, and the maintainer commit above covers the Docker build break, the remaining isolation guards, and a correction to the capacity note.
Summary
AnalyticsEventvariants usingSmolStr,SmallVec<[T; 4]>, and both togetheranalytics_owned_typestarget, leaving the historicalanalytics_eventtarget and its dependency graph unchangedNested.childrenfield heap-backed because inline storage would makeNestedinfinitely sizedThis is the focused first slice discussed in #215. It uses only the existing proto shape; map and singular-message axes can be grounded in separate representative datasets if they are still useful after reviewing these results.
Local benchmark
Apple M4 Max, Darwin arm64, Criterion with 1 s warm-up, 2 s measurement, and 20 samples:
decodemeanmergemeanString+VecSmolStr+VecString+SmallVec<[T; 4]>SmolStr+SmallVec<[T; 4]>These are local directional results, not history baselines. The comparison now runs in its own target, so the existing historical benchmark group and view measurement remain unchanged.
The capacity-4 rows are deliberately spill-heavy on this dataset:
propertieshas a median length of 6.5 and only 28% of messages fit that field inline. They therefore measure this specific capacity/data combination, not a general verdict onSmallVec; a capacity chosen from the distribution would need a separate follow-up measurement.Validation
cargo fmt --all --checkcargo clippy --workspace --all-targets -- -D warnings[email protected](excluding.claude, matching the task)cargo test --workspacecargo clippy --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_types --all-targets -- -D warningscargo test --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_typesRUSTDOCFLAGS='-D warnings' cargo doc --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_types --no-depscargo bench --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_types --no-run(only the comparison target is built)cargo bench --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_event --no-run(only the historical target is built)Part of #215.