Skip to content

bench: compare AnalyticsEvent owned types - #327

Merged
iainmcgin merged 3 commits into
anthropics:mainfrom
ogarciarevett:bench/analytics-owned-types
Jul 21, 2026
Merged

iainmcgin merged 3 commits into
anthropics:mainfrom
ogarciarevett:bench/analytics-owned-types

Conversation

@ogarciarevett

@ogarciarevett ogarciarevett commented Jul 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • generate isolated AnalyticsEvent variants using SmolStr, SmallVec<[T; 4]>, and both together
  • benchmark binary and JSON operations in a dedicated analytics_owned_types target, leaving the historical analytics_event target and its dependency graph unchanged
  • verify every generated variant preserves wire semantics and keep the recursive Nested.children field heap-backed because inline storage would make Nested infinitely sized
  • keep the comparison dependencies optional and use parser-safe benchmark IDs

This is the focused first slice discussed in #215. It uses only the existing proto shape; map and singular-message axes can be grounded in separate representative datasets if they are still useful after reviewing these results.

Local benchmark

Apple M4 Max, Darwin arm64, Criterion with 1 s warm-up, 2 s measurement, and 20 samples:

Owned representation decode mean Change from default merge mean Change from default
String + Vec 452.12 us — 436.97 us —
SmolStr + Vec 306.66 us -32.2% 305.00 us -30.2%
String + SmallVec<[T; 4]> 413.35 us -8.6% 403.58 us -7.6%
SmolStr + SmallVec<[T; 4]> 285.30 us -36.9% 271.24 us -37.9%

These are local directional results, not history baselines. The comparison now runs in its own target, so the existing historical benchmark group and view measurement remain unchanged.

The capacity-4 rows are deliberately spill-heavy on this dataset: properties has a median length of 6.5 and only 28% of messages fit that field inline. They therefore measure this specific capacity/data combination, not a general verdict on SmallVec; a capacity chosen from the distribution would need a separate follow-up measurement.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • tracked Markdown passes [email protected] (excluding .claude, matching the task)
  • cargo test --workspace
  • cargo clippy --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_types --all-targets -- -D warnings
  • cargo test --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_types
  • RUSTDOCFLAGS='-D warnings' cargo doc --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_types --no-deps
  • cargo bench --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_owned_types --no-run (only the comparison target is built)
  • cargo bench --manifest-path benchmarks/buffa/Cargo.toml --no-default-features --features iso,analytics_event --no-run (only the historical target is built)

Part of #215.

@github-actions

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@iainmcgin iainmcgin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude code]

Thanks — the harness work here is careful, and I want to be clear up front that the measurement methodology is sound. The benchmark reuses the existing common.rs unchanged, so every variant runs an identical timed loop over identical input; data generation and message construction sit outside the timed region; outputs go through black_box; and the JSON denominators match across variants. I also confirmed it compiles and links under the bench profile (LTO, codegen-units=1), which matters more than usual because benchmarks/ is outside the cargo workspace and CI never builds it. The round-trip parity test over all 50 payloads is a genuine one, and each variant's build.rs config really does match its label — no copy-paste mismatch.

Four things to fix before I merge it.

1. build.rs:61 has a codename in it. The comment reads:

// ponytail: keep recursive children on Vec; add a heap-backed custom
// list axis only if a recursive-collection benchmark is requested.

ponytail appears exactly once in the entire repository and means nothing to a reader. This is a public repo, so it needs to go. The stated reason is also wrong in a way worth correcting: keeping Nested.children on Vec isn't a scoping decision, it's a hard compile constraint — an inline SmallVec<[Nested; 4]> inside Nested makes Nested infinitely sized. Suggest:

// Nested.children stays Vec: an inline SmallVec<[Nested; 4]> inside Nested
// would be an infinitely sized type.

2. The variants break per-message isolation, which desynchronises the charted history series. This is the one that actually matters. benches/analytics_event.rs opens with "only its own decoder is compiled" and carries a compile_error! guard enforcing it; benchmarks/history/README.md:213 explains why — "only the target shape's decoder is compiled, so no other shape can perturb it via the compiler's inlining." The guard only catches other messages, so it doesn't fire here, but linking three more full AnalyticsEvent codecs into that LTO/cgu=1 binary defeats the property it protects. Since build layout is the dominant noise source in this suite, the existing buffa/analytics_event/decode|encode|json_* rows would stop being comparable to v0.1.0–v0.9.0.

Please move the variants into their own bench target — benches/analytics_owned_types.rs with its own required-features — so the history binary is untouched. That also gives the comparison a natural home.

3. smolstr-smallvec4 is silently dropped by our own parser. It's the only benchmark id in the repo with a hyphen, and benchmarks/history/parse_criterion.py:62 is:

ID_RE = re.compile(r"^[A-Za-z0-9_]+(?:/[A-Za-z0-9_]+)+$")

On a non-match it does current = None; continue, so two of the six new rows vanish from any capture with no error. Rename to smolstr_smallvec4.

4. The inline capacity doesn't match the data, so the headline would misanswer the issue. Across the 50 payloads, properties has a median length of 6.5 and only 28% of instances fit within the inline-4 capacity — so on the widest repeated field, SmallVec<[_; 4]> spills to the heap about 72% of the time, where it is a Vec plus a dead inline buffer. SmallList<[Nested; 4]> on sections also inline-stores four large Nested structs, each carrying its own inline SmallList<[Property; 4]>, which inflates size_of::<AnalyticsEvent>() considerably.

The benchmark measures all of that faithfully — but #215 asks for the optimal owned-type choices, and as built the smallvec4 axis measures SmallVec near its worst case. A reader would conclude "SmallVec doesn't help AnalyticsEvent" when the finding is really "inline-4 is the wrong capacity for this data." Either test a second capacity chosen from the distribution, or document the distribution alongside the results so the number can't be read as a general verdict. Your call which — the first is more useful, the second is cheaper.

None of this is structural; (1) and (3) are one-liners, (2) is a file move plus a Cargo.toml stanza, and (4) is a judgement call I'd rather you make than I. Happy to look again as soon as it's pushed.

One thing that surprised me in a good way: the borsh/cfg_aliases entries in the lockfile look alarming but are provably inert — they're smol_str's optional-dependency closure and never get compiled under the enabled feature set. Worth knowing so nobody rips them out later.

@ogarciarevett

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I pushed f69bbef with all four items addressed:

  1. Replaced the codename comment with the recursive-size constraint.
  2. Moved the comparison into a dedicated analytics_owned_types target/feature; the historical analytics_event build no longer includes the optional codecs or dependencies.
  3. Renamed the combined ID to smolstr_smallvec4; all comparison IDs now match the history parser regex.
  4. Documented the capacity distribution next to the results: median properties length is 6.5 and only 28% fit inline-4, so the row is explicitly spill-heavy rather than a general SmallVec verdict.

The focused variant test, both isolated bench builds, workspace tests, clippy, rustdoc, fmt, and tracked Markdown lint all pass locally.

The owned-type benches add buffa-smolstr as an optional path dependency.
Cargo reads a path dep's manifest during resolution even when the feature is
off, so Dockerfile.bench-buffa, which copies named directories and never
examples/, fails at `cargo bench --no-run` with "failed to load manifest for
dependency buffa-smolstr" before any benchmark runs. That breaks bench-cross
and bench-docker outright. Copy the one directory it needs.

Guard every isolated target against analytics_owned_types, not just the
analytics one. Enabling the feature pulls the AnalyticsEvent codec and three
variant codecs into the shared lib, so any target linking it loses the
byte-identical layout the history series depends on. Selecting a single
target with --bench, which is what the documented procedure and bench-iso
both do, never compiles the one target whose guard already covered this.

Record what the smallvec4 axis actually measures. It is dominated by
inlining four 376-byte Nested values into `sections` (1520 bytes, ~79% of the
128 -> 1904 byte struct growth), not by list-length spill, so a uniformly
larger inline capacity would make it worse rather than better.

No-Verification-Needed: benchmark crate outside the workspace and CI; end-to-end path is the wire-parity test, which passes
@iainmcgin

Copy link
Copy Markdown
Collaborator

[claude code]

Re-reviewed. All three structural fixes landed cleanly, and I verified each by building rather than reading:

  • Isolation restored. benches/analytics_event.rs is byte-identical to pre-PR, every variant item is behind the new analytics_owned_types feature, and the history-procedure build produces a binary with zero smol_str/smallvec symbols. Splitting into a separate target and renaming the group to buffa/analytics_owned_types/* is better than what I asked for — the variant rows now can't be confused with the charted series at all.
  • ponytail is gone, replaced with the correct infinite-size reason for Nested.children.
  • smolstr_smallvec4 now matches parse_criterion.py's id pattern.

I pushed a commit rather than sending it back a second time, because one of the things I found is a hard build break and the rest are small.

The build break. buffa-smolstr is an optional path dependency, and cargo reads a path dep's manifest during resolution even when the feature is off. Dockerfile.bench-buffa copies named directories and never examples/, so task bench-cross and task bench-docker now fail at cargo bench --no-run with failed to load manifest for dependency buffa-smolstr, before any benchmark runs. I reproduced it by replicating the Dockerfile's exact COPY set (exit 101), confirmed one added COPY examples/buffa-smolstr/ line fixes it (exit 0), and pushed that. Nothing in CI would have caught this — no workflow builds benchmarks/ at all.

The isolation guard needed to go on all nine targets, not one. I'd added analytics_owned_types to analytics_event.rs's guard, which was incomplete of me. Enabling the feature pulls the AnalyticsEvent codec plus three variant codecs into the shared lib, so every target linking it loses byte-identical layout. cargo check --bench api_response --features iso,api_response,analytics_owned_types compiled clean before this. It only looked safe because a whole-crate cargo check trips the new target's own guard — and single-target --bench <name> selection, which is exactly what the history procedure and task bench-iso use, never compiles that file. Now guarded in all eight siblings; verified the bad combination errors and all eight normal isolated builds still pass.

On the inline capacity — I got the mechanism wrong the first time, and the corrected version changes the recommendation. I'd written that smallvec4 looks bad because properties (median 6.5, max 10) spills past its 4-element buffer ~72% of the time, and suggested inline-8 as the axis that would show SmallVec winning. Measuring size_of says otherwise:

variant AnalyticsEvent Nested Property
default / smolstr 128 96 72
smallvec4 / both 1904 376 72

The 14.9x growth is dominated by sections, whose element type Nested is 376 bytes: its inline buffer alone is 4 * 376 + 16 = 1520, about 79% of the total. So the configuration inlines the wrong field twice over — it pays 1520 bytes to inline a large message type whose lists are short anyway, while under-sizing properties, the field whose 72-byte elements are actually cheap to inline.

That means inline-8 would make things worse, not better — Nested past 600 bytes and the message past 5 KB. The corrected note is now in the bench header, and the real follow-up is to apply SmallList only to small-element fields sized to their distribution, leaving message-typed lists on Vec. I'll file that against #215.

Two things I deliberately left, both worth doing but not at merge time: the nine hand-maintained compile_error! lists are O(n²) and this is the second time a feature had to be added to all of them by hand — a single "exactly one message feature enabled" count assertion in common.rs would make forgetting impossible and delete ~90 lines. And the owned-type variants generate view code no benchmark reads (~1200 lines × 3), which .generate_views(false) would drop.

Nice detail I want to flag as good: the borsh/cfg_aliases entries in the lockfile look alarming but are provably inert — smol_str's optional-dependency closure, never compiled under the enabled feature set.

Merging. Thanks for the quick turnaround on the restructure — the separate target is the right shape, and the parity test over all 50 payloads is what made it safe to be confident about the variants.

@iainmcgin iainmcgin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude code] Approving on Iain's behalf — the three requested fixes landed, and the maintainer commit above covers the Docker build break, the remaining isolation guards, and a correction to the capacity note.

@iainmcgin
iainmcgin added this pull request to the merge queue Jul 21, 2026
Merged via the queue into anthropics:main with commit 2518c0b Jul 21, 2026
11 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants