Skip to content

Make protobuf the single definition of every format and print JSON only through JsonFormat - #21

Merged
lhotari merged 11 commits into
mainfrom
protobuf-json
Sep 24, 2026
Merged

lhotari merged 11 commits into
mainfrom
protobuf-json

Conversation

@lhotari

@lhotari lhotari commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Only the capture stream's stacks and observations and the stack profile had a schema. Everything else was hand-built JSON with its own conventions, much of it built in Rust:

  • the control records, which the stream carried as JSON strings;
  • the JNI protocol between the agent and the collector, where the C bridge even checked "ok":true by substring;
  • sampling/timeSplit, the manifest and every correlator output.

Three code paths had to agree on key names, hex encodings, null-vs-absent rules and number formats, and two JSON parsers with different strictness read the same data. Backward compatibility is not required, so this makes protobuf the only definition and JSON only a view of a message.

What changes

Schemas

  • jonoffcpu-capture-codec/src/main/proto/jonoffcpu-capture.proto: every stream record is typed. That covers CaptureStart, CaptureEnd (with kernel and userspace counters) and CaptureFinalized (with AnalysisInputs, the async-profiler stats, and the source and JFR artifacts), plus Sampling with oneof admission { none | uniform | proportional }, TimeSplit, VerifiedIdentity, and the wall-clock calibration and signal-environment diagnostics. There are no per-record schema versions; the stream header is format version 2.
  • jonoffcpu-collector.proto: PrepareRequest, EnableRequest, and CollectorReply (a state plus oneof prepared | enabled | stopped | closed | error, with typed error codes).
  • jonoffcpu-agent/src/main/proto/jonoffcpu-manifest.proto: the manifest. It embeds every collector reply as a message.
  • jonoffcpu-correlator/src/main/proto: jonoffcpu-report.proto (the report, classified records, matches and partial-mode files, markers), jonoffcpu-analysis.proto (stacks summary, transforms, top, digest, export rows, run metadata), jonoffcpu-signals.proto (JFR signal rows), and jonoffcpu-profile.proto schema 3, where sampling, time split and the report are typed instead of verbatim JSON strings.

Runtime and JSON

  • The codec module uses the full protobuf runtime plus protobuf-java-util, both digest-pinned and relocated in the shaded JARs; the agent JAR grows by about 0.8 MB.
  • CaptureFormat is the stream framing, shared by both modules.
  • ProtoJson is the only JSON printer and parser, using the proto3 JSON mapping:
    • lowerCamelCase field names;
    • 64-bit integers printed as strings;
    • enums printed by their prefixed UPPER_SNAKE value names;
    • fields without presence always printed, so a zero counter is visible;
    • unknown fields rejected.
  • Gson remains embedded only because JsonFormat needs it at run time; no jonoffcpu code uses it.

Native collector (ABI 2)

  • prepare/enable take encoded requests and every call returns an encoded CollectorReply. The C bridge copies byte[] both ways and parses nothing.
  • Errors carry a typed Failure{code} through anyhow context, which replaces the old matching on message text. Panics are still caught.
  • The collector writes typed records, and the frozen captureEnd still has its never-appended-twice guarantee.
  • The proof binaries print proto3 JSON through pbjson.

Agent

  • Requests and replies are protobuf messages. Echoed sampling, time split, identity and the capture end are checked by message equality, and STOPPING/CLOSING errors keep the handle for a retry.
  • ArtifactVerifier reads typed records.
  • The configuration file keeps its own spellings (blocked, uniform, schedInfo, queued, …) through a strict mapping onto the enums; enum value names are rejected there.

Correlator

  • It reads typed records, and the schema 2/3/4 compatibility paths are gone.
  • Every JSON output is a message printed through ProtoJson. CSV, Markdown and collapsed stacks are rendered from the same messages and are unchanged; the default profile rendering still reproduces the collapsed file byte for byte.
  • export --numbers is removed: JSONL counters are strings, and OFFLINE.md shows the DuckDB casts.
  • summarize takes the capture section from the report embedded in the profile.

Docs

  • AGENTS.md states the contracts: protobuf as the single definition, the binary JNI boundary, and JSON only through ProtoJson.
  • CODING.md says to assert on messages and to build inputs as messages.
  • The READMEs and OFFLINE.md describe the new outputs.

Verification

  • ./gradlew check after a clean build: agent 96 unit, 4 host-native (musl arm64 container, new JNI bundle), 2 packaged-JAR; correlator 183 unit, 19 integration (11 fixture-acceptance skips, which need external recordings), scale, and 3 packaged-JAR, including the DuckDB export test. verifyRuntimeJar passes for both JARs, and a second run reuses the configuration cache.
  • Native: the glibc and musl arm64 bundles build in Docker. cargo test --lib passes 19 tests, with 2 privileged tests ignored, and cargo fmt --check is clean.
  • Not run locally: the privileged-container end-to-end tests (agent → real collector → correlator), the x86-64 bundles and the kernel proof tools. CI covers the first two.

Known gaps

  • JsonFormat escapes some characters in strings, such as = as =. The output is valid JSON but harder to read, for example event=cpu in the manifest's profiler options.
  • jonoffcpu-native/tools/run-known-wait-attribution.py (and the make/build/test-classes steps of the other proof tools) predate the Gradle build and were already broken. Only their JSON reading is updated here.
  • The fixture-acceptance reference recordings (-PjonoffcpuFixtures) need regenerating for the new formats.

…module

Move everything the tests use but that is not a test into each module's
`src/testFixtures`, with Gradle's java-test-fixtures plugin: fixture
builders, JFR event types, the command-line and export helpers, and the
workloads and checks that tests and proof tools launch. Helpers that
other test classes called inside OfflineCorrelatorTest, CommandLineTest
and ExportTest become CorrelationFixture, CommandLineFixture and
ExportFixture. The fixture variants are never published.

The capture stream's Java codec is now generated once, in a new
unpublished jonoffcpu-capture-codec module that owns
src/main/proto/jonoffcpu-capture.proto; the agent and the correlator
embed it through embeddedRuntime and relocate its protobuf runtime as
before, and the correlator generates only its own profile schema from
src/main/proto. The fixture that encodes JSON rows as stream records,
duplicated in both modules, is one CaptureRecordFixture in the codec
module's test fixtures. The Rust collector and both native-bundle
Dockerfiles read the schema from its new location, and the native build
now lists it as an input.

The plain JAR of a shaded module gets a `plain` classifier and declares
its unrelocated libraries, so consumers inside the build no longer
resolve the shaded JAR in its place; only external embedded libraries
are checked against pinned digests. CODING.md documents the fixture
conventions.
@lhotari
lhotari changed the base branch from test-fixtures to main September 24, 2026 12:23
@lhotari
lhotari merged commit bc37f82 into main Sep 24, 2026
5 checks passed
@lhotari
lhotari deleted the protobuf-json branch September 24, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant