Repository navigation
Make protobuf the single definition of every format and print JSON only through JsonFormat - #21
Merged
Merged
Conversation
…module Move everything the tests use but that is not a test into each module's `src/testFixtures`, with Gradle's java-test-fixtures plugin: fixture builders, JFR event types, the command-line and export helpers, and the workloads and checks that tests and proof tools launch. Helpers that other test classes called inside OfflineCorrelatorTest, CommandLineTest and ExportTest become CorrelationFixture, CommandLineFixture and ExportFixture. The fixture variants are never published. The capture stream's Java codec is now generated once, in a new unpublished jonoffcpu-capture-codec module that owns src/main/proto/jonoffcpu-capture.proto; the agent and the correlator embed it through embeddedRuntime and relocate its protobuf runtime as before, and the correlator generates only its own profile schema from src/main/proto. The fixture that encodes JSON rows as stream records, duplicated in both modules, is one CaptureRecordFixture in the codec module's test fixtures. The Rust collector and both native-bundle Dockerfiles read the schema from its new location, and the native build now lists it as an input. The plain JAR of a shaded module gets a `plain` classifier and declares its unrelocated libraries, so consumers inside the build no longer resolve the shaded JAR in its place; only external embedded libraries are checked against pinned digests. CODING.md documents the fixture conventions.
…-af1303a07a0c806b8' into protobuf-json
# Conflicts: # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Only the capture stream's stacks and observations and the stack profile had a schema. Everything else was hand-built JSON with its own conventions, much of it built in Rust:
"ok":trueby substring;sampling/timeSplit, the manifest and every correlator output.Three code paths had to agree on key names, hex encodings, null-vs-absent rules and number formats, and two JSON parsers with different strictness read the same data. Backward compatibility is not required, so this makes protobuf the only definition and JSON only a view of a message.
What changes
Schemas
jonoffcpu-capture-codec/src/main/proto/jonoffcpu-capture.proto: every stream record is typed. That coversCaptureStart,CaptureEnd(with kernel and userspace counters) andCaptureFinalized(withAnalysisInputs, the async-profiler stats, and the source and JFR artifacts), plusSamplingwithoneof admission { none | uniform | proportional },TimeSplit,VerifiedIdentity, and the wall-clock calibration and signal-environment diagnostics. There are no per-record schema versions; the stream header is format version 2.jonoffcpu-collector.proto:PrepareRequest,EnableRequest, andCollectorReply(a state plusoneof prepared | enabled | stopped | closed | error, with typed error codes).jonoffcpu-agent/src/main/proto/jonoffcpu-manifest.proto: the manifest. It embeds every collector reply as a message.jonoffcpu-correlator/src/main/proto:jonoffcpu-report.proto(the report, classified records, matches and partial-mode files, markers),jonoffcpu-analysis.proto(stacks summary, transforms, top, digest, export rows, run metadata),jonoffcpu-signals.proto(JFR signal rows), andjonoffcpu-profile.protoschema 3, where sampling, time split and the report are typed instead of verbatim JSON strings.Runtime and JSON
protobuf-java-util, both digest-pinned and relocated in the shaded JARs; the agent JAR grows by about 0.8 MB.CaptureFormatis the stream framing, shared by both modules.ProtoJsonis the only JSON printer and parser, using the proto3 JSON mapping:JsonFormatneeds it at run time; no jonoffcpu code uses it.Native collector (ABI 2)
prepare/enabletake encoded requests and every call returns an encodedCollectorReply. The C bridge copiesbyte[]both ways and parses nothing.Failure{code}throughanyhowcontext, which replaces the old matching on message text. Panics are still caught.captureEndstill has its never-appended-twice guarantee.Agent
STOPPING/CLOSINGerrors keep the handle for a retry.ArtifactVerifierreads typed records.blocked,uniform,schedInfo,queued, …) through a strict mapping onto the enums; enum value names are rejected there.Correlator
ProtoJson. CSV, Markdown and collapsed stacks are rendered from the same messages and are unchanged; the default profile rendering still reproduces the collapsed file byte for byte.export --numbersis removed: JSONL counters are strings, and OFFLINE.md shows the DuckDB casts.summarizetakes the capture section from the report embedded in the profile.Docs
ProtoJson.Verification
./gradlew checkafter a clean build: agent 96 unit, 4 host-native (musl arm64 container, new JNI bundle), 2 packaged-JAR; correlator 183 unit, 19 integration (11 fixture-acceptance skips, which need external recordings), scale, and 3 packaged-JAR, including the DuckDB export test.verifyRuntimeJarpasses for both JARs, and a second run reuses the configuration cache.cargo test --libpasses 19 tests, with 2 privileged tests ignored, andcargo fmt --checkis clean.privileged-containerend-to-end tests (agent → real collector → correlator), the x86-64 bundles and the kernel proof tools. CI covers the first two.Known gaps
JsonFormatescapes some characters in strings, such as=as=. The output is valid JSON but harder to read, for exampleevent=cpuin the manifest's profiler options.jonoffcpu-native/tools/run-known-wait-attribution.py(and themake/build/test-classessteps of the other proof tools) predate the Gradle build and were already broken. Only their JSON reading is updated here.-PjonoffcpuFixtures) need regenerating for the new formats.