Skip to content

Root the digest at the application and make it readable - #31

Merged
lhotari merged 5 commits into
mainfrom
digest-application-roots
Sep 25, 2026
Merged

lhotari merged 5 commits into
mainfrom
digest-application-roots

Conversation

@lhotari

@lhotari lhotari commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The digest (jonoffcpu-summary.md / .json) is the first thing a reader opens, but it didn't say where to look. On an Apache Pulsar broker capture:

  • It ranked stack leaves, which name the wait mechanism (AbstractQueuedSynchronizer.acquireSharedInterruptibly, a C2 monitor) rather than the code that waited.
  • 99 % of its "busy" time had no application frame (Jetty's ReservedThreadExecutor waiting for work, ZGC's driver), which pushed the 0.3 s of application waits to the tail.
  • It was hard to read. Prose sat between tables, 300-character commands were inline, and it had a table of 800-character stack strings. It also called off-CPU time "busy", which readers take for CPU time.

This PR makes three changes, one commit each (plus a follow-up that removes the root table's heaviest-stack column, which made that table unreadable):

  • roots the digest at the application
  • drops backward compatibility until 1.0.0
  • rewrites the digest to be readable

1. Root the digest at the application (--app)

  • --root-at-unmatched hide (stacks, top): stacks without a --root-at match are left out, and reported as rootAtUnmatchedHidden and by pool. It is the one transform that removes weight, and it's always accounted for.
  • preset:jvm-dispatch: lambda bridges and JDK/Guava executor adapters, for --hide, so that --root-at lands on the work instead of the executor that ran it.
  • top --by root: the flame graph's root boxes, where threads entered the application.
  • top --by app-method: application methods ranked across all stacks, inclusive. Methods always found in the same stacks are grouped into one call-chain row, with self time.
  • Boundary shares: with hiding, --by boundary shares are of the blocked time that has an application frame.
  • correlate and summarize take --app and --hide/--hide-from (default preset:jvm-dispatch). With --app, every digest table comes from the transforms of the application-rooted flame graph.

2. Drop backward compatibility until 1.0.0

AGENTS.md now says the compatibility rule applies from the 1.0.0 release on. Removed:

  • the reserved proto declarations
  • the --dump alias
  • --reason unspecified
  • schema-1 fallbacks

export columns now follow one logical order in CSV and JSON Lines.

3. A readable digest

  • Blocked and waiting replace busy and idle everywhere:
    • Markdown, JSON fields, --waiting/--waiting-from and preset:jvm-waiting. The old spellings are unknown options.
    • With runnable or preempted intervals selected, the first slice is "Blocked or in the run queue".
  • Metadata first. The digest opens with:
    • Recorded: the recording's start and end
    • Analysed: the selected window, or the whole recording
    • Process: PID, JVM and async-profiler versions
    • System: OS, kernel, libc, CPU and container limits
  • Tables first, one sentence each. Where the time went gains share columns. Definitions and caveats move to About this digest, which starts with the terms.
  • Every command is a bash block, wrapped one option per line past 100 characters. top --format md uses the same helper.
  • The heaviest-stacks sections are gone. The flame graph's stacks command stays under How to reproduce.
  • The JFR snapshot is stored in the report, so summarize shows it too. It comes from the JFR pass correlate already makes:
    • the chunk bounds and async-profiler's jdk.ActiveRecording
    • jdk.JVMInformation, jdk.OSInformation, jdk.CPUInformation and jdk.ContainerConfiguration, each kept whole as a protobuf Struct
    • the initial system properties and environment variables
  • --process-details true|false (default false) on correlate and summarize. The command line, system properties and environment variables can hold secrets. Without the flag they're removed from both the report and the digest.

On the key-shared broker fixture, the first table is led by MessageDeduplication.isDuplicateNormal on a C2 monitor (0.087 s, 29.1 % of the time with an application frame). Where the time went reads:

  • blocked: 100.0 % of blocked, 1.1 % of all selected
  • with an application frame: 0.7 %, < 0.1 %
  • without one: 99.3 %, 1.1 %
  • waiting: 98.9 % of all selected

4. preset:* selects every bundled preset meant for an option

  • # options: headers. Each preset's header names the options it is meant for, such as # options: hide. preset:* in a -from option stands for every such preset, in --list-presets order.
  • Keep the presets and add your own: --hide-from 'preset:*' --hide-from FILE keeps jonoffcpu's patterns and adds the consumer's. It also picks up a preset added later for the same purpose.
  • Defaults: an option's default is exactly its preset:*.
  • Options without presets (--include-from, --root-at-from, --leaf-at-from, --app-from) refuse preset:* with exit status 64. An unknown preset is now a usage error (64) too.
  • How it's recorded: summaries and the digest's notes list the presets it expanded to. The digest's reproduce commands repeat 'preset:*', quoted.
  • --list-presets prints each preset's options.

Compatibility

None is kept: this is before 1.0.0, and AGENTS.md now says so.

  • Options:
    • --idle/--idle-from → --waiting/--waiting-from
    • preset:jvm-idle → preset:jvm-waiting
    • --dump → dump
    • --reason unspecified is removed
  • JSON:
    • busy/idle fields are renamed to blocked/waiting
    • the digest's heaviest-stacks fields are removed
    • ExportRow is renumbered, and the export columns are reordered
  • Pulsar launcher: it must pass --waiting-from instead of --idle-from to correlate, and can pass --hide-from 'preset:*' instead of naming preset:jvm-dispatch.

Verification

  • ./gradlew spotlessCheck :jonoffcpu-agent:check :jonoffcpu-correlator:check :jonoffcpu-jfr-converter:check :jonoffcpu-capture-codec:check :jonoffcpu-correlator:readmeTest passes. This includes the agent's privileged end-to-end integration tests (none skipped).
  • Digest tests:
    • golden files for both layouts
    • the reading rules: no busy or idle outside code, one sentence between tables, command blocks at most 100 characters that parse back to their command
    • every reproduce command, run as written, prints its table byte for byte
    • the run-queue label and --process-details gating
    • rendering with commonmark-java, with its tables and heading-anchor extensions, as the launcher does: one language-bash block per command, no long inline code, and every in-page link resolves
  • Correlation tests:
    • the fixture JFR now records the JDK's process events
    • the report carries them, without the command line by default and with it under --process-details true
  • Fixture acceptance (-PjonoffcpuFixtures=…, key-shared fixture):
    • the root, method and boundary rows match the spec's reference computations
    • the share columns and the analysed window match
  • The older 09-23 fixtures predate the current profile schema and fail on main too, so they aren't run here.

The digest never used an application pattern: correlate built it
without one, so it ranked stack leaves, which name the wait mechanism
(AbstractQueuedSynchronizer.acquireSharedInterruptibly, a C2 monitor)
rather than the code that waited. Busy time without an application
frame dominated its tables (40.27 of 40.57 busy seconds on a Pulsar
broker), and --root-at often landed on an executor or a lambda bridge
instead of the work.

stacks and top gain --root-at-unmatched hide, the one transform that
removes weight: unmatched entries leave the lines and rows and are
reported as a total of their own (rootAtUnmatchedHidden in the stacks
summary, a totals row and the pool table in top). preset:jvm-dispatch
hides lambda bridges and JDK/Guava executor adapters so the root is
the task's code. top gains --by root (the flame graph's root, with the
heaviest line under it) and --by app-method (application methods across
all stacks, inclusive, call chains grouped, with self time); with
hiding, boundary shares are of the busy time with an application frame.

correlate and summarize take --app and --hide/--hide-from (default
preset:jvm-dispatch). With --app the digest is computed from the
transforms of the application-rooted flame graph and leads with the
application method that waited, then the roots, the heaviest
application stacks and the application methods, followed by where the
time went and the unattributed busy time by pool, with a note when it
is more than half of the busy time. The Digest message gains
schemaVersion 2 and the new tables. Without --app the Markdown is
unchanged, which a golden file generated from the previous code checks.
Nothing before the 1.0.0 release needs to read old files or accept old
spellings, so the compatibility kept so far is removed rather than
carried: the digest's schemaVersion and its golden file of the layout
without --app, the reserved field declarations of the analysis,
profile and report messages, the deprecated `--dump` alias (the signal
pressure tool now runs `dump`), the schema 1 fallback wording of the
package-name rule and merge, a SourceColumns.add overload without a
switch-out reason, and the digest's 16 KB size bound.

export no longer keeps its pre-0.5.0 columns first: CSV and JSON Lines
share one order, the run and the entry's identity, its counters, then
its stacks, and ExportRow is renumbered to match. --reason no longer
accepts unspecified, which no capture records; all selects blocked,
runnable and preempted.

AGENTS.md now says the compatibility rule applies from 1.0.0 on, and
that until then formats and options change outright.
The digest hid its tables behind prose, printed 300-character commands
inline, carried a table of unreadable stack strings, and called off-CPU
time "busy", which readers take for CPU time. It also did not say when
or on what the capture was recorded.

Busy and idle become blocked and waiting everywhere: the Markdown of
both digest layouts and of top, the analysis messages, --waiting and
--waiting-from, and preset:jvm-waiting; the old spellings are unknown.
With runnable or preempted intervals selected the blocked slice is
"Blocked or in the run queue".

The digest opens with when the capture was recorded, the window
analysed, the process and the system, then one headline line, then the
tables with one sentence each. Where the time went gains shares of the
blocked time and of the selection. Definitions and caveats move to
About this digest, which starts with the terms, and every command is a
bash block wrapped one option per line (a shared Top.shellBlock, also
used by top --format md). The heaviest-stacks sections are removed; the
flame graph's stacks command stays under How to reproduce.

The JFR pass keeps a snapshot of the recording in the report: the chunk
bounds, async-profiler's version, and the first jdk.JVMInformation,
jdk.OSInformation, jdk.CPUInformation and jdk.ContainerConfiguration
events whole, as protobuf Structs, with the initial system properties
and environment variables. The command line, system properties and
environment can hold secrets: the report keeps them, and the digest
shows them, only with --process-details true.
A root row's heaviest transformed line is hundreds of characters long, which
made the root table unreadable in the digest and in top --format md. The flame
graph shows what is under a root better; the column, TopRow.heaviest_stack and
the CSV heaviest_stack column are removed.
A consumer that adds its own patterns to an option with a default preset
had to name the preset to keep it, tying it to today's preset names: a
preset split or added for the same purpose would be missed silently.

Each preset's header now says which options it is meant for (# options:
hide), and preset:* in a -from option stands for every such preset, in
--list-presets order, so --hide-from 'preset:*' --hide-from FILE keeps
jonoffcpu's patterns and adds the consumer's. An option's default is
exactly its preset:*. An option without presets (--include-from,
--root-at-from, --leaf-at-from, --app-from) refuses preset:* with exit
status 64 rather than expanding to nothing, as an unknown preset now is
too. Summaries and the digest's notes record the presets it expanded to;
the digest's reproduce commands repeat 'preset:*' as given, quoted.
--list-presets prints each preset's options.
@lhotari
lhotari merged commit 0730ec9 into main Sep 25, 2026
7 checks passed
@lhotari
lhotari deleted the digest-application-roots branch September 25, 2026 03:25

This branch was successfully deployed

1 active deployment
release — da9b3b00 Deployed Sep 25, 2026 by lhotari via Update README version numbers #10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant