[improve][test] Improve performance experiments, analysis tooling and agent guidance - #26715

…red scenario settings

The performance scenarios' cluster, IoT workload and profiling settings were restructured, and the launcher rejects
settings that no longer exist, so the Key_Shared 500x20 scenarios failed to start.

- iot-key-shared-500x20.yaml sets the brokers' single-copy ledgers under cluster.brokers.env, and the measured
  messages, the payload size, unbatched producers and one application with 20 pods with the nested workload
  settings.
- iot-key-shared-500x20-profile.yaml extends the scenario with the broker and gateway profile files, which profile
  the same components as before.

Assisted-by: Claude Code (claude-opus-5-5)

…the high-rate scenario got a rate limit

iot-telemetry-high-rate.yaml, which the Key_Shared 500x20 scenario extends, now publishes 30,000 messages per second
with at most 10,000 in flight. The Key_Shared 500x20 scenario measures the broker's dispatch at saturation, so it sets
the settings it inherited before: no rate limit, and up to 100,000 messages in flight.

Assisted-by: Claude Code (claude-opus-5-5)

…s' agent quick reference

The host check is optional and applies only to Linux hosts, where it checks the configuration that keeps the
run-to-run variance low, so it goes last instead of first. The metrics stack's row gives the addresses of Grafana
and VictoriaMetrics.

Assisted-by: Claude Code (claude-opus-5-5)

…gent guide names

Link the README's "Read the report" section, the comparing-revisions, run-reports and analyzing-profiles pages,
which the guide named by their paths, and the jafar-perf plugin and the Jafar MCP server.

Assisted-by: Claude Code (claude-opus-5-5)

…, with a map of a run directory's inputs

Separate the guide's concerns into sections with tables: the checks before running, the platforms and what their
results are good for, the rules for using a run's results, the inputs for analysis in a run directory, the
analysis tools, and tuning experiments.

The inputs map every file of a run, a profiled component and a heap dump to what it has and what reads it,
including the off-CPU and async-profiler collapsed stacks, the flame graphs, the JFR recordings, the jonoffcpu
capture and stack profile, the latency logs, the sampled stats, metrics.json and the heap dumps. The tools table
lists what each analysis tool reads, what it is for and where to get it, linking the page that describes it.

Assisted-by: Claude Code (claude-opus-5-5)

… Docker's disk on macOS, with an exit code per kind of failure

"validate" checks that Docker is available and that Docker's disk has space on every operating system, and the
host's configuration on Linux only. Where Docker's data directory isn't on the host, as with Docker in a virtual
machine on macOS, it reads the disk's usage in a container, whose root file system is on the same disk as Docker's
data; df -P replaces GNU df's --output.

Its exit code tells the kinds of failed checks apart, as the sum of 2 for Docker's disk, 4 for Docker being
unavailable and 8 for the host's configuration, so that scripts and agents can act on each. The docs list the exit
codes, and no longer say that validate runs without root.

Assisted-by: Claude Code (claude-opus-5-5)

… guide as a table

List each host operating system and Docker engine with whether it was tested, whether off-CPU profiling works on
it, what configure-perf-test-environment.sh validate checks there and what its results are good for, following the
tested Docker engines in the README.

Assisted-by: Claude Code (claude-opus-5-5)

…ance tests' agent quick reference

Assisted-by: Claude Code (claude-opus-5-5)

…figure-perf-test-environment.sh

Always read how full Docker's disk is with df in a container, whose root file system is on the same disk as Docker's
data and the bookies' ledgers, instead of from Docker's data directory on the host, which isn't there when Docker
runs in a virtual machine. docker info only names the data directory in the output.

Assisted-by: Claude Code (claude-opus-5-5)

…exit code as a bit mask, and link it from the agent guide

Describe the exit code as a bit mask in which each kind of failed check sets its bit, naming each bit by its
position and its value: bit 1 (2) for Docker's disk, bit 2 (4) for Docker being unavailable and bit 3 (8) for the
host's configuration. The table has a heading of its own, Exit codes, and the agent guide refers to it when validate
fails, instead of explaining the bits itself.

Assisted-by: Claude Code (claude-opus-5-5)

Describe the experiment loop, command outputs, artifact and tool inputs, and reproducible evidence handoff. Clarify profiling tasks, measurement validity, and metric interpretation.

Assisted-by: Codex

Clarify unlimited-rate prerequisites, metric windows, profiling and platform constraints, and optional analysis tooling. Add cool-down guidance and distinguish duplicate warnings from failed delivery checks.

Assisted-by: Codex

Expose the correlator and JFR converter CLIs using pinned dependencies and document arbitrary argument passthrough.

Assisted-by: Codex

Describe jfr-converter, flameshow, Inferno and DuckDB quack_flamegraph with usage examples.

Assisted-by: Codex

Move SQL examples into the profile analysis guide, parameterize profile and package filters, and add coverage, call-edge and leaf attribution queries.

Assisted-by: Codex

Link DuckDB and quack_flamegraph from the integration agent guide, clarify collapsed and folded stack terminology, and document JSON and CSV output for automation.

Assisted-by: Codex

Make automated collapsed stack analysis discoverable, improve documentation flow and interpretation guidance, and add command-line quack_flamegraph examples.

Assisted-by: Codex

Assisted-by: Codex