Repository navigation
refactor(adapter): unify JSONL parsing with typed structs and a shared helper - #1327
Conversation
Convert the agent adapters from dynamic `serde_json::Value` parsing plus hand-written `Value::get` navigation to typed `#[derive(Deserialize)]` structs parsed through the shared `adapter::jsonl` helper. Previously each adapter built a full `Value` tree per line (and several allocated a `String` per line via `BufReader::lines()`); they now read the file once, prefilter lines with `memmem`, and deserialize only the fields ccusage consumes. Converted: amp, codebuff, copilot, gemini, goose, kilo, kimi, openclaw, opencode, pi, qwen. Claude and Codex already used typed parsing. Behavior is preserved exactly: the lenient `deserialize_with` helpers match the old `Value` accessor semantics, prefilter markers are chosen to never drop an accepted line (or `None` where no always-present substring exists), and parse/non-object handling keeps the same skip-vs-error outcomes. Adapters with no line-delimited JSON hot path (single-object or SQLite sources) reuse only the leniency helpers. Droid is intentionally left on its existing path: it has no JSONL hot path, so typing it would add edge divergences with no performance benefit. Verified output parity (daily/monthly/weekly/session, --json) against both the previous main build and `ccusage@latest` for every agent with local logs (claude, codex, amp, copilot, gemini, opencode, pi): byte-for-byte identical.
|
no API key found — this repo is configured to use To fix: add the key as a GitHub Actions secret (referenced from your workflow's Open repo secrets → · Configure model → · Setup docs → · Ask in Discord →
|
…refilter Adopt the `LinePrefilter` helper (originally proposed in #1326) so the typed `jsonl::records` path can prefilter on multiple markers with AND/OR semantics instead of a single substring. `records` now takes `Option<&LinePrefilter>`. This restores the precise prefilters that single-marker filtering had to drop: - pi: require both `"usage"` and `"message"` (was `"usage"` only) - kimi: require both `"StatusUpdate"` and `"token_usage"` (was `"token_usage"` only) - openclaw: admit any of `"model_change"` / `"model-snapshot"` / `"usage"` (previously had no prefilter at all) - qwen / copilot: unchanged markers, now via LinePrefilter `LinePrefilter` builds the `memmem` needles once and reuses them across every line. Output remains byte-for-byte identical (verified against ccusage@latest for all agents with local logs). Supersedes #1326 by folding its prefilter into the typed-parsing refactor.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | 6eb56ca | Commit Preview URL Branch Preview URL |
Jun 15 2026, 04:21 PM |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughA new shared ChangesTyped serde refactor across adapter parsers
Sequence Diagram(s)sequenceDiagram
participant FileSystem
participant Adapter
participant LinePrefilter
participant jsonl_records
participant serde_json
participant Parser
FileSystem->>Adapter: fs::read(file)
Adapter->>Adapter: Create LinePrefilter::all/any(markers)
Adapter->>jsonl_records: records::<T>(bytes, Some(prefilter))
loop For each line in bytes
jsonl_records->>Adapter: Split by newline
jsonl_records->>LinePrefilter: matches(line)
alt Line matches prefilter
LinePrefilter-->>jsonl_records: true
jsonl_records->>serde_json: from_slice::<T>(line)
alt Parse succeeds
serde_json-->>jsonl_records: Ok(T)
jsonl_records-->>Parser: Yield T
else Parse fails
serde_json-->>jsonl_records: Err (skip silently)
end
else Line does not match
LinePrefilter-->>jsonl_records: false (skip)
end
end
Parser->>Parser: Process typed T records
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
@coderabbitai review |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
7 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="rust/crates/ccusage/src/adapter/codebuff/parser.rs">
<violation number="1" location="rust/crates/ccusage/src/adapter/codebuff/parser.rs:42">
P2: Invalid UTF-8 chat files are now silently treated as empty data. This can hide corrupted inputs and undercount usage with no error signal.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
| }; | ||
| let Some(messages) = messages.as_array() else { | ||
| let content = fs::read(path)?; | ||
| let Ok(messages) = serde_json::from_slice::<Vec<Value>>(&content) else { |
There was a problem hiding this comment.
P2: Invalid UTF-8 chat files are now silently treated as empty data. This can hide corrupted inputs and undercount usage with no error signal.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At rust/crates/ccusage/src/adapter/codebuff/parser.rs, line 42:
<comment>Invalid UTF-8 chat files are now silently treated as empty data. This can hide corrupted inputs and undercount usage with no error signal.</comment>
<file context>
@@ -38,11 +38,8 @@ struct CodebuffContext {
- };
- let Some(messages) = messages.as_array() else {
+ let content = fs::read(path)?;
+ let Ok(messages) = serde_json::from_slice::<Vec<Value>>(&content) else {
return Ok(Vec::new());
};
</file context>
There was a problem hiding this comment.
This byte-read/skip-gracefully behavior is the PR's explicit, documented design choice (consistent with the Claude loader and all other adapters); the old codebuff hard-errored on invalid UTF-8, and reverting only codebuff would reintroduce cross-adapter inconsistency.
…r-jsonl-parsing # Conflicts: # rust/crates/ccusage/src/adapter/copilot/parser.rs # rust/crates/ccusage/src/adapter/kimi/parser.rs # rust/crates/ccusage/src/adapter/openclaw/parser.rs # rust/crates/ccusage/src/adapter/pi/parser.rs # rust/crates/ccusage/src/adapter/qwen/parser.rs # rust/crates/ccusage/src/fast.rs
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
rust/crates/ccusage/src/adapter/amp/parser.rs (1)
86-96:⚠️ Potential issue | 🟠 Major | ⚡ Quick winFall back to message usage when the ledger yields no entries.
Because
usage_ledgerisSomeeven for an empty/defaulted ledger, this branch returns an empty result and skips valid message-level usage. Parse the ledger first, but only return it when it produced entries; otherwise continue toparse_message_usage.Proposed fix
if let Some(ledger) = thread.usage_ledger.as_ref() { let cache_tokens = cache_tokens_by_message_id(messages); - return Ok(parse_ledger_events( + let ledger_entries = parse_ledger_events( &ledger.events, &cache_tokens, &thread_id, tz, mode, pricing, - )); + ); + if !ledger_entries.is_empty() { + return Ok(ledger_entries); + } } Ok(parse_message_usage(messages, &thread_id, tz, mode, pricing))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/crates/ccusage/src/adapter/amp/parser.rs` around lines 86 - 96, The current code returns immediately when usage_ledger is Some, but this doesn't account for empty or defaulted ledgers. Modify the logic to parse the ledger using parse_ledger_events, but only return that result if it produced non-empty entries. If the parsed ledger result is empty, the code should continue to the fallback logic (parse_message_usage) instead of returning an empty result. Check whether the result from parse_ledger_events contains entries before deciding whether to return it or fall through to the message usage parsing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/crates/ccusage/src/adapter/copilot/parser.rs`:
- Around line 481-489: In the timestamp_from_record function, add an additional
or_else clause to parse record.time as a scalar value using
timestamp_from_scalar, in addition to the existing attempt to parse it as parts
using timestamp_from_parts. This should be inserted after the or_else call for
timestamp_from_parts(record.time.as_ref()) to ensure that both structured and
scalar representations of the time field are tried before falling back to other
timestamp fields. This prevents numeric scalar time values from being ignored
and avoids misdating records.
In `@rust/crates/ccusage/src/adapter/gemini/parser.rs`:
- Around line 144-160: The code currently uses fallback_timestamp as the default
timestamp for records without explicit timestamp/created_at fields, but it
should use session_timestamp instead. The session_timestamp already resolves
startTime/lastUpdated from the session, which is more reliable than falling back
to file mtime. Replace the fallback_timestamp parameter with session_timestamp
in both the parse_direct_event_record call and the parse_stats_events call to
ensure records without timestamps are assigned the correct session-level
timestamp.
In `@rust/crates/ccusage/src/adapter/pi/parser.rs`:
- Around line 55-56: The `cost` field in the `PiLine` struct uses strict object
deserialization, which causes the entire line to be dropped if the upstream data
emits `cost` as a non-object shape (string, number, null, etc.). Modify the
deserialization of the `cost: Option<PiCost>` field to be more lenient and
gracefully handle non-object cost values without failing the deserialization of
the entire `PiLine` record, allowing valid usage data to be retained even when
the cost field is malformed or unexpected.
---
Outside diff comments:
In `@rust/crates/ccusage/src/adapter/amp/parser.rs`:
- Around line 86-96: The current code returns immediately when usage_ledger is
Some, but this doesn't account for empty or defaulted ledgers. Modify the logic
to parse the ledger using parse_ledger_events, but only return that result if it
produced non-empty entries. If the parsed ledger result is empty, the code
should continue to the fallback logic (parse_message_usage) instead of returning
an empty result. Check whether the result from parse_ledger_events contains
entries before deciding whether to return it or fall through to the message
usage parsing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 4acbd115-db78-4b73-9d32-3de421f182c5
📒 Files selected for processing (16)
rust/crates/ccusage/src/adapter/amp/parser.rsrust/crates/ccusage/src/adapter/codebuff/parser.rsrust/crates/ccusage/src/adapter/copilot/parser.rsrust/crates/ccusage/src/adapter/gemini/parser.rsrust/crates/ccusage/src/adapter/goose/parser.rsrust/crates/ccusage/src/adapter/jsonl.rsrust/crates/ccusage/src/adapter/kilo/loader.rsrust/crates/ccusage/src/adapter/kilo/parser.rsrust/crates/ccusage/src/adapter/kimi/parser.rsrust/crates/ccusage/src/adapter/mod.rsrust/crates/ccusage/src/adapter/openclaw/parser.rsrust/crates/ccusage/src/adapter/opencode/loader.rsrust/crates/ccusage/src/adapter/opencode/parser.rsrust/crates/ccusage/src/adapter/pi/parser.rsrust/crates/ccusage/src/adapter/qwen/parser.rsrust/crates/ccusage/src/fast.rs
ccusage
@ccusage/ccusage-darwin-arm64
@ccusage/ccusage-darwin-x64
@ccusage/ccusage-linux-arm64
@ccusage/ccusage-linux-x64
@ccusage/ccusage-win32-x64
commit: |
A malformed non-object nested field (e.g. "cache": 5) made the typed struct fail to deserialize and silently dropped the whole usage record via records().ok(). The pre-refactor Value navigation treated such a field as absent and kept the otherwise usable tokens/cost. Add a shared jsonl::lenient_object helper and apply it to the nested tokens/time/cache fields in the opencode and kilo parsers to restore that behavior. Co-authored-by: Codesmith <[email protected]>
✅ Action performedReview finished.
|
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
…sing' into refactor/unify-adapter-jsonl-parsing
The typed refactor read the gemini record discriminator with the
trimming non_empty_string helper, but the pre-refactor code compared it
with a raw Value::as_str (record.get("type").and_then(Value::as_str) ==
Some("gemini")). Trimming let padded values like " gemini " spuriously
match the "gemini" discriminator. Use the local untrimmed lenient_str
helper, which mirrors Value::as_str: strings verbatim, non-strings to
None without failing the line, so records still fall through to stats
parsing.
Co-authored-by: Codesmith <[email protected]>
…sing' into refactor/unify-adapter-jsonl-parsing
|
no API key found — this repo is configured to use To fix: add the key as a GitHub Actions secret (referenced from your workflow's Open repo secrets → · Configure model → · Setup docs → · Ask in Discord →
|
A non-object `cost` payload made `PiUsage` deserialization fail, which dropped the whole usage record via `jsonl::records().ok()`. The pre-refactor `Value` navigation treated a non-object cost as absent display cost while keeping the record's tokens. Apply `jsonl::lenient_object` to the nested `cost` field to restore that behavior. Addresses a CodeRabbit review finding on PR #1327.
The typed refactor regressed three lenient-navigation behaviors:
- amp messages were strict-deserialized as Vec<AmpMessage>, so a single
non-object element (or a non-array messages field) dropped the entire
thread. Parse them with the new jsonl::lenient_vec so bad elements are
skipped, matching the old Value::as_array navigation.
- amp ledger precedence fired on any usageLedger object (including {}),
so threads with no usable events array stopped falling back to message
usage. Track events as Option<Vec<_>> via jsonl::lenient_array and only
take the ledger branch when the events array is present, and read
usage_ledger via jsonl::lenient_object so a non-object usageLedger no
longer fails the whole thread.
- pi usage.cost was strict-typed, so a non-object cost dropped an
otherwise usable record; read it via jsonl::lenient_object.
Adds jsonl::lenient_array / jsonl::lenient_vec helpers plus regression
tests for each path.
Co-authored-by: Codesmith <[email protected]>
A non-object message field made OpenClawLine deserialization fail, so jsonl::records().ok() dropped the whole line. For a model_change or model-snapshot record that also carried a malformed message, this lost the model/provider state update that subsequent usage entries rely on. The pre-refactor Value navigation treated a non-object message as no message while keeping the line, so apply jsonl::lenient_object to restore that behavior. Co-authored-by: Codesmith <[email protected]>
…sing' into refactor/unify-adapter-jsonl-parsing
`nix flake check`'s treefmt wants the first `matches!` arm in `is_unsupported_nullable_field` split across lines like the rest. Apply the pinned formatter so the check passes; no behavior change.
…sing' into refactor/unify-adapter-jsonl-parsing
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
1 participant
Summary
Unifies how the agent adapters parse their log files and brings every adapter onto the same optimized fast path. Previously the optimization level was inconsistent: the Claude loader read whole files as bytes, prefiltered lines with
memmem, and deserialized into typed structs, while most other adapters parsed every line into a dynamicserde_json::Valueand hand-navigated it withValue::get(and several allocated aStringper line viaBufReader::lines()).Supersedes #1326 — folds that PR's
LinePrefilter(multi-marker AND/OR prefilter) into this broader typed-parsing refactor.What changed
adapter/jsonl.rs:records::<T>(content, prefilter)— iterate byte-lines (no per-lineStringallocation), skip lines rejected by a reusableLinePrefilterbefore any JSON parsing, thenfrom_slicesurvivors into a typed struct (unused fields skipped, no intermediateValuetree).lenient_u64/lenient_i64/lenient_f64/non_empty_string—deserialize_withhelpers that exactly reproduceValue::as_u64/as_i64/as_f64/non_empty_json_stringleniency, so typed structs tolerate unexpectedly encoded fields instead of failing the whole line.fast::LinePrefilter(from perf(adapter): unify JSONL prefilter and byte_lines across agents #1326): builds thememmemneedles once and supports "contains all markers" / "contains any marker" modes. Restores precise prefilters: pi (usage+message), kimi (StatusUpdate+token_usage), openclaw (any ofmodel_change/model-snapshot/usage, previously unfiltered), qwen, copilot.#[derive(Deserialize)]structs + the shared helpers: amp, codebuff, copilot, gemini, goose, kilo, kimi, openclaw, opencode, pi, qwen. (Claude and Codex already used typed parsing.)Valuewhere typing them would change numeric coercion or drop records (documented inline).Why
Valuetree per line).Behavior parity
Output is preserved exactly:
deserialize_withhelpers match the oldValueaccessor semantics.Verified
--jsonoutput fordaily/monthly/weekly/sessionis byte-for-byte identical to both the previousmainbuild andccusage@latest, for every agent with local logs (claude, codex, amp, copilot, gemini, opencode, pi). Adapters without local logs (codebuff, droid, goose, kilo, kimi, openclaw, qwen, hermes) are covered by the existing fixture-backed tests.Note: adapters now read files as bytes (
fs::read) rather thanfs::read_to_string; a file with invalid UTF-8 is now skipped gracefully per-line instead of failing the whole command — a minor, intentional robustness improvement consistent with the Claude loader.Testing
cargo test -p ccusage— 276 passedcargo clippy -p ccusage— cleancargo fmt --check— cleanmainandccusage@latest(see above)Summary by CodeRabbit
Release Notes
New Features
Refactor
Bug Fixes / Tests