Skip to content

Codex reports still read entire session JSONL files into memory #1124

Description

@pretendhigh

Summary

ccusage codex daily on current Rust code still reads each Codex session JSONL file fully into memory before line scanning.

This reproduces the large-session failure mode from #952 for the current unified ccusage codex path. In rust/crates/ccusage/src/codex_loader.rs, visit_codex_session_file used fs::read(path), so a multi-GB Codex session can exhaust memory before the marker filter or aggregation has a chance to run.

Impact

Long-lived Codex Desktop sessions can grow to GB-sized JSONL files. When that happens, npx ccusage@latest codex daily can consume enough memory that macOS starts pausing applications or the process fails before producing a report.

Proposed fix

I opened PR #1123 to stream Codex sessions with BufReader::read_until on newline bytes and reuse the same line buffer, preserving the existing token-count parsing and model tracking behavior while bounding memory to the largest line instead of the whole file.

Local validation

  • cargo fmt --manifest-path rust/Cargo.toml --all --check
  • cargo test --manifest-path rust/Cargo.toml -p ccusage codex_loader::tests::parses_codex_usage_from_streaming_reader
  • cargo test --manifest-path rust/Cargo.toml --workspace
  • cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings
  • Release binary smoke test with a ~95 MB synthetic Codex session fixture via CODEX_HOME=/private/tmp/ccusage-codex-large-fixture rust/target/release/ccusage codex daily --offline --json

Activity

  1. github-actions commented on May 22, 2026

    @github-actions
    Contributor

    This issue was auto-closed. Issues from new contributors are auto-closed by default.

    Maintainers review auto-closed issues and reopen worthwhile ones. Issues that do not meet the quality bar in CONTRIBUTING.md may not be reopened or receive a reply.

    Keep the issue short, concrete, and written in your own voice.

    If a maintainer replies lgtmi, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open.

    See CONTRIBUTING.md.

  2. blain3white commented on Jun 9, 2026

    @blain3white

    I'm the maintainer of Clean My Agent. This is close to its scope: local AI-agent storage scanning across JSONL/SQLite-style sources, with backup and export on top: https://github.com/blain3white/clean-my-agent

  3. ryoppippi commented on Aug 31, 2026

    @ryoppippi
    Member

    Historical audit: this discussion was auto-closed by the legacy contributor gate. That closure did not assess technical importance.

    Audit result: resolved. A later merged change or the current main implementation covers this request. This item is kept for history and does not need to be reopened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage:resolvedResolved by a later change or current implementation.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions