Repository navigation
fix(jsonl): stop the reverse reader looping on a chunk's leading newline - #675
Open
eric-engberg wants to merge 2 commits into
Open
eric-engberg wants to merge 2 commits into
eric-engberg wants to merge 2 commits into
Conversation
iterateJsonlLinesReverseSync walks each chunk backwards with chunk.lastIndexOf(0x0a, end - 1). When the newline it just consumed sat at index 0, end became 0 and the next search ran with an offset of -1, which Buffer.lastIndexOf counts from the end of the buffer. The reader jumped back to the chunk's last newline and yielded the same lines again, forever. That happens whenever a transcript starts with an empty line, or when the byte at size - k * 1 MiB is a newline. Thinking Effort falls back to this reader when the payload's effort.level is present but not a level it accepts (for example ""), so a matching transcript kept the status line spinning at full CPU and never printing. Stop searching once end reaches the start of the chunk.
The reverse reader searched for the previous newline in two places, and only the one inside the loop needed the guard against an offset of -1. lastNewlineBefore() holds the guard and the reason for it, so both searches behave the same and the loop body stays as simple as it was before the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The status line no longer hangs when Thinking Effort falls back to reading the transcript and the transcript has a newline at the start of one of the reader's 1 MiB chunks (for example, a transcript that starts with an empty line). Before, the process spun at full CPU and never printed anything; now it renders as usual.
Why
iterateJsonlLinesReverseSyncreads the file backwards in 1 MiB chunks and walks each chunk withchunk.lastIndexOf(0x0a, end - 1). When the newline it just consumed was at index 0,endbecame 0 and the next search ran with an offset of-1.Buffer.lastIndexOfcounts a negative offset from the end of the buffer, so the reader jumped back to the chunk's last newline and yielded the same lines again, forever.That happens when a transcript starts with an empty line, or when the byte at
size - k × 1 MiBis\n. Thinking Effort uses this reader whenever the payload has aneffort.levelit doesn't accept, such as an empty string:{ "transcript_path": "…/session.jsonl", "model": { "id": "claude-opus-4-6" }, "effort": { "level": "" } }With a 1.1 MB transcript whose last chunk starts with a newline,
mainwas still running after 5 s under both Bun and Node and had to be killed, with no output. The same happened with a three-line transcript starting with\n. With an/effortrecord further back in the file, the reader never got past the last chunk to find it. Session Name's transcript lookup goes through the same reader.How
Once
endreaches the start of the chunk there is nothing left to search, so the reader stops looking for newlines in that chunk instead of passing a negative offset. Everything else stays the same: lines that span chunks, CRLF, the BOM on the first record, and empty lines are handled as before. The forward readers never had this problem: they useindexOf, which only moves forward.Demo
The same payload piped through
mainand this branch, timed and capped at 4 s:effort.levelis""and the 1.1 MB transcript is sample data, generated so that its last 1 MiB chunk starts with a newline and its/effort highrecord is older than that chunk.mainuses all its CPU until it's killed; this branch prints the line in about 0.15 s.Powerline: before
Powerline: after
Plain: before
Plain: after
Testing
jsonl-lines.test.ts: a file that starts with an empty line, and a file just over 1 MiB whose last chunk starts with a newline. Each collects at most 10 lines, so the old behavior fails with repeated lines instead of hanging the suite. Both failed onmain(the same lines repeated ten times) and pass here.bun test: 2784 pass, 0 fail.bun run lintpasses.readFilespy test, which fails the same way onmainbecause Node doesn't allow spying on built-ins.mainis still running at 5 s under both and gets killed; this branch rendersModel: Opus 4.6 | Thinking: high(orThinking: defaultfor the short one) in 0.1–0.25 s.runtime-check.sh -c thinking-effortwith a transcript that doesn't loop: Bun and Node agree in plain and Powerline modes, the output is identical tomain, and the TUI opens and exits cleanly under both.