Repository navigation
[Bug] Branched conversations are fully counted #19
Description
Activity
PRs are welcome
I don't really do TS, but here's the difference from a personal Python tool that dedupes:
you can vibe code to convert your python code to typescript. i didn't get how to fix it. so please send me a PR to fix it. i believe you and your CC can do it!
I don't really vibe code, I'd like to understand the code I'd submit. The core part is just something similar to this for each assistant line you're processing:
processed_requests = set() # This should be declared above the loop, applying across all projects # For each assistant message: request_id = data.get('requestId') message_id = data.get('message', {}).get('id', '') # Create unique identifier from message_id + request_id if request_id and message_id: unique_hash = hashlib.sha256(f"{message_id}:{request_id}".encode()).hexdigest()[:16] if unique_hash not in processed_requests: processed_requests.add(unique_hash) # This is a unique message, count tokens, calculate cost, add to total etc. else: # This is a duplicate from a branch or a duplicate tool use. Skip it, log it, do whatever you want continueI created a draft PR based on that.
I still don't understand exactly how this change will improve the accuracy of costing.Yeah, I was following this conversation a bit in the discord about the duplicated data in the JSONL files. I took a similar swing at this yesterday but my combined key wasn't quite right (cc led me astray). There's definitely a cost difference once those duplicates are pulled out.
This top one is running the PR fix, bottom obviously ccusage v0.6.0 -
Didn't know if I should post this on the PR or the Issue, here's a link: #29 (comment)
I'll be here tomorrow. Thank you for your support!
Good nightI'm not really sure this is still an issue? I have branched many times and couldn't get separate JSONL files to get created with duplicates. I'm also running the PR branch with duplicate removal vs. latest v0.7.0 and the costs are matching. As shown in May there were definitely discrepancies happening, but doesn't seem to be happening now on current cc version:
Still happening for me:
You also have the duplicate request and message id for tool use regardless of branching. Here's an example in a JSONL:[{ "parentUuid": "22f6d3ac-7e0f-4fe8-adaf-56e2698e42aa", "isSidechain": false, "userType": "external", "cwd": "/mnt/g/Claude/Claude_Code_Test", "sessionId": "dd8a5a73-8f9d-499d-ae28-e8099dcee8e5", "version": "1.0.18", "message": { "id": "msg_01W7Fnz943zF65w2rdYBoi9Z", "type": "message", "role": "assistant", "model": "claude-opus-4-20250514", "content": [ { "type": "text", "text": "I see the file might have been moved. Let me check if it's in the current project directory:" } ], "stop_reason": null, "stop_sequence": null, "usage": { "input_tokens": 12, "cache_creation_input_tokens": 213, "cache_read_input_tokens": 69077, "output_tokens": 2, "service_tier": "standard" } }, "requestId": "req_011CQ2mQguC33S1JpgULARTm", "type": "assistant", "uuid": "91220ca1-07e4-40e8-af68-a226832fae16", "timestamp": "2025-06-11T16:41:34.131Z" }, { "parentUuid": "91220ca1-07e4-40e8-af68-a226832fae16", "isSidechain": false, "userType": "external", "cwd": "/mnt/g/Claude/Claude_Code_Test", "sessionId": "dd8a5a73-8f9d-499d-ae28-e8099dcee8e5", "version": "1.0.18", "message": { "id": "msg_01W7Fnz943zF65w2rdYBoi9Z", "type": "message", "role": "assistant", "model": "claude-opus-4-20250514", "content": [ { "type": "tool_use", "id": "toolu_01ASJ42wvnnFtu8VpdaqxHfk", "name": "Bash", "input": { "command": "ls -la /mnt/g/Claude/Claude_Code_Test/Values_Paper_FINAL_FINAL.pdf", "description": "Check if PDF is in current project directory" } } ], "stop_reason": null, "stop_sequence": null, "usage": { "input_tokens": 12, "cache_creation_input_tokens": 213, "cache_read_input_tokens": 69077, "output_tokens": 2, "service_tier": "standard" } }, "requestId": "req_011CQ2mQguC33S1JpgULARTm", "type": "assistant", "uuid": "2028763b-67fb-4116-bbf1-1cbe1ab616e5", "timestamp": "2025-06-11T16:41:35.591Z" }]And here the current discrepancy in cost:
The easiest way to reproduce is taking a long conversation, editing a past message with 2x Esc, exiting Claude Code and then starting that conversation with
claude -rand writing a new message. The whole JSONL still gets copied and fully counted in the current version of ccusage.Reacted by ryoppippiYeah, interestingly I just compared windows vs my mac, and I am seeing duplicates on Windows still created recently, definitely still an issue. Thanks for that detailed info @Richard-Weiss
This is not a duplication on a per-session basis, but a duplication of message, correct?
@Richard-Weiss @ben-vargas Which claude code version you guys tested. if the latest one works fine, we don't need to work on that.
Another question is whether these duplicated messages are actually the result of the Claude API. If they are not used, then they should be removed.
@Richard-Weiss @ben-vargas Which claude code version you guys tested. if the latest one works fine, we don't need to work on that.
It seems it has been happening, at least according to the duplication logic that the PR applies, quite often and on versions at least through v1.0.17.
Here's a script opus created based on the logic in the PR for analyzing duplication based on message_id and request_id concatenation.
#!/usr/bin/env node import { readdir, readFile } from 'node:fs/promises'; import { join } from 'node:path'; import { homedir } from 'node:os'; import { existsSync } from 'node:fs'; const CLAUDE_PATH = join(homedir(), '.claude', 'projects'); // ANSI color codes const colors = { reset: '\x1b[0m', red: '\x1b[31m', green: '\x1b[32m', yellow: '\x1b[33m', blue: '\x1b[34m', magenta: '\x1b[35m', cyan: '\x1b[36m', gray: '\x1b[90m', bold: '\x1b[1m', }; async function getAllJsonlFiles(dir) { const files = []; async function scanDir(currentDir) { try { const entries = await readdir(currentDir, { withFileTypes: true }); for (const entry of entries) { const fullPath = join(currentDir, entry.name); if (entry.isDirectory()) { await scanDir(fullPath); } else if (entry.isFile() && entry.name.endsWith('.jsonl')) { files.push(fullPath); } } } catch (error) { console.error(`Error scanning directory ${currentDir}:`, error.message); } } await scanDir(dir); return files; } async function parseJsonlFile(filePath) { const entries = []; try { const content = await readFile(filePath, 'utf-8'); const lines = content.split('\n').filter(line => line.trim()); for (const line of lines) { try { const data = JSON.parse(line); if (data.message?.id && data.requestId) { entries.push({ messageId: data.message.id, requestId: data.requestId, timestamp: data.timestamp, uuid: data.uuid, parentUuid: data.parentUuid, inputTokens: data.message.usage?.input_tokens || 0, outputTokens: data.message.usage?.output_tokens || 0, cacheCreationTokens: data.message.usage?.cache_creation_input_tokens || 0, cacheReadTokens: data.message.usage?.cache_read_input_tokens || 0, costUSD: data.costUSD, model: data.model, filePath, }); } } catch (e) { // Skip invalid JSON lines } } } catch (error) { console.error(`Error reading file ${filePath}:`, error.message); } return entries; } function formatBytes(bytes) { if (bytes === 0) return '0 B'; const k = 1024; const sizes = ['B', 'KB', 'MB', 'GB']; const i = Math.floor(Math.log(bytes) / Math.log(k)); return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i]; } function getProjectFromPath(filePath) { const relativePath = filePath.replace(CLAUDE_PATH + '/', ''); const parts = relativePath.split('/'); return parts[0] || 'unknown'; } function getSessionFromPath(filePath) { const fileName = filePath.split('/').pop(); return fileName?.replace('.jsonl', '') || 'unknown'; } async function analyzeDuplicates() { console.log(`${colors.bold}${colors.blue}Claude Code Duplicate Analysis${colors.reset}\n`); console.log(`Scanning directory: ${colors.cyan}${CLAUDE_PATH}${colors.reset}\n`); if (!existsSync(CLAUDE_PATH)) { console.error(`${colors.red}Error: Claude projects directory not found at ${CLAUDE_PATH}${colors.reset}`); process.exit(1); } // Get all JSONL files const files = await getAllJsonlFiles(CLAUDE_PATH); console.log(`Found ${colors.green}${files.length}${colors.reset} JSONL files\n`); // Parse all entries const allEntries = []; let totalEntries = 0; for (const file of files) { const entries = await parseJsonlFile(file); allEntries.push(...entries); totalEntries += entries.length; } console.log(`Parsed ${colors.green}${totalEntries}${colors.reset} entries with message IDs and request IDs\n`); // Group by message ID + request ID const duplicateGroups = new Map(); for (const entry of allEntries) { const key = `${entry.messageId}:${entry.requestId}`; if (!duplicateGroups.has(key)) { duplicateGroups.set(key, []); } duplicateGroups.get(key).push(entry); } // Find actual duplicates (more than 1 occurrence) const duplicates = Array.from(duplicateGroups.entries()) .filter(([_, entries]) => entries.length > 1) .sort((a, b) => b[1].length - a[1].length); // Sort by number of duplicates if (duplicates.length === 0) { console.log(`${colors.green}No duplicates found!${colors.reset}`); return; } console.log(`${colors.red}Found ${colors.bold}${duplicates.length}${colors.reset}${colors.red} unique duplicate combinations${colors.reset}\n`); // Calculate impact let totalDuplicateEntries = 0; let totalWastedTokens = { input: 0, output: 0, cacheCreation: 0, cacheRead: 0, }; let totalWastedCost = 0; // Calculate totals first for (const [_, entries] of duplicates) { totalDuplicateEntries += entries.length - 1; for (let i = 1; i < entries.length; i++) { totalWastedTokens.input += entries[i].inputTokens; totalWastedTokens.output += entries[i].outputTokens; totalWastedTokens.cacheCreation += entries[i].cacheCreationTokens; totalWastedTokens.cacheRead += entries[i].cacheReadTokens; if (entries[i].costUSD) { totalWastedCost += entries[i].costUSD; } } } // Calculate additional statistics const maxDuplicates = Math.max(...duplicates.map(([_, entries]) => entries.length)); const avgTimeSpan = []; for (const [_, entries] of duplicates) { if (entries.length > 1) { entries.sort((a, b) => new Date(a.timestamp) - new Date(b.timestamp)); const firstTime = new Date(entries[0].timestamp); const lastTime = new Date(entries[entries.length - 1].timestamp); const timeDiff = (lastTime - firstTime) / 1000; // in seconds avgTimeSpan.push(timeDiff); } } const avgSpan = avgTimeSpan.length > 0 ? avgTimeSpan.reduce((a, b) => a + b, 0) / avgTimeSpan.length : 0; // Show Key Findings console.log(`${colors.bold}${colors.blue}=== KEY FINDINGS ===${colors.reset}\n`); console.log(`${colors.bold}1. Massive Duplicate Problem:${colors.reset}`); console.log(` ${colors.gray}•${colors.reset} ${colors.red}${duplicates.length.toLocaleString()}${colors.reset} unique duplicate combinations found`); console.log(` ${colors.gray}•${colors.reset} ${colors.red}${totalDuplicateEntries.toLocaleString()}${colors.reset} duplicate entries (${colors.red}${((totalDuplicateEntries / totalEntries) * 100).toFixed(1)}%${colors.reset} of all entries!)`); console.log(` ${colors.gray}•${colors.reset} These duplicates would be removed by the deduplication logic`); const totalWasted = totalWastedTokens.input + totalWastedTokens.output + totalWastedTokens.cacheCreation + totalWastedTokens.cacheRead; console.log(`\n${colors.bold}2. Token Waste:${colors.reset}`); console.log(` ${colors.gray}•${colors.reset} ${colors.red}${totalWasted.toLocaleString()}${colors.reset} total wasted tokens due to duplicates`); console.log(` ${colors.gray}•${colors.reset} Most waste is in cache read tokens (${colors.yellow}${(totalWastedTokens.cacheRead / 1000000).toFixed(0)}M${colors.reset}) and cache creation tokens (${colors.yellow}${(totalWastedTokens.cacheCreation / 1000000).toFixed(0)}M${colors.reset})`); if (totalWastedCost > 0) { console.log(` ${colors.gray}•${colors.reset} Estimated wasted cost: ${colors.red}$${totalWastedCost.toFixed(2)}${colors.reset}`); } // Project breakdown const projectDuplicates = new Map(); for (const [_, entries] of duplicates) { for (const entry of entries) { const project = getProjectFromPath(entry.filePath); projectDuplicates.set(project, (projectDuplicates.get(project) || 0) + 1); } } const sortedProjects = Array.from(projectDuplicates.entries()) .sort((a, b) => b[1] - a[1]); console.log(`\n${colors.bold}3. Most Affected Projects:${colors.reset}`); const topProjects = sortedProjects.slice(0, 4); for (const [project, count] of topProjects) { const projectName = project.replace(/-Users-ben-Documents-git-(?:ben-vargas-|unsynced-)?/, ''); const isThisProject = project.includes('ccusage'); console.log(` ${colors.gray}•${colors.reset} ${colors.magenta}${projectName}${colors.reset}: ${colors.yellow}${count.toLocaleString()}${colors.reset} duplicate entries${isThisProject ? ' (this project!)' : ''}`); } console.log(`\n${colors.bold}4. Duplicate Patterns:${colors.reset}`); console.log(` ${colors.gray}•${colors.reset} Duplicates typically occur within seconds (${colors.yellow}${avgSpan.toFixed(0)}-${(avgSpan * 2).toFixed(0)}${colors.reset} second spans)`); console.log(` ${colors.gray}•${colors.reset} Same session files contain multiple duplicates`); console.log(` ${colors.gray}•${colors.reset} Some entries are duplicated up to ${colors.red}${maxDuplicates}${colors.reset} times!`); console.log(`\n${colors.gray}${'─'.repeat(60)}${colors.reset}`); console.log(`\n${colors.bold}${colors.blue}Detailed Statistics:${colors.reset}`); console.log(` ${colors.gray}Total duplicate groups:${colors.reset} ${duplicates.length}`); console.log(` ${colors.gray}Total duplicate entries:${colors.reset} ${totalDuplicateEntries} (these would be removed)`); console.log(` ${colors.gray}Total entries analyzed:${colors.reset} ${totalEntries}`); console.log(` ${colors.gray}Duplicate rate:${colors.reset} ${((totalDuplicateEntries / totalEntries) * 100).toFixed(2)}%`); console.log(`\n${colors.bold}${colors.blue}All Projects with Duplicates:${colors.reset}`); const projectsToShow = Math.min(10, sortedProjects.length); for (let i = 0; i < projectsToShow; i++) { const [project, count] = sortedProjects[i]; console.log(` ${colors.magenta}${project}${colors.reset}: ${count} duplicate entries`); } if (sortedProjects.length > 10) { console.log(` ${colors.gray}... and ${sortedProjects.length - 10} more projects${colors.reset}`); } // Show a few examples console.log(`\n${colors.bold}${colors.yellow}First 10 Duplicate Examples:${colors.reset}\n`); // Display first 10 duplicates const exampleDuplicates = duplicates.slice(0, 10); for (const [key, entries] of exampleDuplicates) { const [messageId, requestId] = key.split(':'); const occurrences = entries.length; // Sort entries by timestamp entries.sort((a, b) => new Date(a.timestamp) - new Date(b.timestamp)); console.log(`${colors.bold}${colors.yellow}Duplicate Found:${colors.reset}`); console.log(` ${colors.gray}Message ID:${colors.reset} ${messageId}`); console.log(` ${colors.gray}Request ID:${colors.reset} ${requestId}`); console.log(` ${colors.gray}Occurrences:${colors.reset} ${colors.red}${occurrences}${colors.reset}`); console.log(` ${colors.gray}Model:${colors.reset} ${entries[0].model || 'unknown'}`); console.log(` ${colors.gray}Tokens:${colors.reset} input=${entries[0].inputTokens}, output=${entries[0].outputTokens}, cache_creation=${entries[0].cacheCreationTokens}, cache_read=${entries[0].cacheReadTokens}`); console.log(`\n ${colors.gray}Timestamps and locations:${colors.reset}`); for (const entry of entries) { const project = getProjectFromPath(entry.filePath); const session = getSessionFromPath(entry.filePath); const timestamp = new Date(entry.timestamp).toLocaleString(); console.log(` ${colors.cyan}${timestamp}${colors.reset} - Project: ${colors.magenta}${project}${colors.reset}, Session: ${colors.blue}${session}${colors.reset}`); } // Show time span if (entries.length > 1) { const firstTime = new Date(entries[0].timestamp); const lastTime = new Date(entries[entries.length - 1].timestamp); const timeDiff = (lastTime - firstTime) / 1000; // in seconds console.log(` ${colors.gray}Time span:${colors.reset} ${timeDiff.toFixed(1)} seconds`); } console.log(''); } } // Run the analysis analyzeDuplicates().catch(console.error);
ben@ben-mbp ccusage % node analyze-duplicates.js Claude Code Duplicate Analysis Scanning directory: /Users/ben/.claude/projects Found 1395 JSONL files Parsed 15396 entries with message IDs and request IDs Found 5299 unique duplicate combinations === KEY FINDINGS === 1. Massive Duplicate Problem: • 5,299 unique duplicate combinations found • 5,804 duplicate entries (37.7% of all entries!) • These duplicates would be removed by the deduplication logic 2. Token Waste: • 391,054,996 total wasted tokens due to duplicates • Most waste is in cache read tokens (363M) and cache creation tokens (27M) 3. Most Affected Projects: • xx-xxx-xxxxxxxx-xxxxxx-xxxx: 3,627 duplicate entries • xx-xxxxxx-xxxx-xxxxxx: 2,109 duplicate entries • xxxxxxx: 1,232 duplicate entries • ccusage: 1,213 duplicate entries (this project!) 4. Duplicate Patterns: • Duplicates typically occur within seconds (6-11 second spans) • Same session files contain multiple duplicates • Some entries are duplicated up to 9 times! ──────────────────────────────────────────────────────────── Detailed Statistics: Total duplicate groups: 5299 Total duplicate entries: 5804 (these would be removed) Total entries analyzed: 15396 Duplicate rate: 37.70% All Projects with Duplicates: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx: 3627 duplicate entries -Users-ben-Documents-git-xxx-xxxxxx-xx-xxxxxx-xxxx-xxxxxx: 2109 duplicate entries -Users-ben-Documents-git-xxxxxxxx-xxxxxxx: 1232 duplicate entries -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxx: 1213 duplicate entries -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxx: 1198 duplicate entries -Users-ben-Documents-git-xxxxxxxx-xxxxxxx: 1009 duplicate entries -Users-ben-Documents-git-xxxxxxxxx-xx-xxxxxxx-xxxx-xxxxxxx-xxxxxxxxxxx: 325 duplicate entries -Users-ben-Documents-git-xxx-xxxxxx-xxxx-xxxxxx-xxxx-xxxxxx: 173 duplicate entries -Users-ben-Documents-git-xxx-xxxxxx-xx-xxxxxx-xxxx-xx: 91 duplicate entries -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxxxxx: 44 duplicate entries ... and 7 more projects First 10 Duplicate Examples: Duplicate Found: Message ID: msg_01K5QNZyadfTHf5uMgaoanTR Request ID: req_011CPxz9DE8LPoX5CC2ddrZV Occurrences: 9 Model: unknown Tokens: input=5, output=756, cache_creation=289, cache_read=13131 Timestamps and locations: 6/9/2025, 10:46:08 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:09 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:10 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:11 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:12 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:13 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:14 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:15 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:15 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 Time span: 7.7 seconds Duplicate Found: Message ID: msg_012sTcMuGMmCb7yPxB4nxX1R Request ID: req_011CPxzC7cd3m8mpAZuKFFty Occurrences: 9 Model: unknown Tokens: input=5, output=574, cache_creation=161, cache_read=16453 Timestamps and locations: 6/9/2025, 10:46:47 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:48 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:49 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:50 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:51 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:53 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:53 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:53 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 6/9/2025, 10:46:54 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 567f22e7-3262-4775-9eec-de7808d5e629 Time span: 6.7 seconds Duplicate Found: Message ID: msg_01AmsUd7GebGYU7aJyr71zit Request ID: req_011CPzyLsL2ZcUekp4NSHqUr Occurrences: 7 Model: unknown Tokens: input=3, output=393, cache_creation=11627, cache_read=0 Timestamps and locations: 6/10/2025, 11:56:51 AM - Project: -Users-ben-Documents-git-xxxxxxxx-xxxxxxx, Session: 3d8cb62a-44df-42fa-96be-99d0935bc8f0 6/10/2025, 11:56:54 AM - Project: -Users-ben-Documents-git-xxxxxxxx-xxxxxxx, Session: 3d8cb62a-44df-42fa-96be-99d0935bc8f0 6/10/2025, 11:56:54 AM - Project: -Users-ben-Documents-git-xxxxxxxx-xxxxxxx, Session: 3d8cb62a-44df-42fa-96be-99d0935bc8f0 6/10/2025, 11:56:55 AM - Project: -Users-ben-Documents-git-xxxxxxxx-xxxxxxx, Session: 3d8cb62a-44df-42fa-96be-99d0935bc8f0 6/10/2025, 11:56:56 AM - Project: -Users-ben-Documents-git-xxxxxxxx-xxxxxxx, Session: 3d8cb62a-44df-42fa-96be-99d0935bc8f0 6/10/2025, 11:56:56 AM - Project: -Users-ben-Documents-git-xxxxxxxx-xxxxxxx, Session: 3d8cb62a-44df-42fa-96be-99d0935bc8f0 6/10/2025, 11:56:56 AM - Project: -Users-ben-Documents-git-xxxxxxxx-xxxxxxx, Session: 3d8cb62a-44df-42fa-96be-99d0935bc8f0 Time span: 4.7 seconds Duplicate Found: Message ID: msg_01H3o7RQ8dLcQxmDXcdX6Nso Request ID: req_011CPwNH8okJiN531eCPodui Occurrences: 6 Model: unknown Tokens: input=4, output=385, cache_creation=12644, cache_read=0 Timestamps and locations: 6/8/2025, 2:15:15 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 7c0d6f3c-0cf9-496f-933a-4df6c78f8a6f 6/8/2025, 2:15:15 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 7c0d6f3c-0cf9-496f-933a-4df6c78f8a6f 6/8/2025, 2:15:17 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 7c0d6f3c-0cf9-496f-933a-4df6c78f8a6f 6/8/2025, 2:15:18 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 7c0d6f3c-0cf9-496f-933a-4df6c78f8a6f 6/8/2025, 2:15:18 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 7c0d6f3c-0cf9-496f-933a-4df6c78f8a6f 6/8/2025, 2:15:19 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 7c0d6f3c-0cf9-496f-933a-4df6c78f8a6f Time span: 4.2 seconds Duplicate Found: Message ID: msg_01VqeSE8X7EGMBziANgYfj84 Request ID: req_011CPwZUNtQyZ6hDTvAtC6FT Occurrences: 6 Model: unknown Tokens: input=0, output=1, cache_creation=653, cache_read=36640 Timestamps and locations: 6/8/2025, 4:42:00 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 4:42:00 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 4:42:01 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 4:42:01 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 4:42:02 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 4:42:03 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 Time span: 3.0 seconds Duplicate Found: Message ID: msg_01WcWFdjkd2rr1vnctKC4BhQ Request ID: req_011CPwfy3UrxRkybsRKVq9sz Occurrences: 6 Model: unknown Tokens: input=4, output=1, cache_creation=11346, cache_read=144619 Timestamps and locations: 6/8/2025, 6:07:12 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 6:07:14 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 6:07:14 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 6:07:15 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 6:07:20 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 6/8/2025, 6:07:21 PM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: 8b1256d5-1066-4abc-bb9b-5b357ed74060 Time span: 8.4 seconds Duplicate Found: Message ID: msg_01XRDWo8wG6MoToyJoTM4sdi Request ID: req_011CQ1zzwiNv2yYn9sw9SXF4 Occurrences: 6 Model: unknown Tokens: input=4, output=1, cache_creation=17271, cache_read=0 Timestamps and locations: 6/11/2025, 12:59:09 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:10 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:11 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:12 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:13 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:13 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f Time span: 3.5 seconds Duplicate Found: Message ID: msg_015a1MMumpievdFmJfTcufdN Request ID: req_011CQ211Z6dBPkdHsAjFrUvg Occurrences: 6 Model: unknown Tokens: input=8, output=1, cache_creation=13152, cache_read=17271 Timestamps and locations: 6/11/2025, 12:59:17 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:18 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:19 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:20 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:20 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:21 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f Time span: 3.9 seconds Duplicate Found: Message ID: msg_01WPCYFfTt7jDaDF6Cbs6Diw Request ID: req_011CQ214VHWNgjA8Zn15sRUH Occurrences: 6 Model: unknown Tokens: input=7, output=356, cache_creation=626, cache_read=13374 Timestamps and locations: 6/11/2025, 12:59:56 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:57 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:58 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:59 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 12:59:59 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f 6/11/2025, 1:00:00 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xx-xxx-xxxxxxxx-xxxxxx-xxxx, Session: c39abe3a-6004-4af0-a08f-421786bf183f Time span: 3.9 seconds Duplicate Found: Message ID: msg_01Awpgmf3qa8ixUVUEceGqKj Request ID: req_011CPrTZRnxY81tysUC9qbrj Occurrences: 6 Model: unknown Tokens: input=6, output=1, cache_creation=2311, cache_read=17445 Timestamps and locations: 6/6/2025, 12:01:32 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxx, Session: efde47cb-a78f-4b45-a240-f8863db1d6ec 6/6/2025, 12:01:33 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxx, Session: efde47cb-a78f-4b45-a240-f8863db1d6ec 6/6/2025, 12:01:35 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxx, Session: efde47cb-a78f-4b45-a240-f8863db1d6ec 6/6/2025, 12:01:36 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxx, Session: efde47cb-a78f-4b45-a240-f8863db1d6ec 6/6/2025, 12:01:36 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxx, Session: efde47cb-a78f-4b45-a240-f8863db1d6ec 6/6/2025, 12:01:36 AM - Project: -Users-ben-Documents-git-xxx-xxxxxx-xxxxxxxx-xxxxxxx, Session: efde47cb-a78f-4b45-a240-f8863db1d6ec Time span: 4.2 seconds
yeah got it thank you!!
I tested in my local
https://x.com/ryoppippi/status/1933100461239795833Interestingly, I thought the number of token would only decrease, but on some days it has increased, which is interesting
- added a commit that references this issue
on Jun 22, 2025
When you branch a conversation in CC, when pressing Esc twice for example, a new JSONL gets created that is very similar to the initial one.
However, obviously it doesn't have the same cost associated with it (previous output tokens are becoming input tokens for example). The combination of message id + request id of assistant message should only be counted once globally, as it heavily skews the numbers otherwise.
You can test that behavior by branching a very long, output intensive conversation multiple times and calculating the price, it's way more than it should be. Only the delta (new assistant message that contains the previous output as input) in the new conversations should be included.