Repository navigation
Parsing bug with SRUM #234
Description
Posting here since issue creation appears to be restricted on this repo. Wanted to flag a reproducible SRUM parsing bug — happy to move this elsewhere if there's a better place for it.
Chainsaw version: 2.16.2 (also reproduced on 2.16.0)
OS: Windows 11, version 25H2
Command:
chainsaw.exe analyse srum --software ./SOFTWARE ./SRUDB.dat --output ./output.json
Error:
[!] Error parsing SRUM database: unable to load the ESE database
Caused by:
c-libesedb: libesedb_file_open: unable to open file: SRUDB.dat.
libesedb_page_read_tags: invalid number of page tags value out of bounds.
libesedb_page_read_values: unable to read page tags.
libesedb_page_read_file_io_handle: unable to read page values.
libesedb_io_handle_read_page: unable to read page.
libfdata_vector_get_element_value_by_index: unable to read element data at offset: 61440 (0x0000f000).
libesedb_catalog_read_file_io_handle: unable to retrieve page: 14.
libesedb_file_open_read: unable to read catalog.
libesedb_file_open_file_io_handle: unable to read from file handle.
libesedb_file_open: unable to open file: SRUDB.dat.
What I've verified:
The SRUDB.dat file itself is confirmed structurally valid: esentutl.exe's own Integrity check mode (/g) reports "Integrity check successful" against the exact same file that Chainsaw fails to open.
The identical error (same page number, same byte offset) occurred across three independent collection methods: a plain esentutl /vss snapshot, esentutl /vssrec (VSS + transaction log replay), and a full copy of the complete SRU* file set (database, checkpoint, all transaction logs) followed by dedicated esentutl /r recovery mode against the collected copy. All three produced the same failure.
SOFTWARE hive was collected via reg save HKLM\SOFTWARE and passed correctly via --software.
Given the file is independently verified valid by Microsoft's own tooling, and the failure is identical regardless of collection method, this looks like it may be a compatibility issue between Chainsaw's SRUM/ESE parser and this specific Windows version's database format, rather than a collection artifact. Happy to provide further diagnostics if useful.
Activity
alexkornitzer commentedon Jul 21, 2026
@ATW136 issue creation should not be restricted but I am no longer admin so if it is there is not much I can do about that.
alexkornitzer commentedon Jul 21, 2026
I have managed to replicate the above but the library that Chainsaw uses does not seem to have been updated in around a year and is linked to a libesedb build from the end of 2023 so that is most likely the cause. I'll look into getting a fresher build of libesedb into Chainsaw on the weekend.
ATW136 commentedon Jul 22, 2026
alexkornitzer commentedon Jul 26, 2026
@ATW136, I have tested locally and everything should now be working with v2.16.3-beta.0. I did have to do quite a big rework on the upstream library sys crate and I don't run Windows, so if you would be able to check the
.exerelease that would be greatly appreciated.al-williams-dev commentedon Jul 26, 2026
Alex,
Thank you for the quick turnaround on this. I tested the beta build (v2.16.3-beta.0) broadly against everything our toolkit actually uses
Chainsaw for, not just the SRUM fix specifically, given you mentioned not being able to test on Windows yourself.
SRUM (the actual fix): Successful. Ran against the exact same collected SRUDB.dat that failed every previous attempt with the "invalid number of page tags" error. It parsed cleanly this time, producing complete, correct data across all ten real SRUM extension tables. I cross-validated the record count for every single table against an independent tool (Eric Zimmerman's SrumECmd, which had separately succeeded against this same file) -- every category matched exactly: App Timeline Provider (55,632), Application Resource Usage Provider (283,810), vfuprov (4,707), Network Connectivity/Data Usage (7,280/72,629), WPN SRUM Provider (19), and Energy Usage (3,259 combined across standard + long-term tables). One honest note: it's slow -- roughly 20 minutes of CPU time on a 148MB database on a modern machine with plenty of free RAM throughout. Not a correctness issue, but a real difference from before.
hunt and analyse gaps: Also tested, both against a real case with a known-good baseline from the previous stable release. Initially saw a
discrepancy (556 vs. 537 detections), but traced it to ruleset drift between when the baseline was recorded and today -- once I re-ran the
stable release against today's identical ruleset for a fair comparison, the beta and stable versions produced byte-for-byte identical results. No regression in either.
Happy to test further if anything specific would help, or if a non-beta release needs the same kind of check.
Al Williams
al-williams-dev commentedon Jul 26, 2026
Alex,
I should add that the test environment was Windows 11 Pro (10.0.26200), 64-bit chainsaw.exe, and 64GB RAM (33GB+ free through out testing).
alexkornitzer commentedon Jul 27, 2026
@al-williams-dev,
Thanks for testing that, does it run as slowly on Linux? Just want to workout if its a configure issue in libesedb (such as multithreading not getting enabled) or if its an issue in Chainsaw. The performance for me on Mac using a hive that is 81MB, and SRUDB that is 4MB was fast, but that dataset is from a VM I don't use all that much so the dataset is probably quite small. I have run chainsaw in that VM and its runs on that data relatively quickly too.
ATW136 commentedon Jul 27, 2026
al-williams-dev commentedon Jul 30, 2026
Alex,
Following up with Linux testing, in addition to the Windows results
I sent earlier -- and a correction to one of those earlier Windows
numbers.
=== RECAP: WINDOWS (previously reported, one correction below) ===
SRUM fix confirmed working, cross-validated against SrumECmd
table-by-table (all ten tables matched exactly). hunt and analyse gaps
confirmed identical to stable once controlled for ruleset drift -- no
regressions.
Environment: Windows 11 Pro (10.0.26200), Intel Core i7-10700K
(8 cores/16 threads), 64-bit chainsaw.exe, 64GB RAM (33GB+ free
throughout testing).
Correction to the earlier "~20 minutes" figure: that was an informal
estimate from periodic Get-Process checks, not a real measurement.
Re-measured properly with Measure-Command: the actual time is ~90
minutes (90m15s under default threading, 91m58s with --num-threads 16
explicitly set -- confirms the default already uses all 16 available
threads correctly, ruling out threading configuration as the cause).
=== NEW: LINUX ===
Environment 1: Xubuntu 26.04 LTS, XFCE desktop, Intel Core i5-480M
(2 cores/4 threads, ~2010-era laptop CPU), 8GB RAM.
hunt and analyse gaps: Identical results to both Windows runs -- 3686
rules loaded (1011 not loaded), 556 detections on 479 documents for
hunt; empty result set for gaps. No platform-specific difference in
either, on old/weak hardware or otherwise.
SRUM -- found one real bug and one real resource limitation, separately:
Relative paths for --software fail silently with a misleading
error. "--software SOFTWARE" produced "unable to load the SOFTWARE
hive / No such file or directory", even though the file was confirmed
present with the correct name. Traced with strace -f -e
trace=open,openat,openat2,stat,newfstatat,access -- the trace showed no
attempt to open anything referencing "SOFTWARE" at all before the
error, meaning it fails before ever reaching a real file lookup. An
absolute path ("--software /home/al/SOFTWARE") resolved it completely,
confirmed by "SOFTWARE hive loaded from..." and "Parsing the SOFTWARE
registry hive..." appearing for the first time.
Once past that, the process was eventually killed by the Linux
kernel's OOM killer on this 8GB machine. Timing from the run that got
furthest: real 54m37s, user 0m58s, sys 1m3s -- under 2 minutes of
actual CPU work inside 54+ minutes of wall time, consistent with severe
swap thrashing. An earlier attempt briefly made the whole system
unresponsive (unresponsive keyboard, continuous disk activity,
unreachable over the network) before recovering on its own -- almost
certainly the same underlying behavior, just reaching the actual OOM
kill this time instead of being interrupted first.
Environment 2 (follow-up): WSL2 (Ubuntu 24.04.2 LTS) on the SAME
physical machine as the Windows test above (Intel Core i7-10700K,
8 cores/16 threads) -- 24GB allocated, well above what failed on the
Xubuntu laptop. This isolates the hardware variable: same CPU, same
machine, only the OS/runtime differing.
Result: completed cleanly in 1m46s real time (user 26.9s + sys 78.9s
almost exactly matching real time -- essentially no swap-wait
overhead). Verified the output given how fast it completed: 411MB,
640,254 total records, every one of the ten real SRUM table counts
matching the Windows run exactly -- genuine three-way agreement now
across SrumECmd, Windows Chainsaw, and Linux Chainsaw, all on the
identical file.
Against the corrected ~90 minute Windows figure, that's roughly a 50x
gap on identical hardware -- confirms this is a genuine platform/build
difference, not a resource-constraint story like the 8GB laptop had.
=== WHERE THE TIME IS ACTUALLY GOING (Windows) ===
Captured a full Process Monitor trace of the Windows run, filtered to
chainsaw.exe: over 130 million total operations for a task producing a
~400MB output file. The large majority were WriteFile calls against
the JSON output specifically -- confirmed both by sampled events
(consistent 1-2 byte writes, sequential incrementing offsets) and by
direct real-time observation during the capture. This looks like the
output is being written essentially unbuffered, one tiny syscall at a
time, rather than accumulated and flushed in efficient large blocks.
Each WriteFile call carries its own fixed kernel-transition overhead
regardless of size, so tens of millions of them writing one file is a
plausible, sufficient explanation for the ~90 minute runtime on its
own -- independent of threading, memory, or hardware, and consistent
with why the same operation is dramatically faster under Linux/WSL2 on
identical hardware.
=== SUMMARY ===
cross-validation.
paths work.
unbuffered, per-byte-or-token output writing on the Windows build
specifically, worth a look at the output-writing code path.
Happy to test further if anything specific would help.
Al
alexkornitzer commentedon Jul 30, 2026
Hi @ATW136,
Thanks for taking the time to do that it has been very helpful, my guess from the above is that the slowdown is due to the aggressive flushing that Chainsaw does, I have tweaked that which I am hoping will solve the issue. I have also fixed the pathing issue too.
Would you be able to try out 2.16.3-beta.1 and see if that improves anything?
al-williams-dev commentedon Aug 1, 2026
Alex,
Tested beta.1
Relative-path fix: WORKS. Re-ran the exact command that failed on
beta.0 ("--software SOFTWARE SRUDB-repair-test.dat", relative paths)
on WSL2 (Ubuntu 24.04.2). Where beta.0 failed immediately with
"unable to load the SOFTWARE hive / No such file or directory",
beta.1 resolved both paths correctly and completed cleanly --
"SOFTWARE hive loaded from..." appearing for the first time with a
relative path. Verified the output too: 640,254 records, matching
every previous successful run exactly.
Flushing fix: doesn't appear to have changed performance. Ran the
same SRUDB-repair-test.dat SRUM parse on Windows four times on beta.1
(same 64GB, i7-10700K machine as before): 2h05m24s (one clear
outlier), then 93m21s and 95m14s on two clean repeats. That's
essentially the same range as beta.0's own two runs (90m15s, 91m58s)
-- no meaningful improvement.
Also captured Process Monitor again on one of the beta.1 runs, filtered
to the process: 136,256,771 total operations, versus 133,982,734 for
beta.0 -- same order of magnitude, not a meaningful reduction. Given
parsing itself has consistently been fast across every version tested
(2.16.2, beta.0, beta.1), and the "Saving output" phase accounts for
effectively the entire runtime in all of them, this looks like it's
still the same underlying bottleneck as before your fix, not something
the flushing change addressed. Wasn't able to get a clean look at the
individual write sizes this time (some tooling issues on my end,
unrelated to chainsaw itself) to confirm whether the specific tiny-
write pattern from beta.0 is still present, but the count and timing
both point the same direction.
Al
alexkornitzer commentedon Aug 2, 2026
Right so upon further testing it appears that this issue only manifests for the
--outputoption and does not impact file redirection. With that being the case the issue was the lack of buffered writing, I have profiled this on Windows and this greatly reduces the file write time. So I am hoping v2.16.3-beta.2 will bring the runtime down for you, if that does not work then its back to the drawing board!al-williams-dev commentedon Aug 4, 2026
Alex,
beta.2 fixed it completely -- great diagnosis.
SRUM timing (same file, same machine as every previous test):
1m25s and 1m15s across two runs. Down from 90-95 minutes on beta.0/
beta.1, and now in the same range as the WSL2 result (1m46s) rather
than dramatically slower than it. Verified both runs produced complete,
correct output -- 640,254 records, 411,315,820 bytes, identical to
every previous successful run.
Nice catch narrowing it to --output specifically vs. redirection --
that lines up exactly with what we'd seen: hunt (always redirected)
was never slow, analyse srum (always --output) was the consistent
bottleneck across every version tested.
Thanks for sticking with this one.
Al
alexkornitzer commentedon Aug 4, 2026
Awesome, thanks for the initial triage and additional testing. I'll get that released later today.
!15848 merge bump-chainsaw-2.16.3 into main