Repository navigation
Use IndexedDB as cache storage on Chromium platform #328
Description
Activity
Could be.
However I found that with IndexedDB (seems to be at
IndexedDB/chrome-extension_[id]_0.indexeddb.leveldbon my side) I could definitely see less issues with garbage collection after loading filter lists or selfie.With a valid selfie, when using
chrome.storage.local, the baseline memory seems stuck at a higher level than expected. I found that with IndexedDB, the baseline memory usage at launch quickly snaps well under 40 MB on my side, something I am not able to get withchrome.storage.local. With a valid selfie (default lists), uBO's baseline memory usage at launch is now lower than it ever has been I am pretty sure.I can only speculate but it seems the browser will keep entries from
chrome.storage.localin memory more aggressively -- even forcing garbage collection in dev tools does not help, while this does not happen with IndexedDB. There is no point to keep cache-related entries in memory once they are loaded.Reacted by uBlock-userMy storage usage on Chromium on
1.17.3rc5was 27 MB and now with the upgrade to1.17.5b0it went down to 10.3 MB, WOW! Is this because of IndexedDB being used now ?Yes, the one listed in settings.html ?
what do you mean ?
@gorhill The storage value seems to be jumping back and forth on my end. Once I see 10 MB in use and then after few minutes/hours, it's 17 MB in use, is this to be expected ?
Probably uBO created a selfie, used for fast loading next time uBO is launched. The selfie is destroyed whenever a filter list is updated, and it will re-create itself after ~17 minutes following the reload of all filter lists.
next time uBO is launched.
So for future use ? Thats odd as I don't close Chromium for hours together(not disabling uBO either) and this happens randomly.
Stable uses more initially, releases excess after restart (very lazy, can take hours).
and what about dev builds as I'm on that ?
I did, I see lots of files inside the first folder, the last two have files name LOCK and other files which are created by chromium I suppose.
I've run a lot of benchmarks and measurements. Here is the summary: IndexedDB + LZ4 is beneficial to uBO in Chromium, even if whatever is stored on disk is independently compressed by Chromium itself. So this will stay.
If ever there is a demonstration that the overhead of LZ4 compression uBO-side is questionable, I can always trivially add code to turn off compression while keeping IndexedDB.
I used the advanced setting
cacheStorageCompressionto turn off compression for my measurements. When toggling this setting, it is important to force an update of all filter lists.Two screenshots of performance benchmark using dev tools to measure launch timings.
No selfie, IndexedDB + compression:
No selfie, IndexedDB + no compression:
Now it seems the "IndexedDB + compression" did better, but we shouldn't draw such conclusion given the small difference. What is important is to be able to conclude that it does not appear using compression adds undue overhead.
Now, memory baseline at launch time (i.e. memory usage after garbage collection, I waited 5 minutes before screenshots).
uBO 1.17.4: selfie + 2nd-gen hntrie + chrome.storage.local:
uBO dev: selfie + 3rd-gen hntrie + chrome.storage.local + no compression:
uBO dev: selfie + 3rd-gen hntrie + IndexedDB + compression:
I tried many different combos and I conclude that what actually reduces baseline memory usage is uBO-side compression (hopefully I didn't lose myself in all the measurements I ran). If I disable compression but still use IndexedDB, I get the 50 MB baseline. Forcing garbage collection in dev tools accomplishes nothing. I don't know the explanation of this but this is what I have observed.
Thus, given that there is seemingly no undue overhead incurred as a result of using compression, and given that the baseline memory usage is much better when using compression, and given that actually reporting a more accurate on-disk bytes used to users (assuming Chromium does compress itself) is desirable, I will keep compression enabled.
Reacted by Irbidan, Evangelos Tsintzas and lukeu
Currently,
IndexedDBis used only on Firefox platform to implement cache storage. Chromium uses extension API storage (chrome.storage.local) as a cache storage, i.e. cache items are sharing the same physical storage as user setting ones.A benefit of using
IndexedDBis that the items stored in it are LZ4-compressable (which is enabled by default for that storage). LZ4 compression is not trivially possible with extension API storage, as this requires the storage to support the saving ofBlobresources. (An approach would be to create a special encoding/decoding step, but that is just not practical.)There is no technical obstacle to also use
IndexedDBon Chromium. The main obstacle of doing so is that there is no code to automatically transfer cached items from one storage to the other, which would cause stale cache items to linger in the old storage should there be a switch to a new one for cache purpose.Provide code to move cache items stored in
chrome.storage.localtoIndexedDBwhen uBO launches and detect thatIndexedDBis being used as cache storage.