Repository navigation
Use a DB for the metadata #31
Description
Activity
It seems that SQLite is slow as well, and it also does not play well with having a dependency and C-code free pak in the future, so I gave up on this. I think the best solution will be to write our own serialization, similarly to the qs package.
RSQLite is not slow, after all: r-dbi/RSQLite#306 :)
OK, this is not so great, because RSQLite depends on Rcpp and C++ is hard to compile statically, so it is hard to create a static installer. The same applies to the qs package.
However, the thor package with embedded lmdb seems nice. It is a different approach, because it is just a key-value store, but it is transactional, and that's exactly what we need, even simpler than switching to a SQL DB. We need to serialize the metadata to a raw vector to store it in the DB.
Also, we don't have to merge the different package sources in the cache, because thor can access all of them within a transaction, and merging them at loading time it quite fast.
thor/lmdb does not have a timeout for locking, and it is also not interruptible, so if a process keeps a write-lock, then other processes just freeze.
Probably not a big issue, but we need to be careful to only lock for a short time, and always abort the transaction in
on.exit(). Or we can use thor'swith_transaction().This will be postponed until after the next release.
Experimenting with embedded sqlite: https://github.com/r-lib/pkgcache/tree/feature/sqlite
