Use a DB for the metadata

[gaborcsardi]

No description provided.

[gaborcsardi]

It seems that SQLite is slow as well, and it also does not play well with having a dependency and C-code free pak in the future, so I gave up on this. I think the best solution will be to write our own serialization, similarly to the qs package.

[gaborcsardi]

RSQLite is not slow, after all: r-dbi/RSQLite#306 :)

[gaborcsardi]

OK, this is not so great, because RSQLite depends on Rcpp and C++ is hard to compile statically, so it is hard to create a static installer. The same applies to the qs package.

However, the thor package with embedded lmdb seems nice. It is a different approach, because it is just a key-value store, but it is transactional, and that's exactly what we need, even simpler than switching to a SQL DB. We need to serialize the metadata to a raw vector to store it in the DB.

Also, we don't have to merge the different package sources in the cache, because thor can access all of them within a transaction, and merging them at loading time it quite fast.

[gaborcsardi]

thor/lmdb does not have a timeout for locking, and it is also not interruptible, so if a process keeps a write-lock, then other processes just freeze.

Probably not a big issue, but we need to be careful to only lock for a short time, and always abort the transaction in on.exit(). Or we can use thor's with_transaction().

[gaborcsardi]

This will be postponed until after the next release.

[gaborcsardi]

Experimenting with embedded sqlite: https://github.com/r-lib/pkgcache/tree/feature/sqlite