Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Access large indexes directly from the file system #10

Open
mxmlnkn opened this issue Jan 16, 2023 · 0 comments
Open

Access large indexes directly from the file system #10

mxmlnkn opened this issue Jan 16, 2023 · 0 comments
Labels
performance Something is slower than it could be

Comments

@mxmlnkn
Copy link
Owner

mxmlnkn commented Jan 16, 2023

gztool has something like this:

windows are not loaded in memory unless they're needed, so the application memory footprint is fairly low

I think it should be fairly easy to implement thanks to the WindowMap class abstraction access to windows. I might have to make it multi-threading safe though. On Linux systems, this abstraction could simply use mmap to implement it.

This also would improve startup time and interleave index loading with decompression.

I think it only makes sense for indexes that are larger than ~32 MiB. Because the memory usage for decompression should at least be in this order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Something is slower than it could be
Projects
None yet
Development

No branches or pull requests

1 participant