Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support compression stream #158

Open
BusyJay opened this issue Nov 24, 2021 · 2 comments
Open

Support compression stream #158

BusyJay opened this issue Nov 24, 2021 · 2 comments
Labels
enhancement New feature or request

Comments

@BusyJay
Copy link
Member

BusyJay commented Nov 24, 2021

Currently compression is done per writes. But there are a lot of duplicated bytes between raft log entries, if compression is enabled for a stream, the compress ratio can be optimized further.

@BusyJay BusyJay added the enhancement New feature or request label Nov 24, 2021
@tabokie
Copy link
Member

tabokie commented Nov 24, 2021

One issue here is the recovery speed. Right now only indexes is read during recovery, #129 shows full scan (large enough read-block-size) can bring nearly 3x regression.
Performance-wise, streaming compression seems to have lower latency and fewer allocation, but compression will be in the critical path inside mutex.

@BusyJay
Copy link
Member Author

BusyJay commented Nov 24, 2021

I'm not sure if such algorithm exists, but compression can collect dict during the whole time, while flushing to disk block by block, perhaps per 64MiB. This can take care of both recover speed and runtime performance.

A log entry is usually only 0~512B as observed in a typical online service.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants