Open
Description
Context
Compaction feels slower in v0.4.0
(may not be, maybe collection is just faster).
However I collected 5.6mil log lines and produced the initial parquet (84,256 files) for these within 1m11s (apologies for delay in obtaining the screen grab)
The compaction stage took from 1m11s -> 4m10s (almost 3 minutes) to compact the 84,256 files to 181
Unsure if changes were made that have slowed down compaction or if in general we could speed it up (parallel processing of folders by tp_date?)
Metadata
Metadata
Assignees
Labels
No labels