Skip to content

Review Compaction Process #431

Open
Open
@graza-io

Description

@graza-io

Context

Compaction feels slower in v0.4.0 (may not be, maybe collection is just faster).

However I collected 5.6mil log lines and produced the initial parquet (84,256 files) for these within 1m11s (apologies for delay in obtaining the screen grab)

Image

The compaction stage took from 1m11s -> 4m10s (almost 3 minutes) to compact the 84,256 files to 181

Image

Unsure if changes were made that have slowed down compaction or if in general we could speed it up (parallel processing of folders by tp_date?)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions