Async support?

[ion-elgreco]

What is the motivation and/or use case?

Is async supported in lakefs spec?

How can we implement this feature?

No response

[AdrianoKF]

Thanks for raising this issue, @ion-elgreco!

Currently, there is no async support in lakeFS-spec, since the lakeFS client does not support it either (only for import operations, which are outside of the scope of lakeFS-spec).
We might consider this feature in the future depending on the overall demand.

Could you elaborate a bit on your use case?

[ion-elgreco]

@AdrianoKF I need to read a couple of hundred thousands text files and parse them with regex. Doing this asynchronous is much faster.

I have currently written it with s3fs and lakefs S3 gateway. Seeing 15-20x performance increase versus sequential reads with lakefs_spec

[leonpawelzik]

Hey @ion-elgreco,

would you be willing to test out our WIP solution: #283?

[ion-elgreco]

@leonpawelzik currently on holidays, and afterwards I won't be using LakeFS anytime soon.

Might be an idea to create small benchmark that compares s3fs vs lakefs-spec async perf