Creating a HF Dataset from lakeFS with S3 storage takes too much time!

[Thunderhead-exe]

Hi,

I’m new to HF dataset and I tried to create datasets based on data versioned in lakeFS (MinIO S3 bucket as storage backend)

Here I’m using ±30000 PIL image from MNIST data however it is taking around 12min to execute, which is a lot!

From what I understand, it is loading the images into cache then building the dataset.
– Please find bellow the execution screenshot –

Is there a way to optimize this or am I doing something wrong?

Thanks!

[Thunderhead-exe]

> Update

The bottleneck, from what I understand, was making one network request per file

For 30k images, this meant 30k separate GET requests to the MinIO server through the S3 API, and that was killing the performance

Using webDataset to transform the large number of files to few .tar files and passing “webdataset” instead of “imagefolder” to the load_dataset function worked perfectly (took only ~11s)