Configure the number of jobs submitted to a ray cluster for Iceberg writes

[akshayah3]

Hi All. I am trying to write to an iceberg table using Daft which is partitioned on a month transform. I am running into issues from AWS when its trying to write the parquet files in S3

I see that ray is parallelizing a number of jobs which I assume are writing data into our s3

Is there a way we can configure the number of tasks that Daft creates so as to overcome the throttling from s3? When the job finishes I see a bunch of failed tasks and there is no data in our underlying iceberg table

[colin-ho]

You can try using into_partitions before the write to reduce the number of partitions that concurrently write to s3. Lmk if this works!

[akshayah3]

@colin-ho This works now without any errors! I am now wondering if I need to set this number on a case to case basis and also how to determine an optimal number of partitions. In my case a dataset of 1.5 Billion records of 20 years worth of data took 45 minutes to be written with a Month Transform using 300 partitions

[colin-ho]

If you're able to know the number of cpu cores in your cluster, i'd generally recommend around 2x the number of cores. You could use something like https://docs.ray.io/en/latest/ray-core/api/doc/ray.cluster_resources.html#ray.cluster_resources to figure it out.

Additionally, I'd recommend fewer partitions when writing to partitioned tables.

Unfortunately though, these are just general recommendations, and will differ on a case by case basis. We're working on better partitioning determinations right now and hopefully some of this can be alleviated