How to batch process/submit a large number of tasks

[npow]

I have ~50k files I need to process. I tried using the map() function to process them in parallel, with a task concurrency limit to batch them. However this seems to still submit all the tasks upfront to the scheduler, which is taking a long time before any tasks are being started. What's the recommended way to go about this use-case?

@flow(flow_run_name="{collection_name}")
def index_audio_files(collection_name):
    fnames = natsorted(glob.glob(f"{input_dir}/**/*.mp3"))
    index_audio_file.map(fnames, collection_name)

[j-tr]

possibly related discussion on slack: https://prefect-community.slack.com/archives/CL09KU1K7/p1682070137534679