Skip to content

Conversation

@GustaMantovani
Copy link

Introduces a workers parameter to both the download_folder function and the CLI, allowing users to specify an integer number of workers or "auto" (which uses os.cpu_count()).

Under the hood, a helper function spawns a configurable number of threads and stores them in a pool. These threads consume from a shared input queue of (file_id, destination_path) tuples, pre-populated by _get_directory_structure().

If skip_download=True, no workers are created. By default, workers=1. When more than one worker is specified, the concurrent threads cause the tqdm progress display to misbehave, so we temporarily enforce quiet=True; this will be addressed in an upcoming PR.

In my tests, parallel downloads substantially increased throughput for folders containing many files.

@GustaMantovani GustaMantovani changed the title Adds support for parallel workers for folder downloads. Add support for parallel workers for folder downloads. May 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant