Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Progress Bar for downloading (and maybe uploading) artifacts #1307

Open
EladDvash opened this issue Aug 6, 2024 · 2 comments
Open

Add Progress Bar for downloading (and maybe uploading) artifacts #1307

EladDvash opened this issue Aug 6, 2024 · 2 comments

Comments

@EladDvash
Copy link

EladDvash commented Aug 6, 2024

Proposal Summary

Add a progress bar to the downloading of artifacts/models/large-files from all remote sources (S3/Azure buckets/GCP buckets/network-drives), consider doing so also for uploading (though that might be an issue when they are non blocking because of terminal spam)

Motivation

When downloading models from previous experiments as artifacts for fine-tuning or as pretrained models it would be great to know how much progress has been made in the download as they can take several minutes to get, currently it just says it started downloading and you have no idea when it might finish or if it's stalled

@eugen-ajechiloae-clearml
Copy link
Collaborator

Hi @EladDvash ! Downloads/uploads made through clearml do have a progress bar that relies on tqdm. You should make sure tqdm is installed on your environment. Also, by default we do not create a progress bar for files/objects smaller than 5MB (configurable here: https://clear.ml/docs/latest/docs/configs/clearml_conf/#sdkstoragelog)

@EladDvash
Copy link
Author

Thanks, I see that clearml supports a progress bar but sadly not for GCP buckets.
In the "download_object" function for "_GoogleCloudStorageDriver" in storage/helper.py there is an argument for a callback but it isn't used and the "download_object_as_stream" function returns a NotImplementedError which are the 2 places where the progressbar callback is used in other places

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants