support max_concurrency in upload_blob and download_blob operations
#420
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
max_concurrency=Nonekwarg in async fs methods that use the azure SDKupload_blobanddownload_blob(which azure then uses to parallelize async chunk uploads/downloads)AzureBlobFileSystem.max_concurrencyattribute which is used whenever method-levelmax_concurrencyis not setfsspec.asyn._get_batch_size()max_concurrencydefaults to 1 for the individual file upload/download.batch_size=...is used to parallelize uploads/downloads at the file level (in something likefs._get()orfs._put()), and no additional parallelization is done for chunks within each file.batch_sizeandmax_concurrency. i.e.fs.get(path, batch_size=4, max_concurrency=2)would download up to 4 files at a time, and up to 2 chunks at a time within each file, giving an overall concurrency of up to 8 async download coroutines being run in the loop at a time.Closes #268 (and supercedes the changes in the
concurrent_iobranch)This PR is incompatible with #329 (but there was discussion in that PR regarding changing the name of the parameter used to something other than
max_concurrencysince it conflicts with the the azure SDK parameter)