[FEATURE REQ] Provide a way to retrieve the etag associated with the final flush request when using OpenWriteAsync

[cool-mist]

Library name

Azure.Storage.Files.DataLake 12.21.0

Please describe the feature.

When disposing the stream obtained from OpenWriteAsync, there isn't a way to get the etag of the final flush request today. This is useful when uploading large amounts of data, and caching the etag. We could send another request to get the Etag, but it is not guaranteed to be the same as the one we just uploaded.

We could write our own stream implementation to do this, however, the value add of using the one from OpenWriteAsync is that the sdk seems to call append in 4MB chunks which matches the smallest billing quanta, and so sdk's implementation is billing efficient.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[amnguye]

Alternatively, if you're open to it you could also use UploadAsync(.., DataLakeUploadOptions,..), with setting the DataLakeUploadOptions.TransferOptions = new() {MaximumTransferSize = 4MB} as well. If OpenWrite is not a necessity, since UploadAsync does return the ETag with PathInfo.

I do agree that the alternative of having to do the stream implementation, is not ideal. However, you can manually ensure that each Append call is done in 4MB chunks.

As for the ask to return the ETag of the Flush of the Stream, may require some kind of API change, due to the fact that the return value is currently a Stream, and the Flush method is overridden from the Stream class. It wouldn't be as straightforward as returning the ETag unfortunately.

We can leave this github issue open to see if there's any traction from other different customers asking for this feature request.