[BUG] Append requests while disposing the stream from OpenWriteAsync does not use custom transport

[cool-mist]

Library name and version

Azure.Storage.Files.DataLake 12.21.0

Describe the bug

When disposing the stream obtained via OpenWriteAsync, any injected custom transport is not used as part of the implicit Append request

Expected behavior

Custom transport of the client should always be used for any requests made by the storage sdk

Actual behavior

Custom transport is not being used for the append request at the very least.

Reproduction Steps

Consider the below code that uploads file via both UploadAsync and OpenWriteAsync. The UploadAsync always succeeds, while the OpenWriteAsync consistently fails. The failure indicates that the delegating handlers in the custom transport were not invoked which can only happen if the custom transport is not invoked at all in that request. I'm not sure how it makes the request? Default HttpClientHandler maybe ?

var uri = "https://<accountName>.dfs.core.windows.net/test/s1";
var clientUri = new Uri(uri);
var tokenCredential = new AzureCliCredential();
var token = await tokenCredential.GetTokenAsync(new TokenRequestContext(new[] { "https://storage.azure.com/.default" }));
var pipeline = new AuthHandler(token.Token);
var client = new DataLakeFileClient(clientUri, new DataLakeClientOptions()
{
    Transport = new HttpClientTransport(new HttpClient(pipeline))
});

await UploadAsync(client);
Console.WriteLine("Check contents");
Console.ReadLine();
await OpenWriteAsync(client);

async Task OpenWriteAsync(DataLakeFileClient client)
{
    using var stream = await client.OpenWriteAsync(true);
    var bytes = Encoding.UTF8.GetBytes("Hello World");
    await stream.WriteAsync(bytes, 0, bytes.Length);
    Console.WriteLine("OpenWrite completed");
}

async Task UploadAsync(DataLakeFileClient client)
{
    await client.UploadAsync(new MemoryStream(Encoding.UTF8.GetBytes("Hello World")), true);
    Console.WriteLine("Upload completed");
}


public class AuthHandler : DelegatingHandler
{
    private readonly string shortLivedToken;

    public AuthHandler(string token) : base(new HttpClientHandler())
    {
        this.shortLivedToken = token;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.Add("Authorization", $"Bearer {this.shortLivedToken}");
        return await base.SendAsync(request, cancellationToken);
    }
}

Status: 401 (Server failed to authenticate the request. Please refer to the information in the www-authenticate header.)
ErrorCode: NoAuthenticationInformation

Content:
{"error":{"code":"NoAuthenticationInformation","message":"Server failed to authenticate the request. Please refer to the information in the www-authenticate header.\nRequestId:03e5784c-201f-0080-6ccb-7aaa08000000\nTime:2025-02-09T08:17:24.3961338Z"}}

Headers:
Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0
x-ms-error-code: NoAuthenticationInformation
x-ms-request-id: 03e5784c-201f-0080-6ccb-7aaa08000000
x-ms-version: 2025-01-05
x-ms-client-request-id: 9b982eac-baf5-4b01-82bb-ea5238f408c6
WWW-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/authorize resource_id=https://storage.azure.com
Date: Sun, 09 Feb 2025 08:17:23 GMT
Content-Length: 250
Content-Type: application/json;charset=utf-8

   at Azure.Storage.Files.DataLake.PathRestClient.AppendData(Stream body, Nullable`1 position, Nullable`1 timeout, Nullable`1 contentLength, Byte[] transactionalContentHash, Byte[] transactionalContentCrc64, String leaseId, Nullable`1 leaseAction, Nullable`1 leaseDuration, String proposedLeaseId, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 flush, String structuredBodyType, Nullable`1 structuredContentLength, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.AppendInternal(Stream content, Nullable`1 offset, UploadTransferValidationOptions validationOptionsOverride, String leaseId, Nullable`1 leaseAction, Nullable`1 leaseDuration, String proposedLeaseId, IProgress`1 progressHandler, Nullable`1 flush, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileWriteStream.AppendInternal(UploadTransferValidationOptions validationOptions, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Shared.StorageWriteStream.AppendAndClearBufferInternal(UploadTransferValidationOptions validationOptions, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Shared.StorageWriteStream.FlushInternal(Boolean async, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(Task task)
   at Azure.Storage.Shared.StorageWriteStream.Flush()
   at Azure.Storage.Shared.StorageWriteStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Program.<<Main>$>g__OpenWriteAsync|0_0(DataLakeFileClient client) in Q:\Repos\stt\Program.cs:line 27
   at Program.<Main>$(String[] args) in Q:\Repos\stt\Program.cs:line 20
 

Environment

No response

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.