TCP errors cause disconnection: write: connection reset by peer. When Staging block.

[justingross-msft]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.30.0~preview.2 but also used 10.29.1

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

2025/08/19 16:21:07 Job-Command copy ./200G https://pstr2816659900356788.blob.core.windows.net/a0-ez88-acg-2025/DataDrop2_2025/200Gtest?se=2025-12-31t15%3A34%3A19z&sig=-REDACTED-&sp=racwdl&spr=https&sr=c&st=2025-07-02t07%3A34%3A19z&sv=2024-11-04

What problem was encountered?

ERROR:
Put "https://pstr2816659900356788.blob.core.windows.net/a0-ez88-acg-2025/DataDrop2_2025%2F200Gtest?blockid=MDAwMDBooatapm9LNEhMlI3k6vQNMDAwMDAwMDAwMDIwNDU0&comp=block&se=2025-12-31T15%3A34%3A19Z&sig=-REDACTED-&sp=racwdl&spr=https&sr=c&st=2025-07-02T07%3A34%3A19Z&sv=2024-11-04": write tcp 134.32.26.23:58476->20.60.140.68:443: write: connection reset by peer
goroutine 1300 [running]:
github.com/Azure/azure-storage-azcopy/v10/ste.stack()
/mnt/vss/_work/1/s/ste/xferLogPolicy.go:100 +0x5e
github.com/Azure/azure-storage-azcopy/v10/ste.logPolicy.Do({{{0xb2d05e00, 0x0}, 0xc0000118d8, 0xc0000118f0}, 0xc00049af30, 0xc00049af60, 0xc00049af90}, 0xc063026780)
/mnt/vss/_work/1/s/ste/xferLogPolicy.go:232 +0xadd
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc063026700)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-storage-azcopy/v10/ste.(*retryNotificationPolicy).Do(0x0?, 0xc063026700)
/mnt/vss/_work/1/s/ste/xferRetryNotificationPolicy.go:64 +0x2a
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc0015e7278)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*retryPolicy).Do(0xc0015e7618?, 0xc063026680)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/runtime/policy_retry.go:147 +0x67c
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc063026640)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-storage-azcopy/v10/ste.(*fileUploadRangeFromURLFixPolicy).Do(0xc0015e7698?, 0xc063026640)
/mnt/vss/_work/1/s/ste/sender-azureFileFromURL.go:118 +0x165
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc063026600)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-storage-azcopy/v10/ste.(*versionPolicy).Do(0xc06302b668?, 0xc063026600)
/mnt/vss/_work/1/s/ste/xferVersionPolicy.go:49 +0x127
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc0630265c0)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*requestIDPolicy).Do(0xc0015e77e0?, 0xc0630265c0)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/runtime/policy_request_id.go:33 +0x145
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc063026580)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.telemetryPolicy.Do({{0xc000700b40?, 0x17648e0?}}, 0xc063026580)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/runtime/policy_telemetry.go:70 +0x1b4
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc063026500)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.includeResponsePolicy(0xc063026500)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/runtime/policy_include_response.go:19 +0x1c
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.PolicyFunc.Do(0xc30803e0a7?, 0x5?)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:214 +0x19
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc063026480)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/request.go:144 +0xf0
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.Pipeline.Do({{0xc0001f9040?, 0x0?, 0x0?}}, 0x12044d9?)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.17.0/internal/exported/pipeline.go:76 +0x45
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.(*BlockBlobClient).StageBlock(0xc001459f08, {0x1b4ff28?, 0xc00071a5d0?}, {0xc0f1dde570?, 0x7f6c2435ac70?}, 0x7f6c6eb863e8?, {0x1b4db70?, 0xc0ad615dd0?}, 0xc07f3b6be0?, 0x0, ...)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.6.0/internal/generated/zz_blockblob_client.go:551 +0xcb
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*Client).StageBlock(0xc001459f20, {0x1b4ff28, 0xc00071a5d0}, {0xc0f1dde570, 0x30}, {0x1b4db70, 0xc0ad615dd0}, 0xc0015e7d38)
/home/cloudtest/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.6.0/blockblob/client.go:221 +0x275
github.com/Azure/azure-storage-azcopy/v10/ste.(*blockBlobUploader).GenerateUploadFunc.(*blockBlobUploader).generatePutBlock.func2()
/mnt/vss/_work/1/s/ste/sender-blockBlobFromLocal.go:104 +0x31a
github.com/Azure/azure-storage-azcopy/v10/ste.(*blockBlobUploader).GenerateUploadFunc.(*blockBlobUploader).generatePutBlock.createSendToRemoteChunkFunc.createChunkFunc.func4(0x5f5e100?)
/mnt/vss/_work/1/s/ste/sender.go:207 +0x294
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).chunkProcessor(0xc000512708, 0x4b2)
/mnt/vss/_work/1/s/ste/mgr-JobMgr.go:1045 +0xab
created by github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).poolSizer in goroutine 82
/mnt/vss/_work/1/s/ste/mgr-JobMgr.go:947 +0x236

2025/08/19 16:53:07 INFO: [P#0-T#0] Early close of chunk in singleChunkReader with context still active
2025/08/19 16:53:07 ERR: [P#0-T#0] UPLOADFAILED: /home/transfer/200G : 000 : Put "https://pstr2816659900356788.blob.core.windows.net/a0-ez88-acg-2025/DataDrop2_2025%2F200Gtest?blockid=MDAwMDBooatapm9LNEhMlI3k6vQNMDAwMDAwMDAwMDIwNDU0&comp=block&se=2025-12-31T15%3A34%3A19Z&sig=-REDACTED-&sp=racwdl&spr=https&sr=c&st=2025-07-02T07%3A34%3A19Z&sv=2024-11-04": write tcp 134.32.26.23:58476->20.60.140.68:443: write: connection reset by peer. When Staging block. X-Ms-Request-Id:

How can we reproduce the problem in the simplest way?

I think this is a product of the environment since this happens when we are transferring data across longer distances, using public internet, and the files are larger (over 20GB we can reproduce this everytime)

Have you found a mitigation/solution?

Running this from a VM in Azure to the storage account even in different regions (far away as well). Running the command over and over in the normal scenario will eventually succeed (7 or 8 tries).

Looking to see if maybe there is better logic that can be built in for retries.

[justingross-msft]

Submitted for another user who stated this was working fine until Q1 of this year.

[tanyasethi-msft]

Hi @justingross-msft, Thanks for reporting this and for the detailed repro steps, The error indicates that the underlying TCP connection is being closed by the remote side (or something in between) while AzCopy is in the middle of staging a block.
AzCopy does have built-in retry logic for transient network failures, but depending on timing and frequency, a reset during an active block write can require multiple retries before the job eventually succeeds.
Could you try adjusting retry-related flags (AZCOPY_CONCURRENCY_VALUE, AZCOPY_BLOCK_SIZE_MB) to reduce the stress per connection and give retries more room? Please also try lowering the concurrency value.