-
Notifications
You must be signed in to change notification settings - Fork 222
Troubleshooting FAQ
If you are new to the tool and have a hard time understanding what is the problem in the first place, you can start by determining whether the job succeeded or not by looking at the exit code:
0 – success
1 – error (check the log file)
2 – panic (check log file if exists else file a bug or reach out to support)
Here is more documentation on how to interpret the log. Once you understand the exact error message, then it becomes much easier to search for the right key words and figure out the solution.
It is common to encounter error 403s. Sometimes it is benign, for example you might see it in AzCopy logs that a HEAD request got 403, which in most cases is okay and the tool was simply checking whether a resource was public.
In other situations, you may have failed transfers, where AzCopy aborts right away when a 403 is encountered. In this case, the request was essential and its failure indicates that the rest of the transfers will most likely fail.
- If you are using a SAS, please verify:
- The expiration and start times are appropriate.
- You selected all the necessary permissions.
- You generated the token using an official SDK or tool. Try Storage Explorer if you haven't already.
- For service to service copies since the copy is performed by the service, IP based SAS is not valid
- If you are using RBAC (i.e. you called
azcopy login), please verify:- You have the right roles assigned to your identity, e.g. Storage Blob Data Contributor.
- If you are using ADLS Gen2 ACLs, please verify:
- The right permissions for your identity are present on all the entities (files and directories) that you intend to manipulate.
Additional documentation available:
- https://docs.microsoft.com/azure/storage/blobs/assign-azure-role-data-access
- https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-access-control
If you encounter authorization failures (403) while performing service-to-service copies and you have firewall or private endpoint enabled, there is a possibility that you haven't configured the network settings correctly, and they are preventing you from transferring data using the client machine where AzCopy is running.
If you are uploading/downloading between the cloud and an on-premise machine, please double check that:
- The machine running AzCopy is able to access the source or destination account. This could mean adding the machine's public IP address to the firewall allow list.
If you are performing service to service copies, please double check that:
- The machine running AzCopy is able to access BOTH the source AND the destination account. This could mean adding the machine's public IP address to both accounts' firewall allow list.
- As long as the above condition is met, then the account to account copy will work (with no other firewall config). Effectively the service will use the IP address of the AzCopy client machine to authorize the source to destination traffic. i.e. it will magically and securely work, without you needing to tell one storage account.
- In case your VM does not or cannot have a public IP address, you should consider using a private endpoint: https://docs.microsoft.com/en-us/azure/storage/common/storage-private-endpoints.
- As both storage account are on different Vnet(Vnet1 and Vnet2), behind different private endpoints(PE1 and PE2), on Azure private DNS Zone, create a record pointing for both Storage account(SA1 and SA2) and then setup peering between Vnet1 and Vnet2. Additional notes about private endpoint:
- PrivateLink is at VNet/Subnet level. If you setup PrivateLink in VNet1/Subnet1 but your VM is in VNet1/Subnet2 the request will NOT be through PrivateLink.
- When an account has private link it does not mean internet traffic is denied. It only means traffic from such VNet will go through such private endpoint.
- PrivateLink / VNet only work in single region. Traffic across region is always regular internet traffic (there are few ways to make this work, such as VNet peering)
If you encounter tcp errors such as "dial tcp: lookup proxy.bla.bla: no such host", it means that your environment is not configured properly to use the right proxy or you are using an advanced proxy that AzCopy does not recognize.
There are two options:
- You can try to update the proxy settings (doc here to reflect the right configurations.
- Or you could bypass the proxy by setting the environment variable NO_PROXY="*".
Here are the endpoints that AzCopy needs to use:
Login endpoints:
- login.microsoftonline.com (global Azure)
- login.chinacloudapi.cn (Azure China)
- login.microsoftonline.de (Azure Germany)
- login.microsoftonline.us (Azure US Government)
Azure Storage endpoints:
- (blob|file|dfs).core.windows.net (global Azure)
- (blob|file|dfs).core.chinacloudapi.cn (Azure China)
- (blob|file|dfs).core.cloudapi.de (Azure Germany)
- (blob|file|dfs).core.usgovcloudapi.net (Azure US Government)
Getting this error is often related to the usage of a proxy, which is using an SSL cert that’s not trusted by the OS. Please verify your settings and make sure that the cert is trusted at the OS level.
We recommend adding the certificate to your machine(s)' root certificate store as that's where the trusted authorities live.
One of the most common issues is to get the error message that your parameters weren't recognized. Don't worry, there are plenty of resources to help you figure out the problem.
First, please double check whether you are using the right version of AzCopy. If you were previously using the deprecated AzCopy V8 or older, you should know that V10 is a complete rewrite and does not share any syntax with the previous versions. Please refer to this migration guide here.
Second, please leverage the built-in help messages, e.g. azcopy cp -h. The same information is available (with a small delay) at the reference documentation website.
Third, we provide an education tool located here, which demonstrates the main commands and the main flags/knobs.
Finally, check out our examples here. If you have any question, please search the Github issues first to see if it was answered already.
If you are getting the following error when invoking the login command:
Failed to perform login command: failed to login with tenantID "common", Azure directory endpoint "https://login.microsoftonline.com", autorest/adal/devicetoken: -REDACTED- AADSTS50005: User tried to log in to a device from a platform (Unknown) that's currently not supported through Conditional Access policy. Supported device platforms are: iOS, Android, Mac, and Windows flavors. Trace ID: -REDACTED- Correlation ID: -REDACTED- Timestamp: 2021-01-05 01:58:28Z
Then it means that your admin has imposed conditional access policy on what kind of device you can login from. AzCopy uses the device code workflow, which cannot guarantee that the machine where you are using the tool is also where you are logging in.
If your device is among the list of supported platforms, then you could potentially use Storage Explorer, which integrates AzCopy for all data transferring (it passes tokens to AzCopy via the secret store) but provides a login workflow that supports passing device information. AzCopy itself also supports MSI and Service Principal, which could be used as an alternative.
If not, please contact your admin for help.
If you are seeing a large number of failed requests with the 503 Server Busy status, you are being throttled by the Storage service. If you are seeing network errors or timeouts, you might be pushing through too much data across your infrastructure and they are not handling it well. In all cases, the workaround is similar.
If you are seeing a large file failing over and over again due to certain chunks failing each time, you should also try the following workaround.
Note: if you are performing a service to service copy using AzCopy, the quality/reliability of the network from where you orchestrate the copy may impact the overall performance, as AzCopy initiates calls for each file to perform the copying of the data on the service side.
You can follow the instructions from this document to limit the concurrent network connections or throughput limit depending on your specific case. The suggested approach is to lower the performance drastically at first, and observe whether it solved the initial problem, then finally ramp back up until an overall balance is achieved.
- Copying data from government clouds to commercial clouds is not supported. The reverse is supported.
- Asynchronous service-side copy is not supported. AzCopy performs synchronous copy only. In other words, by the time the job finishes, the data has actually been moved.
- If you are copying to Azure Files and you forgot to specify the flag --preserve-smb-permissions, and you do not want to transfer the data again, you could consider using Robocopy to bring over the permissions alone.
- Azure Functions has a different endpoint for MSI authentication, which AzCopy does not support yet.
- There is a service issue impacting AzCopy 10.11+ which are using the PutBlobFromURL API to copy blobs smaller than the given block size (whose default is 8MiB). If the user has any firewall (VNet / IP / PL / SE Policy) on the source account, the PutBlobFromURL API may mistakenly give back
409 Copy source blob has been modified. A service side fix is being deployed and should be fully rolled out in a few weeks (status as of 11/12/2021). The workaround is to use AzCopy 10.10.- https://azcopyvnext.azureedge.net/release20210415/azcopy_darwin_amd64_10.10.0.zip
- https://azcopyvnext.azureedge.net/release20210415/azcopy_linux_amd64_10.10.0.tar.gz
- https://azcopyvnext.azureedge.net/release20210415/azcopy_windows_386_10.10.0.zip
- https://azcopyvnext.azureedge.net/release20210415/azcopy_windows_amd64_10.10.0.zip