Skip to content

Commit 95f68c7

Browse files
authored
Merge branch 'main' into src-dest-support
2 parents ee39a9d + 5a7ad77 commit 95f68c7

File tree

186 files changed

+6948
-15473
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

186 files changed

+6948
-15473
lines changed

ChangeLog.md

Lines changed: 48 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,55 @@
11

22
# Change Log
33

4+
## Version 10.31.0
5+
6+
### New Features
7+
1. `--include-root` flag now allows customers to preserve root properties when used in conjunction with `--preserve-XXXX` flags. ([#3163](https://github.com/Azure/azure-storage-azcopy/pull/3163))
8+
9+
### Bug Fixes
10+
1. Fixed a bug to retry on various network errors. ([#3237](https://github.com/Azure/azure-storage-azcopy/pull/3237])) ([#3252](https://github.com/Azure/azure-storage-azcopy/pull/3252))
11+
2. Fixed a bug where remove would not work on paths with encoded characters. ([#2977](https://github.com/Azure/azure-storage-azcopy/issues/2977))
12+
3. Fixed a bug where jobs resume would not produce any output for previously failed jobs. ([#3103](https://github.com/Azure/azure-storage-azcopy/pull/3103))
13+
4. Fixed a bug where FileBlob transfers with EntraID on the source would pass the wrong service version. ([#3242](https://github.com/Azure/azure-storage-azcopy/issues/3242))
14+
15+
## Code Improvements
16+
1. Refactored traverser related code into its own package. ([#3251](https://github.com/Azure/azure-storage-azcopy/pull/3251))
17+
2. Refactored OAuth token manager access to use a client-based pattern instead of global singleton access. ([#3260](https://github.com/Azure/azure-storage-azcopy/pull/3260))
18+
3. Removed unused code related to credential management. ([#3260](https://github.com/Azure/azure-storage-azcopy/pull/3260))
19+
4. Refactored Lifecycle UI code into the cmd package ([#3262](https://github.com/Azure/azure-storage-azcopy/pull/3262)).
20+
5. Error handling code is now injected into JobMgr, or appropriately bubbled upwards instead of using global LCM error handling. ([#3262](https://github.com/Azure/azure-storage-azcopy/pull/3262))
21+
22+
## Version 10.31.0-preview.1
23+
24+
### Dependency updates
25+
1. Golang 1.24.4 -> 1.24.6 ([#3154](https://github.com/Azure/azure-storage-azcopy/issues/3154))
26+
27+
### New Features
28+
1. Azure Files NFS -> Azure Files SMB transfers.
29+
- Transfer from Azure Files NFS to Azure Files SMB. (`--from-to=FileNFSFileSMB`)
30+
2. Azure Files SMB -> Azure Files NFS transfers.
31+
- Transfer from Azure Files SMB to Azure Files NFS. (`--from-to=FileSMBFileNFS`)
32+
3. Symlink support for Azure Files NFS shares.
33+
Introduced support for symbolic links in Azure Files NFS shares.
34+
Symlinks can be preserved, skipped, or followed based on command-line flags.
35+
- Preserve symlinks: `--preserve-symlinks=true`
36+
- Skip symlinks: default behavior when flags are not provided
37+
- Follow symlinks: `--follow-symlinks=true`
38+
4. Added a --check-version flag to make version checking an opt in feature. ([#3173](https://github.com/Azure/azure-storage-azcopy/pull/3173))
39+
40+
### Bug Fixes
41+
1. Fixed a bug to retry on WSAETIMEDOUT on Windows. ([#3195](https://github.com/Azure/azure-storage-azcopy/pull/3195))
42+
2. Fixed a bug with the folder creation tracker which caused folder creation calls to happen more often than necessary. ([#3151](https://github.com/Azure/azure-storage-azcopy/pull/3151))
43+
3. Fixed a bug to redact x-ams-credential from logs. ([#3206](https://github.com/Azure/azure-storage-azcopy/pull/3206))
44+
4. Fixed a bug where powershell login would fail with older versions of Az.Accounts. ([#3191](https://github.com/Azure/azure-storage-azcopy/pull/3191))
45+
5. Fixed a bug where symlink direct targets would be handled as a file instead of a symlink. ([#3222](https://github.com/Azure/azure-storage-azcopy/pull/3222))
46+
47+
### Breaking changes
48+
1. AzCopy no longer checks version by default. ([#3173](https://github.com/Azure/azure-storage-azcopy/pull/3173))
49+
450
## Version 10.30.1
551

6-
### Bug Fixes
52+
### Bug Fixes
753
1. Fixed `--exclude-path` flag not available in remove operations.([PR #3165](https://github.com/Azure/azure-storage-azcopy/pull/3165)) ([GH Issue #3159](https://github.com/Azure/azure-storage-azcopy/issues/3159))
854
2. Fixed regression where AzCopy was not honoring concurrency value in copy operations ([#3192](https://github.com/Azure/azure-storage-azcopy/pull/3192))
955
3. Fixed the incorrect JSON output format of the warning message when there are multiple AzCopy processes running. ([PR #3188](https://github.com/Azure/azure-storage-azcopy/pull/3188)) ([GH Issue #3182](https://github.com/Azure/azure-storage-azcopy/issues/3182))
@@ -13,11 +59,6 @@
1359
### Dependency Updates
1460
1. Golang 1.24.2 -> 1.24.6 (CVE-2025-47907) ([#3154](https://github.com/Azure/azure-storage-azcopy/issues/3154))
1561

16-
## Version 10.31.0-preview.1
17-
18-
### Dependency updates
19-
1. Golang 1.24.4 -> 1.24.6 ([#3154](https://github.com/Azure/azure-storage-azcopy/issues/3154))
20-
2162
## Version 10.30.0
2263
### Breaking changes
2364
1. For transfers involving Azure Files (NFS or SMB), AzCopy will not auto create file shares.
@@ -1246,3 +1287,4 @@ information, including those needed to set the new headers.
12461287
1. excludedBlobType -> excluded-blob-type
12471288
1. outputRaw (in "list" command) -> output
12481289
1. stdIn-enable (reserved for internal use) -> stdin-enable
1290+

azcopy/client.go

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,15 +21,23 @@
2121
package azcopy
2222

2323
import (
24+
"log"
25+
"runtime"
26+
2427
"github.com/Azure/azure-storage-azcopy/v10/common"
2528
"github.com/Azure/azure-storage-azcopy/v10/jobsAdmin"
2629
"github.com/Azure/azure-storage-azcopy/v10/ste"
27-
"log"
28-
"runtime"
30+
)
31+
32+
const (
33+
oauthLoginSessionCacheKeyName = "AzCopyOAuthTokenCache"
34+
oauthLoginSessionCacheServiceName = "AzCopyV10"
35+
oauthLoginSessionCacheAccountName = "AzCopyOAuthTokenCache"
2936
)
3037

3138
type Client struct {
32-
CurrentJobID common.JobID // TODO (gapra): In future this should only be set when there is a current job running. On complete, this should be cleared. It can also behave as something we can check to see if a current job is running
39+
CurrentJobID common.JobID // TODO (gapra): In future this should only be set when there is a current job running. On complete, this should be cleared. It can also behave as something we can check to see if a current job is running
40+
oauthTokenManager *common.UserOAuthTokenManager // OAuth token manager for the current user, used for authentication
3341
}
3442

3543
type ClientOptions struct {
@@ -51,9 +59,28 @@ func NewClient(opts ClientOptions) (Client, error) {
5159
if err != nil {
5260
return c, err
5361
}
62+
// only one UserOAuthTokenManager should exist in azcopy process for current user.
63+
// (a given AzcopyJobPlanFolder is mapped to current user)
64+
if common.AzcopyJobPlanFolder == "" {
65+
panic("invalid state, AzcopyJobPlanFolder should not be an empty string")
66+
}
67+
cacheName := common.GetEnvironmentVariable(common.EEnvironmentVariable.LoginCacheName())
68+
69+
c.oauthTokenManager = common.NewUserOAuthTokenManagerInstance(common.CredCacheOptions{
70+
DPAPIFilePath: common.AzcopyJobPlanFolder,
71+
KeyName: common.Iff(cacheName != "", cacheName, oauthLoginSessionCacheKeyName),
72+
ServiceName: oauthLoginSessionCacheServiceName,
73+
AccountName: common.Iff(cacheName != "", cacheName, oauthLoginSessionCacheAccountName),
74+
})
5475
return c, nil
5576
}
5677

78+
// GetUserOAuthTokenManagerInstance gets or creates OAuthTokenManager for current user.
79+
// Note: Currently, only support to have TokenManager for one user mapping to one tenantID.
80+
func (c *Client) GetUserOAuthTokenManagerInstance() *common.UserOAuthTokenManager {
81+
return c.oauthTokenManager
82+
}
83+
5784
// Ensure we always have more than 1 OS thread running goroutines, since there are issues with having just 1.
5885
// (E.g. version check doesn't happen at login time, if have only one go proc. Not sure why that happens if have only one
5986
// proc. Is presumably due to the high CPU usage we see on login if only 1 CPU, even tho can't see any busy-wait in that code)

azcopy/jobsClean.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ type CleanJobsResult struct {
3939
// CleanJobs removes jobs with a specified status.
4040
// If WithStatus is All, it cleans all jobs and returns the count of jobs cleaned.
4141
// If WithStatus is not All, it cleans jobs with that status and returns the count of jobs cleaned and list of job IDs cleaned.
42-
func (c Client) CleanJobs(opts CleanJobsOptions) (result CleanJobsResult, err error) {
42+
func (c *Client) CleanJobs(opts CleanJobsOptions) (result CleanJobsResult, err error) {
4343
result = CleanJobsResult{}
4444
status := common.IffNil(opts.WithStatus, common.EJobStatus.All())
4545

azcopy/jobsList.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ type ListJobsResponse struct {
4343
Details []JobDetail
4444
}
4545

46-
func (c Client) ListJobs(opts ListJobsOptions) (result ListJobsResponse, err error) {
46+
func (c *Client) ListJobs(opts ListJobsOptions) (result ListJobsResponse, err error) {
4747
status := common.IffNil(opts.WithStatus, common.EJobStatus.All())
4848

4949
resp := jobsAdmin.ListJobs(status)

azcopy/jobsRemove.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ type RemoveJobResult struct {
3636
}
3737

3838
// RemoveJob removes a job with the specified JobID.
39-
func (c Client) RemoveJob(opts RemoveJobOptions) (result RemoveJobResult, err error) {
39+
func (c *Client) RemoveJob(opts RemoveJobOptions) (result RemoveJobResult, err error) {
4040
result = RemoveJobResult{}
4141
if opts.JobID.IsEmpty() {
4242
return result, errors.New("remove job requires the JobID")

azcopy/jobsShow.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ type GetJobSummaryOptions struct {
3535

3636
type JobSummaryResponse common.ListJobSummaryResponse
3737

38-
func (c Client) GetJobSummary(opts GetJobSummaryOptions) (result JobSummaryResponse, err error) {
38+
func (c *Client) GetJobSummary(opts GetJobSummaryOptions) (result JobSummaryResponse, err error) {
3939
if opts.JobID.IsEmpty() {
4040
return JobSummaryResponse{}, errors.New("get job statistics requires the JobID")
4141
}
@@ -56,7 +56,7 @@ type ListJobTransfersOptions struct {
5656
type ListJobTransfersResponse common.ListJobTransfersResponse
5757

5858
// ListJobTransfers lists the transfers for a job with the specified JobID and given transfer status.
59-
func (c Client) ListJobTransfers(opts ListJobTransfersOptions) (result ListJobTransfersResponse, err error) {
59+
func (c *Client) ListJobTransfers(opts ListJobTransfersOptions) (result ListJobTransfersResponse, err error) {
6060
if opts.JobID.IsEmpty() {
6161
return result, errors.New("list job transfers requires the JobID")
6262
}

azcopy/loginlogout.go

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
// Copyright © 2025 Microsoft <[email protected]>
2+
//
3+
// Permission is hereby granted, free of charge, to any person obtaining a copy
4+
// of this software and associated documentation files (the "Software"), to deal
5+
// in the Software without restriction, including without limitation the rights
6+
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
// copies of the Software, and to permit persons to whom the Software is
8+
// furnished to do so, subject to the following conditions:
9+
//
10+
// The above copyright notice and this permission notice shall be included in
11+
// all copies or substantial portions of the Software.
12+
//
13+
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
19+
// THE SOFTWARE.
20+
21+
package azcopy
22+
23+
import (
24+
"context"
25+
"errors"
26+
"fmt"
27+
28+
"github.com/Azure/azure-storage-azcopy/v10/common"
29+
"github.com/Azure/azure-storage-azcopy/v10/ste"
30+
)
31+
32+
type LoginOptions common.LoginOptions
33+
type LoginResponse common.LoginResponse
34+
35+
func (c *Client) Login(opts LoginOptions) (LoginResponse, error) {
36+
resp, err := c.oauthTokenManager.Login(common.LoginOptions(opts))
37+
return LoginResponse(resp), err
38+
}
39+
40+
type GetLoginStatusOptions struct {
41+
}
42+
43+
type LoginStatus struct {
44+
Valid bool
45+
TenantID string
46+
AADEndpoint string
47+
LoginType common.AutoLoginType
48+
}
49+
50+
func (c *Client) GetLoginStatus(_ GetLoginStatusOptions) (LoginStatus, error) {
51+
uotm := c.GetUserOAuthTokenManagerInstance()
52+
53+
// Get current token info and refresh it with GetTokenInfo()
54+
ctx := context.WithValue(context.TODO(), ste.ServiceAPIVersionOverride, ste.DefaultServiceApiVersion)
55+
tokenInfo, err := uotm.GetTokenInfo(ctx)
56+
if err != nil || tokenInfo.IsExpired() {
57+
return LoginStatus{Valid: false}, errors.New("you are currently not logged in, please login using 'azcopy login'")
58+
}
59+
return LoginStatus{
60+
Valid: true,
61+
TenantID: tokenInfo.Tenant,
62+
AADEndpoint: tokenInfo.ActiveDirectoryEndpoint,
63+
LoginType: tokenInfo.LoginType,
64+
}, nil
65+
}
66+
67+
type LogoutOptions struct {
68+
}
69+
70+
type LogoutResponse struct {
71+
}
72+
73+
func (c *Client) Logout(_ LogoutOptions) (LogoutResponse, error) {
74+
uotm := c.GetUserOAuthTokenManagerInstance()
75+
if err := uotm.RemoveCachedToken(); err != nil {
76+
return LogoutResponse{}, fmt.Errorf("failed to perform logout command, %v", err)
77+
}
78+
return LogoutResponse{}, nil
79+
}

azcopy/pathUtils.go

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
// Copyright © 2025 Microsoft <[email protected]>
2+
//
3+
// Permission is hereby granted, free of charge, to any person obtaining a copy
4+
// of this software and associated documentation files (the "Software"), to deal
5+
// in the Software without restriction, including without limitation the rights
6+
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
// copies of the Software, and to permit persons to whom the Software is
8+
// furnished to do so, subject to the following conditions:
9+
//
10+
// The above copyright notice and this permission notice shall be included in
11+
// all copies or substantial portions of the Software.
12+
//
13+
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
19+
// THE SOFTWARE.
20+
21+
package azcopy
22+
23+
import (
24+
"fmt"
25+
"net/url"
26+
27+
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas"
28+
"github.com/Azure/azure-storage-azcopy/v10/common"
29+
)
30+
31+
func GetContainerName(path string, location common.Location) (string, error) {
32+
switch location {
33+
case common.ELocation.Local():
34+
panic("attempted to get container name on local location")
35+
case common.ELocation.Blob(),
36+
common.ELocation.File(),
37+
common.ELocation.FileNFS(),
38+
common.ELocation.BlobFS():
39+
bURLParts, err := sas.ParseURL(path)
40+
if err != nil {
41+
return "", err
42+
}
43+
return bURLParts.ContainerName, nil
44+
case common.ELocation.S3():
45+
baseURL, err := url.Parse(path)
46+
47+
if err != nil {
48+
return "", err
49+
}
50+
51+
s3URLParts, err := common.NewS3URLParts(*baseURL)
52+
53+
if err != nil {
54+
return "", err
55+
}
56+
57+
return s3URLParts.BucketName, nil
58+
case common.ELocation.GCP():
59+
baseURL, err := url.Parse(path)
60+
if err != nil {
61+
return "", err
62+
}
63+
gcpURLParts, err := common.NewGCPURLParts(*baseURL)
64+
if err != nil {
65+
return "", err
66+
}
67+
return gcpURLParts.BucketName, nil
68+
default:
69+
return "", fmt.Errorf("cannot get container name on location type %s", location.String())
70+
}
71+
}

azure-pipelines.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,7 @@ jobs:
159159
displayName: 'Install dependencies'
160160
- script: |
161161
pip install azure-storage-blob==12.12.0
162+
pip install azure-storage-blob six
162163
# set the variable to indicate that the mutex is being acquired
163164
# note: we set it before acquiring the mutex to ensure we release the mutex.
164165
# setting this after can result in an un-broken mutex if someone cancels the pipeline after we acquire the
@@ -291,7 +292,6 @@ jobs:
291292
GOARCH: $(GOARCH)
292293
CGO_ENABLED: $(CGO_ENABLED)
293294
build_name: $(build_name)
294-
azcopy_msi_app_id: $(AZCOPY_MSI_APP_ID)
295295

296296
- job: Additional_E2E_Test_Windows
297297
timeoutInMinutes: 360
@@ -343,7 +343,6 @@ jobs:
343343
GOARCH: $(GOARCH)
344344
CGO_ENABLED: $(CGO_ENABLED)
345345
build_name: $(build_name)
346-
azcopy_msi_app_id: $(AZCOPY_MSI_APP_ID)
347346

348347
#TODO: Add MacOS E2E tests after creating macos pool
349348

0 commit comments

Comments
 (0)