Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azcopy on Mac: too many open files #2746

Open
pkcecw opened this issue Jul 10, 2024 · 2 comments
Open

Azcopy on Mac: too many open files #2746

pkcecw opened this issue Jul 10, 2024 · 2 comments

Comments

@pkcecw
Copy link

pkcecw commented Jul 10, 2024

Which version of the AzCopy was used? ---> AzcopyVersion 10.24.0

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux) ---> Mac

What command did you run? ---> azcopy cp https://xxxpip01.blob.core.windows.net/xxx/PIDxxx/" "/Users/vhcadministrator/Desktop/xxx" --overwrite=ifSourceNewer --recursive=true

What problem was encountered? ---> too many open files

How can we reproduce the problem in the simplest way? running the same command again

Have you found a mitigation/solution? No

Here is the session output:

MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1392
virtual memory (kbytes, -v) unlimited
MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -n
256
MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -n 65356
MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 65356
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1392
virtual memory (kbytes, -v) unlimited

XXX:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ./azcopy cp "https://xxxpip01.blob.core.windows.net/xxx/PIDXXX/?xxx" "/Users/vhcadministrator/Desktop/xxx" --overwrite=ifSourceNewer --recursive=true
INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job c363ca9f-3417-164f-6f71-99b7b284486e has started
Log file is located at: /Users/vhcadministrator/.azcopy/c363ca9f-3417-164f-6f71-99b7b284486e.log

5.1 %, 11973 Done, 67109 Failed, 198388 Pending, 2530 Skipped, 280000 Total (scanning...), 2-sec Throughput (Mb/s): 99.564 panic: open /Users/vhcadministrator/.azcopy/plans/c363ca9f-3417-164f-6f71-99b7b284486e--00028.steV18: too many open files

goroutine 1 [running]:
github.com/Azure/azure-storage-azcopy/v10/common.PanicIfErr(...)
/Users/runner/work/1/s/common/lifecyleMgr.go:724
github.com/Azure/azure-storage-azcopy/v10/ste.JobPartPlanFileName.Map({0xc001f27cc0, 0x32})
/Users/runner/work/1/s/ste/JobPartPlanFileName.go:66 +0x1a6
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).AddJobPart2(0xc000b3c708, 0xc00d92b1d0)
/Users/runner/work/1/s/ste/mgr-JobMgr.go:453 +0x25a
github.com/Azure/azure-storage-azcopy/v10/jobsAdmin.ExecuteNewCopyJobPartOrder({0x0, {0xc363ca9f, 0x3417, 0x164f, {0x6f, 0x71, 0x99, 0xb7, 0xb2, 0x84, ...}}, ...})
/Users/runner/work/1/s/jobsAdmin/init.go:194 +0x2ab
github.com/Azure/azure-storage-azcopy/v10/cmd.inprocSend({0x10a01fa7c, 0x10}, {0x10a2598c0, 0xc00016d888}, {0x10a259980, 0xc016eec018})
/Users/runner/work/1/s/cmd/rpc.go:39 +0xf8
github.com/Azure/azure-storage-azcopy/v10/cmd.init.func4({0x10a01fa7c?, 0xc00076c730?}, {0x10a2598c0?, 0xc00016d888?}, {0x10a259980?, 0xc016eec018?})
/Users/runner/work/1/s/cmd/rpc.go:31 +0x2c
github.com/Azure/azure-storage-azcopy/v10/cmd.addTransfer(, {{0xc00e871cb0, 0x86}, {0xc00e86ca00, 0x91}, 0x0, {0x0, 0xeddbb16a0, 0xc001af63f0}, 0x87c2, ...}, ...)
/Users/runner/work/1/s/cmd/copyEnumeratorHelper.go:24 +0xe3
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).initEnumerator.func5({{0xc00ea21d0f, 0x31}, 0x0, {0x0, 0xeddbb16a0, 0xc001af63f0}, {0x0, 0x0, 0x0}, 0x87c2, ...})
/Users/runner/work/1/s/cmd/copyEnumeratorInit.go:321 +0x585
github.com/Azure/azure-storage-azcopy/v10/cmd.processIfPassedFilters({, _, _}, {{0xc00ea21d0f, 0x31}, 0x0, {0x0, 0xeddbb16a0, 0xc001af63f0}, {0x0, ...}, ...}, ...)
/Users/runner/work/1/s/cmd/zc_enumerator.go:841 +0x97
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).parallelList(0xc00022cbd0, 0xc0000a80a0, {0xc000010c2f, 0x38}, {0xc000010c68, 0xb}, {0x0, 0x0}, 0x0, 0xc000037a40, ...)
/Users/runner/work/1/s/cmd/zc_traverser_blob.go:439 +0x338
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).Traverse(0xc00022cbd0, 0x0, 0xc000037a40, {0x10ac2f140, 0x0, 0x0})
/Users/runner/work/1/s/cmd/zc_traverser_blob.go:309 +0xa45
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CopyEnumerator).enumerate(0xc000149ec0)
/Users/runner/work/1/s/cmd/zc_enumerator.go:789 +0x3a
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).processCopyJobPartOrders(0xc0002ae008)
/Users/runner/work/1/s/cmd/copy.go:1623 +0xf06
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).process(0xc0004a8870?)
/Users/runner/work/1/s/cmd/copy.go:1273 +0x65
github.com/Azure/azure-storage-azcopy/v10/cmd.init.2.func2(0xc0000e9b88?, {0xc000149680?, 0x2?, 0x10a01443a?})
/Users/runner/work/1/s/cmd/copy.go:2028 +0x245
github.com/spf13/cobra.(*Command).execute(0xc0000e9b88, {0xc000149600, 0x4, 0x4})
/Users/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x684
github.com/spf13/cobra.(*Command).ExecuteC(0x10abbf940)
/Users/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x38d
github.com/spf13/cobra.(*Command).Execute(...)
/Users/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/Azure/azure-storage-azcopy/v10/cmd.Execute({0xc00003a340?, 0x10aba5d30?}, {0xc00003e7b0?, 0xc0001450c8?}, 0xc0001450c0?, {0xc363ca9f, 0x3417, 0x164f, {0x6f, 0x71, ...}})
/Users/runner/work/1/s/cmd/root.go:220 +0xec
main.main()
/Users/runner/work/1/s/main.go:79 +0x47b
@pkcecw
Copy link
Author

pkcecw commented Jul 12, 2024

Can someone review the above error help me on this please?

@petereddy
Copy link

petereddy commented Dec 12, 2024 •
edited
Loading

azcopy version 10.27.1

I'm unable to download a large set of files using azcopy on macos. I've tried setting both AZCOPY_CONCURRENT_FILES and AZCOPY_CONCURRENCY_VALUE to various, low settings but no joy. It either dies with too many open files or download failures:

export AZCOPY_CONCURRENT_FILES=30;

0.0 %, 0 Done, 3470 Failed, 36530 Pending, 0 Skipped, 40000 Total (scanning...),

INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job 06654316-5c89-dd48-4bc7-be87ba75f3d4 has started
Log file is located at: /Users/petere/.azcopy/06654316-5c89-dd48-4bc7-be87ba75f3d4.log

0.0 %, 0 Done, 18570 Failed, 61430 Pending, 0 Skipped, 80000 Total (scanning...), panic: couldn't create job part plan file "06654316-5c89-dd48-4bc7-be87ba75f3d4--00008.steV18": open /Users/petere/.azcopy/plans/06654316-5c89-dd48-4bc7-be87ba75f3d4--00008.steV18: too many open files

goroutine 1 [running]:
github.com/Azure/azure-storage-azcopy/v10/ste.JobPartPlanFileName.Create({_, _}, {0x0, {0x6654316, 0x5c89, 0xdd48, {0x4b, 0xc7, 0xbe, 0x87, ...}}, ...})
	github.com/Azure/azure-storage-azcopy/v10/ste/JobPartPlanFileName.go:147 +0x1e14
github.com/Azure/azure-storage-azcopy/v10/jobsAdmin.ExecuteNewCopyJobPartOrder({0x0, {0x6654316, 0x5c89, 0xdd48, {0x4b, 0xc7, 0xbe, 0x87, 0xba, 0x75, ...}}, ...})
	github.com/Azure/azure-storage-azcopy/v10/jobsAdmin/init.go:165 +0x98
github.com/Azure/azure-storage-azcopy/v10/cmd.inprocSend({0x110c53c7a, 0x10}, {0x110f3a100, 0xc00013b188}, {0x110f3a1c0, 0xc0251fcc30})
	github.com/Azure/azure-storage-azcopy/v10/cmd/rpc.go:39 +0xf8
github.com/Azure/azure-storage-azcopy/v10/cmd.init.func6({0x110c53c7a?, 0xc00058abd8?}, {0x110f3a100?, 0xc00013b188?}, {0x110f3a1c0?, 0xc0251fcc30?})
	github.com/Azure/azure-storage-azcopy/v10/cmd/rpc.go:31 +0x2c
github.com/Azure/azure-storage-azcopy/v10/cmd.addTransfer(_, {{0xc0283af6b0, 0x24}, {0xc028b12f80, 0x34}, 0x0, {0x0, 0xedebf03c5, 0xc000b1f180}, 0x54ce, ...}, ...)
	github.com/Azure/azure-storage-azcopy/v10/cmd/copyEnumeratorHelper.go:24 +0xdd
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).initEnumerator.func5({{0xc028c58b14, 0x1f}, 0x0, {0x0, 0xedebf03c5, 0xc000b1f180}, {0x0, 0x0, 0x0}, 0x54ce, ...})
	github.com/Azure/azure-storage-azcopy/v10/cmd/copyEnumeratorInit.go:306 +0x585
github.com/Azure/azure-storage-azcopy/v10/cmd.processIfPassedFilters({_, _, _}, {{0xc028c58b14, 0x1f}, 0x0, {0x0, 0xedebf03c5, 0xc000b1f180}, {0x0, ...}, ...}, ...)
	github.com/Azure/azure-storage-azcopy/v10/cmd/zc_enumerator.go:841 +0x97
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).parallelList(0xc0004dcb60, 0xc00051c1e0, {0xc00059c629, 0x7}, {0xc00059c631, 0x14}, {0x0, 0x0}, 0x0, 0xc0000c43c0, ...)
	github.com/Azure/azure-storage-azcopy/v10/cmd/zc_traverser_blob.go:478 +0x338
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).Traverse(0xc0004dcb60, 0x0, 0xc0000c43c0, {0x1120a93a0, 0x0, 0x0})
	github.com/Azure/azure-storage-azcopy/v10/cmd/zc_traverser_blob.go:323 +0xa45
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CopyEnumerator).enumerate(0xc000308440)
	github.com/Azure/azure-storage-azcopy/v10/cmd/zc_enumerator.go:789 +0x3a
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).processCopyJobPartOrders(0xc000457c08)
	github.com/Azure/azure-storage-azcopy/v10/cmd/copy.go:1645 +0xf06
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).process(0xc000476bd0?)
	github.com/Azure/azure-storage-azcopy/v10/cmd/copy.go:1274 +0x65
github.com/Azure/azure-storage-azcopy/v10/cmd.init.2.func2(0xc000292308?, {0xc0002a4aa0?, 0x2?, 0x110c45f59?})
	github.com/Azure/azure-storage-azcopy/v10/cmd/copy.go:2050 +0x245
github.com/spf13/cobra.(*Command).execute(0xc000292308, {0xc0002a4960, 0xa, 0xa})
	github.com/spf13/[email protected]/command.go:989 +0xa91
github.com/spf13/cobra.(*Command).ExecuteC(0x111fc1fc0)
	github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:1041
github.com/Azure/azure-storage-azcopy/v10/cmd.Execute({0xc000490708?, 0x111f866b0?}, {0xc0004789c0?, 0xc00008e648?}, 0x10fd1a725?, {0x6654316, 0x5c89, 0xdd48, {0x4b, 0xc7, ...}})
	github.com/Azure/azure-storage-azcopy/v10/cmd/root.go:233 +0xec
main.main()
	github.com/Azure/azure-storage-azcopy/v10/main.go:79 +0x47b

When I reset all settings to default, I saw the following in the log file:

2024/12/12 20:56:53 Max open files when downloading: 2147479959 (auto-computed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@petereddy @pkcecw