[BUG] Problems uploading larger pkg using on-prem

[mikaellofgren]

Describe the bug
Problems with uploading larger (1 gb pkg files or larger) to SMB share
Usually it starts with Input/output error and after some retries it works uploading the pkg and usually when the uploads works
it fails in next step with 401 Jamf returned status code '401' - Access denied.
And with another retry it fully works.

To Reproduce
Steps to reproduce the behavior. For example:
Upload larger files to SMB share

• Which Processor JamfPackageUploader

Expected behaviour
Pkg uploaded and policy/patch management created

Log output
Usually it starts like this when I get Input/output error

JamfPackageUploader: Copying Microsoft_Word_365-16.100.25081015.pkg to /Volumes/JamfShare/Packages/Microsoft_Word_365-16.100.25081015.pkg
Traceback (most recent call last):
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
  File "/Library/AutoPkg/autopkglib/__init__.py", line 626, in process
    self.main()
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfPackageUploader.py", line 270, in main
    self.execute()
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfUploaderLib/JamfPackageUploaderBase.py", line 1044, in execute
    self.copy_pkg(smb_url, pkg_path, pkg_name)
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfUploaderLib/JamfPackageUploaderBase.py", line 192, in copy_pkg
    copyfile(pkg_path, destination_pkg_path)
  File "/Library/AutoPkg/Python3/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 260, in copyfile
    _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
  File "/Library/AutoPkg/Python3/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 102, in _fastcopy_fcopyfile
    raise err from None
  File "/Library/AutoPkg/Python3/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 95, in _fastcopy_fcopyfile
    posix._fcopyfile(infd, outfd, flags)
OSError: [Errno 5] Input/output error: '/Users/username/Library/AutoPkg/Cache/autopkg-recipes.jamf.microsoftexcel365/Microsoft_Word_365-16.100.25081015.pkg' -> '/Volumes/JamfShare/Packages/Microsoft_Word_365-16.100.25081015.pkg'
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
[Errno 5] Input/output error: '/Users/username/Library/AutoPkg/Cache/autopkg-recipes.jamf.microsoftexcel365/Microsoft_Word_365-16.100.25081015.pkg' -> '/Volumes/JamfShare/Packages/Microsoft_Word_365-16.100.25081015.pkg'
Failed.
Receipt written to /Users/username/Library/AutoPkg/Cache/autopkg-recipes.jamf.microsoftexcel365/receipts/MicrosoftWord365-receipt-20250818-091010.plist

The following recipes failed:
    MicrosoftWord365.jamf
        Error in autopkg-recipes.jamf.microsoftexcel365: Processor: com.github.grahampugh.jamf-upload.processors/JamfPackageUploader: Error: [Errno 5] Input/output error: '/Users/username/Library/AutoPkg/Cache/autopkg-recipes.jamf.microsoftexcel365/Microsoft_Word_365-16.100.25081015.pkg' -> '/Volumes/JamfShare/Packages/Microsoft_Word_365-16.100.25081015.pkg'

Then after some retries, the upload seems to work but then it fails to create Policy/Patch Management (me guessing API token expire?)

JamfPackageUploader: Package 'Microsoft_Word_365-16.100.25081015.pkg' not found on server
JamfPackageUploader: Package ID: 0
JamfPackageUploader: Number of File Share DPs: 1
JamfPackageUploader: Begin upload to File Share DP smb://jamffs.at..se/JamfShare
JamfPackageUploader: No existing package found
JamfPackageUploader: Expected path: /Volumes/JamfShare/Packages/Microsoft_Word_365-16.100.25081015.pkg
JamfPackageUploader: Copying Microsoft_Word_365-16.100.25081015.pkg to /Volumes/JamfShare/Packages/Microsoft_Word_365-16.100.25081015.pkg
JamfPackageUploader: Package copy successful
Unmount failed for /Volumes/JamfShare
JamfPackageUploader: WARNING! Unmount failed.
JamfPackageUploader: Creating package metadata
JamfPackageUploader: Checking for 'AutoPkg' on https://jamf.at..se:8443
JamfPackageUploader: URL: https://jamf.at..se:8443/api/v1/categories?page=0&page-size=1000&sort=id&filter=name%3D%3D%22AutoPkg%22
JamfPackageUploader: Output file is: /tmp/jamf_upload_n30l6ydp/jamf_upload_lkvclc1c.txt
JamfPackageUploader: Existing cookie found
JamfPackageUploader: curl command: /usr/bin/curl --location --dump-header /tmp/jamf_upload_jamf_at__se_8443/curl_headers_from_jamf_upload.txt https://jamf.at..se:8443/api/v1/categories?page=0&page-size=1000&sort=id&filter=name%3D%3D%22AutoPkg%22 --header User-Agent: JamfUploader/2025.7.10.0 --request GET --silent --show-error --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJiZjIyMjZiZS0xNjEwLTQwYzgtYWU5NC0wZjNhNTUxZjFhZmMiLCJhdWQiOiJiZjIyMjZiZS0xNjEwLTQwYzgtYWU5NC0wZjNhNTUxZjFhZmMiLCJuYmYiOjE3NTU1MDE3OTcsInRva2VuLXV1aWQiOiJhNzA1NmM0My03YmMxLTRlMWYtODZiMy05Njk3OGQ4ZDM2YjgiLCJzdWJqZWN0LXR5cGUiOiJSRUdJU1RFUkVEX0NMSUVOVF9JRCIsImF1dGhlbnRpY2F0aW9uLXR5cGUiOiJDTElFTlRfQ1JFREVOVElBTFMiLCJzY29wZSI6WyJhcGktcm9sZTo2Il0sImlzcyI6Imh0dHBzOi8vamFtZi5hdC5wb2xpc2VuLnNlOjg0NDMvYXBpL29hdXRoL3Rva2VuIiwiZXhwIjoxNzU1NTAxODU3LCJpYXQiOjE3NTU1MDE3OTcsImp0aSI6ImNlMzk1YmQ1LWFmOTMtNGVhMy04Y2U2LTUzN2VjNDAwMmQwMiJ9.f2B1HpB_AM1NQZ_3mwyCemuq6njcERfUBrDgYGekirI --header Accept: application/json --output /tmp/jamf_upload_n30l6ydp/jamf_upload_lkvclc1c.txt --cookie-jar /tmp/jamf_upload_jamf_at__se_8443/curl_cookies_from_jamf_upload.txt --cookie /tmp/jamf_upload_jamf_at__se_8443/curl_cookies_from_jamf_upload.txt
Traceback (most recent call last):
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
  File "/Library/AutoPkg/autopkglib/__init__.py", line 626, in process
    self.main()
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfPackageUploader.py", line 270, in main
    self.execute()
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfUploaderLib/JamfPackageUploaderBase.py", line 1255, in execute
    obj_id = self.update_pkg_metadata_api(
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfUploaderLib/JamfPackageUploaderBase.py", line 614, in update_pkg_metadata_api
    category_id = self.get_category_id(
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfUploaderLib/JamfPackageUploaderBase.py", line 584, in get_category_id
    obj_id = self.get_api_obj_id_from_name(
  File "/Users/username/Library/AutoPkg/RecipeRepos/autopkg-recipes-external/JamfUploaderProcessors/JamfUploaderLib/JamfUploaderBase.py", line 798, in get_api_obj_id_from_name
    raise ProcessorError(
autopkglib.ProcessorError: ERROR: Jamf returned status code '401' - Access denied.
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
ERROR: Jamf returned status code '401' - Access denied.
Failed.

Paste output here or attach a file

Environment (please complete the following information):

• OS: macOS 15.6
• AutoPkg version 2.7.4
• Are you using the processors in autopkg/grahampugh-recipes (stable) = Yes
• Jamf on-prem 11.18.1-t1751293920915
• SMB Share on Windows Server 2019 (1809 os build 17763.7558)

Additional context
Add any other context about the problem here.

[mikaellofgren]

Output from
smbutil statshares -a

                          SERVER_NAME                   jamffs
                          USER_ID                       501
                          SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                          SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                          SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                          SMB_VERSION                   SMB_3.1.1
                          SMB_ENCRYPT_ALGORITHMS        AES_128_CCM_ENABLED
                          SMB_ENCRYPT_ALGORITHMS        AES_128_GCM_ENABLED
                          SMB_ENCRYPT_ALGORITHMS        AES_256_CCM_ENABLED
                          SMB_ENCRYPT_ALGORITHMS        AES_256_GCM_ENABLED
                          SMB_CURR_ENCRYPT_ALGORITHM    OFF
                          SMB_SIGN_ALGORITHMS           AES_128_CMAC_ENABLED
                          SMB_SIGN_ALGORITHMS           AES_128_GMAC_ENABLED
                          SMB_CURR_SIGN_ALGORITHM       AES_128_CMAC
                          COMPRESSION_IO_THRESHOLD      4096
                          COMPRESSION_CHUNK_LEN         262144
                          COMPRESSION_MAX_FAIL_CNT      5
                          WRITE_COMPRESSION_CNT         0
                          WRITE_CNT_LZ77Huff            0
                          WRITE_CNT_LZ77                0
                          WRITE_CNT_LZNT1               0
                          WRITE_CNT_FWD_PATTERN         0
                          WRITE_CNT_BWD_PATTERN         0
                          READ_COMPRESSION_CNT          0
                          READ_CNT_LZ77Huff             0
                          READ_CNT_LZ77                 0
                          READ_CNT_LZNT1                0
                          READ_CNT_FWD_PATTERN          0
                          READ_CNT_BWD_PATTERN          0
                          SMB_SHARE_TYPE                DISK
                          SIGNING_SUPPORTED             TRUE
                          SIGNING_REQUIRED              TRUE
                          EXTENDED_SECURITY_SUPPORTED   TRUE
                          LARGE_FILE_SUPPORTED          TRUE
                          FILE_IDS_SUPPORTED            TRUE
                          DFS_SUPPORTED                 TRUE
                          FILE_LEASING_SUPPORTED        TRUE
                          MULTI_CREDIT_SUPPORTED        TRUE
                          MULTI_CHANNEL_SUPPORTED       TRUE
                          DIR_LEASING_SUPPORTED         TRUE
                          SIGNING_ON                    TRUE
                          SESSION_RECONNECT_TIME        0:0
                          SESSION_RECONNECT_COUNT       0

[mikaellofgren]

Did some tweaks on the SMB share that more or less solved the first issue with Input/output error
using unified log helped:
log stream --predicate 'eventMessage contains "smb"' --info

Set-SmbServerConfiguration -EnableMultiChannel $false
Set-SmbServerConfiguration -EnableLeasing $false
Set-SmbServerConfiguration -EnableOplocks $false

But the main issue remains, if an upload takes a while the API Token is invalid,
and it feels like a checksum check is missing on the pkg to really verify the upload is ok?

Thanks!
Mikael