Skip to content

Releases: s3tools/s3cmd

v2.4.0

12 Dec 02:57
v2.4.0
9d17075
Compare
Choose a tag to compare
  • Added "setversioning" command for versioning configuration (Kuan-Chun Wang)
  • Added "settagging", "gettagging", and "deltagging" commands for bucket/object tagging (Kuan-Chun Wang)
  • Added "setobjectretention" and "setobjectlegalhold" commands (Etienne Adam/Withings SAS)
  • Added "setownership" and "setblockpublicaccess" commands
  • Added "cfinval" command to request Cloudfront to invalidate paths (#1256)
  • Added "--keep-dirs" option to have the folder structure preserved on remote side
  • Added --skip-destination-validation option for "setnotification" command (Kuan-Chun Wang)
  • Added "--max-retries" flag and "max_retries" config option (#914)
  • Added FIPS support (Michael Roth)
  • Added "object ownership" and block public access" values to "info" command output for buckets
  • Added to "ls" command a "DIROBJ" tag for directory objects in S3 remote
  • Added server profiles to run-tests.py to skip tests depending on the server type
  • Fixed "TypeError: sequence item 1: expected str instance, bytes found" error with Python 3.12 (#1343)
  • Fixed a missing return for "object_batch_delete" of S3.py (James Hewitt)
  • Fixed "object is not callable" error because of md5 FIPS test (#1005)
  • Fixed "compute_content_md5 is not defined" error for "setversioning" (#1312) (Gavin John)
  • Fixed list objects to use NextMarker when only prefixes are returned (Albin Parou)
  • Fixed upload to not retry when an S3 compatible server is full
  • Fixed recursive delete of objects named with whitespace (#976)
  • Fixed the mime type when uploading directories to be "application/x-directory"
  • Fixed "string indices must be integers" error for sync when in dry-run mode (#1313) (Shohei Tanaka)
  • Fixed SignatureDoesNotMatch error when modifying an object on Cloudflare R2 (#1332) (Philip C Nilsson)
  • Fixed Cloudfront invalidation issue for paths with wildcard or special characters
  • Fixed Cloudfront crash because of error reporting for retries
  • Fixed Cloudfront "unable to parse URL" error (#1292)
  • Improved the handling of "empty" files on the remote side to sync with local folders
  • Improved "abortmp" command by requiring an object to avoid bad accidents when using Ceph (Joshua Haas)
  • Improved file download by retrying when encountering SlowDown or TooManyRequests errors (Robin Geiger)
  • Improved error messages in case of connection error or host unreachable
  • Improved error messages to be more explicit for upload errors after retries
  • Improved remote2local attributes setting code
  • Improved remote2local with more explicit error messages when setting attributes (#1288)
  • Improved remote2local output messages by using the "mkdir" prefix instead of "make dir"
  • Improved the SortedDict class
  • Improved run-test.py by using "--include" when calling Curl instead of "-include" (Matthew James Kraai)
  • Improved GitHub CI by enabling pip cache in actions/setup-python (Anton Yakutovich)
  • Improved GitHub CI by adding a "codespell" check on push and PRs (Yaroslav Halchenko)
  • Updated GitHub CI tests to use more recent versions of Minio and Python
  • Upgraded GitHub actions (Anton Yakutovich)
  • Cleanup and update of copyright headers, docs, comments and setup.py
  • Cleanup to fix "invalid escape sequence" syntax warnings
  • Many other bug fixes and cleanups

New Contributors

Full Changelog: v2.3.0...v2.4.0

v2.3.0

03 Oct 15:46
v2.3.0
bff5ad5
Compare
Choose a tag to compare
  • Added "getnotification", "setnotification", and "delnotification" commands for notification policies (hrchu)
  • Added support for AWS_STS_REGIONAL_ENDPOINTS (#1218, #1228) (Johan Lanzrein)
  • Added ConnectionRefused [111] exit code to handle connection errors (Salar Nosrati-Ershad)
  • Added support for IMDSv2. Should work automatically on ec2 (Anthony Foiani)
  • Added --list-allow-unordered to list objects unordered. Only supported by Ceph based s3-compatible services (#1269) (Salar Nosrati-Ershad)
  • Fixed --exclude dir behavior for python >= 3.6 (Daniil Tararukhin)
  • Fixed Cloudfront invalidate retry issue (Yuan-Hsiang Lee)
  • Fixed 0 byte cache files crashing s3cmd (#1234) (Carlos Laviola)
  • Fixed --continue behavior for the "get" command (#1009) (Anton Ustyugov)
  • Fixed unicode issue with fixbucket (#1259)
  • Fixed CannotSendRequest and ConnectionRefusedError errors at startup (#1261)
  • Fixed error reporting for object info when the object does not exist
  • Fixed "setup.py test" to do nothing to avoid failure that could be problematic for distribution packaging (#996)
  • Improved expire command to use Rule/Filter/Prefix for LifecycleConfiguration (#1247)
  • Improved PASS/CHECK/INCLUDE/EXCLUDE debug log messages
  • Improved setup.py with python 3.9 and 3.10 support info(Ori Avtalion)
  • Many other bug fixes

v2.2.0

27 Sep 02:24
v2.2.0
5db139e
Compare
Choose a tag to compare
  • Added support for metadata modification of files bigger than 5 GiB
  • Added support for remote copy of files bigger than 5 GiB using MultiPart copy (Damian Martinez, Florent Viard)
  • Added progress info output for multipart copy and current-total info in output for cp, mv and modify
  • Added support for all special/foreign character names in object names to cp/mv/modify
  • Added support for SSL authentication (Aleksandr Chazov)
  • Added the http error 429 to the list of retryable errors (#1096)
  • Added support for listing and resuming of multipart uploads of more than 1000 parts (#346)
  • Added time based expiration for idle pool connections in order to avoid random broken pipe errors (#1114)
  • Added support for STS webidentity authentication (ie AssumeRole and AssumeRoleWithWebIdentity) (Samskeyti, Florent Viard)
  • Added support for custom headers to the mb command (#1197) (Sébastien Vajda)
  • Improved MultiPart copy to preserve acl and metadata of objects
  • Improved the server errors catching and reporting for cp/mv/modify commands
  • Improved resiliency against servers sending garbage responses (#1088, #1090, #1093)
  • Improved remote copy to have consistent copy of metadata in all cases: multipart or not, aws or not
  • Improved security by revoking public-write acl when private acl is set (#1151) (ruanzitao)
  • Improved speed when running on an EC2 instance (#1117) (Patrick Allain)
  • Reduced connection_max_age to 5s to avoid broken pipes as AWS closes https conns after around 6s (#1114)
  • Ensure that KeyboardInterrupt are always properly raised (#1089)
  • Changed sized of multipart copy chunks to 1 GiB
  • Fixed ValueError when using more than one ":" inside add_header in config file (#1087)
  • Fixed extra label issue when stdin used as source of a MultiPart upload
  • Fixed remote copy to allow changing the mime-type (ie content-type) of the new object
  • Fixed remote_copy to ensure that meta-s3cmd-attrs will be set based on the real source and not on the copy source
  • Fixed deprecation warnings due to invalid escape sequences (Karthikeyan Singaravelan)
  • Fixed getbucketinfo that was broken when the bucket lifecycle uses the filter element (Liu Lan)
  • Fixed RestoreRequest XML namespace URL (#1203) (Akete)
  • Fixed PARTIAL exit code that was not properly set when needed for object_get (#1190)
  • Fixed a possible infinite loop when a file is truncated during hashsum or upload (#1125) (Matthew Krokosz, Florent Viard)
  • Fixed report_exception wrong error when LANG env var was not set (#1113)
  • Fixed wrong wiki url in error messages (Alec Barrett)
  • Py3: Fixed an AttributeError when using the "files-from" option
  • Py3: Fixed compatibility issues due to the removal of getchildren() from ElementTree in python 3.9 (#1146, #1157, #1162, # 1182, #1210) (Ondřej Budai)
  • Py3: Fixed compatibility issues due to the removal of encodestring() in python 3.9 (#1161, #1174) (Kentaro Kaneki)
  • Fixed a crash when the AWS_ACCESS_KEY env var is set but not AWS_SECRET_KEY (#1201)
  • Cleanup of check_md5 (Riccardo Magliocchetti)
  • Removed legacy code for dreamhost that should be necessary anymore
  • Migrated CI tests to use github actions (Arnaud J)
  • Improved README with a link to INSTALL.md (Sia Karamalegos)
  • Improved help content (Dmitrii Korostelev, Roland Van Laar)
  • Improvements for setup and build configurations
  • Many other bug fixes

v2.1.0

07 Apr 22:34
v2.1.0
cdc2e26
Compare
Choose a tag to compare
  • Changed size reporting using k instead of K as it a multiple of 1024 (#956)
  • Added "public_url_use_https" config to generate public url using https (#551, #666) (Jukka Nousiainen)
  • Added option to make connection pooling configurable and improvements (Arto Jantunen)
  • Added support for path-style bucket access to signurl (Zac Medico)
  • Added docker configuration and help to run test cases with multiple Python versions (Doug Crozier)
  • Relaxed limitation on special chars for --add-header key names (#1054)
  • Fixed all regions that were automatically converted to lower case (Harshavardhana)
  • Fixed size and alignment of DU and LS output reporting (#956)
  • Fixes for SignatureDoesNotMatch error when host port 80 or 443 is specified, due to stupid servers (#1059)
  • Fixed the useless retries of requests that fail because of ssl cert checks
  • Fixed a possible crash when a file disappears during cache generation (#377)
  • Fixed unicode issues with IAM (#987)
  • Fixed unicode errors with bucked Policy/CORS requests (#847) (Alex Offshore)
  • Fixed unicode issues when loading aws_credential_file (#989)
  • Fixed an issue with the tenant feature of CephRGW. Url encode bucket_name for path-style requests (#1080)
  • Fixed signature v2 always used when bucket_name had special chars (#1081)
  • Allow to use signature v4 only, even for commands without buckets specified (#1082)
  • Fixed small open file descriptor leaks.
  • Py3: Fixed hash-bang in headers to not force using python2 when setup/s3cmd/run-test scripts are executed directly.
  • Py3: Fixed unicode issues with Cloudfront (#1006)
  • Py3: Fixed http.client.RemoteDisconnected errors (#1014) (Ryan Huddleston)
  • Py3: Fixed 'dictionary changed size during iteration' error when using a cache-file (#945) (Doug Crozier)
  • Py3: Fixed the display of file sizes (Vlad Presnyak)
  • Py3: Python 3.8 compatibility fixes (Konstantin Shalygin)
  • Py2: Fixed unicode errors sometimes crashing remote2remote sync (#847)
  • Added s3cmd.egg-info to .gitignore (Philip Dubé)
  • Improved run-test script to not use hard-coded bucket names(#1066) (Doug Crozier)
  • Renamed INSTALL to INSTALL.md and improvements (Nitro, Prabhakar Gupta)
  • Improved the restore command help (Hrchu)
  • Updated the storage-class command help with the recent aws s3 classes (#1020)
  • Fixed typo in the --continue-put help message (Pengyu Chen)
  • Fixed typo (#1062) (Tim Gates)
  • Improvements for setup and build configurations
  • Many other bug fixes

v2.0.2

15 Jul 13:08
v2.0.2
Compare
Choose a tag to compare
  • Fixed unexpected timeouts encountered during requests or transfers due to AWS strange connection short timeouts (#941)
  • Fixed a throttle issue slowing down too much transfers in some cases (#913)
  • Added support for $AWS_PROFILE (#966) (Taras Postument)
  • Added clarification comment for the socket_timeout configuration value OS limit
  • Avoid distutils usage at runtime (Matthias Klose)
  • Python 2 compatibility: Fixed import error of which with fallback code (Gianfranco Costamagna)
  • Fixed Python 3 bytes string encoding when getting IAM credentials (Alexander Allakhverdiyev)
  • Fixed handling of config tri-state bool values (like acl_public) (Brian C. Lane)
  • Fixed V2 signature when restore command is used (Jan Kasiak)
  • Fixed setting full_control on objects with public read access (Matthew Vernon)
  • Fixed a bug when only one path is supplied with Cloudfront. (Mikael Svensson)
  • Fixed signature errors with 'modify' requests (Radek Simko)
  • Fixes #936 - Fix setacl command exception (Robert Moucha)
  • Fixes error reporting if deleting a source object failed after a move (#929)
  • Many other bug fixes (#525, #933, #940, #947, #957, #958, #960, #967)

Important info:
AWS S3 doesn't allow anymore uppercases and underscores in bucket names since march 1, 2018.

v2.0.1

21 Oct 18:38
Compare
Choose a tag to compare
  • Support for Python 3 is now stable
  • Fixed signature issues due to upper cases in hostname (#920)
  • Improved support for Minio Azure gateway (Julien Maitrehenry, Harshavardhana)
  • Added signurl_use_https option to use https prefix for signurl (Julien Recurt)
  • Fixed a lot of remaining issues and regressions for Python 3 (#922, #921, #908)
  • Fixed --configure option with Python 3
  • Fixed non string cmdline parameters being ignored
  • Windows support fixes (#922)
  • Don't force anymore to have a / on last parameter for the "modify" command (#886)
  • Removed the python3 support warning
  • Detect and report error 403 in getpolicy for info command (#894)
  • Added a specific error message when getting policy by non owner (#885)
  • Many other bug fixes (#905, #892, #890, #888, #889, #887)

v2.0.0

26 Jun 22:26
Compare
Choose a tag to compare
  • Added support for Python 3 (Shaform, Florent Viard)
  • Added getlifecycle command (Daniel Gryniewicz)
  • Added --cf-inval for invalidating multiple CF distributions (Joe Mifsud)
  • Added --limit to "ls" and "la" commands to return the specified number of objects (Masashi Ozawa)
  • Added --token-refresh and --no-token-refresh and get the access token from the environment (Marco Jakob)
  • Added --restore-priority and --restore-days for S3 Glacier (Robert Palmer)
  • Fixed requester pays header with HEAD requests (Christian Rodriguez)
  • Don't allow mv/cp of multiple files to single file (Guy Gur-Ari)
  • Generalize wildcard certificate forgiveness (Mark Titorenko)
  • Multiple fixes for SSL connections and proxies
  • Added support for HTTP 100-CONTINUE
  • Fixes for s3-like servers
  • Big cleanup and many unicode fixes
  • Many other bug fixes

V1.6.1

20 Jan 18:17
Compare
Choose a tag to compare
  • Added --host and --host-bucket
  • Added --stats
  • Fix for newer python 2.7.x SSL library updates
  • Many other bug fixes

v1.6.0

18 Sep 04:57
Compare
Choose a tag to compare

s3cmd-1.6.0 - 2015-09-18

  • Support signed URL content disposition type
  • Added 'ls -l' long listing including storage class
  • Added --limit-rate=RATE
  • Added --server-side-encryption-kms-id=KEY_ID
  • Added --storage-class=CLASS
  • Added --requester-pays, [payer] command
  • Added --[no-]check-hostname
  • Added --stop-on-error, removed --ignore-failed-copy
  • Added [setcors], [delcors] commands
  • Added support for cn-north-1 region hostname checks
  • Output strings may have changed. Scripts calling s3cmd expecting
    specific text may need to be updated.
  • HTTPS is now the default
  • Many unicode fixes
  • Many other bug fixes

v1.5.2

18 Mar 02:17
Compare
Choose a tag to compare
Tag v1.5.2