S3 Upload Error (Ceph) #431
Description
Version ABS: Aerospike Backup Service version=v3.3.0 commit=e48a13a0cc431384a5037bb428a1039fd552fe89 buildTime=2025-08-21T11:15:34Z
Version Aerospike: Aerospike Community Edition build 7.0.0.9
Configuration file ABS:
service: http: address: 0.0.0.0 port: 8080 logger: level: TRACE format: PLAIN stdout-writer: true aerospike-clusters: uat_n03: max-parallel-scans: 1 seed-nodes: - host-name: 10.10.10.10 port: 3002 storage: minio: s3-storage: access-key-id: "test" # Optional. Access Key ID for S3 StaticCredentialsProvider. bucket: "uat-backup-remote-backup" max-async-connections: 1 min-part-size: 50_000_000 path: s3://uat-backup-remote-backup/backup/ # Mandatory. The root path for the backup repository within the bucket. s3-endpoint-override: "https://s3.corp.ru" # Optional. Alternative endpoint for the S3 SDK to communicate (e.g., http://host.docker.internal:9000). s3-log-level: TRACE # Optional. Log level of the AWS S3 SDK (OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE). Default: FATAL. # s3-profile: default s3-region: "ru-msk-avt-1" # Mandatory. The S3 region where the bucket resides (e.g., eu-central-1). secret-access-key: "test2" # Optional. Secret Access Key for S3 StaticCredentialsProvider. storage1: local-storage: path: /backup backup-policies: defaultPolicy_1: parallel: 1 sealed: true file-limit: 1000 retry-policy: base-timeout: 3_000 max-retries: 10 multiplier: 1 retention: full: 10 incremental: 10 backup-routines: fullBackup1: backup-policy: defaultPolicy_1 source-cluster: uat_n03 storage: minio interval-cron: '@hourly' namespaces: [namespace1] fullBackup2: backup-policy: defaultPolicy_1 source-cluster: uat_n03 storage: storage1 interval-cron: '@hourly' namespaces: []
After manually starting backup creation :
curl -k -v -X POST http://localhost:8080/v1/backups/schedule/fullBackup1
Error messages in the logs:
time=2025-09-03T08:20:24.210Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/config_applier.go:67 msg="Delete scheduled jobs" keys=[] time=2025-09-03T08:20:24.210Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/backup_scheduler.go:125 msg=Schedule routine=fullBackup1 type=full nextRun=2025-09-03T09:00:00.000Z time=2025-09-03T08:20:24.210Z level=DEBUG source=github.com/reugn/[email protected]/quartz/scheduler.go:345 msg="Successfully added job" key=scheduled_fullBackup1-full time=2025-09-03T08:20:24.211Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/backup_scheduler.go:125 msg=Schedule routine=fullBackup2 type=full nextRun=2025-09-03T09:00:00.000Z time=2025-09-03T08:20:24.211Z level=DEBUG source=github.com/reugn/[email protected]/quartz/scheduler.go:345 msg="Successfully added job" key=scheduled_fullBackup2-full time=2025-09-03T08:20:24.211Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/running_backups_registry.go:91 msg="Start backup history synchronization" routines="[fullBackup2 fullBackup1]" len=2 time=2025-09-03T08:20:24.211Z level=TRACE source=github.com/reugn/[email protected]/quartz/scheduler.go:621 msg="Next tick" job=scheduled_fullBackup1-full after=39m35.788169764s time=2025-09-03T08:20:24.213Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/running_backups_registry.go:142 msg="Last backup time scan completed" routine=fullBackup2 duration=1.040989ms lastRun="Full: 2025-09-03T08:16:09Z, Incremental: never" time=2025-09-03T08:20:25.932Z level=DEBUG source=github.com/reugn/[email protected]/quartz/scheduler.go:345 msg="Successfully added job" key=ad-hoc_fullBackup1-adhoc-1756887625932 time=2025-09-03T08:20:25.932Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/internal/server/middleware/logger.go:74 msg="request success" method=POST path=/v1/backups/schedule/fullBackup1 status=202 duration=207.564µs ip=127.0.0.1:56456 time=2025-09-03T08:20:25.932Z level=TRACE source=github.com/reugn/[email protected]/quartz/scheduler.go:573 msg="Interrupted waiting for next tick" time=2025-09-03T08:20:25.932Z level=TRACE source=github.com/reugn/[email protected]/quartz/scheduler.go:621 msg="Next tick" job=ad-hoc_fullBackup1-adhoc-1756887625932 after=0s time=2025-09-03T08:20:25.933Z level=TRACE source=github.com/reugn/[email protected]/quartz/scheduler.go:569 msg=Tick time=2025-09-03T08:20:25.933Z level=INFO source=github.com/reugn/[email protected]/quartz/scheduler.go:747 msg="Job exited the execution loop" key=ad-hoc_fullBackup1-adhoc-1756887625932 error="trigger has expired" time=2025-09-03T08:20:25.933Z level=DEBUG source=github.com/reugn/[email protected]/quartz/scheduler.go:644 msg="Job is about to be executed" key=ad-hoc_fullBackup1-adhoc-1756887625932 time=2025-09-03T08:20:25.933Z level=TRACE source=github.com/reugn/[email protected]/quartz/scheduler.go:621 msg="Next tick" job=scheduled_fullBackup1-full after=39m34.06678441s time=2025-09-03T08:20:34.210Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/aerospike/client_manager.go:214 msg="Aerospike client closed" hosts="[{Name:10.10.10.10 TLSName: Port:3002}]" len=0 id=447d934a32513e6d911d9813eea5618ad2af6b2041c71eff37f6cab0368db7e9 time=2025-09-03T08:20:34.569Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/running_backups_registry.go:142 msg="Last backup time scan completed" routine=fullBackup1 duration=10.357256157s lastRun="Full: never, Incremental: never" time=2025-09-03T08:20:34.569Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/running_backups_registry.go:111 msg="History synchronization completed" routines="[fullBackup2 fullBackup1]" len=2 duration=10.357673695s time=2025-09-03T08:20:36.591Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/aerospike/client_manager.go:134 msg="Created new backup client" len=1 id=447d934a32513e6d911d9813eea5618ad2af6b2041c71eff37f6cab0368db7e9 time=2025-09-03T08:20:46.913Z level=DEBUG source=github.com/aerospike/[email protected]/internal/metrics/collector.go:96 msg=metrics backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.name=rps backup.message="backup metrics 4fa7375f-395f-491c-8616-5c258a0a6eed" backup.enabled=true time=2025-09-03T08:20:46.913Z level=DEBUG source=github.com/aerospike/[email protected]/internal/metrics/collector.go:96 msg=metrics backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.name=kbps backup.message="backup metrics 4fa7375f-395f-491c-8616-5c258a0a6eed" backup.enabled=true time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/processor_record_reader.go:53 msg="created new records reader processor" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:71 msg="created new file writer processor" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/shared.go:47 msg="job starting" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:133 msg="created new file writers" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writersNumber=1 time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=0 backup.writer.id=df3a1c6a-6836-45f2-ab9a-b5841c979834 backup.writer.type=token time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=af04853a-2739-4e18-af08-07a863df5d5c backup.writer.type=token_stats time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:98 msg="created data writers" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.count=1 time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/sindex_reader.go:47 msg="created new sindex reader" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=9e09e0aa-8797-4774-bdb4-7d66e9d65cf8 backup.reader.type=sindex time=2025-09-03T08:20:46.914Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=96387efe-2bd9-4908-b9e6-fc4053a7e59c backup.writer.type=token time=2025-09-03T08:20:46.915Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=c1026048-f66e-4451-949d-6e6616043d02 backup.writer.type=token_stats time=2025-09-03T08:20:46.915Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/sindex_reader.go:63 msg="fetching all secondary indexes" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=9e09e0aa-8797-4774-bdb4-7d66e9d65cf8 backup.reader.type=sindex time=2025-09-03T08:20:47.092Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=96387efe-2bd9-4908-b9e6-fc4053a7e59c backup.writer.type=token time=2025-09-03T08:20:47.092Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:168 msg="closed token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=96387efe-2bd9-4908-b9e6-fc4053a7e59c backup.writer.type=token time=2025-09-03T08:20:47.092Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/udf_reader.go:47 msg="created new udf reader" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=434c9b72-3e83-43df-a771-f6e6dff20e9c backup.reader.type=udf time=2025-09-03T08:20:47.092Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=a1846442-f1d4-4974-ac28-a3eb249b20a3 backup.writer.type=token time=2025-09-03T08:20:47.093Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=dfb25486-3b37-425b-bc69-19b90952c66a backup.writer.type=token_stats time=2025-09-03T08:20:47.093Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/udf_reader.go:62 msg="fetching all UDFs" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=434c9b72-3e83-43df-a771-f6e6dff20e9c backup.reader.type=udf time=2025-09-03T08:20:47.521Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=a1846442-f1d4-4974-ac28-a3eb249b20a3 backup.writer.type=token time=2025-09-03T08:20:47.521Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:168 msg="closed token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=a1846442-f1d4-4974-ac28-a3eb249b20a3 backup.writer.type=token time=2025-09-03T08:20:47.521Z level=DEBUG source=github.com/aerospike/[email protected]/internal/processors/void_time.go:41 msg="created new VoidTime processor" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.processor.id=0342e32c-7f86-4a89-a489-e71dd46e7882 backup.processor.type=void_time time=2025-09-03T08:20:47.521Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:122 msg="created new aerospike record reader" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=0e5e0d96-3b20-4593-adfc-09210d49ddb3 backup.reader.type=record backup.toTime=2025-09-03T08:20:25.932Z backup.partitionFilter=0-4096 backup.scanLimiter=true backup.namespace=namespace1 backup.setList=[] time=2025-09-03T08:20:47.521Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:139 msg="scan started" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=0e5e0d96-3b20-4593-adfc-09210d49ddb3 backup.reader.type=record time=2025-09-03T08:20:47.522Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:271 msg="starting partition scan" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=0e5e0d96-3b20-4593-adfc-09210d49ddb3 backup.reader.type=record backup.set="" backup.begin=0 backup.count=4096 time=2025-09-03T08:21:18.889Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:157 msg="scan finished" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=0e5e0d96-3b20-4593-adfc-09210d49ddb3 backup.reader.type=record time=2025-09-03T08:21:33.695Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.writer=0 backup.writer.id=df3a1c6a-6836-45f2-ab9a-b5841c979834 backup.writer.type=token time=2025-09-03T08:21:46.841Z level=DEBUG source=github.com/aerospike/[email protected]/shared.go:51 msg="job failed" backup.client.id=635 backup.handler.id=4fa7375f-395f-491c-8616-5c258a0a6eed backup.handler.type=backup backup.handler.storage=s3 backup.error="write error: failed to write data: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx00000c47d3655df02cf3f-0068b7fa7e-1d36552-ru-msk-avt-1.s3.corp.ru, HostID: 1d36552-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError, close error: failed to close token writer: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx000009dbc62b77efadce1-0068b7fa8d-1d35681-ru-msk-avt-1.s3.corp.ru, HostID: 1d35681-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError" time=2025-09-03T08:21:47.180Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/backup_backend_service.go:324 msg="Deleted folder" path=fullBackup1/backup/1756887625932 routine=fullBackup1 time=2025-09-03T08:21:47.180Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/retry_service.go:59 msg="Execution failed, retrying..." routine=fullBackup1 label=backup attempt=1 maxAttempts=10 retryInterval=3s error="backup failed: write error: failed to write data: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx00000c47d3655df02cf3f-0068b7fa7e-1d36552-ru-msk-avt-1.s3.corp.ru, HostID: 1d36552-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError, close error: failed to close token writer: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx000009dbc62b77efadce1-0068b7fa8d-1d35681-ru-msk-avt-1.s3.corp.ru, HostID: 1d35681-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError" time=2025-09-03T08:21:50.343Z level=DEBUG source=github.com/aerospike/[email protected]/internal/metrics/collector.go:96 msg=metrics backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.name=rps backup.message="backup metrics 0ee25f2d-d544-495d-8151-bb250a7082d8" backup.enabled=true time=2025-09-03T08:21:50.343Z level=DEBUG source=github.com/aerospike/[email protected]/internal/metrics/collector.go:96 msg=metrics backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.name=kbps backup.message="backup metrics 0ee25f2d-d544-495d-8151-bb250a7082d8" backup.enabled=true time=2025-09-03T08:21:50.343Z level=DEBUG source=github.com/aerospike/[email protected]/processor_record_reader.go:53 msg="created new records reader processor" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:21:50.343Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:71 msg="created new file writer processor" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:21:50.343Z level=DEBUG source=github.com/aerospike/[email protected]/shared.go:47 msg="job starting" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:21:50.343Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:133 msg="created new file writers" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writersNumber=1 time=2025-09-03T08:21:50.344Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=0 backup.writer.id=0df2d98b-d72f-46b8-b168-c843cd5ef79b backup.writer.type=token time=2025-09-03T08:21:50.344Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=cd3d1e18-a8f4-43b0-99da-d5e68d11d1c6 backup.writer.type=token_stats time=2025-09-03T08:21:50.344Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:98 msg="created data writers" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.count=1 time=2025-09-03T08:21:50.344Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/sindex_reader.go:47 msg="created new sindex reader" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=f2b5a322-a9ee-4308-9f37-d07d713b66f7 backup.reader.type=sindex time=2025-09-03T08:21:50.344Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=177d84c9-84ed-48cc-aa95-d3e4e0b69de9 backup.writer.type=token time=2025-09-03T08:21:50.344Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=dce45633-137b-4245-8fc1-0591ede9d0b8 backup.writer.type=token_stats time=2025-09-03T08:21:50.344Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/sindex_reader.go:63 msg="fetching all secondary indexes" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=f2b5a322-a9ee-4308-9f37-d07d713b66f7 backup.reader.type=sindex time=2025-09-03T08:21:50.513Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=177d84c9-84ed-48cc-aa95-d3e4e0b69de9 backup.writer.type=token time=2025-09-03T08:21:50.513Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:168 msg="closed token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=177d84c9-84ed-48cc-aa95-d3e4e0b69de9 backup.writer.type=token time=2025-09-03T08:21:50.513Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/udf_reader.go:47 msg="created new udf reader" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=4bfbaca8-1a1b-4e22-ad72-170df17709bc backup.reader.type=udf time=2025-09-03T08:21:50.513Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=4cdf9ec0-e9d5-4ea9-952f-e3ac6f35e8f2 backup.writer.type=token time=2025-09-03T08:21:50.513Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=68911e09-b87f-4fa6-8487-bf969a453195 backup.writer.type=token_stats time=2025-09-03T08:21:50.513Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/udf_reader.go:62 msg="fetching all UDFs" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=4bfbaca8-1a1b-4e22-ad72-170df17709bc backup.reader.type=udf time=2025-09-03T08:21:50.762Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=4cdf9ec0-e9d5-4ea9-952f-e3ac6f35e8f2 backup.writer.type=token time=2025-09-03T08:21:50.763Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:168 msg="closed token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=4cdf9ec0-e9d5-4ea9-952f-e3ac6f35e8f2 backup.writer.type=token time=2025-09-03T08:21:50.763Z level=DEBUG source=github.com/aerospike/[email protected]/internal/processors/void_time.go:41 msg="created new VoidTime processor" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.processor.id=0650de1f-0886-473b-8e00-00e4b7182f0b backup.processor.type=void_time time=2025-09-03T08:21:50.763Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:122 msg="created new aerospike record reader" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=9c2d1594-f4b4-4a44-b70f-00c94f71c487 backup.reader.type=record backup.toTime=2025-09-03T08:20:25.932Z backup.partitionFilter=0-4096 backup.scanLimiter=true backup.namespace=namespace1 backup.setList=[] time=2025-09-03T08:21:50.763Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:139 msg="scan started" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=9c2d1594-f4b4-4a44-b70f-00c94f71c487 backup.reader.type=record time=2025-09-03T08:21:50.763Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:271 msg="starting partition scan" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=9c2d1594-f4b4-4a44-b70f-00c94f71c487 backup.reader.type=record backup.set="" backup.begin=0 backup.count=4096 time=2025-09-03T08:22:05.659Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:157 msg="scan finished" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=9c2d1594-f4b4-4a44-b70f-00c94f71c487 backup.reader.type=record time=2025-09-03T08:22:19.737Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.writer=0 backup.writer.id=0df2d98b-d72f-46b8-b168-c843cd5ef79b backup.writer.type=token time=2025-09-03T08:22:32.009Z level=DEBUG source=github.com/aerospike/[email protected]/shared.go:51 msg="job failed" backup.client.id=635 backup.handler.id=0ee25f2d-d544-495d-8151-bb250a7082d8 backup.handler.type=backup backup.handler.storage=s3 backup.error="write error: failed to write data: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx000001ebf1f8c3064654d-0068b7faad-1d367af-ru-msk-avt-1.s3.corp.ru, HostID: 1d367af-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError, close error: failed to close token writer: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx00000bcb866eee9fcdcda-0068b7fabb-1d356ea-ru-msk-avt-1.s3.corp.ru, HostID: 1d356ea-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError" time=2025-09-03T08:22:32.355Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/backup_backend_service.go:324 msg="Deleted folder" path=fullBackup1/backup/1756887625932 routine=fullBackup1 time=2025-09-03T08:22:32.355Z level=INFO source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/retry_service.go:59 msg="Execution failed, retrying..." routine=fullBackup1 label=backup attempt=2 maxAttempts=10 retryInterval=3s error="backup failed: write error: failed to write data: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx000001ebf1f8c3064654d-0068b7faad-1d367af-ru-msk-avt-1.s3.corp.ru, HostID: 1d367af-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError, close error: failed to close token writer: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx00000bcb866eee9fcdcda-0068b7fabb-1d356ea-ru-msk-avt-1.s3.corp.ru, HostID: 1d356ea-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError" time=2025-09-03T08:22:35.579Z level=DEBUG source=github.com/aerospike/[email protected]/internal/metrics/collector.go:96 msg=metrics backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.name=rps backup.message="backup metrics 71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a" backup.enabled=true time=2025-09-03T08:22:35.579Z level=DEBUG source=github.com/aerospike/[email protected]/internal/metrics/collector.go:96 msg=metrics backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.name=kbps backup.message="backup metrics 71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a" backup.enabled=true time=2025-09-03T08:22:35.579Z level=DEBUG source=github.com/aerospike/[email protected]/processor_record_reader.go:53 msg="created new records reader processor" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:22:35.579Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:71 msg="created new file writer processor" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:22:35.579Z level=DEBUG source=github.com/aerospike/[email protected]/shared.go:47 msg="job starting" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 time=2025-09-03T08:22:35.579Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:133 msg="created new file writers" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writersNumber=1 time=2025-09-03T08:22:35.580Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=0 backup.writer.id=b56fef5c-de78-4674-8317-fb41f33c7acd backup.writer.type=token time=2025-09-03T08:22:35.580Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=99c53252-7b51-43c2-bcf2-913669659145 backup.writer.type=token_stats time=2025-09-03T08:22:35.580Z level=DEBUG source=github.com/aerospike/[email protected]/processor_file_writer.go:98 msg="created data writers" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.count=1 time=2025-09-03T08:22:35.580Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/sindex_reader.go:47 msg="created new sindex reader" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=23291698-ae85-4c88-89c7-200fec7180e6 backup.reader.type=sindex time=2025-09-03T08:22:35.580Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=5036c609-aef3-4be5-aea5-6cbf12bc0a49 backup.writer.type=token time=2025-09-03T08:22:35.580Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=947f28ed-5861-4e11-bfdc-fdcb82e99e60 backup.writer.type=token_stats time=2025-09-03T08:22:35.580Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/sindex_reader.go:63 msg="fetching all secondary indexes" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=23291698-ae85-4c88-89c7-200fec7180e6 backup.reader.type=sindex time=2025-09-03T08:22:35.752Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=5036c609-aef3-4be5-aea5-6cbf12bc0a49 backup.writer.type=token time=2025-09-03T08:22:35.752Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:168 msg="closed token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=sindex backup.writer.id=5036c609-aef3-4be5-aea5-6cbf12bc0a49 backup.writer.type=token time=2025-09-03T08:22:35.752Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/udf_reader.go:47 msg="created new udf reader" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=ef197d7e-9dfa-4aa1-84e5-36fb88779df7 backup.reader.type=udf time=2025-09-03T08:22:35.752Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:121 msg="created new token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=b04f10e5-478b-4093-bc9f-6fb99bdd7e45 backup.writer.type=token time=2025-09-03T08:22:35.753Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:52 msg="created new token stats writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer.id=cf8c098a-077d-4342-8ff4-f160f638e498 backup.writer.type=token_stats time=2025-09-03T08:22:35.753Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/udf_reader.go:62 msg="fetching all UDFs" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=ef197d7e-9dfa-4aa1-84e5-36fb88779df7 backup.reader.type=udf time=2025-09-03T08:22:36.006Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=b04f10e5-478b-4093-bc9f-6fb99bdd7e45 backup.writer.type=token time=2025-09-03T08:22:36.006Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:168 msg="closed token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=udf backup.writer.id=b04f10e5-478b-4093-bc9f-6fb99bdd7e45 backup.writer.type=token time=2025-09-03T08:22:36.007Z level=DEBUG source=github.com/aerospike/[email protected]/internal/processors/void_time.go:41 msg="created new VoidTime processor" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.processor.id=a660ab2b-2b4e-484e-b49b-74868b7f87a0 backup.processor.type=void_time time=2025-09-03T08:22:36.007Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:122 msg="created new aerospike record reader" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=e774be14-a456-408c-bd13-83cc23bffc66 backup.reader.type=record backup.toTime=2025-09-03T08:20:25.932Z backup.partitionFilter=0-4096 backup.scanLimiter=true backup.namespace=namespace1 backup.setList=[] time=2025-09-03T08:22:36.007Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:139 msg="scan started" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=e774be14-a456-408c-bd13-83cc23bffc66 backup.reader.type=record time=2025-09-03T08:22:36.007Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:271 msg="starting partition scan" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=e774be14-a456-408c-bd13-83cc23bffc66 backup.reader.type=record backup.set="" backup.begin=0 backup.count=4096 time=2025-09-03T08:22:46.666Z level=DEBUG source=github.com/aerospike/[email protected]/io/aerospike/record_reader.go:157 msg="scan finished" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.reader.id=e774be14-a456-408c-bd13-83cc23bffc66 backup.reader.type=record time=2025-09-03T08:23:04.369Z level=DEBUG source=github.com/aerospike/[email protected]/token_writers.go:162 msg="try to close token writer" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.writer=0 backup.writer.id=b56fef5c-de78-4674-8317-fb41f33c7acd backup.writer.type=token ^Ctime=2025-09-03T08:23:06.482Z level=DEBUG source=github.com/aerospike/aerospike-backup-service/v3/cmd/backup/main.go:183 msg="Got system signal" time=2025-09-03T08:23:06.482Z level=INFO source=github.com/reugn/[email protected]/quartz/scheduler.go:546 msg="Closing the scheduler" time=2025-09-03T08:23:06.482Z level=INFO source=github.com/reugn/[email protected]/quartz/scheduler.go:577 msg="Exit the execution loop" time=2025-09-03T08:23:06.483Z level=ERROR source=github.com/aerospike/aerospike-backup-service/v3/pkg/service/backup_routine_orchestrator.go:95 msg="Full backup failed" routine=fullBackup1 error="backup failed: namespace namespace1: context canceled" time=2025-09-03T08:23:06.483Z level=DEBUG source=github.com/aerospike/[email protected]/shared.go:51 msg="job failed" backup.client.id=635 backup.handler.id=71011a7c-5f10-4fd9-8ddb-e61ed24dbb3a backup.handler.type=backup backup.handler.storage=s3 backup.error="write error: failed to write data: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: tx000001fd654c1e1542862-0068b7fad6-1d35573-ru-msk-avt-1.s3.corp.ru, HostID: 1d35573-ru-msk-avt-1.s3.corp.ru-ru-msk-avt-1, api error XAmzContentSHA256Mismatch: UnknownError, close error: failed to close token writer: failed to upload part: failed to upload part: operation error S3: UploadPart, https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled"
S3 logs contain errors, part size is less than 5Mb:
Jul 24 21:27:45 prxmskav3 haproxy[4132307]: 10.10.10.102:51074 [24/Jul/2025:21:27:45.041] frontend-data~ backend-s3.corp.ru/cphmskav60.msk.ru 0/0/0/10/10 200 533 - - ---- 433/432/37/1/0 0/0 {s3.corp.ru} "POST /uat-backup-remote-backup/asuatn03/fullBackup1/backup/1753381650000/data/namespace1/0_namespace1_1.asb?uploads= HTTP/1.1"