0.1.25 scylla-bench image pull error while executing some longevity tests for Scylla Operator (disrupt_truncate_large_partition nemesis)

[grzywin]

During SCT longevity tests execution for Scylla Operator one of the nemesis (disrupt_truncate_large_partition) is failing with error:
https://argus.scylladb.com/tests/scylla-cluster-tests/cc29b0a8-f215-4d69-abb8-fc6698b8849d

Traceback (most recent call last):
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5720, in wrapper
    result = method(*args[1:], **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2189, in disrupt_truncate_large_partition
    self.tester.verify_stress_thread(bench_thread, error_handler=self._nemesis_stress_failure_handler)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 2354, in verify_stress_thread
    error_handler(thread_pool, errors)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2141, in _nemesis_stress_failure_handler
    raise NemesisStressFailure(
sdcm.exceptions.NemesisStressFailure: Aborting 'SisyphusMonkey' nemesis as stress command failed with the following errors:
 on node 'sct-loaders-eu-west-2-0': ['Stress command execution failed with: (400)\nReason: Bad Request\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e37a585a-860f-42d6-983b-6dc445870981\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Fri, 02 May 2025 08:03:47 GMT\', \'Content-Length\': \'230\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \\\\"loader\\\\" in pod \\\\"sct-loaders-eu-west-2-0-pod-7\\\\" is waiting to start: trying and failing to pull image","reason":"BadRequest","code":400}\\n\'\n']
 on node 'sct-loaders-eu-west-2-1': ['Stress command execution failed with: (400)\nReason: Bad Request\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'21b319ae-a5a8-45d3-a134-a90848787594\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Fri, 02 May 2025 08:03:51 GMT\', \'Content-Length\': \'230\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \\\\"loader\\\\" in pod \\\\"sct-loaders-eu-west-2-1-pod-4\\\\" is waiting to start: trying and failing to pull image","reason":"BadRequest","code":400}\\n\'\n']

After some futher investigation I found out in logs that there is some problem with pulling scylla-bench 0.1.25 image:

< t:2025-04-22 10:22:33,892 f:base.py         l:225  c:LocalCmdRunner       p:DEBUG > <ip-35-177-15-185>:     state:
< t:2025-04-22 10:22:33,892 f:base.py         l:225  c:LocalCmdRunner       p:DEBUG > <ip-35-177-15-185>:       waiting:
< t:2025-04-22 10:22:33,892 f:base.py         l:225  c:LocalCmdRunner       p:DEBUG > <ip-35-177-15-185>:         message: 'Back-off pulling image "scylladb/scylla-bench:0.1.25": ErrImagePull:
< t:2025-04-22 10:22:33,892 f:base.py         l:225  c:LocalCmdRunner       p:DEBUG > <ip-35-177-15-185>:           rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/scylladb/scylla-bench:0.1.25":
< t:2025-04-22 10:22:33,892 f:base.py         l:225  c:LocalCmdRunner       p:DEBUG > <ip-35-177-15-185>:           no match for platform in manifest: not found'
< t:2025-04-22 10:22:33,892 f:base.py         l:225  c:LocalCmdRunner       p:DEBUG > <ip-35-177-15-185>:         reason: ImagePullBackOff

As adviced by @dimakr I added extra param to the test run to use scylla-bench 0.1.24 instead of 0.1.25 and after this change the nemesis passed without error, so there is probably something wrong with 0.1.25 image.
SCT_STRESS_IMAGE={"scylla-bench":"scylladb/hydra-loaders:scylla-bench-v0.1.24"}
Passing run:
https://argus.scylladb.com/tests/scylla-cluster-tests/6550b2c7-cda1-4db6-9058-ca9f3984a1a6

[dimakr]

< t:2025-04-22 10:22:33,892 f:base.py l:225 c:LocalCmdRunner p:DEBUG > : no match for platform in manifest: not found'

@CodeLieutenant could it be that after the new CI release process was implemented we're missing some arch platform, specific for k8s backend(s)?

[CodeLieutenant]

< t:2025-04-22 10:22:33,892 f:base.py l:225 c:LocalCmdRunner p:DEBUG > : no match for platform in manifest: not found'

@CodeLieutenant could it be that after the new CI release process was implemented we're missing some arch platform, specific for k8s backend(s)?

Yes, that could be a problem

What arch we need, current CI is building for X86_64 and ARM v8

[dimakr]

Yes, that could be a problem

What arch we need, current CI is building for X86_64 and ARM v8

@CodeLieutenant From what I see in the test logs, the AWS c6i.xlarge type of instance is used:

10:42:40  < t:2025-05-02 08:42:40,270 f:sct_config.py   l:2218 c:sdcm.sct_config      p:INFO  > instance_type_loader: c6i.xlarge

This is X86_64, which we have covered in the CI.

Also checked the manifests for image v0.1.25 (failing one) and v0.1.24 (which is OK):

❯ docker manifest inspect docker.io/scylladb/scylla-bench:0.1.25
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:bb952eb90a348e967a7fca3f2fab7ca2aa3e332ff3be815dda0036542d80f2c7",
         "platform": {
            "architecture": "amd64",
            "os": "linux",
            "variant": "v3"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:404bac7d1125ed1b71ae768cbc69bb66aa41da93b72643cb4461dbe27fb3b562",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

❯ docker manifest inspect scylladb/hydra-loaders:scylla-bench-v0.1.24
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 2021,
         "digest": "sha256:6bcfc9bd02ea58d975fffd08ee5b5d3d3d9d155c03000cd08e8080d31b7cb8b2",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 2021,
         "digest": "sha256:a9838a974303a2f5e2bbfd42bcaa9826c22b856b91bbebb97827a3174f2bc585",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:80e5124fb4ae55284abf071316af28bab207f7f09238f518a0bf06df34750fdf",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:78d11b0979a245f44db80712bd5a62967fd205e768197f59ba8f146e607ef4c5",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

The supported platforms look similar, with only minor difference that in v0.1.25 the amd64 platform entry specifies "variant": "v3".
Could it be the cause of runtime failing to match the platform?

[CodeLieutenant]

Could be

[fruch]

Could be

mediaType is different, working one uses oci images, and non working ones is using docker.distribution.manifest.list

[CodeLieutenant]

S-b 0.1.x is out of support, closing this issue. If the same issue arrises we can reopen it

[fruch]

Also we removed this v3 variant, @abramche might get to confirm it at some point