[Question] Switch Docker image for running registry (due to Docker Hub introduced pull limits)

[mzaikin]

Hi All:

We got into a complicated situation: we used for about half-year Apicurio Registry (with JPA for persistence) on IBM Public cloud OpenShift. Apicurio was installed via OperatorHub (Apicurio Registry Operator 0.0.4-v1.3.2.final provided by Apicurio).

At some point there started image errors due to Docker Hub pull limits (added Nov 2020), and we got cluster overfilled with Crashed pods, Pods number exceeded the limit (110 PODs per worker node), and now the whole cluster working unstable.

The typical error is like that:

labor-schema-registry-deployment-bkddz-1-xt56z
NamespaceNSkafka-labor
Nov 12, 2021, 12:22 AM
Generated from kubelet on 10.183.105.11
5 times in the last 43 minutes
Failed to pull image "docker.io/apicurio/apicurio-registry-jpa@sha256:44eeddd3562ca3ac3f5fb440e5f68880ab9b4d6caa0b34541a9178c4a039b1f4": rpc error: code = Unknown desc = Error reading manifest sha256:44eeddd3562ca3ac3f5fb440e5f68880ab9b4d6caa0b34541a9178c4a039b1f4 in docker.io/apicurio/apicurio-registry-jpa: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

labor-schema-registry-deployment-w2p92-1-nbs7b
NamespaceNSkafka-labor
Nov 12, 2021, 1:02 AM
Generated from kubelet on 10.183.105.11
13 times in the last 1 day
Error: ErrImagePull
PodPkafka-staging-labor-cluster-kafka-0
NamespaceNSkafka-labor
Nov 12, 2021, 1:02 AM
Generated from kubelet on 10.183.105.2
2087 times in the last 1 day
Readiness probe failed:

I tried to pull docker.io image to OpenShift local ImageStream and resolve by editing YAML for the created ApicurioRegistry instance:

spec:
  configuration:
    dataSource:
      password: xxxx
      url: 'jdbc:postgresql://postgresql.kafka-labor.svc:5432/apicuriodb'
      userName: xxxx
  deployment:
    host: >-
      labor-schema-registry.kafka-labor.bmslaborcluster-wdc04-b-78fe44b0e75a09a49dbea67133e7b33a-0000.us-east.containers.appdomain.cloud
  deployment.replicas: 1
  image:
    name: >-
      image-registry.openshift-image-registry.svc:5000/kafka-labor/apicurio-registry-jpa@sha256:44eeddd3562ca3ac3f5fb440e5f68880ab9b4d6caa0b34541a9178c4a039b1f4
    persistence: jpa

But this image is ignored and I still see in Events attempts to reach docker.io

Question: How can I refresh all installed Apicurio Schema Registry related objects (replication controllers, deployments, etc..) to switch to new Docker image to avoid pull errors?

[jsenko]

Hi @mzaikin

We have an Apicurio organization in quay.io as well, so you can try using quay.io/apicurio/apicurio-registry-jpa:1.3.2.Final.

The 0.x Operator supports specifying the image in the CR (as you're doing), so you should be able to set spec/image/name, but persistence option should be defined in spec/configuration/persistence instead of spec/image/persistence. Please check if that solves the issue.

Second option is to edit the Deployment of the Operator pods, and change the values in the environment variables that specify the images, by replacing docker.io with quay.io.