initImage fails when using hashlocked operator image

[james-callahan]

Report

initImage doesn't match that of operator resulting in failure

More about the problem

I hash-lock (i.e. use a digest) to run the operator, i.e. the operator deployment has image: percona/percona-xtradb-cluster-operator:1.15.0@sha256:6f7d8d4e472b8c4d166573cc7bb714bbb0fdf1535142b6138c62fdecbf881df9

When the operator attempts to figure out the init image to use (at

percona-xtradb-cluster-operator/pkg/k8s/utils.go

Lines 45 to 56 in fc46e36

	operatorPod, err := OperatorPod(ctx, cli)
	if err != nil {
	return "", errors.Wrap(err, "get operator deployment")
	}
	imageName, err := operatorImageName(&operatorPod)
	if err != nil {
	return "", err
	}
	if cr.CompareVersionWith(version.Version) != 0 {
	imageName = strings.Split(imageName, ":")[0] + ":" + cr.Spec.CRVersion
	}
	return imageName, nil

) it looks at it's own image and tries to figure out what init image to use.
With the above image, that results in a different image (probably takes the imageName = strings.Split(imageName, ":")[0] + ":" + cr.Spec.CRVersion path)

However, my cluster has security rules that prevent unhashlocked images. So the pod fails to get created.

admission webhook "validate.kyverno.svc-fail" denied the request: 

resource StatefulSet/fleet/fleet-database-pxc was blocked due to the following policies 

require-image-checksum:
  autogen-require-image-checksum: 'validation error: Images must use checksums rather
    than tags. rule autogen-require-image-checksum failed at path /spec/template/spec/initContainers/0/image/'

create newStatefulSetNode
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).deploy
	/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:643
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile
	/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:330
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1695

Steps to reproduce

1. hash-lock your operator image (e.g. image: percona/percona-xtradb-cluster-operator:1.15.0@sha256:6f7d8d4e472b8c4d166573cc7bb714bbb0fdf1535142b6138c62fdecbf881df9)
2. look at the initContainer image for you database statefulset
3. observe that it doesn't match

Versions

1. Kubernetes
2. Operator
3. Database

Anything else?

No response

[hors]

hi @james-callahan the workaround is to set custom initImage via CR

[james-callahan]

hi @james-callahan the workaround is to set custom initImage via CR

Yes I have had to use that as a workaround for now.
However it introduces unwanted coupling between deploying the CR and the version of the operator in use.

[hors]

hi @james-callahan the workaround is to set custom initImage via CR

Yes I have had to use that as a workaround for now. However it introduces unwanted coupling between deploying the CR and the version of the operator in use.

I see your point. I think we can improve it but you will need to have *:1.15.0* in your tag. Without it operator can't understand the operator version. I will create jira task for this improvement.

[hors]

@james-callahan I have created jira task https://perconadev.atlassian.net/browse/K8SPXC-1469