Trying to upgrade from 0.16.2 to 0.22.1

[maurolscla]

I'm trying to upgrade strimzi from version 0.16.2 to 0.22.1. But once I'm done with the upgrade the operator is not able to start.
Logs show the following:

+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
++ sed -E -n 's/.* version "([0-9]*).*$/\1/p'
++ java -version
+ JAVA_MAJOR_VERSION=11
+ '[' 11 -ge 9 ']'
+ JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.20.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.4.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.crd-annotations-0.20.1.jar:lib/io.fabric8.openshift-model-operator-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.8.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.keycloak.keycloak-core-11.0.2.jar:lib/io.fabric8.kubernetes-model-apps-4.12.0.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.fabric8.kubernetes-model-storageclass-4.12.0.jar:lib/org.keycloak.keycloak-common-11.0.2.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-coordination-4.12.0.jar:lib/io.fabric8.kubernetes-model-settings-4.12.0.jar:lib/io.strimzi.kafka-oauth-client-0.6.1.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.strimzi.operator-common-0.20.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.4.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/io.fabric8.kubernetes-model-events-4.12.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-discovery-4.12.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.fabric8.kubernetes-model-apiextensions-4.12.0.jar:lib/io.fabric8.kubernetes-client-4.12.0.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-4.12.0.jar:lib/io.fabric8.openshift-model-monitoring-4.12.0.jar:lib/io.strimzi.kafka-oauth-common-0.6.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.config-model-0.20.1.jar:lib/io.fabric8.kubernetes-model-metrics-4.12.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.kubernetes-model-common-4.12.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-server-0.6.1.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.kubernetes-model-core-4.12.0.jar:lib/io.strimzi.certificate-manager-0.20.1.jar:lib/io.fabric8.kubernetes-model-networking-4.12.0.jar:lib/io.fabric8.openshift-model-4.12.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.kafka.kafka-clients-2.6.0.jar:lib/io.fabric8.openshift-model-operatorhub-4.12.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/io.strimzi.api-0.20.1.jar:lib/io.fabric8.kubernetes-model-batch-4.12.0.jar:lib/io.fabric8.kubernetes-model-extensions-4.12.0.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/io.fabric8.kubernetes-model-scheduling-4.12.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.4.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/io.fabric8.openshift-model-console-4.12.0.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.4.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-4.12.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.kubernetes-model-admissionregistration-4.12.0.jar:lib/io.fabric8.kubernetes-model-policy-4.12.0.jar:lib/io.fabric8.kubernetes-model-rbac-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.8.jar:lib/io.fabric8.openshift-client-4.12.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.65.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar io.strimzi.operator.cluster.Main
2021-04-13 23:18:41 INFO  Main:60 - ClusterOperator 0.20.1 is starting
2021-04-13 23:18:43 INFO  Main:80 - Environment facts gathered: ClusterOperatorConfig(KubernetesVersion=1.16,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false,OpenShiftDeploymentConfigs=false)
2021-04-13 23:18:43 INFO  Util:272 - Using config:
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	SCHEMA_REGISTRY_API_SERVICE_HOST: 172.20.36.164
	STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: strimzi/operator:0.20.1
	KAFDROP_PORT_9000_TCP_ADDR: 172.20.245.41
	JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny
	KAFDROP_SERVICE_PORT_HTTP: 9000
	SCHEMA_REGISTRY_API_PORT_8080_TCP_PORT: 8080
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	SCHEMA_REGISTRY_API_PORT_8080_TCP_PROTO: tcp
	KUBERNETES_PORT_443_TCP: tcp://172.20.0.1:443
	JAVA_MAIN: io.strimzi.operator.cluster.Main
	KAFDROP_PORT_9000_TCP: tcp://172.20.245.41:9000
	STRIMZI_VERSION: 0.20.1
	STRIMZI_DEFAULT_JMXTRANS_IMAGE: strimzi/jmxtrans:0.20.1
	STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.22.1
	KAFKA_CONNECT_UI_PORT_8080_TCP_PORT: 8080
	STRIMZI_NAMESPACE: kafka
	KAFDROP_PORT_9000_TCP_PROTO: tcp
	KAFKA_CONNECT_UI_PORT: tcp://172.20.228.58:8080
	SCHEMA_REGISTRY_API_PORT_8080_TCP_ADDR: 172.20.36.164
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	SHLVL: 0
	STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: strimzi/kafka-bridge:0.19.0
	TINI_SHA256: 12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855
	STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0

	KUBERNETES_PORT: tcp://172.20.0.1:443
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	KUBERNETES_SERVICE_HOST: 172.20.0.1
	KAFDROP_PORT: tcp://172.20.245.41:9000
	JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.20.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.4.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.crd-annotations-0.20.1.jar:lib/io.fabric8.openshift-model-operator-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.8.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.keycloak.keycloak-core-11.0.2.jar:lib/io.fabric8.kubernetes-model-apps-4.12.0.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.fabric8.kubernetes-model-storageclass-4.12.0.jar:lib/org.keycloak.keycloak-common-11.0.2.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-coordination-4.12.0.jar:lib/io.fabric8.kubernetes-model-settings-4.12.0.jar:lib/io.strimzi.kafka-oauth-client-0.6.1.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.strimzi.operator-common-0.20.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.4.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/io.fabric8.kubernetes-model-events-4.12.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-discovery-4.12.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.fabric8.kubernetes-model-apiextensions-4.12.0.jar:lib/io.fabric8.kubernetes-client-4.12.0.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-4.12.0.jar:lib/io.fabric8.openshift-model-monitoring-4.12.0.jar:lib/io.strimzi.kafka-oauth-common-0.6.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.config-model-0.20.1.jar:lib/io.fabric8.kubernetes-model-metrics-4.12.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.kubernetes-model-common-4.12.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-server-0.6.1.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.kubernetes-model-core-4.12.0.jar:lib/io.strimzi.certificate-manager-0.20.1.jar:lib/io.fabric8.kubernetes-model-networking-4.12.0.jar:lib/io.fabric8.openshift-model-4.12.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.kafka.kafka-clients-2.6.0.jar:lib/io.fabric8.openshift-model-operatorhub-4.12.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/io.strimzi.api-0.20.1.jar:lib/io.fabric8.kubernetes-model-batch-4.12.0.jar:lib/io.fabric8.kubernetes-model-extensions-4.12.0.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/io.fabric8.kubernetes-model-scheduling-4.12.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.4.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/io.fabric8.openshift-model-console-4.12.0.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.4.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-4.12.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.kubernetes-model-admissionregistration-4.12.0.jar:lib/io.fabric8.kubernetes-model-policy-4.12.0.jar:lib/io.fabric8.kubernetes-model-rbac-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.8.jar:lib/io.fabric8.openshift-client-4.12.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.65.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar
	KAFKA_CONNECT_UI_SERVICE_PORT: 8080
	STRIMZI_KAFKA_CONNECT_S2I_IMAGES: 2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0

	SCHEMA_REGISTRY_API_SERVICE_PORT: 8080
	KAFKA_CONNECT_UI_PORT_8080_TCP_ADDR: 172.20.228.58
	TINI_VERSION: v0.18.0
	STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: strimzi/kafka:0.20.1-kafka-2.6.0
	STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: strimzi/kafka:0.20.1-kafka-2.6.0
	KAFKA_CONNECT_UI_PORT_8080_TCP: tcp://172.20.228.58:8080
	STRIMZI_OPERATION_TIMEOUT_MS: 300000
	KUBERNETES_PORT_443_TCP_ADDR: 172.20.0.1
	SCHEMA_REGISTRY_API_PORT_8080_TCP: tcp://172.20.36.164:8080
	STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0

	KAFDROP_SERVICE_PORT: 9000
	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: strimzi/operator:0.20.1
	KAFDROP_SERVICE_HOST: 172.20.245.41
	KUBERNETES_SERVICE_PORT: 443
	STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: strimzi/kafka:0.20.1-kafka-2.6.0
	STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: strimzi/operator:0.20.1
	KAFKA_CONNECT_UI_PORT_8080_TCP_PROTO: tcp
	SCHEMA_REGISTRY_API_PORT: tcp://172.20.36.164:8080
	HOSTNAME: strimzi-cluster-operator-66ff46677f-6dfm8
	STRIMZI_KAFKA_CONNECT_IMAGES: 2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0

	STRIMZI_KAFKA_IMAGES: 2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0

	KAFKA_CONNECT_UI_SERVICE_HOST: 172.20.228.58
	KUBERNETES_PORT_443_TCP_PORT: 443
	STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: strimzi/kafka:0.20.1-kafka-2.6.0
	HOME: /home/strimzi
	STRIMZI_OPERATOR_NAMESPACE: kafka
	MALLOC_ARENA_MAX: 2
	KAFDROP_PORT_9000_TCP_PORT: 9000

2021-04-13 23:18:43 INFO  Main:121 - The KafkaConnectS2I custom resource definition can only be used in environment which supports OpenShift build, image and apps APIs. These APIs do not seem to be supported in this environment.
2021-04-13 23:18:43 INFO  ClusterOperator:81 - Creating ClusterOperator for namespace kafka
2021-04-13 23:18:43 INFO  ClusterOperator:98 - Starting ClusterOperator for namespace kafka
2021-04-13 23:18:43 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-13 23:18:43 INFO  AbstractOperator:442 - Watcher closed in namespace kafka
2021-04-13 23:18:43 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-13 23:18:43 INFO  AbstractOperator:442 - Watcher closed in namespace kafka
2021-04-13 23:18:43 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-13 23:18:43 INFO  AbstractOperator:442 - Watcher closed in namespace kafka
2021-04-13 23:18:44 WARN  WatchConnectionManager:353 - Executor didn't terminate in time after shutdown in close(), killing it in: io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@15bfa4f5
2021-04-13 23:18:44 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-13 23:18:44 INFO  AbstractOperator:442 - Watcher closed in namespace kafka
2021-04-13 23:18:44 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-13 23:18:44 INFO  AbstractOperator:442 - Watcher closed in namespace kafka
2021-04-13 23:18:45 WARN  WatchConnectionManager:353 - Executor didn't terminate in time after shutdown in close(), killing it in: io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@13dc5b3c
2021-04-13 23:18:45 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-13 23:18:45 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-04-13 23:18:45 ERROR Main:156 - Cluster Operator verticle in namespace kafka failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found

	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:202) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
	Suppressed: java.lang.Throwable: waiting here
		at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:154) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:341) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:818) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:791) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:84) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
		at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:63) ~[io.strimzi.operator-common-0.20.1.jar:0.20.1]
		at io.strimzi.operator.common.AbstractOperator.lambda$createWatch$16(AbstractOperator.java:431) ~[io.strimzi.operator-common-0.20.1.jar:0.20.1]
		at io.strimzi.operator.common.Util.lambda$async$0(Util.java:53) ~[io.strimzi.operator-common-0.20.1.jar:0.20.1]
		at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
		at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
		at java.lang.Thread.run(Thread.java:834) [?:?]

I have doubled checked and CRDs are there. And the namespace kafka as well.
When I revert it back to 0.16.2 it is able to start with no issues.
Any idea what might be happening here?

[tomncooper]

We don't support upgrades across non-consecutive versions. The safest option is to upgrade from one version to the next and upgrade the Kafka version as you go. A less safe option is to upgrade to the highest version that still supports your current kafka version, upgrade kafka and then repeat. That may allow you to skip some versions but again that is not recommended.

[scholzj]

We do support it from 0.22.1. Although I'm sure nobody tested it specifically with 0.16.2. The error here has nothing to do with the version from which you are upgrading. The 404 is normally missing CRDs. So you should double check if they are all installed and if they are installed in the right versions (there are CRDs in 0.22 which did not exist in 0.16 I guess).