need help to deploy scylladb connector #5117

clemensk1 · 2021-06-11T09:34:24Z

clemensk1
Jun 11, 2021

Hi,

I`m trying to install the connector for scylladb. https://github.com/scylladb/kafka-connect-scylladb.git
I did a "mvn clean install" like in the quickstart guide what ended with some jar files. I put these jar files into a tgz and uploaded it to a server to provide a link in my KafkaConnect resource. The logs of the build container:

INFO[0001] Retrieving image manifest quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0001] Retrieving image quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0001] Retrieving image manifest quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0001] Retrieving image quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0002] Retrieving image quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0003] Retrieving image manifest quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0003] Retrieving image quay.io/strimzi/kafka:0.21.1-kafka-2.7.0 
INFO[0004] Executing 0 build triggers                   
INFO[0004] Unpacking rootfs as cmd RUN mkdir -p /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && curl -L --output /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX       && tar xvfz /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX -C /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && rm -vf /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX requires it. 
INFO[0029] USER root:root                               
INFO[0029] cmd: USER                                    
INFO[0029] RUN mkdir -p /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && curl -L --output /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX       && tar xvfz /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX -C /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && rm -vf /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX 
INFO[0029] Taking snapshot of full filesystem...        
INFO[0034] cmd: /bin/sh                                 
INFO[0034] args: [-c mkdir -p /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && curl -L --output /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX       && tar xvfz /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX -C /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && rm -vf /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX] 
INFO[0034] util.Lookup returned: &{Uid:0 Gid:0 Username:root Name: HomeDir:/root} 
INFO[0034] performing slow lookup of group ids for root 
INFO[0034] Running: [/bin/sh -c mkdir -p /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && curl -L --output /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX       && tar xvfz /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX -C /opt/kafka/plugins/kafka-connect-scylladb/e00b99f4       && rm -vf /opt/kafka/plugins/kafka-connect-scylladb/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX] 
INFO[0034] Taking snapshot of full filesystem...        
INFO[0034] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0034] USER 1001                                    
INFO[0034] cmd: USER

feel free to download the tgz package and inspect it
https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX

If I create a KafkaConnector resource, the logs of KafkaConnect are:

ERROR Uncaught exception in REST call to /connectors/kafka-connect-scylladb/config (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper) [qtp1658704411-22]
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.connect.scylladb.ScyllaDbSinkConnector, available connectors are: ....

So it seems the plugin doesn't get installed. What am I doing wrong? Do you have a hint?

Here are my yaml files:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  namespace: my-kafka-project
  name: my-connect-cluster
  annotations:
#  # use-connector-resources configures this KafkaConnect
#  # to use KafkaConnector resources to avoid
#  # needing to call the Connect REST API directly
    strimzi.io/use-connector-resources: "true"
spec:
  #version: 2.8.0
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap.my-kafka-project:9093
  #tls:
    #trustedCertificates:
      #- secretName: my-cluster-cluster-ca-cert
        #certificate: ca.crt
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  build:
    output:
      type: docker
      image: chargeiq/kafka-connect:latest
      pushSecret: my-registry-credentials
    plugins: 
      - name: kafka-connect-scylladb
        artifacts:
          - type: tgz
            url: https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX
  template:
    pod:
      imagePullSecrets:
        - name: my-registry-credentials

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: kafka-connect-scylladb
  namespace: my-kafka-project
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: io.connect.scylladb.ScyllaDbSinkConnector
  tasksMax: 1
  config:
    camel.source.path.timerName: timer
    topics: scyllatest1
    scylladb.keyspace: test
    scylladb.contact.points: scylla.scylla-eu-central-1-eu-central-1c-0

Strimzi version: 0.21.1

Many Thanks!

Answered by scholzj

Jun 11, 2021

So, what plugins are listed when you do kubectl get kc -o yaml -n my-kafka-project?

View full answer

scholzj · 2021-06-11T10:36:41Z

scholzj
Jun 11, 2021
Maintainer

So, what plugins are listed when you do kubectl get kc -o yaml -n my-kafka-project?

3 replies

clemensk1 Jun 11, 2021
Author

this is the output:

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta1
  kind: KafkaConnect
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kafka.strimzi.io/v1beta1","kind":"KafkaConnect","metadata":{"annotations":{"strimzi.io/use-connector-resources":"true"},"name":"my-connect-cluster","namespace":"my-kafka-project"},"spec":{"bootstrapServers":"my-cluster-kafka-bootstrap.my-kafka-project:9093","build":{"output":{"image":"chargeiq/kafka-connect:latest","pushSecret":"my-registry-credentials","type":"docker"},"plugins":[{"artifacts":[{"type":"tgz","url":"https://drive.google.com/uc?export=download\u0026id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX"}],"name":"kafka-connect-scylladb"}]},"config":{"config.storage.replication.factor":-1,"config.storage.topic":"connect-cluster-configs","group.id":"connect-cluster","offset.storage.replication.factor":-1,"offset.storage.topic":"connect-cluster-offsets","status.storage.replication.factor":-1,"status.storage.topic":"connect-cluster-status"},"replicas":1,"template":{"pod":{"imagePullSecrets":[{"name":"my-registry-credentials"}]}}}}
      strimzi.io/use-connector-resources: "true"
    creationTimestamp: "2021-06-11T08:58:45Z"
    generation: 1
    name: my-connect-cluster
    namespace: my-kafka-project
    resourceVersion: "18609062"
    selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/my-kafka-project/kafkaconnects/my-connect-cluster
    uid: 0cbbb967-9eda-48bb-a63d-08eb15eaa565
  spec:
    bootstrapServers: my-cluster-kafka-bootstrap.my-kafka-project:9093
    build:
      output:
        image: chargeiq/kafka-connect:latest
        pushSecret: my-registry-credentials
        type: docker
      plugins:
      - artifacts:
        - type: tgz
          url: https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX
        name: kafka-connect-scylladb
    config:
      config.storage.replication.factor: -1
      config.storage.topic: connect-cluster-configs
      group.id: connect-cluster
      offset.storage.replication.factor: -1
      offset.storage.topic: connect-cluster-offsets
      status.storage.replication.factor: -1
      status.storage.topic: connect-cluster-status
    replicas: 1
    template:
      pod:
        imagePullSecrets:
        - name: my-registry-credentials
  status:
    conditions:
    - lastTransitionTime: "2021-06-11T09:00:36.734041Z"
      status: "True"
      type: Ready
    connectorPlugins:
    - class: org.apache.kafka.connect.file.FileStreamSinkConnector
      type: sink
      version: 2.7.0
    - class: org.apache.kafka.connect.file.FileStreamSourceConnector
      type: source
      version: 2.7.0
    - class: org.apache.kafka.connect.mirror.MirrorCheckpointConnector
      type: source
      version: "1"
    - class: org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
      type: source
      version: "1"
    - class: org.apache.kafka.connect.mirror.MirrorSourceConnector
      type: source
      version: "1"
    labelSelector: strimzi.io/cluster=my-connect-cluster,strimzi.io/name=my-connect-cluster-connect,strimzi.io/kind=KafkaConnect
    observedGeneration: 1
    replicas: 1
    url: http://my-connect-cluster-connect-api.my-kafka-project.svc:8083
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

scholzj Jun 11, 2021
Maintainer

I think there are two problems:

The first is that it looks to me like the URL on Google Drive is not something what can be downloaded just with CURL. When you try curl -L -v https://drive.google.com/uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX, it does not seem to work. I'm not exactly sure why, but I think Google does there something what makes it not a direct download URL and you might need a real browser to download it. I'm not sure we can do much about it.
The second problem seems to be on our end ... we try to name the downloaded file based on last segment from the URL which in our case is uc?export=download&id=1sEZHDMskmwV31vTx7nUI_yoJM7AgdpXX and I think that fails as well. But I will have a look, this should be fairly easy to fix I think.

clemensk1 Jun 11, 2021
Author

You are totally right! I just thought if it works with a browser it works with curl, I should have figured it out myself...
I used another file server and now it works just fine :)

Many Thanks, and have a nice day!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

need help to deploy scylladb connector #5117

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

need help to deploy scylladb connector #5117

Uh oh!

clemensk1 Jun 11, 2021

Replies: 1 comment · 3 replies

Uh oh!

scholzj Jun 11, 2021 Maintainer

Uh oh!

clemensk1 Jun 11, 2021 Author

Uh oh!

scholzj Jun 11, 2021 Maintainer

Uh oh!

clemensk1 Jun 11, 2021 Author

clemensk1
Jun 11, 2021

Replies: 1 comment 3 replies

scholzj
Jun 11, 2021
Maintainer

clemensk1 Jun 11, 2021
Author

scholzj Jun 11, 2021
Maintainer

clemensk1 Jun 11, 2021
Author