AzureKeyVault Integration not working

[david-fischer]

I would like to integrate my self-hosted kafka-connector with Azure Keyvault via your secret provider.

Setup

I installed the plugin in the Dockerfile I am using.

FROM confluentinc/cp-kafka-connect:7.6.0 
# ...
RUN confluent-hub install --no-prompt confluentinc/csid-secrets-provider-azure:latest
# ...

In the docker-compose I now set:

  connect: 
     # ...
        CONNECT_CONFIG_PROVIDERS: keyVault
        CONNECT_CONFIG_PROVIDERS_KEYVAULT_CLASS: io.confluent.csid.config.provider.azure.KeyVaultConfigProvider
        CONNECT_CONFIG_PROVIDERS_KEYVAULT_PARAM_VAULT_URL: ${KEY_VAULT_URL}
        CONNECT_CONFIG_PROVIDERS_KEYVAULT_PARAM_CREDENTIAL_TYPE: ClientSecret
        CONNECT_CONFIG_PROVIDERS_KEYVAULT_PARAM_CLIENT_SECRET: ${KEY_VAULT_CLIENT_SECRET}
        CONNECT_CONFIG_PROVIDERS_KEYVAULT_PARAM_TENANT_ID: ${KEY_VAULT_TENANT_ID}
        CONNECT_CONFIG_PROVIDERS_KEYVAULT_PARAM_CLIENT_ID: ${KEY_VAULT_CLIENT_ID}

(I also tried ...KEY_VAULT... but this also did not work)

and in the respective config file (minimal working example):

{
  "config": {
    "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
    "file": "/home/appuser/${keyVault:test-secret:test_key}.txt",
    "tasks.max": "1",
    "topics": "connect-test"
  },
  "name": "file_sink_local"
}

Problem

The${keyVault:test-secret:test_key} part is never resolved and a file is created with the literal name ${keyVaulttest-secrettest_key}.txt is created.
No error is thrown and I did not notice a warning or info that the Azure Key Vault Config Provider encountered an issue in the logs.

Troubleshooting

• I tried my credentials to the Azure Keyvault with a Python script and was able to obtain the secret.
• I did not even see requests for the secret from my connector in the logs of the KeyVault
• docker compose logs connect | grep KeyVault gave following result:

connect  | [2024-03-14 17:54:16,644] INFO Added plugin 'io.confluent.csid.config.provider.azure.KeyVaultConfigProvider' (org.apache.kafka.connec
t.runtime.isolation.DelegatingClassLoader)
connect  | [2024-03-14 17:54:16,650] INFO Added alias 'KeyVault' to plugin 'io.confluent.csid.config.provider.azure.KeyVaultConfigProvider' (org
.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
connect  | [2024-03-14 17:54:16,651] INFO Added alias 'KeyVaultConfigProvider' to plugin 'io.confluent.csid.config.provider.azure.KeyVaultConfig
Provider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

• I read here that the config provider needs to be available in the CLASSPATH so I added a symlink:

ln -s  \
     /usr/share/confluent-hub-components/confluentinc-csid-secrets-provider-azure/lib/  \
     /usr/share/java/kafka/csid-secrets-provider-azure

Do you have an idea why this is not working? I would appreciate any input!

[rasifmahmud]

I figured out the issue. It does not work because the provider name keyVault has an upper case letter in it.

The config CONNECT_CONFIG_PROVIDERS_KEYVAULT_CLASS gets transformed to config.providers.keyvault.class which is all lower case. The docker utils here do this transformation:

https://github.com/confluentinc/kafka-images/blob/b0a7821097b353d26aafbc0b4bc434bfa2d8a9fb/kafka-connect-base/include/etc/confluent/docker/configure#L54C1-L55C1

https://github.com/confluentinc/confluent-docker-utils/blob/master/confluent/docker_utils/dub.py

Due to this case mismatch, the plugin is never loaded on the worker:

https://github.com/confluentinc/kafka/blob/2e23f9fb218a31fa12681425f23019559117ba3d/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/Plugins.java#L489

So, either they need to update the provider name to be all lower case or confluent's docker utils need to handle the case.

I have found several workarounds.

Approach 1

Manually set those params tweaking/tricking the connect docker image. What I did was replace the run file of the image with this:

#!/usr/bin/env bash
#
# Copyright 2016 Confluent Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

. /etc/confluent/docker/bash-config

. /etc/confluent/docker/mesos-setup.sh
. /etc/confluent/docker/apply-mesos-overrides

echo "===> User"
id

echo "===> Configuring ..."
/etc/confluent/docker/configure

echo "===> Running preflight checks ... "
/etc/confluent/docker/ensure

echo "===> Launching ... "

echo "config.providers=keyVault" >> "/etc/kafka-connect/kafka-connect.properties"
echo "config.providers.keyVault.class=io.confluent.csid.config.provider.azure.KeyVaultConfigProvider" >> "/etc/kafka-connect/kafka-connect.properties"
echo "config.providers.keyVault.param.vault.url=https://example.com" >> "/etc/kafka-connect/kafka-connect.properties"

exec /etc/confluent/docker/launch

Create the above file named "run" in the same directory as your Dockerfile and add all the needed config.providers.* configs. And then replace the run file in your Dockerfile:

FROM confluentinc/cp-server-connect:7.6.0
RUN confluent-hub install --no-prompt confluentinc/csid-secrets-provider-azure:latest
COPY ./run /etc/confluent/docker/

Approach 2

Directly specify those configs in the connector config. But as a warning, there are many connectors out there that fails to resolve secrets. For instance, we got it resolved for mongo here: https://jira.mongodb.org/browse/KAFKA-414

Approach 3

You can build a custom jar of this library replacing keyVault with keyvault here:

csid-secrets-providers/azure/src/main/java/io/confluent/csid/config/provider/azure/KeyVaultConfigProvider.java

Line 169 in 1537b41

@ConfigProviderKey("keyVault")

Havent tested this approach but should work in theory.