Skip to content

Unable to utilize EnvVarConfigProvider #1121

@RuthDevlaeminck

Description

@RuthDevlaeminck

I am using the confluentinc/cp-kafka-connect:7.9.1 image and have installed the snowflake connector via

confluent-hub install --no-prompt --verbose snowflakeinc/snowflake-kafka-connector:3.2.0

I have set the following environmental variables:

    CONNECT_CONFIG_PROVIDERS: "env,file,directory"
    CONNECT_CONFIG_PROVIDERS_ENV_CLASS: "org.apache.kafka.common.config.provider.EnvVarConfigProvider"
    CONNECT_CONFIG_PROVIDERS_FILE_CLASS: 'org.apache.kafka.common.config.provider.FileConfigProvider'
    CONNECT_CONFIG_PROVIDERS_DIRECTORY_CLASS: "org.apache.kafka.common.config.provider.DirectoryConfigProvider"

When I attempt to start a job using the file config provider it works fine

{
    "name": "name",
    "config": {
        "connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector",
        ....
        "snowflake.user.name": "${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_USER_NAME}",
        "snowflake.private.key": "${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_PRIVATE_KEY}",
        "snowflake.database.name": "${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_DATABASE_NAME}",
        ....
    }
}

so ${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_USER_NAME} is translated to 'myUserName'

However when I try to use the env config provider it fails to translate the string using the config provider and attempts to use it directly

{
    "name": "name",
    "config": {
        "connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector",
        ....
        "snowflake.user.name": "${env:SNOWFLAKE_USER_NAME}",
        "snowflake.private.key": "${env:SNOWFLAKE_PRIVATE_KEY}",
        "snowflake.database.name": "${env:SNOWFLAKE_DATABASE_NAME}",
        ....
    }
}

so the username that the connector attempts to use is '${env:SNOWFLAKE_USER_NAME}' rather than what is stored in that environmental variable.

I have been successful in using the same base image using the environment config provider to connect to mongodb.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions