-
Notifications
You must be signed in to change notification settings - Fork 106
Description
I am using the confluentinc/cp-kafka-connect:7.9.1 image and have installed the snowflake connector via
confluent-hub install --no-prompt --verbose snowflakeinc/snowflake-kafka-connector:3.2.0
I have set the following environmental variables:
CONNECT_CONFIG_PROVIDERS: "env,file,directory"
CONNECT_CONFIG_PROVIDERS_ENV_CLASS: "org.apache.kafka.common.config.provider.EnvVarConfigProvider"
CONNECT_CONFIG_PROVIDERS_FILE_CLASS: 'org.apache.kafka.common.config.provider.FileConfigProvider'
CONNECT_CONFIG_PROVIDERS_DIRECTORY_CLASS: "org.apache.kafka.common.config.provider.DirectoryConfigProvider"
When I attempt to start a job using the file config provider it works fine
{
"name": "name",
"config": {
"connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector",
....
"snowflake.user.name": "${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_USER_NAME}",
"snowflake.private.key": "${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_PRIVATE_KEY}",
"snowflake.database.name": "${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_DATABASE_NAME}",
....
}
}
so ${file:/opt/secrets/sf-credentials.properties:SNOWFLAKE_USER_NAME} is translated to 'myUserName'
However when I try to use the env config provider it fails to translate the string using the config provider and attempts to use it directly
{
"name": "name",
"config": {
"connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector",
....
"snowflake.user.name": "${env:SNOWFLAKE_USER_NAME}",
"snowflake.private.key": "${env:SNOWFLAKE_PRIVATE_KEY}",
"snowflake.database.name": "${env:SNOWFLAKE_DATABASE_NAME}",
....
}
}
so the username that the connector attempts to use is '${env:SNOWFLAKE_USER_NAME}' rather than what is stored in that environmental variable.
I have been successful in using the same base image using the environment config provider to connect to mongodb.