AWS systems_manager ParameterNotFound error catching

[nsepetys]

Hi,

I am wondering if anyone has information as to why AWS secrets handling for the ParameterNotFound client exception does not get raised? This error should be something Airflow administrators have taken care of to avoid in the first place but accidents happen and these parameters can end up removed or the names changes. The result is that tasks don't fail and the error slips through the cracks causing a downstream trickle of unexpected situations. This is not expected primarily because when choosing the default Airflow secrets back-end we get a different experience when the key is missing- the issue is raised.

• Variables.get for default raises KeyError
  • To my best understanding AWS secrets back-end passes through this method but outcome is different depending on whether or not AWS is used
• Connections.get_connection_from_secrets raises AirflowNotFoundException
  • Again, according to my best understanding, the secrets back-end (regardless of whether it's AWS or default) comes through here but the raise never happens here because of the ParameterNotFound handling earlier on.

@kaxil - You appear to be the authority on what is there (basing this on one of the latest PRs in AWS systems manager sphere). Thank you for your work on all this. It has made our org happy to have a better method for handling secrets. I am curious though if we had any reason why we have the approach we currently have for this particular error catch. I am somewhat new to Airflow internals and also somewhat new to Python, so I'm hoping, if anything, you can help educate me.

[kaxil]

Hi @nsepetys,

Glad that you found the secrets backend useful.

The main reason we don't raise ParameterNotFound for AWS secrets backend (SystemsManagerParameterStoreBackend and SecretsManagerBackend) is so that we can fallback to getting variables or Connections via Environment Variables and then Metadata Database.

For example, for Airflow Variables:

We do a similar thing for Hashicorp Vault and GoogleCloud Secrets Backend but as they don't raise an error, we just return None (e.g

airflow/airflow/providers/google/cloud/secrets/secret_manager.py

Lines 148 to 158 in ffe3bd2

	def get_config(self, key: str) -> Optional[str]:
	"""
	Get Airflow Configuration
	:param key: Configuration Option Key
	:return: Configuration Option Value
	"""
	if self.config_prefix is None:
	return None
	return self._get_secret(self.config_prefix, key)

)

We loop across all the 3 backends (Secrets Backend in the airflow.cfg, Environment Variables and then MetastoreBackend)

airflow/airflow/models/connection.py

Lines 343 to 354 in a9ac2b0

	def get_connection_from_secrets(cls, conn_id: str) -> 'Connection':
	"""
	Get connection by conn_id.
	:param conn_id: connection id
	:return: connection
	"""
	for secrets_backend in ensure_secrets_loaded():
	conn = secrets_backend.get_connection(conn_id=conn_id)
	if conn:
	return conn
	raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")

airflow/airflow/configuration.py

Lines 1017 to 1033 in ffe3bd2

	def initialize_secrets_backends() -> List[BaseSecretsBackend]:
	"""
	* import secrets backend classes
	* instantiate them and return them in a list
	"""
	backend_list = []
	custom_secret_backend = get_custom_secret_backend()
	if custom_secret_backend is not None:
	backend_list.append(custom_secret_backend)
	for class_name in DEFAULT_SECRETS_SEARCH_PATH:
	secrets_backend_cls = import_string(class_name)
	backend_list.append(secrets_backend_cls())
	return backend_list

We only raise an error when Connection or Variable is not found in all the 3 backends

[nsepetys]

Thanks for taking the time to explain @kaxil . That provided some good details. I think this new understanding has given me a theory as to what is going on with our instance of Airflow. You take care!