Airflow Worker container is crashing with error "ModuleNotFoundError: No module named 'postgresql'"

[dineshkumar20]

Hi all
I'm running airflow(2.8.3 version) on Kubernetes with CeleryExecutor. All Airflow components are working fine except worker. It throws error like this. Can someone please help me to resolve this issue?

[2024-03-23T10:23:28.497+0000] {configuration.py:2066} INFO - Creating new FAB webserver config file in: /opt/airflow/webserver_config.py 
 Traceback (most recent call last): 
   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/celery/cli/celery_command.py", line 60, in wrapper 
     providers_configuration_loaded(func)(*args, **kwargs)
   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function 
     return func(*args, **kwargs)
   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/celery/cli/celery_command.py", line 162, in worker 
     if hasattr(celery_app.backend, "ResultSession"): 
   File "/home/airflow/.local/lib/python3.9/site-packages/celery/app/base.py", line 1301, in backend 
     self._backend = self._get_backend() 
   File "/home/airflow/.local/lib/python3.9/site-packages/celery/app/base.py", line 966, in _get_backend 
     backend, url = backends.by_url(
   File "/home/airflow/.local/lib/python3.9/site-packages/celery/app/backends.py", line 68, in by_url 
     return by_name(backend, loader), url 
   File "/home/airflow/.local/lib/python3.9/site-packages/celery/app/backends.py", line 48, in by_name 
     cls = symbol_by_name(backend, aliases)
   File "/home/airflow/.local/lib/python3.9/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name 
     module = imp(module_name, package=package, **kwargs)
   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module 
     return _bootstrap._gcd_import(name[level:], package, level)
   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import 
   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load 
   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked 
 ModuleNotFoundError: No module named 'postgresql'
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last): 
   File "/home/airflow/.local/bin/airflow", line 8, in <module>
     sys.exit(main()) 
   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/__main__.py", line 57, in main 
     args.func(args)
   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/cli_config.py", line 49, in command 
     return func(*args, **kwargs)
   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 114, in wrapper 
     return f(*args, **kwargs)
   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/celery/cli/celery_command.py", line 64, in wrapper 
     raise AirflowOptionalProviderFeatureException(
 airflow.exceptions.AirflowOptionalProviderFeatureException: Failed to import providers_configuration_loaded. This feature is only available in Airflow versions >= 2.8.0 
 Stream closed EOF for airflow/airflow-worker-0 (worker)

[potiuk]

You seem to not have postgresql dependency installed in your image. You seem to have result backend configured that require postgresql package to be installed, so you need to make sure whatever you have as your result backend set has the appropriate driver

https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#result-backend

By default if you do not set it, the same configuration is used as for airflow connection:

https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-postgresql-database

Which uses psycopg2 driver:

postgresql+psycopg2://<user>:<password>@<host>/<db>

Likely you have some backend result configuration that requires postgresql package to be installed. You need to review your configuration and make sure you have the right drivers in your image matching the right configuration.

[BurkovBA]

Ran into this on 2.9.0. In my config I had result_backend:

postgresql://<user>:<password>@<host>/<db>

After changing this to:

postgresql+psycopg2://<user>:<password>@<host>/<db>

The problem is gone. Apparently it is necessary to explicitly specify psycopg2 driver in Celery connection line. Same is true for sql_alchemy_conn - it requires exactly db+postgresql+psycopg2://, otherwise will fail to store the result backend and will crash the scheduler.

Thank you.

[skiedude]

Another possible solution:
I ran into this and a similar issue with psycopg2 package missing because I switched to the slim-* base docker image last minute and didn't realize it was missing all the connector packages for redis, postgres, celery. Switched back to the non-slim and everything was fine.