Update Airflow to 2.10.5

[ividito]

Summary:

Airflow (+related libraries) version upgrade. This prompted some other changes and improvements, as several things broke with the updated libraries.

• old SQS image is outdated and had to be replaced
• switched to uv and dropped conda to fix library build issues
• example DAGs use outdated providers, I opted to delete instead of refactor
• LazyXcomAccess -> LazyXcomSequence
• stactools DAGs were scheduled by default - this has been fixed

Manual step required to make deployment work with new version - python scripts/run_task.py --vpc-id vpc-xxx --security-group xxx --task-definition xxx --cluster xxx--command 'db init'

^ No longer applies, instead we must set the airflow_version variable for SM2A - NASA-IMPACT/self-managed-apache-airflow#23

Testing/action items

• SIT deployment tested (deploy works, but services require manual db upgrade)
• tested raster ingestion with veda-dataset-pipeline
• tested vector ingestion with veda-vector-pipeline - works, but SIT cannot connect to dev features DB to submit generated items

[botanical]

Why are we using LazyXComSelectSequence instead of LazyXComAccess?

I'm seeing this error when I run make sm2a-local-init 😯

(editing to share error message!)

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/airflow/dags/generate_dags.py
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 346, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/airflow/dags/generate_dags.py", line 9, in <module>
    from veda_data_pipeline.veda_discover_pipeline import get_discover_dag
  File "/opt/airflow/dags/veda_data_pipeline/veda_discover_pipeline.py", line 6, in <module>
    from veda_data_pipeline.groups.discover_group import discover_from_s3_task, get_files_task
  File "/opt/airflow/dags/veda_data_pipeline/groups/discover_group.py", line 7, in <module>
    from airflow.models.xcom import LazyXComSelectSequence
ImportError: cannot import name 'LazyXComSelectSequence' from 'airflow.models.xcom' (/home/airflow/.local/lib/python3.11/site-packages/airflow/models/xcom.py)
/home/airflow/.local/lib/python3.11/site-packages/airflow_multi_dagrun/operators.py:3 DeprecationWarning: This module is deprecated. Please use `airflow.api.common.trigger_dag` instead.
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/airflow/dags/veda_data_pipeline/veda_promotion_pipeline.py
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 346, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/airflow/dags/veda_data_pipeline/veda_promotion_pipeline.py", line 10, in <module>
    from veda_data_pipeline.groups.discover_group import discover_from_s3_task, get_dataset_files_to_process
  File "/opt/airflow/dags/veda_data_pipeline/groups/discover_group.py", line 7, in <module>
    from airflow.models.xcom import LazyXComSelectSequence
ImportError: cannot import name 'LazyXComSelectSequence' from 'airflow.models.xcom' (/home/airflow/.local/lib/python3.11/site-packages/airflow/models/xcom.py)
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/airflow/dags/veda_data_pipeline/veda_vector_pipeline.py
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 346, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/airflow/dags/veda_data_pipeline/veda_vector_pipeline.py", line 9, in <module>
    from veda_data_pipeline.groups.discover_group import discover_from_s3_task, get_files_task
  File "/opt/airflow/dags/veda_data_pipeline/groups/discover_group.py", line 7, in <module>
    from airflow.models.xcom import LazyXComSelectSequence
ImportError: cannot import name 'LazyXComSelectSequence' from 'airflow.models.xcom' (/home/airflow/.local/lib/python3.11/site-packages/airflow/models/xcom.py)
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/airflow/dags/veda_data_pipeline/veda_dataset_pipeline.py
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 346, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/airflow/dags/veda_data_pipeline/veda_dataset_pipeline.py", line 4, in <module>
    from veda_data_pipeline.groups.discover_group import discover_from_s3_task, get_files_task
  File "/opt/airflow/dags/veda_data_pipeline/groups/discover_group.py", line 7, in <module>
    from airflow.models.xcom import LazyXComSelectSequence
ImportError: cannot import name 'LazyXComSelectSequence' from 'airflow.models.xcom' (/home/airflow/.local/lib/python3.11/site-packages/airflow/models/xcom.py)
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/airflow/dags/veda_data_pipeline/veda_discover_pipeline.py
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 346, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/airflow/dags/veda_data_pipeline/veda_discover_pipeline.py", line 6, in <module>
    from veda_data_pipeline.groups.discover_group import discover_from_s3_task, get_files_task
  File "/opt/airflow/dags/veda_data_pipeline/groups/discover_group.py", line 7, in <module>
    from airflow.models.xcom import LazyXComSelectSequence
ImportError: cannot import name 'LazyXComSelectSequence' from 'airflow.models.xcom' (/home/airflow/.local/lib/python3.11/site-packages/airflow/models/xcom.py)
/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/db.py:741 DeprecationWarning: The load_default_connections option in [core] has been moved to the load_default_connections option in [database] - the old setting has been used, but please update your config.

[ividito]

Why are we using LazyXComSelectSequence instead of LazyXComAccess?

I'm seeing this error when I run make sm2a-local-init 😯

I wasn't able to reproduce this error, but I've seen similar errors caused by version mismatches. The LazyXCom models were renamed between Airflow 2.8 and 2.10. Could you try running make sm2a-local-build before make sm2a-local-init?

[botanical]

Why are we using LazyXComSelectSequence instead of LazyXComAccess?
I'm seeing this error when I run make sm2a-local-init 😯

I wasn't able to reproduce this error, but I've seen similar errors caused by version mismatches. The LazyXCom models were renamed between Airflow 2.8 and 2.10. Could you try running make sm2a-local-build before make sm2a-local-init?

With make sm2a-local-build I get

 => ERROR [airflow-worker stage-0 11/17] RUN uv venv                                                               0.2s
------
 > [airflow-worker stage-0 11/17] RUN uv venv:
0.152   × Failed to inspect Python interpreter from managed installations at
0.152   │ `/home/airflow/.local/share/uv/python/cpython-3.11.12-linux-aarch64-gnu/bin/python3.11`
0.152   ├─▶ Failed to query Python interpreter at
0.152   │   `/home/airflow/.local/share/uv/python/cpython-3.11.12-linux-aarch64-gnu/bin/python3.11`
0.152   ╰─▶ No such file or directory (os error 2)
------
failed to solve: process "/bin/sh -c uv venv" did not complete successfully: exit code: 1

Maybe this is an issue I am experiencing personally from having an incorrect python version or something 🤔 Which version are you using @ividito ?

[ividito]

My docker version output shows version 28.1.1

Client:
 Version:           28.1.1
 API version:       1.49
 Go version:        go1.23.8
 Git commit:        4eba377
 Built:             Fri Apr 18 09:51:06 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop 4.0.0 ()
 Engine:
  Version:          28.1.1
  API version:      1.49 (minimum version 1.24)
  Go version:       go1.23.8
  Git commit:       01f442b
  Built:            Fri Apr 18 09:52:57 2025
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I'm a bit stumped on how to debug this if it's not a docker version issue. I tried a fresh build with no caching, and still didn't run into any errors.

[botanical]

⭐