Add localstack Breeze integration (#54050) (#57668)

* Add localstack breeze integration

* Fix Breeze unit test

* Disbale test for localstack and export envs

* Add docs for Running External System Integrations with Breeze

* Fix output-commands.svg

(cherry picked from commit 84bd22c)

Author: jason810496

Dockerfile.ci

@@ -1208,6 +1208,27 @@ function environment_initialization() {
         ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
     fi
 
+    if [[ ${INTEGRATION_LOCALSTACK:-"false"} == "true" ]]; then
+        echo
+        echo "${COLOR_BLUE}Configuring LocalStack integration${COLOR_RESET}"
+        echo
+
+        # Define LocalStack AWS configuration
+        declare -A localstack_config=(
+            ["AWS_ENDPOINT_URL"]="http://localstack:4566"
+            ["AWS_ACCESS_KEY_ID"]="test"
+            ["AWS_SECRET_ACCESS_KEY"]="test"
+            ["AWS_DEFAULT_REGION"]="us-east-1"
+        )
+
+        # Export each configuration variable and log it
+        for key in "${!localstack_config[@]}"; do
+            export "$key"="${localstack_config[$key]}"
+            echo "  * ${COLOR_BLUE}${key}:${COLOR_RESET} ${localstack_config[$key]}"
+        done
+        echo
+    fi
+
     cd "${AIRFLOW_SOURCES}"
 
     # Temporarily add /opt/airflow/providers/standard/tests to PYTHONPATH in order to see example dags

contributing-docs/testing/integration_tests.rst

@@ -70,6 +70,8 @@ core or provider type of test.
 +--------------+-------------------------------------------------------+
 | keycloak     | Integration for manual testing of multi-team Airflow. |
 +--------------+-------------------------------------------------------+
+| localstack   | Integration that emulates AWS services locally.       |
++--------------+-------------------------------------------------------+
 | mongo        | Integration required for MongoDB hooks.               |
 +--------------+-------------------------------------------------------+
 | mssql        | Integration required for mssql hooks.                 |

dev/breeze/doc/03_developer_tasks.rst

@@ -392,6 +392,29 @@ These are all available flags of ``start-airflow`` command:
   :width: 100%
   :alt: Breeze start-airflow
 
+Running External System Integrations with Breeze
+------------------------------------------------
+
+You can run Airflow alongside external systems in Breeze, such as Kafka, Cassandra, MongoDB, and more.
+
+To start Airflow with an integration, use the following command:
+
+.. code-block:: bash
+
+    breeze --python 3.10 --backend postgres --integration <integration_name>
+
+For example, to run Airflow with Kafka:
+
+.. code-block:: bash
+
+    breeze --python 3.10 --backend postgres --integration kafka
+
+Check the available integrations by running:
+
+.. code-block:: bash
+
+    breeze --integration --help
+
 Launching multiple terminals in the same environment
 ----------------------------------------------------