Make it easier and faster to iterate on task-sdk-integration-tests

There were a few of things that made task-sdk-tests iteration
a bit unobvious and slower. With this PR, we should be able
to iterate over task-sdk-integration-tests WAY faster and get
more contributors involved in contributing to those.

* It was not clear that prerequisite of running the tests was building
  PROD image for Pyton 3.10. This is now clear in the documentation.

* PROD images can be built in two different modes - from sources
  with --installation-method equal to . or from packages with
  the --installatio-method equal to "apache-airflow". This was
  not clearly communicated during build and it is now printed at
  output

* It was not clear that when you build PROD images from sources,
  you should first compile ui assets, because otehrwise the
  assets are not added as part of the image. With this PR the
  `breeze prod-image build` command checks if the .vite manifest
  is present in the right `dist` folders and will error out,
  suggesting to run `breeze compile-ui-assets` before.

* when building PROD images from sources, it is faster to rebuild
  the images with `uv` than with `pip`. the --use-uv parameter now
  defaults to False when building from packages and to True when
  building from sources.

* There was an error in .dockerignore where generated dist files
  were not added to context when PROD image was built from sources.
  This resulted in "permission denied' when such PROD images were used
  to run tests.

* The test compose had fallback of Airflow 3.0.3 which would be
  misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory

* We are now mounting sources of Airflow to inside the image by default
  and skip it in CI. This mounting happens in local environment where PROD
  image is built usually from sources, and it is disabled in CI by using
  --skip-mounting-local-volumes flag. We also do not stop docker compose
  by default when runnig it locally in order to make fast iteration the
  default.

* We pass host operating system when starting the compose, and we only
  change ownership on Linux - this is a long running operation on MacOS
  because mounted filesystem is slow, but it's also not needed on MacOS
  because the file system also maps ownershipt and files created by
  Airflow are created with local user id.

* We pass local user id to containers to make sure that the files
  created on linux are created by the local user (logs and the like).

* We are now detecting whether docker-compose is running and when we run
  with locally mounted sources, we reuse those running containers. When
  we don't mount local sources, we shut-down the compose before running
  to make sure we do not have sources mounted - and we close the compose
  by default when we do not mount local sources.

* When sources are mounted we are only enabling DEV_MODE inside the containers
  so that components are hot-reloading (new feature added in apache#57741
  last weeks. This way you do not have to restart anything when sources
  are changed and you can re-run the tests when docker compose is running.

* The environment are passsed now via .env file so that you can easily
  reproduce docke compose command locally

* The docker compose files are not copied any more, they are moved directly
  to the top of 'task-sdk-integraiton-tests' and used from there.

* Additional diagnostics added to show what's going on.

* Handling verbose option from breeze by adding more debugging informatio

* Updated documentation about the tests

* Small QOL immprovement - if expected dags are not yet parsed by dag file
  processor, when test starts, getting their status will return 404 "Not
  Found". In such case our tests implemented a short retry scheme with tenacity

Author: potiuk

.dockerignore

@@ -140,6 +140,7 @@ airflow/www/node_modules
 
 # But ensure UI dist files are included
 !airflow-core/src/airflow/ui/dist
+!airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/dist
 !providers/fab/src/airflow/providers/fab/www/dist
 !providers/edge3/src/airflow/providers/edge3/plugins/www/dist
 

.github/workflows/additional-prod-image-tests.yml

@@ -190,7 +190,7 @@ jobs:
           make-mnt-writeable-and-cleanup: true
         id: breeze
       - name: "Run Task SDK integration tests"
-        run: breeze testing task-sdk-integration-tests
+        run: breeze testing task-sdk-integration-tests --skip-mounting-local-volumes
 
   test-e2e-integration-tests-basic:
     name: "Test e2e integration tests with PROD image"

.gitignore

@@ -281,3 +281,6 @@ _e2e_test_report.json
 
 # UV cache
 .uv-cache/
+
+# Allow logs in task-sdk-integration-tests
+!/task-sdk-integration-tests/logs/

contributing-docs/testing/task_sdk_integration_tests.rst

@@ -68,13 +68,50 @@ Why Task SDK Integration?
 Running Task SDK Integration Tests
 ----------------------------------
 
-There are multiple ways to run Task SDK Integration Tests depending based on your preferences.
+Prerequisite - build PROD image
+...............................
+
+.. note::
+
+   The task-sdk integration tests are using locally build production images started in docker-compose by
+   Pytest. This means that while the tests are running in the environment that you start it from (usually
+   local development environment), you need to first build the images that you want to test against.
+
+You also need to make sure that your assets are built first.
+.. code-block:: bash
+
+   # From the Airflow repository root
+   breeze compile-ui-assets
+
+Then, you should build the base image once before running the tests. You can do it using Breeze:
+
+.. code-block:: bash
+
+   # From the Airflow repository root
+   breeze prod-image build --python 3.10
+
+This will build the ``ghcr.io/apache/airflow/main/prod/python3.10.latest`` image that will be used by
+default to run the tests. The ``breeze prod image build`` command by default - when run from sources of
+airflow - will use the local sources and build the image using ``uv`` to speed up the build process. Also when
+building from sources it will check if the assets are built and will error if they are not. However it will
+not check if the assets are up to date - so make sure to run the ``breeze compile-ui-assets`` command
+above if you have changed any UI sources and did not build your assets after that.
+
+Note that you do not have to rebuild the image every time you run the tests and change Python sources - because
+the docker-compose setup we use in tests will automatically mount the local Python sources into the
+container, so you can iterate quickly without rebuilding the image. However, if you want to test changes
+that affect the image (like modifying dependencies, system packages, rebuilding UI etc.) you will need to
+rebuild the image with the same command as above.
+
+After you build the image, there are several ways to run Task SDK Integration Tests,
+depending based on your preferences.
 
 Using Breeze
 ............
 
 The simplest way to run Task SDK Integration Tests is using Breeze, which provides CI like
-reproducibility:
+reproducibility
+
 
 .. code-block:: bash
 
@@ -87,36 +124,74 @@ reproducibility:
    # Run with custom Docker image
    DOCKER_IMAGE=my-custom-airflow-image:latest breeze testing task-sdk-integration-tests
 
-Running in Your Current Virtual Environment
-...........................................
+Using uv
+........
 
-Since you're already working in the Airflow repository, you can run Task SDK Integration Tests
-directly:
+Since you're already working in the Airflow repository, you can run Task SDK Integration Tests directly:
 
 **Run Tests**
 
 .. code-block:: bash
 
    # Navigate to task-sdk-integration-tests directory and run tests
-   cd task-sdk-integration-tests/
+   cd task-sdk-integration-tests
    uv run pytest -s
 
    # Run specific test file
-   cd task-sdk-integration-tests/
+   cd task-sdk-integration-tests
    uv run pytest tests/task_sdk_tests/test_task_sdk_health.py -s
 
-   # Keep containers running for debugging
-   cd task-sdk-integration-tests/
-   SKIP_DOCKER_COMPOSE_DELETION=1 uv run pytest -s
-
 **Optional: Set Custom Docker Image**
 
 .. code-block:: bash
 
    # Use a different Airflow image for testing
-   cd task-sdk-integration-tests/
+   cd task-sdk-integration-tests
    DOCKER_IMAGE=my-custom-airflow:latest uv run pytest -s
 
+By default when you run your tests locally, the Docker Compose deployment is kept between the sessions,
+your local sources are mounted into the containers and the Airflow services are restarted automatically
+(hot reloaded) when Python sources change.
+
+This allows for quick iterations without rebuilding the image or restarting the containers.
+
+Stopping docker-compose
+.......................
+
+When you finish testing locally (or when you updated dependencies and rebuild your images),
+you likely want to stop the running containers. You can stop the the running containers by running:
+
+.. code-block:: bash
+
+   # Stop and remove containers
+   cd task-sdk-integration-tests
+   docker-compose down -v
+
+Docker compose will be automatically started again next time you run the tests.
+
+Running tests in the way CI does it
+....................................
+
+Our CI runs the tests in a clean environment every time without mounting local sources. This means that
+any changes you have locally will not be visible inside the containers. You can reproduce it locally by adding
+--skip-mounting-local-volumes to breeze command or by setting SKIP_MOUNTING_LOCAL_VOLUMES=1 in your
+environment when running tests locally. Before that however make sure that your PROD image is rebuilt
+using latest sources. When you disable mounting local volumes, the containers will be stopped by default
+when the tests end, you can disable that by setting SKIP_DOCKER_COMPOSE_DELETION=1 in your environment
+or passing --skip-docker-compose-deletion to breeze command.
+
+.. code-block:: bash
+
+   # Keep containers running for debugging
+   cd task-sdk-integration-tests
+   SKIP_MOUNTING_LOCAL_VOLUMES=1 uv run pytest -s
+
+or
+
+.. code-block:: bash
+
+  # Using Breeze to keep containers running
+  breeze testing task-sdk-integration-tests --skip-mounting-local-volumes
 
 Debugging Failed Tests
 ......................
@@ -127,33 +202,23 @@ and the Docker Compose deployment is shut down. To debug issues more effectively
 .. code-block:: bash
 
    # Run with maximum verbosity
-   cd task-sdk-integration-tests/
+   cd task-sdk-integration-tests
    uv run pytest tests/task_sdk_tests/ -vvv -s --tb=long
 
-   # Keep containers running for inspection (local environment)
-   cd task-sdk-integration-tests/
-   SKIP_DOCKER_COMPOSE_DELETION=1 uv run pytest tests/task_sdk_tests/test_task_sdk_health.py::test_task_sdk_health
-
-   # Keep containers running for inspection (using Breeze)
-   breeze testing task-sdk-integration-tests --skip-docker-compose-deletion
-
-   # Inspect container logs (when containers are still running)
-   cd task-sdk-integration-tests/docker
-   docker-compose logs airflow-apiserver
-   docker-compose logs airflow-scheduler
-   docker-compose logs postgres
+   # Inspect container logs (when containers are still running - which is default)
+   # The -f flag follows the logs in real-time so you can open several terminals to monitor different services
+   cd task-sdk-integration-tests
+   docker-compose logs -f airflow-apiserver
+   docker-compose logs -f airflow-scheduler
+   docker-compose logs -f postgres
 
    # Access running containers for interactive debugging
-   docker-compose exec airflow-apiserver bash
-
-.. tip::
-   **Container Cleanup Control**: By default, the Docker Compose deployment is deleted after tests
-   complete to keep your system clean. To keep containers running for debugging:
+   docker-compose exec -it airflow-apiserver bash
 
-   - **Local environment**: Export ``SKIP_DOCKER_COMPOSE_DELETION=1`` before running tests
-   - **Breeze environment**: Use the ``--skip-docker-compose-deletion`` flag
+Every time you save airflow source code, the components running inside the container will be restarted
+automatically (hot reloaded). You can disable this behaviour by setting SKIP_MOUNTING_LOCAL_VOLUMES=1
+as described above (but then your sources will not be mounted).
 
-   Remember to manually clean up containers when done: ``cd task-sdk-integration-tests/docker && docker-compose down -v``
 
 Testing Custom Airflow Images
 ..............................