Add CLI hot-reload support via --dev flag #57741
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jason810496
changed the title
amoghrajesh
reviewed
Nov 3, 2025
potiuk
reviewed
Nov 4, 2025
potiuk
approved these changes
Nov 4, 2025
Member
potiuk
left a comment
potiuk
left a comment
There was a problem hiding this comment.
Looks good - the only thing I miss is integration with breeze start-airflow --dev-mode
potiuk
approved these changes
Nov 4, 2025
potiuk
force-pushed
the
feature/cli/add-dev-flag
branch
from
9e4e7b0 to
7f4acd5
Compare
jason810496
force-pushed
the
feature/cli/add-dev-flag
branch
from
7f4acd5 to
9aefac2
Compare
amoghrajesh
approved these changes
Nov 5, 2025
Lee-W
approved these changes
Nov 5, 2025
Member
Lee-W
left a comment
Lee-W
left a comment
There was a problem hiding this comment.
a few nits, but overall looks good
Co-authored-by: jason810496 <[email protected]>
Co-authored-by: jason810496 <[email protected]>
Co-authored-by: jason810496 <[email protected]>
…rror messages Co-authored-by: jason810496 <[email protected]>
Co-authored-by: jason810496 <[email protected]>
… dag-processor support Co-authored-by: jason810496 <[email protected]>
Co-authored-by: jason810496 <[email protected]>
Co-authored-by: jason810496 <[email protected]>
Co-authored-by: jason810496 <[email protected]>
- remove logger in triggerer command - ensure type annotation in hot-reload module - fix test hot-reload
jason810496
force-pushed
the
feature/cli/add-dev-flag
branch
from
9aefac2 to
dc41c5c
Compare
potiuk
approved these changes
Nov 10, 2025
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging informatio * Updated documentation about the tests * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging informatio * Updated documentation about the tests * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging informatio * Updated documentation about the tests * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging informatio * Updated documentation about the tests * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging informatio * Updated documentation about the tests * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging informatio * Updated documentation about the tests * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging informatio * Updated documentation about the tests * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * If the PROD image has not been built before, breeze will propose to build it and even do it automatically if the answer is not provided within 20 seconds. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * A `--down` flag is added to breeze testing task-sdk-integration-tests that tears down running docker compose. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging information * Updated documentation about the tests to be more comprehensive about the options, updated file structure etc. * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * If the PROD image has not been built before, breeze will propose to build it and even do it automatically if the answer is not provided within 20 seconds. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * A `--down` flag is added to breeze testing task-sdk-integration-tests that tears down running docker compose. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging information * Updated documentation about the tests to be more comprehensive about the options, updated file structure etc. * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 12, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * If the PROD image has not been built before, breeze will propose to build it and even do it automatically if the answer is not provided within 20 seconds. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * A `--down` flag is added to breeze testing task-sdk-integration-tests that tears down running docker compose. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging information * Updated documentation about the tests to be more comprehensive about the options, updated file structure etc. * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 13, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * If the PROD image has not been built before, breeze will propose to build it and even do it automatically if the answer is not provided within 20 seconds. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * A `--down` flag is added to breeze testing task-sdk-integration-tests that tears down running docker compose. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging information * Updated documentation about the tests to be more comprehensive about the options, updated file structure etc. * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 13, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * If the PROD image has not been built before, breeze will propose to build it and even do it automatically if the answer is not provided within 20 seconds. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * A `--down` flag is added to breeze testing task-sdk-integration-tests that tears down running docker compose. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging information * Updated documentation about the tests to be more comprehensive about the options, updated file structure etc. * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 13, 2025
There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * If the PROD image has not been built before, breeze will propose to build it and even do it automatically if the answer is not provided within 20 seconds. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in apache#57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * A `--down` flag is added to breeze testing task-sdk-integration-tests that tears down running docker compose. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging information * Updated documentation about the tests to be more comprehensive about the options, updated file structure etc. * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
potiuk
added a commit
that referenced
this pull request
Nov 13, 2025
…58231) There were a few of things that made task-sdk-tests iteration a bit unobvious and slower. With this PR, we should be able to iterate over task-sdk-integration-tests WAY faster and get more contributors involved in contributing to those. * It was not clear that prerequisite of running the tests was building PROD image for Pyton 3.10. This is now clear in the documentation. * PROD images can be built in two different modes - from sources with --installation-method equal to . or from packages with the --installatio-method equal to "apache-airflow". This was not clearly communicated during build and it is now printed at output * It was not clear that when you build PROD images from sources, you should first compile ui assets, because otehrwise the assets are not added as part of the image. With this PR the `breeze prod-image build` command checks if the .vite manifest is present in the right `dist` folders and will error out, suggesting to run `breeze compile-ui-assets` before. * If the PROD image has not been built before, breeze will propose to build it and even do it automatically if the answer is not provided within 20 seconds. * when building PROD images from sources, it is faster to rebuild the images with `uv` than with `pip`. the --use-uv parameter now defaults to False when building from packages and to True when building from sources. * There was an error in .dockerignore where generated dist files were not added to context when PROD image was built from sources. This resulted in "permission denied' when such PROD images were used to run tests. * The test compose had fallback of Airflow 3.0.3 which would be misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory * We are now mounting sources of Airflow to inside the image by default and skip it in CI. This mounting happens in local environment where PROD image is built usually from sources, and it is disabled in CI by using --skip-mounting-local-volumes flag. We also do not stop docker compose by default when runnig it locally in order to make fast iteration the default. * We pass host operating system when starting the compose, and we only change ownership on Linux - this is a long running operation on MacOS because mounted filesystem is slow, but it's also not needed on MacOS because the file system also maps ownershipt and files created by Airflow are created with local user id. * We pass local user id to containers to make sure that the files created on linux are created by the local user (logs and the like). * We are now detecting whether docker-compose is running and when we run with locally mounted sources, we reuse those running containers. When we don't mount local sources, we shut-down the compose before running to make sure we do not have sources mounted - and we close the compose by default when we do not mount local sources. * When sources are mounted we are only enabling DEV_MODE inside the containers so that components are hot-reloading (new feature added in #57741 last weeks. This way you do not have to restart anything when sources are changed and you can re-run the tests when docker compose is running. * The environment are passsed now via .env file so that you can easily reproduce docke compose command locally * The docker compose files are not copied any more, they are moved directly to the top of 'task-sdk-integraiton-tests' and used from there. * A `--down` flag is added to breeze testing task-sdk-integration-tests that tears down running docker compose. * Additional diagnostics added to show what's going on. * Handling verbose option from breeze by adding more debugging information * Updated documentation about the tests to be more comprehensive about the options, updated file structure etc. * Small QOL immprovement - if expected dags are not yet parsed by dag file processor, when test starts, getting their status will return 404 "Not Found". In such case our tests implemented a short retry scheme with tenacity
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
related: #47595
Why
Improve development speed by enabling hot-reload, so components don’t need to be manually restarted when making changes in airflow-core.
What
--devflag support for all component CLIs (except the API server, which already supports hot-reload via uvicorn).watchfilespackage to detect file changes and usepsutilto recursively clean up processes.airflow serve-logprocess was not reliably terminated.pyproject.tomlis required becausewatchfilesis already included through FastAPI’s dependency tree.Demo
hot-reload-demo.mov