Move SQS message queue to Amazon provider

[potiuk]

The common.messaging abstraction should discover common messaging queue providers using the same mechanism as we have for other core extensions. Previously common.messaging had the optional (but not really) dependencies to other providers, but that was not needed and introduced unnecessary coupling.

By switching to our built-in discovery mechanism we get immediately all the niceties of provider discovery mechanisms:

• queue is provided by the actual provider where the service or integration already is implemented (sqs -> amazon provider, in the future kafka -> kafka provider)
• queues are discovered from installed providers
• there is no coupling or imports between common.messaging and the providers that implement messaging, the dependency is in the other way - providers that implement messaging depend on common.messaging
• airflow providers queues CLI and providers core extensions documentation is automatically generated

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

Few things that this PR leverages from our existing framework for provider's capabiliy discovery:

• airflow providers queues CLI - showing automatically all message queues discovered from the "get_provider_info" extension points of installed providers:

• Added "Core concept" page describing Message Queue concept:

• Added Summary Page in providers summary doc - with summary of all autoamaticaly discovered message queues from provider.yaml

• Added Message Queuees on the ist of core extensions:

All that comes out-of-the-box after tapping-in the standard provider's discovery mechanism,

BTW. The new uv run --group doc build-docs --autobuild workflow is FANTASTIC - I really started to like editing docuementation, it likely speeds up documentation iteration more than order of magnitude. I was able to modify and move things around super-quickly and preview things while they are changed, see the errors in generated documentation when there were errors etc.

And I would really appreciate some suggestions (or even fixups) to the documentation I created. I slapped things together quickly from AIP-82 and few other places, but I am pretty sure we can do way better and more "catchy" description - maybe even add some images from the AIP-82 design etc.

[potiuk]

That was rather straightforward and quick as I thought - I even added missing unit tests for the SQS provider.

[potiuk]

I'd also appreciate some "real-life" testing @vincbeck - I do not want to setup all the SQS queue for me and I fixed all the unit tests (I think) - so I am quite positive it will work, but well ... you never know.

[potiuk]

If you look closely - our tests had shown that there are a couple of consequences of "fixing the common.messaging" . So I think it would be good to summarize it here. Also for @kaxil and @eladkal -> to agree and realise the impact it will have on Airflow and provider's releases:

• this is definitely the "way to go" for common.messaging - > the coupling introduced by the previous approach would be very difficult to justify and manage, and baking it in the "Providers manger" interface brings in all the niceties we have there and follows all other integrations

• we have to add lower-binding to the common.messaging>=1.1.0 in amazon (and in kafka next) - as the implementation expects to be discovered by the common.messaging discovery mechanism

• also common.messaging should have apache-airflow>=3.0.1 - because the discovery mechanism for common messaging is going to be only avaiable in Airflow 3.0.1 (assuming we will merge that one to 3.0.1).

That leaves a little bit of doubt - should we merge it to 3.0.1 ("bugfix") or should it be 3.1.0 ("feature") of airflow to discover common.messaging providers. I'd lean towards "bug-fix" and merging it with 3.0.1, in a way this is fixing the way how common.messaging works and Airflow 3.0.0 has a lot more things that all but guarantee literally no-one will use it. And we had a precedent for that - Airflow 2.0.0 had a bug in provider's manager interface that would prevent evolution of providers and we implemented a fix in 2.0.1 (or even 2.0.2 - can't remember) that fixed it by adding essentially new thing to airflow. This is not really a "user" facing feature, and intention is to "fix" things - so for me this is reeally a "fix" and merging it to 3.0.1 is something I would strongly recommend. That also will allow us to implement a lot more common.messaging interfaces way quicker without waiting for 3.1.0 (like the Kafka one)

A small consequence of that is that we have to add `"common.messaging" to exclude it for comatibility matrix tests - for Airflow 3.0.0, but that's fine - and we will remove it from that matrix once we bump the compatibility tests to 3.0.1.

I am also torn on 1.0.1 vs. 1.1.0 for common-messaging - It can go with either - we are far more relaxed about feature versions for providers. But I will leave it to @eladkal to decide.

So all seems on a clear path to be cherry-picked to 3.0.1 I think (and the code changed in airflo is really a copy&paste /extension of the other provider manager's features, so that is rather safe to cherry-pick.

[vincbeck]

Overall it looks really good! It is indeed a way better implementation than the previous one, thank you!

[vincbeck]

I'd also appreciate some "real-life" testing @vincbeck - I do not want to setup all the SQS queue for me and I fixed all the unit tests (I think) - so I am quite positive it will work, but well ... you never know.

I just tested it and it works like a charm. No modification on my DAG was needed.

[potiuk]

I'd also appreciate some "real-life" testing @vincbeck - I do not want to setup all the SQS queue for me and I fixed all the unit tests (I think) - so I am quite positive it will work, but well ... you never know.

I just tested it and it works like a charm. No modification on my DAG was needed.

❤️ ❤️ ❤️ ❤️ ❤️

[kaxil]

That leaves a little bit of doubt - should we merge it to 3.0.1 ("bugfix") or should it be 3.1.0 ("feature") of airflow to discover common.messaging providers. I'd lean towards "bug-fix" and merging it with 3.0.1

Yeah, 3.0.1 is fine imo

[kaxil]

I am also torn on 1.0.1 vs. 1.1.0 for common-messaging - It can go with either - we are far more relaxed about feature versions for providers. But I will leave it to @eladkal to decide.

Yeah since it is fully backwards compat it can go in 1.0.1 imo

[potiuk]

I am also torn on 1.0.1 vs. 1.1.0 for common-messaging - It can go with either - we are far more relaxed about feature versions for providers. But I will leave it to @eladkal to decide.

Yeah since it is fully backwards compat it can go in 1.0.1 imo

Agreed. After @vincbeck comment I am also for 1.0.1 on this one too. And I also added a few more >= for other pieces (fab/git etc.) as they are generated automatically now in pre-commit, so that when we cherry-pick to v3-0-test we will make sure 3.0.1 by default will have all the min provider versions. We will just have to make sure to make RC out before we make 3.0.1 rc out. The current release process should seamlessly use such .rc candidates when they are in PyPI and we have protection in place that we won't be able to build the image before those RC`s graduate to be relesed and we update constraints.

[potiuk]

The special tests behave strangely. I added a bit better diagnostics as well to see what's going on - now we will have more detailed logging showing what providers are registred and whether any providers are registered when we cannot find the right provider. Let's see what the tests will show.

[potiuk]

Ough. Diagnostics help. I think we have some side-effects where provider's Manager is cleaned and not restored in some other tests I will look for this - and add protection.

[potiuk]

Ah... I see now - > also we have to exclude that test for latest botocore, because it requires aiobotocore 🤦

[github-actions]

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker 4f962f4 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue