Skip to content

Delete MWAA #352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
May 21, 2025
Merged

Delete MWAA #352

merged 13 commits into from
May 21, 2025

Conversation

ividito
Copy link
Contributor

@ividito ividito commented Apr 15, 2025

Summary

Starts to address #315

  • Remove MWAA infrastructure
  • Remove MWAA CI
  • Rework Makefile and scripts to match new folder structure (no more copying dags folder)
  • SM2A moved to top level of repo
  • Initial documentation refactor in Restructure SM2A documentation #363

TODO

  • Workflows API was managed with MWAA terraform. Should we keep it? We could move it into the SM2A infra easily enough, not sure what's needed for the runtime.
  • The SM2A README can be combined with the root README, but I haven't done this yet Restructure SM2A documentation #363
  • Merge https://github.com/NASA-IMPACT/veda-deploy/pull/113 when we merge this in
  • Some of the diffs don't look right (AFAIK many of the sm2a infra files that got marked as deleted were just moved, probably not an issue but I will double-check that the end result is what we intended)

This also unblocks additional work on #351 , where we can significantly improve Docker build times.

@smohiudd
Copy link
Contributor

I think we should fully deprecate workflows API and remove it. We're not promoting it anymore as a way to ingest data

@smohiudd
Copy link
Contributor

Should we merge #272 and #297 first before merging this PR to minimize conflicts?

@ividito
Copy link
Contributor Author

ividito commented Apr 16, 2025

@smohiudd #272 should definitely go in first, #297 we can discuss at tag-up (I know we were unsure about it - I tested that the feature flag works on SIT, I think we can merge it and decide whether to enable it or not later on)

@ividito ividito marked this pull request as ready for review April 30, 2025 16:16
@ividito ividito mentioned this pull request May 12, 2025
ividito and others added 7 commits May 12, 2025 15:59
Co-authored-by: Jennifer Tran <[email protected]>
- add tags we should be using
- direct readers to DAG run templates, as well as veda-data examples
@ividito
Copy link
Contributor Author

ividito commented May 12, 2025

We'll deploy this to SIT using veda-deploy to test the CI, if that works this should be good to merge

Copy link
Member

@botanical botanical left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some small nits. Thanks for tackling such a large PR

Co-authored-by: Jennifer Tran <[email protected]>
@ividito ividito merged commit 7b5cf7e into dev May 21, 2025
6 checks passed
@ividito ividito deleted the prune/mwaa branch May 21, 2025 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants