Conversation
ividito
force-pushed
the
feat/transactions-api
branch
from
ad3fac7 to
cccda9e
Compare
amarouane-ABDELHAK
left a comment
amarouane-ABDELHAK
left a comment
There was a problem hiding this comment.
Thank you for the PR. I just left some small comments
|
This branch has been deployed to SIT, but testing is currently blocked because the dev STAC API has switched over to using veda-keycloak and we still need to sort out how programatic auth will work. |
|
This branch was tested by switching on and off the |
sm2a/infrastructure/main.tf
Outdated
| CLOUDFRONT_TO_INVALIDATE = var.cloudfront_to_invalidate | ||
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate | ||
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate, | ||
| TRANSACTIONS_ENDPOINT_ENABLED = var.transactions_endpoint_enabled==true ? "True" : null |
There was a problem hiding this comment.
I don't think tf bools are serialized in airflow in python. I used this hack but open if there's a better way.
|
|
||
| group_kwgs = {"group_id": "Process", "tooltip": "Process"} | ||
|
|
||
| airflow_vars = Variable.get("aws_dags_variables") |
There was a problem hiding this comment.
If we set deserialize_json=True, do you know if we still need to do:
var.transactions_endpoint_enabled==true ? "True" : null
in main.tf?
| airflow_vars = Variable.get("aws_dags_variables") | |
| airflow_vars = Variable.get("aws_dags_variables", deserialize_json=True) |
And maybe we refactor this file a bit so we call Variable.get in tasks rather than in the top level, since the docs seem to imply this is best practice: https://airflow.apache.org/docs/apache-airflow/2.8.4/best-practices.html#airflow-variables
There was a problem hiding this comment.
I would love a refactor for this (we call the same set of Variable.get("aws_dags_variables") lines 16 times in our DAGs), but I think that's a different issue. .airflowignore keeps the DAG processor from parsing the groups and utils folders, so there's no performance loss by having these calls here.
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate | ||
| CLOUDFRONT_TO_INVALIDATE = var.cloudfront_to_invalidate, | ||
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate, | ||
| TRANSACTIONS_ENDPOINT_ENABLED = var.transactions_endpoint_enabled==true ? "True" : null, |
There was a problem hiding this comment.
Related fix that removes the need for the ternary here: NASA-IMPACT/self-managed-apache-airflow#18
- corrected type hint - corrected exception in auth method - removed unused cognito var
ividito
force-pushed
the
feat/transactions-api
branch
from
f9b9efc to
bef1371
Compare
|
^^ rebased to pull in changes from dev |
|
Based on conversations with stac-fastapi maintainers, they recommend we don't use bulk ingestion. I tried instead to use the PUT {collection_id}/items/{item_id} endpoint for create and update operations but stac-fastapi only supports updates. The submit stac util will need to be refactored to handle both creating new items via POST and updating items via PUT. @ividito I suggest we proceed with mwaa refactor PR since this transactions PR will need more time. |
4 participants
Proposed implementation for #288
Original ingest API support is maintained through an Airflow variable. I learned while setting this up that this is not a recommended approach, but it is consistent with what we do elsewhere.