Submit STAC using transactions API #297
Conversation
ividito
force-pushed
the
feat/transactions-api
branch
from
ad3fac7 to
cccda9e
Compare
- corrected type hint - corrected exception in auth method - removed unused cognito var
|
This branch has been deployed to SIT, but testing is currently blocked because the dev STAC API has switched over to using veda-keycloak and we still need to sort out how programatic auth will work. |
|
This branch was tested by switching on and off the |
sm2a/infrastructure/main.tf
Outdated
| @@ -104,7 +104,9 @@ module "sma-base" { | |||
| ASSUME_ROLE_WRITE_ARN = var.assume_role_write_arn, | |||
| SM2A_BASE_URL = module.sma-base.airflow_url, | |||
| CLOUDFRONT_TO_INVALIDATE = var.cloudfront_to_invalidate | |||
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate | |||
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate, | |||
| TRANSACTIONS_ENDPOINT_ENABLED = var.transactions_endpoint_enabled==true ? "True" : null | |||
There was a problem hiding this comment.
I don't think tf bools are serialized in airflow in python. I used this hack but open if there's a better way.
|
|
||
| group_kwgs = {"group_id": "Process", "tooltip": "Process"} | ||
|
|
||
| airflow_vars = Variable.get("aws_dags_variables") |
There was a problem hiding this comment.
If we set deserialize_json=True, do you know if we still need to do:
var.transactions_endpoint_enabled==true ? "True" : null
in main.tf?
| airflow_vars = Variable.get("aws_dags_variables") | |
| airflow_vars = Variable.get("aws_dags_variables", deserialize_json=True) |
And maybe we refactor this file a bit so we call Variable.get in tasks rather than in the top level, since the docs seem to imply this is best practice: https://airflow.apache.org/docs/apache-airflow/2.8.4/best-practices.html#airflow-variables
There was a problem hiding this comment.
I would love a refactor for this (we call the same set of Variable.get("aws_dags_variables") lines 16 times in our DAGs), but I think that's a different issue. .airflowignore keeps the DAG processor from parsing the groups and utils folders, so there's no performance loss by having these calls here.
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate | ||
| CLOUDFRONT_TO_INVALIDATE = var.cloudfront_to_invalidate, | ||
| CLOUDFRONT_PATH_TO_INVALIDATE = var.cloudfront_path_to_invalidate, | ||
| TRANSACTIONS_ENDPOINT_ENABLED = var.transactions_endpoint_enabled==true ? "True" : null, |
There was a problem hiding this comment.
Related fix that removes the need for the ternary here: NASA-IMPACT/self-managed-apache-airflow#18
Proposed implementation for #288
Original ingest API support is maintained through an Airflow variable. I learned while setting this up that this is not a recommended approach, but it is consistent with what we do elsewhere.