…24) * work to update to newer airflow version * work to update to newer airflow version #2 * removed the use of PREINGESTION_AMI for PipelinesSingleEMRConfig * make the job schedule config optional * make the job schedule config optional * optional indexing for ingestion * optional indexing for ingestion #2 * cope with empty s3 path * cope with empty s3 path * remove dataset from solr before reindexing * remove dataset from solr before reindexing #2 * switch to using unzip to avoid "java.util.zip.ZipException: only DEFLATED entries" errors * rewritten delete dag to use boto3 instead of aws cli commands to remove the requuirement to have aws cli installed on running container * rewritten delete dag to use boto3 instead of aws cli commands to remove the requirement to have aws cli installed on running container #2 * rewritten delete dag to use boto3 instead of aws cli commands to remove the requirement to have aws cli installed on running container #2 * rewritten delete dag to use boto3 instead of aws cli commands to remove the requirement to have aws cli installed on running container #2 * airflow 2.10.1 composer update * Change devcontainer name to 2.10.1 * Changed update gbif metadata to use aws s3 sync * return response as the output for call_url * Support SDS generate XML DAG * Jar validation added * Added java support for devcontainer * Some refactoring in SDS generate XML * renamed to ..._dag.py * Enable debugpy helper added * Added Tags to the EMR clusters * reformatted pre-ingest dag * Changed the devcontainer image to have python 3.11 same as MWAA * Enable test of the connection in Airflow devcontainer * did a fix in launch.json for vscode to use enable_debugpy in doco * Added slack_notification var into env variables * Added slack notifications to DAG default args * formatted * Updated dependencies in requirements * Added slack notification channel as a variable * Created callback function for failure and success method for notification * Added success notification operator as last task to DAGs * Added success notification operator as last task to DAGs * Added success notification operator as last task to DAGs * Added success notification operator as last task to DAGs * Fixed the slack notification fail for hobs with no success in the past * Added a test launch for update collection alias cli * Mainly changed for IDs having https in them * Requirement has airflow constraint file inclusion * Fixed Redis version for devcontainer * Added git to the image for devcontainer * WIP * Feature/emr upgrade issue (#21) * #20 - Fix emr application provisioning error when using higher versions of EMR. * Add commit 7276315 which had been missed in branch 2_10_1 * Fix to set the pipelines config * Set EMR to not terminate for debugging purpose * Upgrade Airflow version to 2.10.5 and update devcontainer configuration * Update Airflow version to 2.10.5 and modify postCreateCommand; add airflow-webserver.pid and airflow.db to .gitignore * Add SSH support to Docker setup and install git * Added pre-sync count for checking the asserion records * implemented the pre-sync assertion count step to full-index and update alias DAG * Fixed openlineage issue for devcontainer * Enhance error handling and add timeouts for HTTP requests; improve docstrings for clarity * Add error output redirection to bash commands in EMR steps * Improve docstrings for bootstrap actions and configuration * Enhance documentation for EMR cluster functions and configurations; add detailed docstrings for clarity and usage * Refactor DAG scripts to streamline bash command imports and improve readability; update EMR job flow overrides to include dataset references * Add SSH tunnel command to docker-compose for Solr Test access in worker nodes * Refactor PythonOperator calls to streamline lambda functions and improve readability in multiple DAGs * Add AWS credentials volume to docker-compose for Airflow access * Refactor pre_ingest_drs_dag.py * Redirect stderr to stdout in step command arguments for improved logging in EMR steps * Refactor load_dataset_dag.py to improve readability and streamline metadata retrieval * Enhance metadata handling in pre_ingest_drs_dag.py and ala_helper.py by adding update_checked_date task and supporting functions for fetching and updating dataset metadata * Remove debugpy import and call from update_checked_date function in pre_ingest_drs_dag.py * Add deploy script for S3 bucket deployment of Airflow DAGs * Fix S3 deployment script to correctly reference local dags directory and improve directory renaming logic * Add AWS CLI feature to devcontainer configuration * Uupdate EMRConfig Name property to use ENVIRONMENT_TYPE from ala_config * Refactor EMR cluster creation to use PythonOperator and streamline setup process across multiple DAGs * Add AWS configuration to docker-compose for local development * #23 - generate parquet using pyspark * Fixed error with preingest DAG * Removed versions from requirements.txt to be compatible by constraints file * Add 'run_indexing' parameter to task context in load_dataset DAG * Refactor DAGs to pass 'dataset_ids' parameter to run_large_emr function for improved dataset handling * Add .env to .gitignore to prevent environment configuration files from being tracked * Update Dockerfile and devcontainer configuration; enhance dataset download script and DAGs for improved functionality * Fix script reference in elastic_dataset_dag to use bootstrap-index-actions.sh for dataset indexing * Reorder task dependencies in full_index_to_solr DAG for Assertion Sync record count * Refactor update_gbif_metadata function to simplify HTTP request handling using ala_helper.call_url * Remove outdated constraints file for Python 3.11 compatibility * Fix logging message to correctly reference dataset identifier in pre_ingest_drs_dag * Enhance Solr commit monitoring by adding count change checks and increasing max count for status checks * Enhance call_url function by adding timeout parameter and updating get_assertion_records_count to use call_url for improved error handling * Add strtobool function and refactor delete_drs_dag to improve object deletion logic * Remove debugpy import and initialization from delete_drs_dag.py * Refactor SOLR indexing DAG to generate collection name in a step * Add sanitize_name function for EMR cluster name validation and update EMRConfig to use it * Refactor ingest_all_datasets DAG to improve dataset partitioning logic and streamline task dependencies * Add EMR cluster node count and total threshold variables to ala_config and update ingest_all_datasets DAG to use them * Move shell scripts into `sh` folder * Add CloudWatch Agent bootstrap script and update EMRConfig to include it; refactor S3 paths in various bootstrap scripts * Refactor ingest_all_datasets DAG to enhance task grouping and dynamic mapping for xlarge datasets; streamline dependency management and improve code clarity. --------- Co-authored-by: Dave Martin <djtfmartin@gmail.com> Co-authored-by: Patricia Koh <patricia.koh@csiro.au>

Launch ingest dag with different emr version failed #20

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions