Bug: `download_gtfs_schedule_v2.email_download_failures` fails to send email

[erikamov]

Describe the bug
When download_gtfs_schedule_v2 DAG runs
It return [Errno 99] Cannot assign requested address error on email_download_failures task.

It was temporarily disabled to reduce all the noise on errors in the Pipeline.

To Reproduce
1- On local or Staging Airflow, trigger a new run on download_gtfs_schedule_v2.
2 - See the log

Expected behavior
Expected to see no errors.

Additional context

Full Log:

default-hostname
*** Reading remote logs from Cloud Logging.
[2025-10-06, 03:05:55 UTC] {taskinstance.py:1160} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: download_gtfs_schedule_v2.email_download_failures scheduled__2025-10-05T03:00:00+00:00 [queued]>
[2025-10-06, 03:05:55 UTC] {taskinstance.py:1160} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: download_gtfs_schedule_v2.email_download_failures scheduled__2025-10-05T03:00:00+00:00 [queued]>
[2025-10-06, 03:05:55 UTC] {taskinstance.py:1362} INFO - Starting attempt 2 of 2
[2025-10-06, 03:05:55 UTC] {taskinstance.py:1383} INFO - Executing <Task(PythonOperator): email_download_failures> on 2025-10-05 03:00:00+00:00
[2025-10-06, 03:05:55 UTC] {standard_task_runner.py:57} INFO - Started process 2823 to run task
[2025-10-06, 03:05:55 UTC] {standard_task_runner.py:84} INFO - Running: ['airflow', 'tasks', 'run', 'download_gtfs_schedule_v2', 'email_download_failures', 'scheduled__2025-10-05T03:00:00+00:00', '--job-id', '3025448', '--raw', '--subdir', 'DAGS_FOLDER/dags.py', '--cfg-path', '/tmp/tmpt560a_yo']
[2025-10-06, 03:05:55 UTC] {standard_task_runner.py:85} INFO - Job 3025448: Subtask email_download_failures
[2025-10-06, 03:05:55 UTC] {task_command.py:422} INFO - Running <TaskInstance: download_gtfs_schedule_v2.email_download_failures scheduled__2025-10-05T03:00:00+00:00 [running]> on host airflow-worker-hpljr
[2025-10-06, 03:05:55 UTC] {taskinstance.py:1664} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='hello@calitp.org' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='download_gtfs_schedule_v2' AIRFLOW_CTX_TASK_ID='email_download_failures' AIRFLOW_CTX_EXECUTION_DATE='2025-10-05T03:00:00+00:00' AIRFLOW_CTX_TRY_NUMBER='2' AIRFLOW_CTX_DAG_RUN_ID='scheduled__2025-10-05T03:00:00+00:00'
[2025-10-06, 03:05:55 UTC] {warnings.py:110} WARNING - /opt/python3.11/lib/python3.11/site-packages/airflow/utils/context.py:314: AirflowContextDeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_start' or 'logical_date' instead.
  warnings.warn(_create_deprecation_warning(k, replacements))

[2025-10-06, 03:05:55 UTC] {base.py:73} INFO - Using connection ID 'smtp_postmark' for task execution.
[2025-10-06, 03:05:55 UTC] {email.py:270} INFO - Email alerting: attempt 1
[2025-10-06, 03:05:55 UTC] {taskinstance.py:1939} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/python3.11/lib/python3.11/site-packages/airflow/operators/python.py", line 192, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.11/lib/python3.11/site-packages/airflow/operators/python.py", line 209, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/gcs/dags/download_gtfs_schedule_v2/email_download_failures.py", line 41, in email_failures
    send_email(
  File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/email.py", line 80, in send_email
    return backend(
           ^^^^^^^^
  File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/email.py", line 154, in send_email_smtp
    send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
  File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/email.py", line 272, in send_mime_email
    smtp_conn = _get_smtp_connection(smtp_host, smtp_port, smtp_timeout, smtp_ssl)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/email.py", line 316, in _get_smtp_connection
    return smtplib.SMTP(host=host, port=port, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.11/lib/python3.11/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.11/lib/python3.11/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.11/lib/python3.11/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.11/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/opt/python3.11/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address
[2025-10-06, 03:05:55 UTC] {warnings.py:110} WARNING - /opt/python3.11/lib/python3.11/configparser.py:1272: FutureWarning: section/key [core/dag_concurrency] has been deprecated, you should use[core/max_active_tasks_per_dag] instead. Please update your `conf.get*` call to use the new name
  if not self._parser.has_option(self._name, key):

[2025-10-06, 03:05:55 UTC] {warnings.py:110} WARNING - /opt/python3.11/lib/python3.11/configparser.py:1274: FutureWarning: section/key [core/dag_concurrency] has been deprecated, you should use[core/max_active_tasks_per_dag] instead. Please update your `conf.get*` call to use the new name
  return self._parser.get(self._name, key)

[2025-10-06, 03:05:55 UTC] {warnings.py:110} WARNING - /opt/python3.11/lib/python3.11/configparser.py:1272: FutureWarning: section/key [api/auth_backend] has been deprecated, you should use[api/auth_backends] instead. Please update your `conf.get*` call to use the new name
  if not self._parser.has_option(self._name, key):

[2025-10-06, 03:05:55 UTC] {warnings.py:110} WARNING - /opt/python3.11/lib/python3.11/configparser.py:1274: FutureWarning: section/key [api/auth_backend] has been deprecated, you should use[api/auth_backends] instead. Please update your `conf.get*` call to use the new name
  return self._parser.get(self._name, key)

[2025-10-06, 03:05:55 UTC] {warnings.py:110} WARNING - /opt/python3.11/lib/python3.11/configparser.py:1272: FutureWarning: section/key [celery/worker_log_server_port] has been deprecated, you should use[logging/worker_log_server_port] instead. Please update your `conf.get*` call to use the new name
  if not self._parser.has_option(self._name, key):

[2025-10-06, 03:05:55 UTC] {warnings.py:110} WARNING - /opt/python3.11/lib/python3.11/configparser.py:1274: FutureWarning: section/key [celery/worker_log_server_port] has been deprecated, you should use[logging/worker_log_server_port] instead. Please update your `conf.get*` call to use the new name
  return self._parser.get(self._name, key)

[2025-10-06, 03:05:55 UTC] {taskinstance.py:1401} INFO - Marking task as FAILED. dag_id=download_gtfs_schedule_v2, task_id=email_download_failures, execution_date=20251005T030000, start_date=20251006T030555, end_date=20251006T030555
[2025-10-06, 03:05:55 UTC] {logging_mixin.py:154} INFO - Skipping email to slack channel. No CALITP_SLACK_URL in environment
[2025-10-06, 03:05:55 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 3025448 for task email_download_failures ([Errno 99] Cannot assign requested address; 2823)
[2025-10-06, 03:05:55 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2025-10-06, 03:05:55 UTC] {taskinstance.py:2781} INFO - 0 downstream tasks scheduled from follow-on schedule check

[erikamov]

Use dds.app.notify@dot.ca.gov for email alerts.