DPS Job failing due to connection error

[mccabete]

I have a job that has been submitting automatically for a while now. I noticed that my last few submissions failed with this error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ops/verdi/ops/hysds-1.1.10/hysds/dataset_ingest_bulk.py", line 900, in publish_datasets
    bulk_index_dataset(app.conf.GRQ_UPDATE_URL_BULK, prod_jsons)
  File "/home/ops/verdi/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/ops/verdi/ops/hysds-1.1.10/hysds/dataset_ingest_bulk.py", line 748, in bulk_index_dataset
    r = requests.post(grq_update_url, verify=False, json=json.dumps(update_jsons))
  File "/opt/conda/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/conda/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='x.x.x.x', port=8878): Max retries exceeded with url: /api/v0.2/grq/dataset/index (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbfe557a400>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Here are some of the job IDs that have failed this way:

3f1dbe45-79a1-4968-8bb3-6c7f787dd8ef
0a7a7604-eda5-4a59-adfa-f77310687903
1bc2a855-caab-4b61-b707-c116955d0a1c

Any idea what is going on? Thanks!

[sujen1412]

Seems like our dataset ingestion service is not responding. Checking...

[mccabete]

@sujen1412 did you find anything?

[sujen1412]

No, I was able to test that the service was up and running and many others jobs are completing successfully.