Failed to query Vespa

[guohai]

dAnswer(commit 0553062) works okay with default configurations. But when I switched to "text-embedding-3-large", which will trigger the re-indexing. Issue happens

`
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 233, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
raise exc from None
File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
response = connection.handle_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
return self._connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
raise exc
File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
) = self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
event = self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
data = self._network_stream.read(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 124, in read
with map_exceptions(exc_map):
File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/app/danswer/document_index/vespa/chunk_retrieval.py", line 199, in _get_chunks_via_visit_api
response = http_client.get(url, params=filtered_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1054, in get
return self.request(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 827, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1015, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 232, in handle_request
with map_httpcore_exceptions():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/app/danswer/background/indexing/run_indexing.py", line 255, in _run_indexing
new_docs, total_batch_chunks = indexing_pipeline(
^^^^^^^^^^^^^^^^^^
File "/app/danswer/indexing/indexing_pipeline.py", line 148, in index_doc_batch_with_handler
r = index_doc_batch(
^^^^^^^^^^^^^^^^
File "/app/danswer/utils/timing.py", line 31, in wrapped_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/danswer/indexing/indexing_pipeline.py", line 373, in index_doc_batch
insertion_records = document_index.index(chunks=access_aware_chunks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/danswer/document_index/vespa/index.py", line 339, in index
delete_vespa_docs(
File "/app/danswer/document_index/vespa/deletion.py", line 61, in delete_vespa_docs
future.result()
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/retry/api.py", line 73, in retry_decorator
return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/retry/api.py", line 33, in __retry_internal
return f()
^^^
File "/app/danswer/document_index/vespa/deletion.py", line 23, in _delete_vespa_doc_chunks
doc_chunk_ids = get_all_vespa_ids_for_document_id(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/retry/api.py", line 73, in retry_decorator
return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/retry/api.py", line 33, in __retry_internal
return f()
^^^
File "/app/danswer/document_index/vespa/chunk_retrieval.py", line 241, in get_all_vespa_ids_for_document_id
document_chunks = _get_chunks_via_visit_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/danswer/document_index/vespa/chunk_retrieval.py", line 210, in _get_chunks_via_visit_api
raise httpx.HTTPError(error_base) from e
httpx.HTTPError: Failed to query Vespa
`

[rkuo-danswer]

Hi! are you able to inspect the CPU and memory usage of your index container while this is happening? The first thought would be to increase the amount of memory and cpu allocated to the index container.

[VishDev12]

Hey @rkuo-danswer, thank you for the suggestion!

I'm facing this issue too and it fails to meaningfully progress with the reindexing.

For reference:

• I'm using the Helm chart.
• I changed from the default of nomic -> GCP VertexAI text-embedding-004. I also changed multipass indexing from false to true.
• The memory and CPU consumption of the indexing pod remains well within requests since it's only making calls to GCP.
• Vespa resources:
  • Used:
    • CPU : 8400m
    • Memory: <4-5 GB
  • Allocated (for testing the upper bounds of this issue):
    • CPU: 24000m
    • Memory: 12GB
• Env vars:
  • CELERY_WORKER_INDEXING_CONCURRENCY: 8
  • NUM_INDEXING_WORKERS: 8
  • MIN_THREADS_ML_MODELS: 1
  • VESPA_SEARCHER_THREADS -> unmodified; default value

Specific, repeating error that's thrown in the background pod; with different documents, etc.

ERROR:    12/30/2024 02:12:30 PM             chunk_retrieval.py  203: [CC Pair: 5] [Index Attempt: 768] Failed to query Vespa:
Request URL: http://da-vespa-0.vespa-service:8081/document/v1/default/danswer_chunk_text_embedding_004/docid?selection=danswer_chunk_text_embedding_004.document_id%3D%3D%27https%3A%2F%2Fx.atlassian.net%2Fbrowse%2FXYZ-123%27&wantedDocumentCount=1000&fieldSet=danswer_chunk_text_embedding_004%3Adocument_id
Request Headers: Headers({'host': 'da-vespa-0.vespa-service:8081', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.27.0'})
Request Payload: {'selection': "danswer_chunk_text_embedding_004.document_id=='https://x.atlassian.net/browse/XYZ-123'", 'continuation': None, 'wantedDocumentCount': 1000, 'fieldSet': 'danswer_chunk_text_embedding_004:document_id'}
Exception: timed out

Version of Vespa used: nightly-latest-20241230

Do you have any recommendations for tuning Vespa to prevent this? Any insights about what's causing the load and why the CPU usage on Vespa doesn't grow beyond 8-9CPUs and timeouts still occur on the client?

Thanks in advance 🙂

[github-actions]

This issue is stale because it has been open 75 days with no activity. Remove stale label or comment or this will be closed in 15 days.

[github-actions]

This issue was closed because it has been stalled for 90 days with no activity.