Code

Upgrade upload process#70

Code

Merged

StreetLamb merged 20 commits intomasterStreetLamb/tribe:masterfrom upgrade-upload-processStreetLamb/tribe:upgrade-upload-processCopy head branch name to clipboardJul 12, 2024

+565-197

Commits

Commits on Jul 9, 2024

• Upgrade qdrant-client version to 1.10.1 to fix bug when using prefer_grpc=True with hybrid search

  StreetLamb

  committed
  1d89deb

  Copy full SHA for 1d89deb

• Set prefer_grpc=true in QdrantClient for better performance

  StreetLamb

  committed
  ac8f36d

  Copy full SHA for ac8f36d

• Upgrade langchain dependencies

  StreetLamb

  committed
  11aeceb

  Copy full SHA for 11aeceb

Commits on Jul 11, 2024

• Use grpc port for QDRANT_URL and add FASTEMBED_CACHE_PATH.

  StreetLamb

  committed
  91880b9

  Copy full SHA for 91880b9

• Set default MAX_WORKERS to 1 in .env.example

  StreetLamb

  committed
  44feb54

  Copy full SHA for 44feb54

Commits on Jul 12, 2024

• Add celery and redis

  StreetLamb

  committed
  886f6ae

  Copy full SHA for 886f6ae

• Update Dockerfile to create non-root user for celery worker. Update huggingface cache directory so non-root user can access it.

  StreetLamb

  committed
  d5fa5e5

  Copy full SHA for d5fa5e5

• Add docker-compose configs for celery and redis

  StreetLamb

  committed
  9262e60

  Copy full SHA for 9262e60

• Update upload routes to trigger celery worker to process and upload files to vector store instead of doing it in backend to improve performance

  StreetLamb

  committed
  4aea69d

  Copy full SHA for 4aea69d

• Switch back to pymupdf as pymupdf4llm is cpu and memory intensive

  StreetLamb

  committed
  84bd0ce

  Copy full SHA for 84bd0ce

• Dont show success modal for adding and deleting upload in frontend

  StreetLamb

  committed
  70ac5d8

  Copy full SHA for 70ac5d8

• Update build images command in local-deployment guide

  StreetLamb

  committed
  8c89bba

  Copy full SHA for 8c89bba

• Set MAX_WORKERS as optional in .env.example

  StreetLamb

  committed
  190c1d9

  Copy full SHA for 190c1d9

• Add celery stubs

  StreetLamb

  committed
  2b8f421

  Copy full SHA for 2b8f421

• Uninstall pymupdf4llm and install pymupdf

  StreetLamb

  committed
  e5b780a

  Copy full SHA for e5b780a

• Fix mypy issues

  StreetLamb

  committed
  9ad6528

  Copy full SHA for 9ad6528

• Refactor

  StreetLamb

  committed
  3a5bcf7

  Copy full SHA for 3a5bcf7

• Update upload tests

  StreetLamb

  committed
  63d9efb

  Copy full SHA for 63d9efb

• Set MAX_WORKERS default to 1

  StreetLamb

  committed
  c06eace

  Copy full SHA for c06eace

• Fix embedding models and upload permission issues

  StreetLamb

  committed
  7365661

  Copy full SHA for 7365661