[QUESTION] Can i use Pathway for DocumentStore only

[sayymeer]

What is your question or problem? Please describe.
I want to create a application which has following function via REST API

1. Create new vector store
2. Ingest a file in vector store
3. query a vector store

Describe what you would like to happen
I would like to know if this is possible in pathway. One way is I create a DocumentStore and connect it with a local folder and i upload files in that folder with parent name of folder is equal to name of the vector store. And when querying a vector store i will use metadata filter to query files only on that vector store.

[zxqfd555-pw]

Hi @sayymeer!

What you're describing is very similar to a typical use case for the llm-app framework, which we have available in a separate repository.

The main difference is that we create a single vector store for the whole run, whereas it seems from the issue text that you'd like to create multiple ones. Fundamentally, in Pathway, we specify all data sources in advance, as the computation graph also needs to be defined before the run.

So one option is to create multiple vector stores beforehand, but that’s more resource-intensive. Overall, I recommend following the approach you mentioned with a metadata filter. Please let us know the specific issues you're facing with the implementation so we can assist you more precisely.

[Anish2811]

Yes, you can use Pathway just as a document store or vector store without needing to run complex pipelines or LLM integrations.