Better citation management

[danieldekay]

Currently we are asking the LLM to cite using a markdown hyperlink, formatted in APA or whatever format name we use.

This works well in some cases, but fails in other cases. Usually you don't know, because you don't click every link and see, but since we ask for a hyperlink failed cases are always hallucinated, so only a check of the URL for a 404 or something else works.

Would it not be better, if we cite with metadata from the documents or chunks, and generate the hyperlinks/table of references more algorithmically?

@assafelovic - have you experimented with something like that?

We could use the metadata in the langchain documents for the citations, like a universal key for each. This way we can even add "page 7 on PDF file ABC" for citations.

Crazy version: In my own project I convert the final markdown to PDF using pandoc and my own latex template -- i bet we could make fancy footnotes and citations here as well, if we would create an intermediate bibtex file...

[assafelovic]

This is super interesting, but would result in a lot of verification steps. I wonder if maybe this could be solved with better prompt engineering? Pushing the model to only include sources that it exist, etc?

[danieldekay]

I am reading what Langchain write about this, and maybe that is an easy quick win:

https://python.langchain.com/docs/how_to/qa_citations/
https://github.com/langchain-ai/langchain/blob/master/cookbook/qa_citations.ipynb

They basically ask the model to return structured output that includes an ID for the citation. But it requires post-processing -- which is good, because you want to have a list of references only for what actually gets used, no?

[danieldekay]

@assafelovic , as you may have seen on the tickets, I am currently digging into document-only work, and there wen currently have very bad citations, and empty list of URLs. Both "local" and "hybrid" need to treat document chunks more like URLs for the existing mechanism to work.

[ElishaKay]

Sup @danieldekay

❤ it!

I also agree there are a lot of interesting things we can do with the metadata of the Langchain Documents.

@assafelovic, we've had a lot of cases where users are also confused because the documents flow begins with a web search.

I'd be happy to think through the upgrades with @danieldekay to the documents and vector store modules

Something I've also been thinking about is treating vector stores as another swappable component (like we do with LLM's and Retrievers)

Happy to hear your thoughts 🙏

[danieldekay]

This week I will bring my own vectorstore and play a bit with it. Will let you know how it goes.

[danieldekay]

@ElishaKay - I failed. Yes, it would use my pdfs (academic literature), and it creates some output with citations. But it started to cite works that are cited in my documents, and when I checked the actual log, I saw it using the table of references a lot to create its content. This seems to be a know problem with using academic literature, which is highly condensed, but also references not only its own ideas but summarizes the work of others.

What I have not tried is to actually extract the chapters from these papers, and only use, e.g. results and discussion, but not the citations or the introduction.

[ElishaKay]

Sup @danieldekay

Failure is relative

Perhaps Langgraph could be a good fit for feeding the input of one step into the next

[danieldekay]

Yes, iterations and different pathways are required for better content processing

https://docs.unstructured.io/open-source/concepts/document-elements Could be useful to extract and annotate the content of documents better.