docs_to_knowledge_graph

Build Real-Time Knowledge Graph For Documents with LLM

We will process a list of documents and use LLM to extract relationships between the concepts in each document. We will generate two kinds of relationships:

1. Relationships between subjects and objects. E.g., "CocoIndex supports Incremental Processing"
2. Mentions of entities in a document. E.g., "core/basics.mdx" mentions CocoIndex and Incremental Processing.

You can find a step by step blog for this project here

Please drop Cocoindex on Github a star to support us if you like our work. Thank you so much with a warm coconut hug 🥥🤗.

Prerequisite

• Install Postgres if you don't have one.
• Install Neo4j.
• Install / configure LLM API. In this example we use Ollama, which runs LLM model locally. You need to get it ready following this guide. Alternatively, you can also follow the comments in source code to switch to OpenAI, and configure OpenAI API key before running the example.

Documentation

You can read the official CocoIndex Documentation for Property Graph Targets here.

Run

Build the index

Install dependencies:

pip install -e .

Update index:

cocoindex update main

Browse the knowledge graph

After the knowledge graph is built, you can explore the knowledge graph.

• If you're using Neo4j, you can open the explorer at http://localhost:7474, with username neo4j and password cocoindex. You can run the following Cypher query to get all relationships:

MATCH p=()-->() RETURN p

CocoInsight

I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. It just connects to your local CocoIndex server, with Zero pipeline data retention. Run following command to start CocoInsight:

cocoindex server -ci main

And then open the url https://cocoindex.io/cocoinsight.