RAG Ingest pipeline with PDF metadata for metadata

[markoviksten]

Hi @dosu I would like to build RAG Ingest pipeline which detect PDF metadata for page number, title and url for each chunk so I can return source of orginal document for end user? Can you creat example JSON which I can download for both 1) Ingest Pipeline (ie use pdf as source and chroma as target) 2) Search same chroma db and return vector search results + orginal docuement page number, title and url metadata

[markoviksten]

@dosu or it can be astra database, could you generate some part of example for above

[dosubot[bot]]

Here’s an example of how the JSON structure looks for both ingesting PDF chunks with metadata and returning search results, whether you use Chroma or Astra as your vector database. Each chunk is represented as a Data object, with the main text and metadata fields (like page number, title, and url) included in the data dictionary. This format is used for both ingestion and search results, and the metadata is preserved throughout the pipeline (reference).

Example: Ingest Pipeline (PDF → Chroma or Astra)

[
  {
    "data": {
      "text": "This is the content of page 1.",
      "page_number": 1,
      "title": "Introduction",
      "url": "https://example.com/mydoc.pdf"
    }
  },
  {
    "data": {
      "text": "This is the content of page 2.",
      "page_number": 2,
      "title": "Background",
      "url": "https://example.com/mydoc.pdf"
    }
  }
]

You would pass this array of objects to the vector store (Chroma or Astra). The metadata fields (page_number, title, url) are extracted by the PDF loader (e.g., UnstructuredLoader with chunking_strategy: by_page or by_title) and included automatically if present (reference).

Example: Search Results (Chroma or Astra)

[
  {
    "data": {
      "text": "This is the content of page 2.",
      "page_number": 2,
      "title": "Background",
      "url": "https://example.com/mydoc.pdf"
    },
    "score": 0.89
  }
]

The search result includes the same metadata fields, so you can display the source (page number, title, url) to the end user (reference).

This structure is the same for both Chroma and Astra, as both accept and return metadata in this format. If you need a more detailed example or want to see how to configure the loader or vector store in Langflow’s UI, let me know!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}