How to use ImageReader?

[douglasjunior]

Description

I'm trying to use ImageReader with LlamaIndex TS to create embeddings for images, but I receive the following error:

Error: Invalid node type: IMAGE

I couldn't find any TypeScript example in the documentation showing how to use ImageReader with embeddings and vector stores. My goal is to index both PDF (text) and image files in the same vector store for retrieval.

Code snippet:

const llmOpenApi = openai({
  apiKey: OPENAI_API_KEY,
  model: 'gpt-4o',
});

Settings.llm = llmOpenApi;
Settings.embedModel = new HuggingFaceEmbedding({
  modelType: 'sentence-transformers/all-MiniLM-L6-v2',
});

const pdfReader = new PDFReader();
const imageReader = new ImageReader();
const vectorStore = new QdrantVectorStore({
  url: QDRANT_URL,
});

const pdfFiles = [path.join(dataDir, 'my-document.pdf')];
const imageFiles = [path.join(dataDir, 'my-image.png')];

let documents: Document[] = [];
for (const file of pdfFiles) {
  documents = documents.concat(await pdfReader.loadData(file));
}
for (const file of imageFiles) {
  documents = documents.concat(await imageReader.loadData(file));
}

const index = await VectorStoreIndex.fromDocuments(documents, {
  vectorStores: {
    TEXT: vectorStore,
    IMAGE: vectorStore,
  },
});

Expected behavior

• I expect to be able to embed and index both text and image documents using the same or separate vector stores, as suggested by the multimodal support in the docs.

Actual behavior

• The error Invalid node type: IMAGE is thrown at index creation.

Versions

"@llamaindex/huggingface": "^0.1.13",
"@llamaindex/openai": "^0.4.3",
"@llamaindex/qdrant": "^0.1.19",
"@llamaindex/readers": "^3.1.7",
"@llamaindex/workflow": "^1.1.7",
"llamaindex": "^0.11.6"

Additional context

• I couldn't find any working TypeScript example for multimodal indexing (text + image).
• I suspect the issue is related to embedding model selection or how ImageReader creates nodes, but the error message isn't clear.
• What is the recommended way to embed and index images in LlamaIndex TS? Is there a supported embedding model for images in TypeScript?

[douglasjunior]

If I use the storageContextFromDefaults, I got different error:

  const storageContext = await storageContextFromDefaults({ vectorStore });

  const index = await VectorStoreIndex.fromDocuments(documents, {
    storageContext,
  });

Error:

Error: Cannot insert nodes of type IMAGE without assigned vector store

[douglasjunior]

Tryied with SimpleVectorStore and the error keeps the same:

  const llmOpenApi = openai({
    apiKey: OPENAI_API_KEY,
    model: 'gpt-4o',
  });
  const embeddingModel = new HuggingFaceEmbedding({
    modelType: 'sentence-transformers/all-MiniLM-L6-v2',
  });

  Settings.llm = llmOpenApi;
  Settings.embedModel = embeddingModel;

  const textVectorStore = new QdrantVectorStore({
    url: QDRANT_URL,
    embeddingModel,
  });
  const imageVectorStore = await SimpleVectorStore.fromPersistDir(
    path.join(storageDir, 'images'),
    new ClipEmbedding(),
  );

  // load documents ....

  const storageContext = await storageContextFromDefaults({
    persistDir: storageDir,
    vectorStores: {
      TEXT: textVectorStore,
      IMAGE: imageVectorStore,
    },
  });
  const index = await VectorStoreIndex.fromDocuments(documents, {
    storageContext,
  });

Error:

Error: Invalid node type: IMAGE
    at jsonToNode (.../node_modules/@llamaindex/core/schema/dist/index.js:636:19)
    at ImageNode.clone (.../node_modules/@llamaindex/core/schema/dist/index.js:514:16)
    at VectorStoreIndex.insertNodesToStore (.../node_modules/llamaindex/indices/dist/index.js:2509:55)
    at addNodesToVectorStores (.../node_modules/llamaindex/indices/dist/index.js:251:23)
    at async VectorStoreIndex.insertNodes (.../node_modules/llamaindex/indices/dist/index.js:2523:9)
    at async VectorStoreIndex.buildIndexFromNodes (.../node_modules/llamaindex/indices/dist/index.js:2417:9)
    at async VectorStoreIndex.init (.../node_modules/llamaindex/indices/dist/index.js:2367:13)
    at async VectorStoreIndex.fromDocuments (.../node_modules/llamaindex/indices/dist/index.js:2445:20)
    at async main (/Users/douglas.junior/projects/DGS/software-architect-agent/src/example.ts:69:17)

[douglasjunior]

I think I found the issue, just adding IMAGE to this switch/case make it work:

LlamaIndexTS/packages/core/src/schema/node.ts

Lines 335 to 353 in a4d394f

	export function jsonToNode(json: any, type?: ObjectType) {
	if (!json.type && !type) {
	throw new Error("Node type not found");
	}
	const nodeType = type || json.type;
	switch (nodeType) {
	case ObjectType.TEXT:
	return new TextNode(json);
	case ObjectType.INDEX:
	return new IndexNode(json);
	case ObjectType.DOCUMENT:
	return new Document(json);
	case ObjectType.IMAGE_DOCUMENT:
	return new ImageDocument(json);
	default:
	throw new Error(`Invalid node type: ${nodeType}`);
	}
	}

Possible fix:

export function jsonToNode(json: any, type?: ObjectType) {
  if (!json.type && !type) {
    throw new Error("Node type not found");
  }
  const nodeType = type || json.type;

  switch (nodeType) {
    case ObjectType.TEXT:
      return new TextNode(json);
    case ObjectType.INDEX:
      return new IndexNode(json);
    case ObjectType.DOCUMENT:
      return new Document(json);
    case ObjectType.IMAGE_DOCUMENT:
    case ObjectType.IMAGE: // <-- here
      return new ImageDocument(json);
    default:
      throw new Error(`Invalid node type: ${nodeType}`);
  }
}

[marcusschiesser]

@douglasjunior thanks, I think you're on the right track - can you test #2017?

[douglasjunior]

@marcusschiesser thank you for your time.

Seems that the Error: Invalid node type: IMAGE is gone with your fix version, but I still can't query about images.

When querying about PDF files the LLM response is satisfactory, ex:

The combined budget of San Francisco for community health and public protection in the fiscal year 2024-25 is approximately $4.08 billion.

But when querying about Images is not, ex:

It seems there was an issue retrieving the information about the relations between subcomponents inside the portfolio module of the Health Panel project. Let me try again to get the details for you.I am currently unable to retrieve the information about the relations between subcomponents inside the portfolio module of the Health Panel project due to a technical issue. If you have any other questions or need assistance with something else, feel free to ask!

I have created a minimal repro project here:

• main branch with oficial llamaindex version: https://github.com/douglasjunior/learning-llamaindex-ts/tree/main
• pt-2017 branch with your fix: https://github.com/douglasjunior/learning-llamaindex-ts/tree/pr-2017

The PDF and image files used in this example are inside data folder

[marcusschiesser]

Can you try using the multi modal response synthesizer for your image query tool? Here's how to use it:

LlamaIndexTS/examples/multimodal/rag.ts

Line 32 in fffe93f

responseSynthesizer: getResponseSynthesizer("multi_modal"),

[douglasjunior]

Hi @marcusschiesser , thanks again for your response.

I have tried this RAG example, but I can't reproduce with success.

First, I can't understand how the images are read in the example, so I tried to change to use the VectorStoreIndex.fromDocuments instead of VectorStoreIndex.init

https://github.com/douglasjunior/learning-llamaindex-ts/blob/023a5e5eb7987cade2a7b9e52b8046df49fb43f4/src/multimodal.ts#L72-L78

Then it required to set an Settings.embbedModel:

https://github.com/douglasjunior/learning-llamaindex-ts/blob/023a5e5eb7987cade2a7b9e52b8046df49fb43f4/src/multimodal.ts#L29-L29

Then I start to receive this error in the line 84:

dtype not specified for "text_model". Using the default dtype (fp32) for this device (cpu).
Retrieved 1 nodes for query: What's the relations between subcomponents inside the portfolio module of Health Panel project?
Retrieved 1 nodes for query: What's the relations between subcomponents inside the portfolio module of Health Panel project?
Error: ENOENT: no such file or directory, open '/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/8a28b52d-8ab3-4511-ba43-5c7a9b7bb5e3'
    at async open (node:internal/fs/promises:639:25)
    at async Object.readFile (node:internal/fs/promises:1242:14)
    at async Object.imageToDataUrl (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/utils/dist/index.cjs:93:28)
    at async /Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/response-synthesizers/dist/index.cjs:61:30
    at async Promise.all (index 0)
    at async createMessageContent (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/response-synthesizers/dist/index.cjs:76:29)
    at async MultiModal.getResponse (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/response-synthesizers/dist/index.cjs:316:24)
    at async MultiModal.synthesize (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/response-synthesizers/dist/index.cjs:33:24)
    at async RetrieverQueryEngine.query (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/query-engine/dist/index.cjs:407:26)
    at async main (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/src/multimodal.ts:84:18) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/8a28b52d-8ab3-4511-ba43-5c7a9b7bb5e3'
}

Seems that it was trying to read the image in the wrong place.

Finally, I tried to replace the SimpleVectorStore.fromPersistDir for the new QdrantVectorStore:

https://github.com/douglasjunior/learning-llamaindex-ts/blob/023a5e5eb7987cade2a7b9e52b8046df49fb43f4/src/multimodal.ts#L48-L67

And then the error changed:

ApiError: Bad Request
    at Object.fun [as queryPoints] (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@qdrant/openapi-typescript-fetch/dist/cjs/fetcher.js:172:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async QdrantClient.query (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@qdrant/js-client-rest/dist/cjs/qdrant-client.js:1410:26)
    at async QdrantVectorStore.query (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/qdrant/dist/index.cjs:203:24)
    at async VectorIndexRetriever.retrieveQuery (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/llamaindex/indices/dist/index.cjs:2608:32)
    at async VectorIndexRetriever._retrieve (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/llamaindex/indices/dist/index.cjs:2583:54)
    at async VectorIndexRetriever.retrieve (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/retriever/dist/index.cjs:29:24)
    at async RetrieverQueryEngine.retrieve (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/query-engine/dist/index.cjs:459:23)
    at async RetrieverQueryEngine._query (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/query-engine/dist/index.cjs:425:32)
    at async RetrieverQueryEngine.query (/Users/douglas.junior/projects/Learning/learning-llamaindex-ts/node_modules/@llamaindex/core/query-engine/dist/index.cjs:408:26) {
  headers: Headers {
    'transfer-encoding': 'chunked',
    'content-encoding': 'gzip',
    'content-type': 'application/json',
    vary: 'accept-encoding, Origin, Access-Control-Request-Method, Access-Control-Request-Headers',
    date: 'Thu, 12 Jun 2025 17:20:25 GMT'
  },
  url: 'http://localhost:6333/collections/image-collection/points/query',
  status: 400,
  statusText: 'Bad Request',
  data: {
    status: {
      error: 'Format error in JSON body: Expected some form of vector, id, or a type of query at line 1 column 13422'
    },
    time: 0
  }
}