LlamaIndex Open Source Roadmap

[logan-markewich]

Last Updated: January 7, 2024

This is a living document that presents our 3-6 month roadmap for LlamaIndex. This will give you a sense of where the framework is evolving and the use cases you can plug it into!

We first define our high-level goals, and then outline our tasks/relative priorities and categorize which goal(s) they fit in.

Additional Notes:

• We’ll be updating this every 1-3 months.
• If you’re interested in contributing to LlamaIndex, check out our contribution guide for more details!
• The document primarily references our Python library but our Typescript library will follow a similar roadmap.
• Hop on our Discord if you want to share thoughts/feedback/suggestions for roadmap items.

High-Level Goals

1. Make the Framework Production-Ready: LlamaIndex already powers production LLM stacks at both startups and large enterprises, but there’s still room for improvement. We want to make sure that core modules in the framework are bug-free, well-tested and robust to large amounts of compute/data usage. Interfaces are stable and well-documented. Changes have proper backwards compatibility and deprecation notices. Security issues are squashed.
2. Make the Framework accessible to all Users: One of the core strengths of LlamaIndex is the different levels of abstraction that cater to both standard user and advanced users. Standard users want declarative, out-of-the-box modules that work well. Advanced users want arbitrary customizability/composability with their existing workflows. Our goal as a framework is to serve the entire spectrum of these users.
3. Add Advanced Capabilities: A core strength of LlamaIndex is the breadth and depth of advanced capabilities that let LLMs interact with user data. We want to continue evolving that. This includes more/better file-specific data ingestion modules, advanced data structures like knowledge graphs, advanced retrieval methods to handle multi-documents/complex data types, and agentic/step-wise reasoning. This also includes more end-to-end templates tying together these concepts to solve more use cases.
4. Staying on top of the Ecosystem: The AI ecosystem evolves very quickly. Every week there are new models, open-source projects/tools, services released. LlamaIndex already has 150+ data loaders, 30+ LLM integrations, 40+ vector database integrations, and more. We want to make sure that you’re able to build software with the latest and greatest models and surrounding tooling.

Priorities

Given these goals, here’s our set of priorities. The priorities are outlined below with tags (P0, P1, etc.).

1. [P0] Deprecate and remove ServiceContext
   • The current service context is extremely clunky, and leads to a lot of frustration when trying to use non-OpenAI LLMs and embeddings
   • This would be replaced with a proper global settings object for models, and proper interfaces to accept component level overrides
   • Goals: [1]
2. [P0] Proper integration testing for key components
   • LlamaIndex has a ton of integrations — most are not tested regularly. LLMs, vector databases, etc. can break, and we rely on the community to tell us.
   • We will be setting up dedicated testing for the most popular integrations, and communicating in our documentation which are covered by these tests
   • Goals: [1]
3. [P0] Every Core Component is Easy to Subclass
   • For advanced users and use-cases, you will often need a custom component somewhere in your project
   • All core-components (retrievers, query engines, agents, etc.) should be trivial to subclass, requiring minimal code to modify and implement behavior
   • Goals: [2]
4. [P1] Improved Documentation
   • Our API documentation is under-documented, incomplete, and confusing to read
   • Our module guides can be more detailed
   • More use-cases should be featured
   • Goals: [1, 2]
5. [P1] Improved Composability
   • Easier ways to combine and structure different modules
   • This means ways to combine retrievers, query engines, etc. should be easier
   • We have a recursive retriever, but it is also difficult to use
   • Goals: [2, 3]
6. [P1] Advanced Data Ingestion Modules
   • This includes better file-based parsing, splitting, and ingestion
   • Some files require complex setups to work well (i.e. recursive retriever), we can help make these setups less intimidating
   • Goals: [3]
7. [P1] Separate and version integrations
   • It will be more scalable to have each integration be an independent pip package
   • This will help with managing dependencies, as well as helping surface our integrations on tools like llama-hub
   • Goals: [1]
8. [P2] Complex Data Understanding
   • Better modules for querying structured, semi-structured, unstructured data
   • Goals: [3]
9. [P2] Large Data Understanding
   • Better modules that can scalably ingest and accurately query large volumes of data
   • Goals: [1, 3]
10. [P2] Improved Multimodal Abstractions and Capabilities
    • Goals: [3]
11. [P2] Support cool stuff that comes out
    • This goal is intentionally vague. The intention is to support whatever the AI community is interested in at the given moment in time.
    • Goals: [4]

[sumitsahoo]

Looks like 2024 will be amazing 🙏🏻

[sandangel]

I feel like many improvements we planned here are pretty similar to haystack v2 design. we can learn a lot from it. It was super easy for me to create a custom component in haystack v2, and they separated all integrations into one repo, each integration has its own package and version.

[sandangel]

Sound great. Really look forward to it.

[jamesbraza]

Hello! I dig the roadmap.

If you're thinking about going with separate packages for everything, consider using native namespace packaging so all packages are installed to the same namespace from llama_index.*. It basically gives the benefit of separate "plug-in" style packages, without having to make many new namespaces. The only tradeoff is native namespace packaging doesn't work with __init__.py shortcut imports.

LangChain missed the boat here, and now I have a bunch of from langchain_core, from langchain_openai, from langchain_community, etc and it's a mess. Ideally they'd all be just be from langchain.

[logan-markewich]

@jamesbraza I really like the namespaces idea. However, using namespaces essentially destroys shortcut imports right? i.e. instead of from llama_index.llms import OpenAI you'd have to do from llama_index.llms.openai import OpenAI -- would be largely breaking for any existing code/users.

We are considering having a) a conversion script for existing code and b) from llama_index.legacy to keep the old paths for a certain period, similar to pydantic. Thoughts on this?

[jamesbraza]

Moving to namespaces can possibly break shortcut imports, but not necessarily. What you can do is have the __init__.py in the core package, and nowhere else. Then the core package's __init__.py can attempt imports while wrapped in try-except. If the relevant submodule isn't installed, the ImportError is suppressed.

An alternate solution, how LangChain solved their decomposition (for a short period) was having langchain depend on langchain-community, and moving the __init__.py in langchain to import from langchain-community.

Yes namespace packages impose a different set of constraints, so I will understand if you don't want to go down this route 👍

[logan-markewich]

I'm case anyone missed it, we did this 👏

[vlasvlasvlas]

niiiiice

[victormimo]

Building in public 🚀

[AkimfromParis]

Congratulations on this great roadmap! 🍾

From the perspective of robust pipelines and accessibility for all users, it will be great to have a language-agnostic implementation for prompts and query engines.

For example, in my latest notebook for Japanese LLM, I could not use sub_question_query_engine because the prompt is in English. The Japanese LLM based on Llama2 always replies in English. I am currently writing a new notebook about Llama-Index using Mixtral7xB for financial reports in English, French, and Italian. And quantized models of Mixtral MoE are already confused between languages. : )

I think that Spacy did a great job with language support with a simple language code. If you put that in place, I would be happy to help translate it into French and maybe Japanese. 🙇

[logan-markewich]

Yup, this is definitely on the road map :) I think the first step is, we need to centralize all our prompts -- they are scattered throughout the codebase right now.

Once they are centralized, there are a lot of things we can do!

[fengyunzaidushi]

great!

[sandangel]

Another suggestion: Having more consistent naming and usage between non-stream/stream/async/sync. For example, in this code https://github.com/run-llama/llama_index/blob/main/llama_index/chat_engine/types.py#L157-L168, we have async_response_gen() which is a method, and response_gen is a property.

Also in many places we call arun, run, achat, chat etc... but in the above code we use async_response_gen instead of aresponse_gen.
I wonder what should we do moving forward for more consistent naming convention and usage.

[logan-markewich]

I agree -- imo it should all be function() or afunction()... except for functions that start with an a already

The response_gen thing was a consequence of one being a property (to match the query engine response object) vs. an async function (you can't have async properties... right?)

[sandangel]

https://pypi.org/project/async-property/

I think you can, but not with the built-in @Property decorator

[jamesbraza]

Is there any sort of fluent interface for prompt building, that also knows about "placement" in the prompt? I find often times I am making prompts templates using triple quoted strings, and am manually comprehending the U-shaped attention given to parts of the prompt.

Maybe this is outside of the scope of this discussion, but it would really useful (and unique) if LlamaIndex had a lightweight and fluent abstraction for prompt building:

>>> prompt_template = PromptTemplate()
>>> prompt_template.append_intro("I am an overview.")
>>> prompt_template.append_positive_example("I am a desirable example.")  # Gets added in an examples section
>>> prompt_template.schema(PydanticModel) # Auto-infers to use JSON
>>> prompt_template.append_schema_info("Field XYZ is case sensitive.")  # Gets added after schema
>>> prompt_template.append_intro("And someone later reused me and expanded upon the intro.")
>>> print(prompt_template)
I am an overview. And someone later reused me and expanded upon the intro.

Good examples include:
- I am a desirable example.

Please format the response in JSON according to this schema <pydantic JSON schema>. Field XYZ is case sensitive.

I only have like 10 prompts in use, and I hand crafted all of them. I think LlamaIndex team has much greater insight to variance of prompts, so maybe they can think of better abstraction than this.

[wzhgoj]

Great Roadmap, thank you for your contributions @logan-markewich !

If I might add my 2 cents here as a user:
The biggest obstacle currently for me is the lack of compatibility between pydantic v1 and v2, even with the bridge.pydantic classes my largely pydantic v2 code breaks. As I had to implement Delegators for the LLMs and Embeddings, this gets broken in unexpected ways too by pydantic v1.

So in order to achieve "[P0] Every Core Component is Easy to Subclass", I think pydantic v1 has to go. Is that implicit in your roadmap?

[logan-markewich]

Definitely! I would like to remove this soon as well

[amcegan]

Do ye plan to enable metadata filtering on Azure Cognitive Search?
Is there a blocker?

[logan-markewich]

It should already be available in v0.10.x

[sandangel]

I would suggest an integration with DSPy would be great.

[logan-markewich]

Its in the queue 💪

[sandangel]

Hi, Is there any update to the current roadmap?