👨🏻💻 LLM Engineer Toolkit This repository contains a curated list of 120+ LLM libraries category wise.
LLM Training and Fine-Tuning
Library
Description
Link
unsloth
Fine-tune LLMs faster with less memory.
Link
PEFT
State-of-the-art Parameter-Efficient Fine-Tuning library.
Link
TRL
Train transformer language models with reinforcement learning.
Link
Transformers
Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.
Link
Axolotl
Tool designed to streamline post-training for various AI models.
Link
LLMBox
A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.
Link
LitGPT
Train and fine-tune LLM lightning fast.
Link
Mergoo
A library for easily merging multiple LLM experts, and efficiently train the merged LLM.
Link
Llama-Factory
Easy and efficient LLM fine-tuning.
Link
Ludwig
Low-code framework for building custom LLMs, neural networks, and other AI models.
Link
Txtinstruct
A framework for training instruction-tuned models.
Link
Lamini
An integrated LLM inference and tuning platform.
Link
XTuring
xTuring provides fast, efficient and simple fine-tuning of open-source LLMs, such as Mistral, LLaMA, GPT-J, and more.
Link
RL4LMs
A modular RL library to fine-tune language models to human preferences.
Link
DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Link
torchtune
A PyTorch-native library specifically designed for fine-tuning LLMs.
Link
PyTorch Lightning
A library that offers a high-level interface for pretraining and fine-tuning LLMs.
Link
LLM Application Development Frameworks
Library
Description
Link
LangChain
LangChain is a framework for developing applications powered by large language models (LLMs).
Link
Llama Index
LlamaIndex is a data framework for your LLM applications.
Link
HayStack
Haystack is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more.
Link
Prompt flow
A suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications.
Link
Griptape
A modular Python framework for building AI-powered applications.
Link
Weave
Weave is a toolkit for developing Generative AI applications.
Link
Llama Stack
Build Llama Apps.
Link
Multi API Access
Library
Description
Link
LiteLLM
Library to call 100+ LLM APIs in OpenAI format.
Link
AI Gateway
A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
Link
Routers
Library
Description
Link
RouteLLM
Framework for serving and evaluating LLM routers - save LLM costs without compromising quality. Drop-in replacement for OpenAI's client to route simpler queries to cheaper models.
Link
Memory
Library
Description
Link
mem0
The Memory layer for your AI apps.
Link
Memoripy
An AI memory layer with short- and long-term storage, semantic clustering, and optional memory decay for context-aware applications.
Link
Interface
Library
Description
Link
Streamlit
A faster way to build and share data apps. Streamlit lets you transform Python scripts into interactive web apps in minutes
Link
Gradio
Build and share delightful machine learning apps, all in Python.
Link
AI SDK UI
Build chat and generative user interfaces.
Link
AI-Gradio
Create AI apps powered by various AI providers.
Link
Simpleaichat
Python package for easily interfacing with chat apps, with robust features and minimal code complexity.
Link
Chainlit
Build production-ready Conversational AI applications in minutes.
Link
Low Code
Library
Description
Link
LangFlow
LangFlow is a low-code app builder for RAG and multi-agent AI applications. It’s Python-based and agnostic to any model, API, or database.
Link
Cache
Library
Description
Link
GPTCache
A Library for Creating Semantic Cache for LLM Queries. Slash Your LLM API Costs by 10x 💰, Boost Speed by 100x. Fully integrated with LangChain and LlamaIndex.
Link
Library
Description
Link
FastGraph RAG
Streamlined and promptable Fast GraphRAG framework designed for interpretable, high-precision, agent-driven retrieval workflows.
Link
Chonkie
RAG chunking library that is lightweight, lightning-fast, and easy to use.
Link
RAGChecker
A Fine-grained Framework For Diagnosing RAG.
Link
RAG to Riches
Build, scale, and deploy state-of-the-art Retrieval-Augmented Generation applications.
Link
BeyondLLM
Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of Retrieval-Augmented Generation (RAG) systems.
Link
SQLite-Vec
A vector search SQLite extension that runs anywhere!
Link
fastRAG
fastRAG is a research framework for efficient and optimized retrieval-augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval.
Link
FlashRAG
A Python Toolkit for Efficient RAG Research.
Link
Llmware
Unified framework for building enterprise RAG pipelines with small, specialized models.
Link
Rerankers
A lightweight unified API for various reranking models.
Link
Vectara
Build Agentic RAG applications.
Link
Library
Description
Link
LLM Compressor
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment.
Link
LightLLM
Python-based LLM inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Link
vLLM
High-throughput and memory-efficient inference and serving engine for LLMs.
Link
torchchat
Run PyTorch LLMs locally on servers, desktop, and mobile.
Link
TensorRT-LLM
TensorRT-LLM is a library for optimizing Large Language Model (LLM) inference.
Link
WebLLM
High-performance In-browser LLM Inference Engine.
Link
Library
Description
Link
Langcorn
Serving LangChain LLM apps and agents automagically with FastAPI.
Link
LitServe
Lightning-fast serving engine for any AI model of any size. It augments FastAPI with features like batching, streaming, and GPU autoscaling.
Link
LLM Data Extraction
Library
Description
Link
Crawl4AI
Open-source LLM Friendly Web Crawler & Scraper.
Link
ScrapeGraphAI
A web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).
Link
Docling
Docling parses documents and exports them to the desired format with ease and speed.
Link
Llama Parse
GenAI-native document parser that can parse complex document data for any downstream LLM use case (RAG, agents).
Link
PyMuPDF4LLM
PyMuPDF4LLM library makes it easier to extract PDF content in the format you need for LLM & RAG environments.
Link
Crawlee
A web scraping and browser automation library.
Link
MegaParse
Parser for every type of document.
Link
ExtractThinker
Document Intelligence library for LLMs.
Link
Library
Description
Link
DataDreamer
DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows.
Link
fabricator
A flexible open-source framework to generate datasets with large language models.
Link
Promptwright
Synthetic Dataset Generation Library.
Link
EasyInstruct
An Easy-to-use Instruction Processing Framework for Large Language Models.
Link
Library
Description
Link
CrewAI
Framework for orchestrating role-playing, autonomous AI agents.
Link
LangGraph
Build resilient language agents as graphs.
Link
Agno
Build AI Agents with memory, knowledge, tools, and reasoning. Chat with them using a beautiful Agent UI.
Link
Agents SDK
Build agentic apps using LLMs with context, tools, hand off to other specialized agents.
Link
AutoGen
An open-source framework for building AI agent systems.
Link
Smolagents
Library to build powerful agents in a few lines of code.
Link
Pydantic AI
Python agent framework to build production grade applications with Generative AI.
Link
gradio-tools
A Python library for converting Gradio apps into tools that can be leveraged by an LLM-based agent to complete its task.
Link
Composio
Production Ready Toolset for AI Agents.
Link
Atomic Agents
Building AI agents, atomically.
Link
Memary
Open Source Memory Layer For Autonomous Agents.
Link
Browser Use
Make websites accessible for AI agents.
Link
OpenWebAgent
An Open Toolkit to Enable Web Agents on Large Language Models.
Link
Lagent
A lightweight framework for building LLM-based agents.
Link
LazyLLM
A Low-code Development Tool For Building Multi-agent LLMs Applications.
Link
Swarms
The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework.
Link
ChatArena
ChatArena is a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social interactions.
Link
Swarm
Educational framework exploring ergonomic, lightweight multi-agent orchestration.
Link
AgentStack
The fastest way to build robust AI agents.
Link
Archgw
Intelligent gateway for Agents.
Link
Flow
A lightweight task engine for building AI agents.
Link
AgentOps
Python SDK for AI agent monitoring.
Link
Langroid
Multi-Agent framework.
Link
Agentarium
Framework for creating and managing simulations populated with AI-powered agents.
Link
Upsonic
Reliable AI agent framework that supports MCP.
Link
Library
Description
Link
Ragas
Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications.
Link
Giskard
Open-Source Evaluation & Testing for ML & LLM systems.
Link
DeepEval
LLM Evaluation Framework
Link
Lighteval
All-in-one toolkit for evaluating LLMs.
Link
Trulens
Evaluation and Tracking for LLM Experiments
Link
PromptBench
A unified evaluation framework for large language models.
Link
LangTest
Deliver Safe & Effective Language Models. 60+ Test Types for Comparing LLM & NLP Models on Accuracy, Bias, Fairness, Robustness & More.
Link
EvalPlus
A rigorous evaluation framework for LLM4Code.
Link
FastChat
An open platform for training, serving, and evaluating large language model-based chatbots.
Link
judges
A small library of LLM judges.
Link
Evals
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Link
AgentEvals
Evaluators and utilities for evaluating the performance of your agents.
Link
LLMBox
A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.
Link
Opik
An open-source end-to-end LLM Development Platform which also includes LLM evaluation.
Link
Library
Description
Link
MLflow
An open-source end-to-end MLOps/LLMOps Platform for tracking, evaluating, and monitoring LLM applications.
Link
Opik
An open-source end-to-end LLM Development Platform which also includes LLM monitoring.
Link
LangSmith
Provides tools for logging, monitoring, and improving your LLM applications.
Link
Weights & Biases (W&B)
W&B provides features for tracking LLM performance.
Link
Helicone
Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc.
Link
Evidently
An open-source ML and LLM observability framework.
Link
Phoenix
An open-source AI observability platform designed for experimentation, evaluation, and troubleshooting.
Link
Observers
A Lightweight Library for AI Observability.
Link
Library
Description
Link
PCToolkit
A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models.
Link
Selective Context
Selective Context compresses your prompt and context to allow LLMs (such as ChatGPT) to process 2x more content.
Link
LLMLingua
Library for compressing prompts to accelerate LLM inference.
Link
betterprompt
Test suite for LLM prompts before pushing them to production.
Link
Promptify
Solve NLP Problems with LLMs & easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify.
Link
PromptSource
PromptSource is a toolkit for creating, sharing, and using natural language prompts.
Link
DSPy
DSPy is the open-source framework for programming—rather than prompting—language models.
Link
Py-priompt
Prompt design library.
Link
Promptimizer
Prompt optimization library.
Link
Library
Description
Link
Instructor
Python library for working with structured outputs from large language models (LLMs). Built on top of Pydantic, it provides a simple, transparent, and user-friendly API.
Link
XGrammar
An open-source library for efficient, flexible, and portable structured generation.
Link
Outlines
Robust (structured) text generation
Link
Guidance
Guidance is an efficient programming paradigm for steering language models.
Link
LMQL
A language for constraint-guided and efficient LLM programming.
Link
Jsonformer
A Bulletproof Way to Generate Structured JSON from Language Models.
Link
Library
Description
Link
JailbreakEval
A collection of automated evaluators for assessing jailbreak attempts.
Link
EasyJailbreak
An easy-to-use Python framework to generate adversarial jailbreak prompts.
Link
Guardrails
Adding guardrails to large language models.
Link
LLM Guard
The Security Toolkit for LLM Interactions.
Link
AuditNLG
AuditNLG is an open-source library that can help reduce the risks associated with using generative AI systems for language.
Link
NeMo Guardrails
NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Link
Garak
LLM vulnerability scanner
Link
Library
Description
Link
Sentence-Transformers
State-of-the-Art Text Embeddings
Link
Model2Vec
Fast State-of-the-Art Static Embeddings
Link
Text Embedding Inference
A blazing fast inference solution for text embeddings models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5.
Link
Library
Description
Link
Text Machina
A modular and extensible Python framework, designed to aid in the creation of high-quality, unbiased datasets to build robust models for MGT-related tasks such as detection, attribution, and boundary detection.
Link
LLM Reasoners
A library for advanced large language model reasoning.
Link
EasyEdit
An Easy-to-use Knowledge Editing Framework for Large Language Models.
Link
CodeTF
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM.
Link
spacy-llm
This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks.
Link
pandas-ai
Chat with your database (SQL, CSV, pandas, polars, MongoDB, NoSQL, etc.).
Link
LLM Transparency Tool
An open-source interactive toolkit for analyzing internal workings of Transformer-based language models.
Link
Vanna
Chat with your SQL database. Accurate Text-to-SQL Generation via LLMs using RAG.
Link
mergekit
Tools for merging pretrained large language models.
Link
MarkLLM
An Open-Source Toolkit for LLM Watermarking.
Link
LLMSanitize
An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs).
Link
Annotateai
Automatically annotate papers using LLMs.
Link
LLM Reasoner
Make any LLM think like OpenAI o1 and DeepSeek R1.
Link
Please consider giving a star, if you find this repository useful.
