|
| 1 | +Metadata-Version: 2.2 |
| 2 | +Name: diragnosis |
| 3 | +Version: 0.0.0.post1 |
| 4 | +Summary: diRAGnosis - Diagnose the performance of your RAG! |
| 5 | +Author-email: Clelia Astra Bertelli < [email protected]> |
| 6 | +Project-URL: Homepage, https://github.com/AstraBert/RAGnosis |
| 7 | +Project-URL: Issues, https://github.com/AstraBert/RAGnosis/issues |
| 8 | +Classifier: Programming Language :: Python :: 3 |
| 9 | +Classifier: License :: OSI Approved :: MIT License |
| 10 | +Classifier: Operating System :: OS Independent |
| 11 | +Requires-Python: >=3.10 |
| 12 | +Description-Content-Type: text/markdown |
| 13 | +Requires-Dist: llama-index |
| 14 | +Requires-Dist: llama-index-embeddings-openai |
| 15 | +Requires-Dist: llama-index-embeddings-huggingface |
| 16 | +Requires-Dist: llama-index-embeddings-cohere |
| 17 | +Requires-Dist: llama-index-embeddings-mistralai |
| 18 | +Requires-Dist: llama-index-llms-openai |
| 19 | +Requires-Dist: llama-index-llms-cohere |
| 20 | +Requires-Dist: llama-index-llms-groq |
| 21 | +Requires-Dist: llama-index-llms-mistralai |
| 22 | +Requires-Dist: llama-index-llms-anthropic |
| 23 | +Requires-Dist: llama-index-vector-stores-qdrant |
| 24 | +Requires-Dist: fastembed |
| 25 | +Requires-Dist: qdrant_client |
| 26 | +Requires-Dist: pydantic |
| 27 | +Requires-Dist: pandas |
| 28 | + |
| 29 | +<h1 align="center">diRAGnosis🩺</h1> |
| 30 | + |
| 31 | +<h2 align="center">Diagnose the performance of your RAG</h2> |
| 32 | + |
| 33 | +<div align="center"> |
| 34 | + <h3>If you find diRAGnosis useful, please consider to donate and support the project:</h3> |
| 35 | + <a href="https://github.com/sponsors/AstraBert"><img src="https://img.shields.io/badge/sponsor-30363D?style=for-the-badge&logo=GitHub-Sponsors&logoColor=#EA4AAA" alt="GitHub Sponsors Badge"></a> |
| 36 | +</div> |
| 37 | +<br> |
| 38 | +<div align="center"> |
| 39 | + <img src="https://raw.githubusercontent.com/AstraBert/diRAGnosis/main/logo.png" alt="diRAGnosis Logo" width=300 height=300> |
| 40 | +</div> |
| 41 | + |
| 42 | +**diRAGnosis** is a lightweight framework, built with [LlamaIndex](https://llamaindex.ai), that allows you to evaluate the performance of LLMs and retrieval models in RAG frameworks with your documents. It can be used as an application (thanks to [FastAPI](https://fastapi.tiangolo.com/) + [Gradio](https://gradio.app)) running locally on your machine, or as a python package. |
| 43 | + |
| 44 | +## Installation and usage |
| 45 | + |
| 46 | +### As an application |
| 47 | + |
| 48 | +Clone the application: |
| 49 | + |
| 50 | +```bash |
| 51 | +git clone https://github.com/AstraBert/diRAGnosis.git |
| 52 | +cd diRAGnosis/ |
| 53 | +``` |
| 54 | + |
| 55 | +**Docker (recommended)**🐋 |
| 56 | + |
| 57 | +> _Required: [Docker](https://docs.docker.com/desktop/) and [docker compose](https://docs.docker.com/compose/)_ |
| 58 | + |
| 59 | +- Launch the Docker application: |
| 60 | + |
| 61 | +```bash |
| 62 | +# If you are on Linux/macOS |
| 63 | +bash run_services.sh |
| 64 | +# If you are on Windows |
| 65 | +.\run_services.ps1 |
| 66 | +``` |
| 67 | + |
| 68 | +Or, if you prefer: |
| 69 | + |
| 70 | +```bash |
| 71 | +docker compose up db -d |
| 72 | +docker compose up dashboard -d |
| 73 | +``` |
| 74 | + |
| 75 | +You will see the application running on http://localhost:8000/dashboard and you will be able to use it. Depending on your connection and on your hardware, the set up might take some time (up to 30 mins to set up) - but this is only for the first time your run it! |
| 76 | + |
| 77 | + |
| 78 | +**Source code**🗎 |
| 79 | + |
| 80 | +> _Required: [Docker](https://docs.docker.com/desktop/), [docker compose](https://docs.docker.com/compose/) and [conda](https://anaconda.org/anaconda/conda)_ |
| 81 | + |
| 82 | +- Set up diRAGnosis app using the dedicated script: |
| 83 | + |
| 84 | +```bash |
| 85 | +# For MacOs/Linux users |
| 86 | +bash setup.sh |
| 87 | +# For Windows users |
| 88 | +.\setup.ps1 |
| 89 | +``` |
| 90 | + |
| 91 | +- Or you can do it manually, if you prefer: |
| 92 | + |
| 93 | +```bash |
| 94 | +docker compose up db -d |
| 95 | + |
| 96 | +conda env create -f environment.yml |
| 97 | + |
| 98 | +conda activate eval-framework |
| 99 | + |
| 100 | +cd scripts/ |
| 101 | +uvicorn main:app --host 0.0.0.0 --port 8000 |
| 102 | + |
| 103 | +conda deactivate |
| 104 | +``` |
| 105 | + |
| 106 | +You will see the application running on http://localhost:8000/dashboard and you will be able to use it. |
| 107 | + |
| 108 | +### As a python package |
| 109 | + |
| 110 | +As a python package, you will be able to install diRAGnosis using `pip`: |
| 111 | + |
| 112 | +```bash |
| 113 | +pip install diRAGnosis |
| 114 | +``` |
| 115 | + |
| 116 | +Once you have installed it, you can import the four functions ([detailed in the dedicated reference file](https://github.com/AstraBert/diRAGnosis/tree/main/REFERENCE.md)) available for diRAGnosis like this: |
| 117 | + |
| 118 | +```python |
| 119 | +from diRAGnosis.evaluation import generate_question_dataset, evaluate_llms, evaluate_retrieval, display_available_providers |
| 120 | +``` |
| 121 | +Once you imported them, this is an example of how you can use them: |
| 122 | + |
| 123 | +```python |
| 124 | +from qdrant_client import QdrantClient, AsyncQdrantClient |
| 125 | +import asyncio |
| 126 | +import os |
| 127 | +from dotenv import load_dotenv |
| 128 | +import json |
| 129 | + |
| 130 | +load_dotenv() |
| 131 | +# import your API keys (in this case, only OpenAI) |
| 132 | +openai_api_key = os.environ["OPENAI_API_KEY"] |
| 133 | +# define your data |
| 134 | +input_files = ["file1.pdf", "file2.pdf"] |
| 135 | +# create a Qdrant client (asynchronous and synchronous) |
| 136 | +qdrant_client = QdrantClient("http://localhost:6333") |
| 137 | +qdrant_aclient = AsyncQdrantClient("http://localhost:6333") |
| 138 | +# display available LLM and Embedding model providers |
| 139 | +display_available_providers() |
| 140 | +async def main(): |
| 141 | + # generate dataset |
| 142 | + question_dataset, docs = await generate_question_dataset(input_files = input_files, llm = "OpenAI", model="gpt-4o-mini", api_key = openai_api_key, questions_per_chunk = 10, save_to_csv = "questions.csv", debug = True) |
| 143 | + # evaluate LLM performance |
| 144 | + binary_pass, scores = await evaluate_llms(qc = qdrant_client, aqc = qdrant_aclient, llm = "OpenAI", model="gpt-4o-mini", api_key = openai_api_key, docs = docs, questions = question_dataset, embedding_provider = "HuggingFace", embedding_model = "Alibaba-NLP/gte-modernbert-base", enable_hybrid = True, debug = True) |
| 145 | + print(json.dumps(binary_pass, indent=4)) |
| 146 | + print(json.dumps(scores, indent=4)) |
| 147 | + # evaluate retrieval performance |
| 148 | + retrieval_metrics = await evaluate_retrieval(qc = qdrant_client, aqc = qdrant_aclient, input_files = input_files, llm = "OpenAI", model="gpt-4o-mini", api_key = openai_api_key, embedding_provider = "HuggingFace", embedding_model = "Alibaba-NLP/gte-modernbert-base", questions_per_chunk = 5, enable_hybrid = True, debug = True) |
| 149 | + print(json.dumps(retrieval_metrics, indent=4)) |
| 150 | + |
| 151 | +if __name__ == "__main__": |
| 152 | + asyncio.run(main()) |
| 153 | +``` |
| 154 | + |
| 155 | +## How it works |
| 156 | + |
| 157 | +<div align="center"> |
| 158 | + <img src="https://raw.githubusercontent.com/AstraBert/diRAGnosis/main/workflow.png" alt="diRAGnosis Workflow"> |
| 159 | +</div> |
| 160 | + |
| 161 | +diRAGnosis takes care of the evaluation of LLM and retrieval model performance on your documents in a completely automated way: |
| 162 | + |
| 163 | +- Once your documents are uploaded, they are converted into a synthetic question dataset (either for Retrieval Augmented Generation or for retrieval only) by an LLM of your choice |
| 164 | +- The documents are also chunked and uploaded to a vector database served by [Qdrant](https://qdrant.tech) - you can choose a semantic search only or an hybrid search setting |
| 165 | +- The LLMs are evaluated, with binary pass and with scores, on the faithfulness and relevancy of their answers based on the questions they are given and on the retrieved context that is associated to each question |
| 166 | +- The retrieval model is evaluated according to hit rate (retrieval of the correct document as first document) and to MRR (Mean Reciprocal Ranking, i.e. the positioning of the correct document in the ranking of the retrieved documents) |
| 167 | +- The metrics are returned to the user |
| 168 | + |
| 169 | +## Contributing |
| 170 | + |
| 171 | +Contributions are always welcome! Follow the contributions guidelines reported [here](https://github.com/AstraBert/diRAGnosis/tree/main/CONTRIBUTING.md). |
| 172 | + |
| 173 | +## License and rights of usage |
| 174 | + |
| 175 | +The software is provided under MIT [license](https://github.com/AstraBert/diRAGnosis/tree/main/LICENSE). |
0 commit comments