The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place.

Evaluate your LLM's response with Prometheus and GPT4 💯

🤠 Agent-as-a-Judge and DevAI dataset

[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

CodeUltraFeedback: aligning large language models to coding preferences

This is the repo for the survey of Bias and Fairness in IR with LLMs.

Official implementation for "MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?"

Code and data for "Timo: Towards Better Temporal Reasoning for Language Models" (COLM 2024)

The official repository for our EMNLP 2024 paper, Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability.

Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"

xVerify: Efficient Answer Verifier for Large Language Model Evaluations

Harnessing Large Language Models for Curated Code Reviews

Root Signals Python SDK

A set of examples demonstrating how to evaluate Generative AI augmented systems using traditional information retrieval and LLM-As-A-Judge validation techniques

A comprehensive study of the LLM-as-a-judge paradigm in a controlled setup that reveals new results about its strengths and weaknesses.

The official repository for our ACL 2024 paper: Are LLM-based Evaluators Confusing NLG Quality Criteria?

LLM-as-judge evals as Semantic Kernel Plugins

A set of tools to create synthetically-generated data from documents

Notebooks for evaluating LLM based applications using the Model (LLM) as a judge pattern.

Controversial Questions for Argumentation and Retrieval