METR

All

37 repositories

inspect_k8s_sandbox
Public
A Kubernetes sandbox environment for use with inspect_ai
Python
•
MIT License
•7•1•0•0•Updated Aug 2, 2025Aug 2, 2025
CoT-faithfulness-and-monitorability
Public
Python
•0•0•1•1•Updated Aug 2, 2025Aug 2, 2025
task-assets
Public
Python
•0•0•1•0•Updated Jul 31, 2025Jul 31, 2025
vivaria
Public
Vivaria is METR's tool for running evaluations and conducting agent elicitation research.
ai elicitation ai-evaluation evals
TypeScript
•
MIT License
•34•100•220•7•Updated Jul 31, 2025Jul 31, 2025
inspect-tasks-public
Public
Python
•
Other
•4•0•13•1•Updated Jul 30, 2025Jul 30, 2025
inspect_ai
Public
Inspect: A framework for large language model evaluations
Python
•
MIT License
•270•1•1•0•Updated Jul 29, 2025Jul 29, 2025
inspect-agents
Public
METR's wrapper around the inspect react agent. Intended to allow consistent usage and customization.
Python
•0•0•0•0•Updated Jul 29, 2025Jul 29, 2025
Measuring-Early-2025-AI-on-Exp-OSS-Devs
Public
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Python
•0•9•0•0•Updated Jul 21, 2025Jul 21, 2025
cross-domain-horizon
Public
Estimate the time horizon of AIs over time on various domains like knowledge and vision
Python
•0•1•0•0•Updated Jul 14, 2025Jul 14, 2025
autonomy-evals-guide
Public
SCSS
•
MIT License
•4•3•0•2•Updated Jul 8, 2025Jul 8, 2025
triframe_inspect
Public
Python
•0•0•6•0•Updated Jul 7, 2025Jul 7, 2025
inspect-tasks-public-fork
Public
Python
•
Other
•4•0•0•0•Updated Jun 19, 2025Jun 19, 2025
mcp-reward-hack
Public
TypeScript
•1•0•0•2•Updated Jun 13, 2025Jun 13, 2025
terraform-aws-lambda
Public
Terraform module, which takes care of a lot of AWS Lambda/serverless tasks (build dependencies, packages, updates, deployments) in countless combinations 🇺🇦
HCL
•
Apache License 2.0
•723•0•0•0•Updated May 10, 2025May 10, 2025
RE-Bench
Public
Python
•
Other
•8•94•4•0•Updated Apr 24, 2025Apr 24, 2025
eval-analysis-public
Public
Public repository containing METR's DVC pipeline for eval data analysis
Python
•19•84•5•1•Updated Apr 6, 2025Apr 6, 2025
inspect_evals
Public
Collection of evals for Inspect AI
Python
•
MIT License
•154•1•0•0•Updated Mar 19, 2025Mar 19, 2025
task-protected-scoring
Public
Python
•1•0•3•2•Updated Mar 18, 2025Mar 18, 2025
uplift_clone_hypothesis
Public
Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
Python
•
Other
•618•0•2•1•Updated Mar 18, 2025Mar 18, 2025
hcast-public
Public
HTML
•
Other
•2•10•2•0•Updated Mar 17, 2025Mar 17, 2025
public-tasks
Public
HTML
•
Other
•10•99•1•2•Updated Mar 16, 2025Mar 16, 2025
agent-prs-on-vivaria
Public
Vivaria is METR's tool for running evaluations and conducting agent elicitation research.
TypeScript
•
MIT License
•34•0•0•32•Updated Feb 19, 2025Feb 19, 2025
SWE-bench-fork
Public
[ICLR 2024] SWE-bench: Can Language Models Resolve Real-world Github Issues?
Python
•
MIT License
•574•0•0•0•Updated Feb 11, 2025Feb 11, 2025
KernelBenchFiltered
Public
Python
•1•10•1•0•Updated Feb 11, 2025Feb 11, 2025
viv-task-dev
Public
Shell
•1•0•6•0•Updated Feb 6, 2025Feb 6, 2025
task-standard
Public
METR Task Standard
TypeScript
•
MIT License
•33•156•6•2•Updated Feb 3, 2025Feb 3, 2025
agent-fork-of-eval-analysis-public
Public archive
For of eval-analysis-public repo intended for agent PRs
Python
•19•0•0•0•Updated Jan 28, 2025Jan 28, 2025
llm-foundry
Public
LLM training code for Databricks foundation models
Python
•
Apache License 2.0
•573•0•0•0•Updated Jan 28, 2025Jan 28, 2025
metr-task-boilerplate
Public
A Cookiecutter template for developing tasks according to the METR Task Standard
evaluations safety llm
TypeScript
•0•1•0•0•Updated Jan 22, 2025Jan 22, 2025
task-artifacts
Public
Python
•0•0•0•0•Updated Jan 21, 2025Jan 21, 2025