⚔️OpenHands PR Arena⚔️ is a platform for evaluating and benchmarking agentic coding assistants through paired pull request (PR) generations. PR Arena enables developers to compare multiple LLMs in real-world issue resolution by presenting side-by-side pull requests and allowing users to select the better fix.

7 LLMs ready to enter the Arena!

Setup PR Arena on GitHub

For installation & usage instructions, see our Setup Guideline.

Privacy Notification

1. The only code we collect is the git_diff generated during issue resolution. We never access or store the entire codebase, access GitHub secrets, or release any user data.
2. Important: Installing this App will automatically add a workflow file named pr-arena-workflow.yml to your repository. This file redirects to the actual resolver workflow located at: https://github.com/neulab/pr-arena/blob/main/.github/workflows/pr-arena-resolver.yml. If you are concerned about repository workflows, we encourage you to review the resolver workflow to understand the operations it performs.
3. Do not modify the injected workflow. Any modifications will prevent it from being triggered.
4. Please install and use this app only on repositories where you consent to having code snippets (i.e., git_diff) processed by the LLM provider.
5. The following metadata is collected for research purpose:
   • User info: owner, repo, repo URL
   • Model info: user preference on model, duration of an attempt
   • Code info: agent code (git_diffs), commit hash, repository language composition

Security & Permission

This GitHub App requires the following permissions:

• Read & Write access to Issues and Pull Requests — to analyze issues and generate PRs
• Workflow execution — to trigger automated fixes via GitHub Actions
• Access to repository contents — to apply code changes and submit pull requests

No user secrets or sensitive information are stored in your repository. All sensitive operations are securely handled through our backend infrastructure.

Support and Acknowledgment

👐 We welcome your feedback. Feel free to fill out the google form or open an issue on this repository.
👐 If you have any issues on using OpenHands PR Arena, please send an email.
👐 This project is built upon OpenHands GitHub Backlog Resolver and inspired by Copilot Arena, an open source AI coding assistant that provides paired autocomplete completions from different LLMs.

Maintainer