metriq-gym

metriq-gym is a Python framework for implementing and running standard quantum benchmarks on different quantum devices by different providers.

• Open – Open-source since its inception and fully developed in public.
• Transparent – All benchmark parameters are defined in a schema file and the benchmark code is reviewable by the community.
• Cross-platform – Supports running benchmarks on multiple quantum hardware providers (integration powered by qBraid-SDK)
• User-friendly – Provides a simple command-line interface for dispatching, monitoring, and polling benchmark jobs (you can go on with your life while your job waits in the queue).

Data generated by metriq-gym is intended to be published on https://metriq.info.

Join the conversation!

• For code, repo, or theory questions, especially those requiring more detailed responses, submit a Discussion.
• For casual or time sensitive questions, chat with us on Discord's #metriq channel.

Installation

Quick Start (Recommended)

Install metriq-gym directly from PyPI:

pip install metriq-gym

This will install the latest version of the package available for your environment and all its dependencies, making the mgym command available in your terminal.

Prerequisites

• Python (version 3.12 or newer)

Developer Setup

These instructions are for setting up a development environment if you plan to contribute to metriq-gym or run the latest version from source.

Prerequisites

Before you begin, ensure you have the following installed:

• Python (version 3.12 or newer)
• Poetry for managing dependencies

Cloning the Repository

First, clone the repository and navigate to the project directory:

git clone --recurse-submodules https://github.com/unitaryfoundation/metriq-gym.git
cd metriq-gym

If you've already cloned the repo (with or without submodules), run:

git pull --recurse-submodules

to keep it up to date.

Installation

Once you have poetry installed and the repository cloned, run:

poetry install

from the root folder of the project, in order to install the project dependencies.

We recommend doing this in an isolated virtual environment. See Poetry documentation for more information on managing virtual environments.

If you use pyenv, here is a quick start guide to set up the environment and install all dependencies:

pyenv install 3.13  
pyenv local 3.13 
poetry install
eval $(poetry env activate)

All Python commands below should be run in the virtual environment.

Running benchmarks

Workflow

You can dispatch benchmark jobs by specifying one or more configuration files for the benchmarks you wish to run.

mgym dispatch <BENCHMARK_CONFIG_1> <BENCHMARK_CONFIG_2> ... --provider <PROVIDER> --device <DEVICE>

Refer to the schemas/examples/ directory for example configuration files for supported benchmarks.

If running on quantum cloud hardware, the jobs will be added to a polling queue. The status of the queue can be checked with

mgym poll --job_id <METRIQ_GYM_JOB_ID>

where <METRIQ_GYM_JOB_ID> is the assigned job ID of the job that was dispatched as provided by metriq-gym.

Alternatively, the poll action can be used without the --job_id flag to view all dispatched jobs, and select the one that is of interest.

mgym poll

In order to export results to a JSON file, you can use the --json flag with the poll action.

mgym poll --job_id <METRIQ_GYM_JOB_ID> --json

This will create a JSON file with the results and the metadata of the job identified by <METRIQ_GYM_JOB_ID>. By default, the JSON file will be saved in the current working directory with the name <METRIQ_GYM_JOB_ID>.json.

Using local simulators

For quick testing without access to cloud hardware, metriq-gym can dispatch jobs to a local simulator. At the moment the Qiskit Aer simulator is supported. Specify the local provider and the aer_simulator device. Example (from the project root directory):

mgym dispatch metriq_gym/schemas/examples/qml_kernel.example.json --provider local --device aer_simulator

You can also create a noisy simulator based on an IBM backend by passing the backend name as the device:

mgym dispatch metriq_gym/schemas/examples/qml_kernel.example.json --provider local --device ibm_<BACKEND>

Polling local simulator jobs works the same way:

mgym poll --job_id <METRIQ_GYM_JOB_ID>

Credential management

To run on hardware, each hardware provider offers API tokens that are required to interact with their quantum devices. In order to run on these devices, you will need to follow the instructions on the respective pages of the providers and obtain API keys from them.

The .env.example file illustrates how to specify the API keys once you have acquired them. You will need to create a .env file in the same directory as .env.example and populate the values of these variables accordingly.

View jobs

You can view all the jobs that have been dispatched by using the view action. This will display basic information about each job, including its ID, backend, job type, provider, and device.

mgym view

In order to view the details of a specific job (e.g., the parameters the job was launched with), you can use the view action with the --job_id flag or select the job by index from the list of all dispatched jobs.

mgym view --job_id <METRIQ_GYM_JOB_ID>

Example: Benchmarking Bell state effective qubits (BSEQ) on IBM hardware

The following example is for IBM, but the general workflow is applicable to any of the supported providers and benchmarks.

To run on IBM hardware, you will also require an IBM token. To obtain this, please visit the IBM Quantum Platform and include the API token in the local .env file as previously described.

The schemas/examples/ directory houses example JSON configuration files that define the benchmark to run. In this case, we use the bseq.example.json file as we want to run a BSEQ job. The following dispatches a job on the ibm-sherbrooke device for BSEQ.

mgym dispatch metriq_gym/schemas/examples/bseq.example.json --provider ibm --device ibm_sherbrooke

We should see logging information in our terminal to indicate that the dispatch action is taking place:

INFO - Starting job dispatch...
INFO - Dispatching BSEQ benchmark from metriq_gym/schemas/examples/bseq.example.json on ibm_sherbrooke...
...
INFO - Job dispatched with ID: 93a06a18-41d8-475a-a030-339fbf3accb9

We can confirm that the job has indeed been dispatched and retrieve the associated metriq-gym job ID (along with other pieces of metadata).

+--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+
| Metriq-gym Job Id                    | Provider   | Device                                               | Type           | Dispatch time (UTC)        |
+======================================+============+======================================================+================+============================+
| 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm        | ibm_sherbrooke                                       | BSEQ           | 2025-03-05T10:21:18.333703 |
+--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+

We can use the "poll" action to check the status of our job:

mgym poll --job_id 93a06a18-41d8-475a-a030-339fbf3accb9

Doing so gives us the results of our job (if it has completed):

INFO - Polling job...
BSEQResult(largest_connected_size=100, fraction_connected=0.7874015748031497)

In the event where the job has not completed, we would receive the following message instead

INFO - Polling job...
INFO - Job is not yet completed. Current status:
INFO - - d0wtyfhvx7bg008203b0: QUEUED (position 3)
INFO - Please try again later.

As a convenience, while we could supply the metriq-gym job ID, we can also poll the job by running mgym poll and then selecting the job to poll by index from our local metriq-gym jobs database.

Available jobs:
+----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+
|    | Metriq-gym Job Id                    | Provider   | Device                                               | Type           | Dispatch time (UTC)         |
+====+======================================+============+======================================================+================+=============================+
| 0  | 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm        | ibm_sherbrooke                                        | BSEQ           | 2025-03-05T10:21:18.333703 |
+----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+
Select a job index: 

Entering the index (in this case, 0), polls the same job.

Select a job index: 0
INFO - Polling job...

Running multiple benchmarks

metriq-gym supports running multiple benchmarks by specifying multiple configuration files. This allows you to obtain a comprehensive performance profile of a quantum device with full control over the parameters of each benchmark.

Multiple benchmark types

To run different types of benchmarks on a device, specify multiple configuration files:

mgym dispatch bseq_config.json clops_config.json qv_config.json --provider ibm --device ibm_sherbrooke

Same benchmark with different parameters

You can run the same benchmark type multiple times with different configurations to test various parameter ranges:

mgym dispatch bseq_small.json bseq_large.json --provider ibm --device ibm_sherbrooke

The system will process each configuration file as a separate job, giving you full control over the benchmark parameters and allowing for comprehensive device characterization.

Contributing

First, follow the Developer Setup instructions above.

Style guide

We don't have a style guide per se, but we recommend that both linter and formatter are run before each commit. In order to guarantee that, please install the pre-commit hook with

poetry run pre-commit install

immediately upon cloning the repository.

Tests

The entire suite of tests can be run with

poetry run pytest

Unit tests only can be run with

poetry run pytest -m "not e2e"

End-to-end tests only can be run with

poetry run pytest -m e2e

Type checking

The project uses mypy for static type checking. To run mypy, use the following command:

poetry run mypy

Documentation

The project uses Sphinx to generate documentation. To build the HTML documentation:

1. Navigate to the docs/ directory:

cd docs/

Run the following command to build the HTML files:

make html

Open the generated index.html file located in the _build/html/ directory to view the documentation.