Quantum Benchmark Zoo

← All benchmarks

Arline Benchmarks

Software stack historical

Automated head-to-head comparison of quantum compilers (Qiskit, Tket, Cirq, Quilc, PyZX) on gate counts, depth, circuit cost, and runtime, with auto-generated LaTeX reports

Measures
Post-compilation gate count, circuit depth, hardware-dependent circuit cost, and per-stage compiler runtime for each compilation framework
Introduced by
Kharkov, Ivanova, Mikhantiev & Kotelnikov (Arline / Turation Ltd) (2020)
Also known as
arline_benchmarks

Arline Benchmarks was an automated benchmarking platform for quantum compilers, released in mid-2020 by the quantum software startup Arline (named for Arline Greenbaum Feynman) and described in a 2022 preprint. Like the other compiler benchmarks in this catalog, it scores software rather than quantum hardware: it answered, in one reproducible run, which compilation framework produced the cheapest circuits for a given target.

How it works

The platform compiles a shared suite of circuits (random circuits plus structured workloads such as VQE ansätze, Trotter decompositions, Grover search, and option pricing via amplitude estimation) through several frameworks (IBM Qiskit, CQC Tket, Google Cirq, Rigetti Quilc, PyZX) to fixed hardware gate sets. For each run it records post-optimization gate counts, circuit depth, hardware-dependent circuit cost functions, and compiler runtime broken down by compilation stage, then assembles everything into an auto-generated LaTeX report. It also introduced composite pipelines that chain optimization subroutines from different compilers, letting users hunt for cross-compiler combinations that beat any single framework.

Strengths and limitations

Arline was among the first tools to make compiler comparison push-button and apples-to-apples, and it saw early industrial use: Oxford Quantum Circuits used it for compiler testing, per the project README. Its published comparisons, however, reflect 2020–2022 framework versions and are long stale. The paper remains an arXiv preprint, and the platform is unmaintained: the last GitHub push was March 2022, the last PyPI release June 2020, and the company domain no longer resolves. Note that Arline Quantum, the companion open-source circuit library released alongside it, is a distinct package.

Notable results

The platform is of historical interest today. Its niche, standardized and automated quantum-SDK performance comparison, is now served by IBM’s Benchpress (2024), while curated circuit suites such as MQT Bench cover the workload side.

Key papers

Reference implementations

  • Benchpress : IBM's open-source suite of 1,000+ timed tests comparing how fast quantum SDKs construct, manipulate, and transpile circuits, plus the quality of the compiled output.
  • MQT Bench : Library of 70,000+ benchmark circuits, each served at four abstraction levels, so quantum compilers, simulators, and verifiers can be compared on common workloads.