Quantum Benchmark Zoo

← All benchmarks

Volumetric benchmarking

System-level

Sandia framework that generalizes Quantum Volume from square circuits to the full width × depth plane, mapping the frontier of circuit shapes a device can execute.

Measures
The region of circuit shapes (width w, depth d) at which a device passes a chosen test suite's success criterion, plotted as capability regions with a Pareto frontier
Introduced by
Blume-Kohout & Young (Sandia) (2019)

Volumetric benchmarking is a framework introduced by Robin Blume-Kohout and Kevin Young at Sandia in 2019 that generalizes IBM’s Quantum Volume from square circuits to the whole width × depth plane. Instead of compressing a processor’s power into one number, it maps which circuit shapes (how many qubits, for how many layers) the device can execute, exposing the space-time trade-offs that any single-number benchmark hides.

How it works

A volumetric benchmark is a mapping from circuit shapes (width w, depth d) to test suites plus a pass criterion: for each shape it specifies an ensemble of test circuits, a success metric for each circuit, and a threshold. The device runs suites over a grid of shapes, each shape is marked pass or fail, and the results are plotted as a region in the width × depth plane whose boundary is a Pareto frontier of achievable shapes. The framework deliberately subsumes many circuit families (random, periodic, and algorithm-inspired), and Quantum Volume is recovered as the special case of square random circuits with the heavy-output criterion. The defining paper was published as Quantum 4, 362 (2020).

Strengths and limitations

Its strength is showing the shape of capability: two devices with equal QV can have opposite frontiers (one wide and shallow, the other narrow and deep), and volumetric plots put random-circuit tests and algorithm-inspired suites on the same axes. The flexibility is also the caveat: a volumetric result is defined only relative to the chosen circuit family, success metric, and threshold, so plots from different studies are not comparable unless the full specification matches. Frontiers also depend strongly on the ensemble: Sandia’s mirror-circuit experiments found structured test circuits failing at shapes up to an order of magnitude smaller than random ones (Proctor et al.).

Notable results

The framework has become a standard presentation for whole-device results: Sandia’s randomized mirror circuits mapped capability regions of twelve IBM and Rigetti processors, the QED-C application suite plots its results volumetrically, and IonQ’s algorithmic qubits (#AQ) is reported in this framing.

Key papers

Reference implementations

  • Quantum Volume : Single-number, full-system benchmark that scores a device by the largest random square circuit it can run while still generating heavy outputs reliably.
  • Randomized Mirror Circuits : Sandia's scalable whole-processor test that runs self-inverting random circuits across the width × depth plane and maps where a device still returns the right bitstring.
  • QED-C Application-Oriented Benchmarks : Open-source consortium suite that scores devices on ~16 algorithm workloads, from Bernstein-Vazirani to VQE and MaxCut, via normalized fidelity on width-by-depth volumetric plots.