Quantum Benchmark Zoo

← All benchmarks

qml-benchmarks

Application-level

Xanadu's suite for benchmarking quantum machine-learning models against out-of-the-box classical baselines, which won on every task tested.

Measures
Test accuracy (and, since v0.2, generative-model performance) of quantum models versus out-of-the-box classical baselines
Introduced by
Bowles, Ahmed & Schuld (Xanadu) (2024)
Also known as
Better than classical?, Xanadu QML benchmarks

qml-benchmarks is Xanadu’s open-source suite for testing whether quantum machine-learning models actually beat classical baselines. Built by Bowles, Ahmed and Schuld for the 2024 paper “Better than classical?”, it is as much a benchmarking methodology (and a critique of sloppy QML comparisons) as a scoring suite: its headline finding is that out-of-the-box classical models outperformed every quantum model tested.

How it works

The suite re-implements 12 quantum classifiers from the literature and pits them against standard classical models across six binary-classification task families that generate 160 datasets of varying difficulty. Every model, quantum or classical, goes through the same systematic hyperparameter search, and everything runs in classical simulation on PennyLane, JAX, and scikit-learn: the object under test is the model, not a quantum processor. Version 0.2 (March 2025) extended the framework to classical and quantum generative models and added Ising and spin-blob dataset families.

Strengths and limitations

Its discipline (same data, same tuning budget, honest classical baselines) is the main contribution: the paper found that removing entanglement from the quantum models often matches or improves performance, and it is widely cited as evidence against near-term QML advantage claims rather than used as a leaderboard. The limitations are the flip side: experiments are small-scale and purely simulated, so it says nothing about hardware performance (hardware-oriented application suites like SupermarQ measure that), and the tasks are deliberately simple. The paper remains arXiv-only as of mid-2026, and later third-party quantum-versus-classical studies that build on its methodology are distinct works.

Notable results

Xanadu maintains the repository: v0.1 (June 2024) reproduces the paper, and v0.2 (March 2025) added generative-model benchmarking. The paper is a standard reference in QML-benchmarking discussions, a modern successor to single-task generative scores like the qBAS score.

Key papers

Reference implementations

  • qBAS score : Early hybrid quantum-classical benchmark scoring how well a trained shallow circuit samples the bars-and-stripes distribution, reported as an F1 score of precision and recall.
  • SupermarQ : Scalable, hardware-agnostic suite of eight application-level benchmarks, with a six-dimensional feature vector that profiles how each workload stresses a device.