Quantum Benchmark Zoo

← All benchmarks

QUARK

Application-level

BMW Group's open-source framework for defining, orchestrating, and reproducing application-level quantum benchmarks drawn from industry use cases.

Measures
Per-application solution quality and runtime of quantum hardware, simulators, and algorithms on industry-derived workloads, executed through configurable, reproducible benchmarking pipelines
Introduced by
Finžgar, Ross, Klepsch & Luckow (BMW Group) (2022)
Also known as
QUantum computing Application benchmaRK, QUARK framework

QUARK (QUantum computing Application benchmaRK) is an open-source framework introduced by BMW Group researchers for designing, implementing, executing, and analyzing quantum application benchmarks, presented at IEEE QCE 2022. It approaches benchmarking from an industrial angle: rather than fixing a suite of circuits, it standardizes the pipeline by which application benchmarks are defined, run, and reproduced. It is strictly a framework rather than a benchmark, though often tabulated alongside scored suites.

How it works

A QUARK benchmark is assembled from pluggable modules: an application, a mapping into a solver-ready formulation, a solver or algorithm, and a target quantum device or simulator. Reference application workloads include robot-path optimization (PVC sealing), vehicle-option optimization (auto carrier loading), TSP, maximum independent set, set cover, MaxSAT, and quantum generative learning. From a configuration file, the framework executes every requested combination, records the results reproducibly, and reports per-application metrics for solution quality and runtime. It deliberately omits an aggregate device score.

Strengths and limitations

QUARK’s workloads descend from real industrial use cases rather than textbook kernels, and its emphasis on orchestration and reproducibility addresses a genuine weakness of ad-hoc application benchmarking. The corresponding limitation is comparability: results are per application and depend on the configured modules, so cross-device numbers only make sense under identical configurations. Two citation pitfalls: the author list varies by arXiv version (v1 and v2 list four authors; Leonhard Hölscher was added in v3, the version published at IEEE QCE 2022), and no paper is titled “QUARK 2.0”: the 2.x line’s generative-learning extensions are described in Kiwit et al. 2023 and a follow-up scalability and noise-resilience study. It is unrelated to other things named Quark, such as the lightweight cryptographic hash family.

Notable results

Development is active: the original repository (Apache-2.0) released v2.1.7 in July 2025 and is now labeled the legacy code base, while a rewritten core module for QUARK 3.0+ is in pre-release development. Fixed-suite counterparts include the QED-C benchmarks and SupermarQ; QOBLIB standardizes optimization problem instances on the adjacent library side.

Key papers

Reference implementations

  • QED-C Application-Oriented Benchmarks : Open-source consortium suite that scores devices on ~16 algorithm workloads, from Bernstein-Vazirani to VQE and MaxCut, via normalized fidelity on width-by-depth volumetric plots.
  • QOBLIB : Open library of ten classically hard, practically motivated optimization problem classes (the Intractable Decathlon) with common reporting rules for quantum and classical solvers.
  • SupermarQ : Scalable, hardware-agnostic suite of eight application-level benchmarks, with a six-dimensional feature vector that profiles how each workload stresses a device.