Quantum Benchmark Zoo

← All benchmarks

Chook

Platform-specific

Open-source Python suite that generates Ising, QUBO, and higher-order optimization problems with planted, a-priori-known ground states and tunable hardness across five planting schemes.

Measures
Solver performance on generated instances with planted optima; the scoring metric (success probability, time-to-solution, residual energy) is chosen per study
Introduced by
Perera, Katzgraber & colleagues (2020)

Chook is the reference open-source suite for generating binary optimization problems with planted solutions, published by Perera, Katzgraber and colleagues in 2020. Because the ground state is known by construction, quantum annealers, Ising machines, and classical heuristics can be scored end-to-end at sizes where exact classical solving is impossible. It targets optimization platforms rather than gate-model QPUs (hence its filing here), and it is an instance generator rather than a scored protocol.

How it works

Chook unifies five planting schemes introduced in earlier papers by overlapping authors: tile planting (2D/3D lattices composed from solved subproblem tiles), the Wishart planted ensemble (fully connected, with a tunable hardness peak), equation planting (k-regular k-XORSAT systems verifiable by Gaussian elimination), deceptive cluster loops for Chimera-graph annealers, and k-local planting for higher-order (HOBO) cost functions. A pip-installable Python tool emits Ising/QUBO instance files together with the planted ground-state energy; each study then picks its own figure of merit (success probability, time-to-solution, or residual energy) for recovering the known optimum.

Strengths and limitations

Planted optima provide an answer key at any scale, and the hardness knobs let studies dial difficulty. But Chook’s value is comparable inputs, not comparable scores: it defines no metric or pass/fail protocol, so results across different planting schemes, parameters, and metrics are not directly comparable. Hardness realism is also debated: the posiform-planting line of work (2023–2025) was motivated partly by limitations of existing planted generators, and a 2024 Communications Physics study argues such planted ensembles cannot characterize the physical systems’ evolution. Development is dormant (last release v0.2.0, March 2021; the paper remains arXiv-only), yet use continues: a 2024 benchmarking study generated its Wishart-planted instances with Chook to score a coherent-Ising-machine simulator against parallel tempering, and recent planting papers still cite it as the reference generator.

Key papers

Reference implementations

  • Frustrated cluster loops : Planted-solution Ising benchmark of the D-Wave 2000Q era: frustrated loops laid over ferromagnetic qubit clusters with tunable ruggedness, scored by time-to-solution against classical solvers.
  • Deceptive cluster loops : Frustrated cluster loops with a tunable coupling scale λ that conceals the planted structure, built to test whether annealer speedups survive against structure-exploiting classical solvers.
  • Time-to-solution : The standard quantum-annealing benchmark: wall-clock time to find the ground state at least once with 99% probability, compared against optimized classical solvers to test for quantum speedup.