Benchmarks Game suite

benchmarks/game/ holds ten programs from the Computer Language Benchmarks Game with a Plenty version of each, so that changes to generated code can be measured against fixed programs and a recorded baseline. It is a regression reference: it records where Plenty stands, and a slower run after a compiler change is a finding to explain. To make a program faster, follow Analyzing emitted code.

Layout

Each benchmark directory contains:

The reference programs are distributed under the Benchmarks Game’s revised BSD license, reproduced in benchmarks/game/LICENSE-benchmarks-game.txt. Each manifest names the program’s page and contributors.

Each reference is one C or Rust program from the site. None is marked with an asterisk, the site’s mark for a program that uses a different algorithm, so each benchmark has one idiomatic Plenty program that follows the reference’s algorithm. Where the reference uses threads and the Plenty program does not, the table says so; compare CPU time (user_s) as well as wall time for those.

Running

The runner, benchmarks/game/bench.py, uses only the Python standard library. It needs the release compiler (cargo build --locked --release), clang, rustc and cargo for the references, and optionally perf, taskset and GNU time. regex-redux’s reference is a Cargo package that uses the regex version pinned in the workspace lockfile; it builds offline from the local registry.

python3 -B benchmarks/game/bench.py list
python3 -B benchmarks/game/bench.py check                      # build all, verify check sizes
python3 -B benchmarks/game/bench.py run --mode quick n-body    # one benchmark, small size
python3 -B benchmarks/game/bench.py run --mode full --perf --output results.json \
    --compare benchmarks/game/baseline.json
python3 -B benchmarks/game/bench.py compare benchmarks/game/baseline.json results.json --metric instructions

Builds, generated inputs and outputs go under target/bench-game/. run:

1. builds the references with the commands in their manifests, which match the site’s flags except -march=native (-C target-cpu=native) instead of the site’s ivybridge;
2. builds each Plenty variant with plenty --compile and the variant’s flags;
3. checks every program’s output at the check size against expected/;
4. generates the fasta inputs for k-nucleotide, reverse-complement and regex-redux with the fasta reference;
5. runs each program --repeats times (default 3), pinned with taskset -c 0,2,4,6,8,10,12,14 (one thread per physical core on the reference laptop), recording wall time, CPU time and peak RSS, and reports the median and minimum;
6. compares each Plenty program’s output at the timed size with the reference’s output from the same run;
7. with --perf, adds one perf stat run for instructions:u and cycles:u;
8. writes JSON with the commit, tool versions, build commands and machine.

n-body and spectral-norm outputs compare within 1e-8; the others must be byte-identical.

compare (or run --compare) reports each Plenty program whose metric grew by more than --threshold (default 0.10) and exits 1 if any did. Metrics are time (median wall time), ratio (median relative to the reference in the same run, which transfers better between machines) and instructions (the least noisy on one machine). Benchmarks run at different sizes are skipped.

Sizes and baseline

quick sizes finish in about half a minute in total. full sizes are the site’s except reverse-complement, which uses the 25,000,000 fasta input shared with k-nucleotide (about 250 MB) instead of the site’s 100,000,000 (about 1 GB). benchmarks/game/baseline.json holds a full run on the reference laptop (AMD Ryzen 7 7840HS, performance governor):

Times are median wall-clock seconds. The k-nucleotide and regex-redux references are multithreaded while the idiomatic Plenty programs run one thread; user_s in the JSON gives CPU time (36.2 s for the k-nucleotide reference, 0.65 s for regex-redux).

k-nucleotide’s parallel variant postdates this baseline and has no entry in it yet; record it with the next full run. It counts the seven frame lengths on seven pool workers, as the reference does, and is the suite’s threaded workload for reference-count changes (issue #751): each job owns a Shared handle to the sequence, and its dictionary, with heap-string keys for the 12- and 18-byte frames, returns through a future and is released on the main thread.

n-body’s clist variant also postdates the baseline. It stores the bodies in a clist[Body], one column per field, instead of a list[Body], and tracks the cost of indexed clist field access in loops (issue #759): at 1,000,000 steps it runs 5.66 billion instructions against 2.95 billion for the list[Body] program and 0.23 billion for the reference.

n-body’s unchecked variant also postdates the baseline. It is the idiomatic program with the index arithmetic of energy and advance in unchecked: suites, built with --unchecked assume, so it measures what integer overflow checks cost. List bounds checks remain, since no build setting removes them. At 1,000,000 steps it runs 2.85 billion instructions against 2.95 billion for the idiomatic program.

Blocked benchmarks

A benchmark whose Plenty program needs a missing feature keeps its reference, expected output and sizes, with "status": "blocked" and the reason. The runner prints the reason instead of timing it, and tests/test_benchmarks_game.rs asserts it stays blocked with a phase-guard comment naming the backlog item. pidigits waits on std.bigint (backlog B38, issue #481). Its reference also needs GMP’s gmp.h and -lgmp.

Investigating a regression

dump writes the Plenty program’s optimized Cranelift IR and assembly, and the reference’s disassembly, under target/bench-game/dump/NAME/:

python3 -B benchmarks/game/bench.py dump n-body --filter advance

From there, follow Analyzing emitted code and its emitted-code agent skill (.agents/skills/emitted-code/SKILL.md): isolate the hot loop as a kernel next to an equivalent C kernel, as in experiments/kernels/, and attribute each extra instruction before changing the compiler. run --perf --metric instructions confirms the result on the whole program.

Tests

tests/test_benchmarks_game.rs compiles every Plenty variant without optimization and checks it at the check size, so a compiler change that breaks a benchmark fails cargo test. It does not time anything or build the references. benchmarks/game/test_bench.py tests the runner:

cargo test --locked --test integration test_benchmarks_game::
python3 -B -m unittest discover -s benchmarks/game -p 'test_*.py'

Adding a benchmark

1. Copy the reference program verbatim from its site page into reference/ and its site output into expected/.
2. Write benchmark.json following an existing one, then the Plenty program.
3. Add the benchmark to ready_benchmarks! (or BLOCKED) in tests/test_benchmarks_game.rs and to the table above.
4. Run bench.py check, then a full run, and update baseline.json.