Conformance kit#

Your hardware against the reference, bit by bit.

The kit checks a new GPU (Mac, AMD, ARM or any other) against reference signatures generated on NVIDIA, Intel, llvmpipe and numpy, which agree on every case.

pip install ".[proof]"
vision conformance --output result.json --certificate certificate.html

It runs 15 cases:

  • 12 .vis programs: diffusion, a failure cascade, the wrap edge, all deterministic functions, 262 144 inputs per math function, sin/cos/tan/atan2 with arguments up to 2¹²⁷, the newer functions (inverse trigonometric, hyperbolic, erf, normcdf, …), measures, a plant loaded from a PNG, integer channels with flux(), an ensemble of 40 scenarios, and 3-D canvases (wrap and zero edges, 7-point stencils, reads between 2-D and 3-D canvases);

  • 3 tensor cases: matrix product, training a small GPT, and int8 inference;

  • plus a SHA-256 computed on the GPU.

The kit also measures speed (diffusion and matrix product). --output collects everything into one file to send back, and --certificate writes a readable HTML page with every case, the signatures and what the certificate does and does not claim.

On this machine: 60 checks on 4 devices, all agreeing, also with the package installed outside the repository. On Apple Silicon (Metal, macOS 15.7, GitHub’s macOS arm64 runner): all 13 cases agree as well, on the macOS runner in the continuous integration.

What the kit has found#

The kit and the verification campaign found real differences between vendors that earlier tests missed: a driver that computed the signed integer remainder wrongly (the wrap edge read the wrong pixel on that GPU), GPUs that read very small inputs as zero, a GPU that rounds very small products differently from the others, and a platform with fewer buffer slots per kernel. Vision handles all of them, with the same bits on every device, and the kit checks each one on any new machine.

On this page