How determinism works#

Every GPU. One answer.

Why the same float32 gives different bits#

The same float32 program does not give the same bits on every GPU. The compilers of each vendor are free to fuse, reorder and simplify operations, the hardware handles very small numbers differently, and every vendor ships its own exp, sin and the rest. The differences are small, but they grow during a long simulation or a training run, and they make a result impossible to check bit by bit on another machine.

What the auditable mode guarantees#

In the auditable mode (the default), every operation of a .vis program and of vision.tensor has one defined result, the same on every supported device:

  • every float operation is rounded the same way, whatever the compiler of the driver would do;

  • every value is finite: an overflow gives the largest float with the same sign, and every function has a finite result for every input;

  • the 46 built-in functions are Vision’s own, not the vendor’s (see Built-in functions);

  • conversions and integer operations follow fixed rules, also where drivers disagree;

  • every sum has a fixed order and nothing depends on the timing of the GPU (no atomics);

  • no decision depends on the device, so tuning cannot change a result.

Vision handles each of the differences it has found between vendors without any per-vendor setting: a new device either gives the same bits or is reported by the conformance kit.

What has been tested#

Test

Result

Every math function, all 4 294 967 296 float32 inputs

identical bits on NVIDIA, Intel, llvmpipe and numpy for all 30 variants

900 random .vis programs with overflows, values near 2⁻¹²⁶, division by zero and x - x

identical on numpy, NVIDIA, Intel and llvmpipe

Heat diffusion up to 4096² and 20 000 steps

identical on all four

Each protection switched off, one at a time (mutation testing)

every one detected by the tests

GPT training, 30 steps, on three software stacks

the same SHA-256 of the weights

Apple Silicon passes the conformance kit (Metal, on GitHub’s macOS runner: 16 of 16 cases). AMD GPUs have not been tested yet; the conformance kit is ready for that.

What it costs#

The finite-value rule costs about 19% on a 1024² diffusion on the RTX 4050 (which fits in cache), about 3% on 2048² (memory-bound) and about 2% on the Intel GPU. All protections together make the auditable mode 1.3–1.4× slower than native arithmetic on the RTX 4050 and 1.04–1.4× on the Intel GPU.

Outside the guarantee#

  • The fast mode of vision.tensor (fast), which uses native functions and FMA.

  • Tensor cores, whose internal summation order varies between vendors; the auditable mode does not use them.

  • Programs that call code outside Vision.