BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

Quantizing a trained LLM straight to FP4 lets Blackwell GPUs use native FP4 matrix multiplies, but accuracy drops noticeably. The common fix, SVDQuant, stores an extra BF16 low-rank correction, which costs storage and competes with the main computation for the same tensor cores. This paper from the Massachusetts Institute of Technology proposes BiLoCo (binary low-rank correction): every vector in the correction is replaced by +1 and −1 entries, so applying the correction needs only additions and subtractions and can run on the otherwise idle CUDA cores, concurrently with the FP4 matrix multiply. The same storage holds 16 times the rank, perplexity matches or beats SVDQuant on 4B to 72B models, and at batch size 1 the whole kernel is only 6% to 11% slower than plain FP4.

Teams deploying LLM inference and evaluating FP4 should read this. The starting point is practical: during decoding, the matrix multiply is bound by reading weights from memory, the tensor cores wait for data, and the CUDA cores sit idle, so the correction is designed to fit exactly those idle units. The method is simple and the code is public. But it only pays off for small-batch decoding; at larger batches the advantage disappears, so check your serving profile first.

What problem does it solve?

Post-training quantization (PTQ) means compressing weights and activations to low precision after training, without fine-tuning. Blackwell GPUs natively support NVFP4 matrix multiplies: every 16 values share one FP8 scale and each value has 4 bits, about 4.5 bits per weight on average. Both storage and compute are low precision, so it is cheap.

The trouble is that 4 bits can only represent 15 values: 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4 and ±6. If a block contains a few very large values, the shared scale must accommodate them and everything else in the block loses precision. Quantizing directly to NVFP4 therefore raises perplexity visibly.

A natural fix is to split the weight into two parts: a residual R stored in 4 bits plus a low-rank matrix stored at higher precision, which absorbs the large directions 4 bits cannot represent. SVDQuant does this with a rank-32 BF16 correction obtained by SVD. That correction has two costs: each rank stores two BF16 vectors, which is a lot of bits, and it is itself a matrix multiply that competes with the FP4 computation for tensor cores.

The core idea: corrections made only of +1 and −1

BiLoCo writes the correction as a sum of r rank-one terms, each a coefficient α times two vectors containing only +1 and −1. At inference it computes α·u·(v·x). The dot product v·x needs only additions and subtractions: add the matching entry of x where v is +1, subtract it where v is −1.

A worked example: if x is [0.5, −1.2, 2.0, 0.3] and v is [+1, −1, +1, −1], then v·x = 0.5 + 1.2 + 2.0 − 0.3 = 3.4, with no multiplications at all. Multiply by α, then use the signs of u to add or subtract that number into each output position.

The storage arithmetic is favorable. Ignoring the coefficients, a pair of sign vectors costs 1 bit per entry while a pair of BF16 vectors costs 16, so the same storage holds 16 times the rank. For a 5120×5120 matrix, a BF16 rank-32 correction and a binary rank-512 correction both take about 5.2 million bits, while the FP4 weight itself takes about 120 million bits, so the correction adds only about 4% storage.

Is more rank better than more precise rank?

At equal storage, should you spend bits on a few precise directions or many coarse ones? It depends on the singular value spectrum. If the first few directions dominate, a few precise ranks suffice; if the spectrum is flat, covering more directions matters more.

The authors measured 189 projection matrices from three models and found the leading spectrum is flat: the first singular value is only about twice the 32nd. They measure capture as the ratio of the Frobenius norm of the low-rank part to that of the original matrix. By this measure, binary rank 512 captures 47% to 53%, while BF16 rank 32 captures only 26% to 29%.

At every storage budget the binary correction captures more. On LLM weights, spending limited storage on covering more directions beats making each direction more precise.

Finding the sign vectors

The optimal binary decomposition cannot be read off an SVD. The authors use an existing greedy procedure that finds one rank-one component at a time: Hoot says▾

Hoot saysA 3 × 2 matrix by hand: two greedy sign components, then a joint refitClose ✕

Alternating signs finds the directions, but when two components overlap their separately fitted coefficients interfere; one joint refit can take the error straight to zero.

Greedy, one by one: W rows (1, 1), (1, 1), (4, 4)

component 1

v = (+, +) → Rv = (2, 2, 8) → u = (+, +, +), Rᵀu = (6, 6), v unchanged

α₁

12 ÷ 6 = 2

residual

(−1, −1), (−1, −1), (2, 2)

component 2

u = (−, −, +), v = (+, +); α₂ = 8 ÷ 6 ≈ 1.33

error left

sum of squares ≈ 1.33

Each α is fitted on the residual of its moment and ignores later components.

Joint refit

the overlap

u₁ · u₂ = −1 ≠ 0 and the v's match

rows 1, 2 need

α₁ − α₂ = 1

row 3 needs

α₁ + α₂ = 4

solution

α₁ = 2.5, α₂ = 1.5, error 0

Same two sign components, only new coefficients, and the error drops from 1.33 to 0; more components and more overlap make this step worth more.

W is an illustrative toy, not a weight from the paper; the real procedure quantises the leftover residual to NVFP4 at the end. α is the error-minimising value, which for sign vectors is uᵀRv divided by the number of entries. Source: paper §4: greedy rank-one components one at a time (u = sign(Rv), v = sign(Rᵀu), 5 alternations), error-minimising coefficient subtracted from the residual; all coefficients refit jointly because components overlap; leftover residual quantised to NVFP4

1. Set the residual R to the original weight W.
2. Initialize a random sign vector v and alternate: u takes the signs of Rv, v takes the signs of Rᵀu. Five rounds are enough to settle.
3. Compute the error-minimizing coefficient α, subtract α·u·vᵀ from the residual, and find the next component.
4. After all components are found, refit all coefficients jointly, because components overlap and individually fitted coefficients are not optimal.
5. Quantize what remains of the residual to NVFP4.

It is fast: at rank 512, one B300 handles Qwen3-4B in about 1 minute and Qwen3-32B in about 5 minutes.

How the kernel adds almost no time

Additions alone are not enough; the correction must truly overlap the FP4 multiply. The authors wrote a fused kernel for the B300, which has 148 streaming multiprocessors (SMs, the GPU's independently scheduled compute units).

At launch, some thread blocks run the FP4 multiply as usual, each owning one 128×64 output tile. The remaining thread blocks compute the intermediate z = α·(v·x) on CUDA cores and write it to a small buffer in memory. The epilogue threads of each GEMM block, which are idle during the main loop anyway, wait for z, expand it with u into the correction for their tile, and add it before writing the result. Compared with running the GEMM and then launching a separate correction kernel, this uses a single launch, no extra stream, and no write-out and read-back of the full output; the only extra memory traffic is the tiny z buffer.

Why does the correction cost so little? It reads very little data. At rank 128 it reads about 0.16 MiB of sign bits while the FP4 path reads about 12.5 MiB of weights, an 80x gap. Because data reading is nearly free, the correction's time is set by four steps that must happen in order: accumulate the signed sums over the input dimension, publish z to the buffer, have the GEMM block wait until z is ready, and expand with u. As long as these finish before the GEMM main loop ends, the correction is fully hidden.

Result one: accuracy

The table shows WikiText-2 perplexity (lower is better), with BiLoCo and SVDQuant at equal storage. On 4B, 8B and 14B BiLoCo is better on both WikiText-2 and C4; on 32B and 72B it ties or is slightly better. With a quarter of the storage (rank 128), BiLoCo trails SVDQuant by at most 0.05.

Perplexity gaps look small; the distance to the original model's output distribution says more. The authors computed per-token KL divergence between each quantized model and the BF16 model on 200 FineWeb documents, reporting both the mean and the 99th percentile, the value at the 99% position when all token KLs are sorted. BiLoCo has a lower 99th-percentile KL on all five models; on Qwen3-14B it is 0.130 versus 0.167, 22% lower. So the gain is not just on typical tokens but larger on the tokens most likely to go wrong.

Downstream gains are much smaller: compared with SVDQuant at equal storage, mean WinoGrande and HellaSwag accuracy is 0.1 to 0.7 points higher on the four Qwen models and 0.3 points lower on Llama-3-8B.

Result two: speed

The test shape is K = N = 5120, close to an attention projection in a 14B model, compared against the fastest plain FP4 kernel measured in the same run.

At batch 1, every rank from 16 to 512 adds 6% to 11%, or 0.34 to 0.64 microseconds. But correction work grows with both batch and rank, while FP4 GEMM time barely changes up to batch 128 because it is spent reading weights. So larger batches expose the correction. With a 20% overhead limit, rank 128 works up to batch 8 and rank 512 only up to batch 2.

One easy-to-miss result: a dense BF16 correction of the same rank inside the same kernel takes the same time as the binary one. The advantage of the binary format is not faster arithmetic but more rank per byte.

Limitations as I see them

First, the scope is narrow. It only pays off for small-batch decoding; if your service often runs at batch 16 or more, overhead doubles or worse, and the authors themselves recommend a tensor-core correction there.

Second, the kernel does not support the MLP gate and up projections. Those layers are so wide that they need more output tiles than the B300 has SMs, while the kernel requires all GEMM blocks to run at once. These are exactly the largest layers, so end-to-end speedups need another solution for them.

Third, speed is measured only per kernel, with no end-to-end decode latency for a full model. The accuracy gains are also modest: downstream tasks differ by fractions of a point, and the clearest perplexity gains are on smaller models.

How a team could try it

1. Confirm your deployment is small-batch decoding, such as on-device, low-concurrency private deployment, or latency-sensitive single-user serving.
2. Quantize your model with the public code; it takes minutes. Check perplexity and KL against direct NVFP4.
3. If accuracy improves meaningfully, time the kernel over your usual batch range and pick a rank within 20% overhead.
4. For gate and up projections, keep an SVDQuant-style correction or plain quantization, and use BiLoCo for the other layers.

Bottom line

During decoding the tensor cores wait on memory while the CUDA cores sit idle; making the FP4 correction out of ±1 vectors that need only additions buys 16 times the rank for the same storage at almost no extra time at small batch sizes.