From Ampere onward, NVIDIA GPUs support 2:4 sparsity: in a matmul, one operand keeps 2 nonzeros per 4 elements along the accumulate axis, and tensor cores can nominally run twice as fast. It stacks with quantization, so the practical deployment question is: at the current precision, is the next step to add 2:4, or to drop one precision tier? This independent researcher's paper compares, on Llama-3.1-8B-Instruct and Qwen3-8B, under "matched storage bits (including sparse position indices)" and "matched nominal compute tiers," all six matmul operand roles in one transformer layer, with speed measured on L40S, H100, and RTX PRO 6000. Conclusion: as long as a lower precision tier is still available, adding 2:4 is both less accurate and slower; only at the hardware's lowest supported precision (FP8 on Ada/Hopper) does 2:4 pay off—on H100, a vLLM-served 2:4 + FP8 model raises prefill throughput 27% over dense FP8 with matched GSM8K, but only after retraining.
1. vLLM already removed 2:4 support—is it still worth using?
On one side every NVIDIA GPU generation keeps sparse tensor cores, and Blackwell adds sparse FP4; on the other, in 2026 vLLM and LLM Compressor removed 2:4 support for lack of users. Prior prune-versus-quantize comparisons mostly use "nominal compression," focus on weights, and do not put "add 2:4" versus "drop one precision" head-to-head on real hardware.
The authors flag four easy-to-miss points:
- 2:4 stores a position index beside each kept value, so bits per original element exceed half;
- both claim 2× throughput, but datasheets do not say how much you actually get;
- at inference most operands (activations, Q, K, V, attention probabilities) are computed on the fly and must be pruned/compressed before every matmul—time invisible if you only time the matmul;
- hardware dictates which operand may be sparse and along which axis, so tensor-wide statistics (e.g. kurtosis) do not show what 2:4 actually drops.
2. Method: compare at matched cost
2.1 Matched bits
Each format is billed by bits actually stored per original element—values, indices, and scales. Dense INTk uses one FP16 scale per 128 elements: $k+0.125$ bits/element; 2:4 keeps half the elements but adds 1 index bit per original element. So 2:4 + INT8 costs the same as dense INT5; 2:4 + INT4 equals INT3; 2:4 + FP16 costs more than FP8.
At matched storage the real choice is "drop half the elements and store the rest at 2× precision" versus "keep every element and add 1 bit each."
2.2 Matched compute tiers
Pair by datasheet nominal throughput: 2× tier is 2:4 + FP16 versus dense FP8/INT8; 4× tier is 2:4 + FP8 versus dense 4-bit float (Blackwell only).
2.3 How to measure
Accuracy is operator-level relative error versus an FP32 reference, summarized as $R=$ (best 2:4 config error) / (best quantization config error); $R<1$ means 2:4 wins. Each side picks the best among equally effortful methods: 2:4 tries magnitude pruning, Wanda, SparseGPT, etc.; quantization tries round-to-nearest, GPTQ, AWQ; all use C4 calibration. Speed uses cuSPARSELt sparse matmul; on-the-fly operands count online prune or online quantize time on both sides. Over 220K measured configs.
3. Experiments
3.1 Above the lowest precision: dropping a precision tier always wins
This checks whether adding 2:4 can ever be more accurate or faster while a lower precision is still available.
Accuracy. At matched bits, 2:4 loses at every storage budget on both models: at 8 bits the error gap is more than an order of magnitude; the gap shrinks as bits fall but does not vanish; under a 3.1-bit budget the closest comparison (Llama's K versus INT3) still has slightly above 1. In the 2× tier, 2:4 + FP16 error is 5–16× dense FP8 on every operand.
Speed. Table below is "opponent time / 2:4 time"; >1 means 2:4 is faster (Llama-3.1-8B projection layers, median over 2K–16K tokens):
W = sparse weights, X = sparse activations (including online preprocess). Within each tier, the 2:4 side is slower. The authors' explanation: the energy in the 2 values 2:4 drops per 4 elements (sum of squares) cannot be recovered by giving the kept values more bits, while quantization error halves with each extra bit. On Gaussian data, elementwise 2:4 removes 13.2% of energy.
3.2 At the lowest precision: 2:4 can speed up, but needs retraining
This checks how much 2:4 helps at the FP8 or NVFP4 floor and at what cost. Linear-layer weights are 2:4; attention stays BF16; model is Llama-3.1-8B base.
vLLM end-to-end (single H100), 2:4 model = Sparse-Llama (2:4 Llama retrained on 13B tokens):
2:4 + FP8 beats dense FP8 on both prefill and decode, with no significant GSM8K difference. Serving the same weights with dense kernels removes the speedup, so the gain comes from sparse kernels. Under BF16, vLLM's 2:4 path is not faster than dense.
Retraining cost. One-shot pruning is unusable at every floor. Table: WikiText-2 perplexity / GSM8K accuracy (250 questions):
The authors retrain by distilling from the dense model on 1B FineWeb-Edu tokens: perplexity approaches Sparse-Llama, but GSM8K is only about half. Perplexity overestimates how much retraining has recovered; judging whether retraining is done requires reasoning tasks.
3.3 Blackwell sparse FP4
Blackwell sparse FP4 is not elementwise 2:4; it keeps 2 of 4 adjacent pairs per 8 elements. The kernel is 1.4× dense NVFP4, but removes 20.8% of energy on Gaussian data; one-shot pruning yields perplexity 108.83—five times elementwise 2:4; Sparse-Llama retrained for elementwise 2:4 cannot convert directly. Within the authors' 1B-token retrain budget, GSM8K recovers only to 0.132.
4. Limitations
Only two 8B models; Blackwell 4:8 conclusions are limited to 1B tokens and one untuned retrain recipe—whether longer retraining recovers is untested. The 4-bit integer weight floor (mainly for decode) has no speed numbers because vLLM removed the sparse kernels. Production often adds a Hadamard rotation before low-bit quantization; the authors note that makes 2:4 worse on most operands, so these unrotated results are optimistic for 2:4.
5. Takeaways
If the hardware still supports a lower precision (e.g. H100 BF16→FP8), drop precision first—do not add 2:4. If you already serve FP8, prefill throughput is the bottleneck, and you can get a fully retrained 2:4 model, 2:4 can give real speedups—but vLLM mainline removed support, so you must maintain it yourself. When evaluating a retrained sparse model, do not trust perplexity alone.