Dense Matrix Multiplication’s Dominance Is Being Challenged — And the Numbers Back It Up

Feedforward layers sit at the heart of every large language model, consuming over two-thirds of model parameters and more than 80% of total floating-point operations. Researchers from Sakana AI and NVIDIA have now published an ICML 2026 paper introducing TwELL — a tile-wise ELLPACK sparse format with fused CUDA kernels — that squeezes 20.5% faster inference and 21.9% faster training out of those layers without changing the underlying model architecture. The paper is available on arXiv under identifier 2603.23198.

The core insight is deceptively simple: for any given token processed by an LLM, only a tiny fraction of hidden neurons actually fire. The rest output zero after the activation function — a phenomenon called activation sparsity. NVIDIA GPUs are engineered to excel at dense matrix multiplications, so they have historically ignored that sparsity entirely, treating zeros as if they were meaningful values. TwELL is built to stop doing that, and the measured speedups suggest the wasted compute was substantial.

What makes this announcement technically credible is its scope. Prior sparse LLM kernels — including TurboSparse, ProSparse, and Q-Sparse — addressed memory-bound GEMV (General Matrix-Vector) operations suited to single- or few-token inference. TwELL targets compute-bound GEMM (General Matrix-Matrix Multiplication) operations in a batched setting with thousands of tokens, covering the regimes that actually matter for training and high-throughput production inference.

Why Activation Sparsity Has Been a Locked Door Until Now

Activation sparsity is not a new observation. Engineers have known for years that LLM feedforward layers produce large numbers of zeros after activation functions. The problem has always been exploitation: historically, sparse operations on modern GPUs ran slower than their dense counterparts, making the theoretical savings unreachable in practice. Traditional sparse formats also impose a conversion overhead — the cost of reorganizing data into a sparse representation — that frequently negates any compute savings before a single multiply-accumulate is skipped.

The conventional ELLPACK format, for instance, packs non-zero values row-by-row. Constructing it from tiled matrix-multiplication output requires a separate kernel launch, a global memory read, and synchronization across Compute Thread Arrays (CTAs) — three sequential bottlenecks that collectively erode the benefit of skipping zero-valued computations. This is why prior approaches, despite targeting the right problem, could not generalize beyond narrow single-token inference scenarios.

TwELL sidesteps this by using tile-wise packing instead. It is built directly within the epilogue of the matmul kernel itself, and it partitions columns into horizontal tiles that match the matmul kernel’s tile size T_n. The result, as documented in the ICML 2026 paper, is that the sparse format is constructed without the separate kernel launch, global memory round-trip, or CTA synchronization that made traditional ELLPACK impractical. The fused CUDA kernel design is what converts a theoretically attractive idea into a measured 20.5% inference speedup and 21.9% training speedup on NVIDIA GPUs.

The Harder Question: What This Approach Gives Up

The speedups are real and independently corroborated — documentation from multiple sources confirms TwELL achieves over 20% faster LLM training and inference on NVIDIA GPUs without altering model architecture. But the approach makes a deliberate trade: it abandons the highly optimized dense computation pathways that NVIDIA’s Tensor Cores are specifically designed to exploit. Dense GEMM on modern NVIDIA hardware benefits from decades of compiler tuning, cuBLAS optimizations, and hardware-level support. TwELL introduces kernel complexity that sits outside that established stack.

The unstructured nature of the sparsity is also a genuine engineering challenge. Unstructured sparsity — meaning zeros appear at arbitrary positions rather than in regular patterns — is fundamentally harder to exploit efficiently than structured sparsity, where hardware can predict memory access patterns. Managing irregular sparse data in a compute-bound batched setting, at the scale of thousands of tokens, requires careful kernel design to avoid introducing new memory bottlenecks that replace the compute bottleneck being eliminated.

The prior art comparison is instructive here. TurboSparse, ProSparse, and Q-Sparse each addressed a narrower, more tractable version of the same problem — memory-bound single-token inference — and still did not achieve widespread production adoption. TwELL’s extension to batched GEMM is a meaningful technical step, but the gap between a published speedup on a research benchmark and a production-grade kernel that ships in inference frameworks is not trivial to close.

Analysis based on reporting by MarkTechPost. Original article here. Additional sources consulted: Github Repository — github.com/Benjamin-Dobell/nvidia-update; Independent Source — di.gg/ai/psdt91ak; Github Repository — github.com/nvidia.