Mamba-3 State Space Model Introduced
Carnegie Mellon University, Princeton University, Together AI, and Cartesia AI have jointly introduced Mamba-3, a new State Space Model (SSM) designed with an ‘inference-first’ approach. This development addresses the growing importance of inference-time compute in Large Language Model performance, aiming to overcome the computational bottlenecks associated with traditional Transformer architectures. Mamba-3 replaces exponential-Euler discretization with exponential-trapezoidal discretization and incorporates complex-valued SSMs, a significant departure from earlier real-valued linear models that struggled with ‘state-tracking’ tasks.
A key innovation in Mamba-3 is its transition from a single-input, single-output (SISO) to a multiple-input, multiple-output (MIMO) structure. This is achieved by increasing the rank R of the input and output projections, transforming the state update from an outer product to a matrix-matrix multiplication. This shift can increase decoding FLOPs by up to 4x relative to Mamba-2 at a fixed state size. Despite this computational increase, MIMO architecture improves modeling quality and perplexity while maintaining comparable wall-clock decode latency due to the computation being overlaid with existing memory I/O for state updates.
Architectural Enhancements in Mamba-3
The Mamba-3 block adopts a layout similar to Llama, alternating with SwiGLU blocks. The model integrates RMS normalization to the B and C projections, alongside the addition of Head-Specific Biases to these components, which are designed to induce convolution-like behavior. For hybrid architectures, a pre-gate, grouped RMSNorm is utilized. Furthermore, BC/QK Normalization has been implemented to stabilize training, allowing for the removal of the post-gate RMSNorm used in previous versions. Evaluations of Mamba-3 were conducted on the FineWeb-Edu dataset across four model scales, ranging from 180M to 1.5B parameters.
The theoretical equivalence between discretized complex SSMs and real-valued SSMs utilizing data-dependent Rotary Positional Embeddings (RoPE) on the B and C projections has been established by the research team. Optimized Triton for prefill and CuTe DSL for decode kernels ensure that the added mathematical components in Mamba-3 remain lightweight.
Performance and Efficiency Gains
The SISO variant of Mamba-3 demonstrates superior performance compared to both Mamba-2 and Gated DeltaNet (GDN). Specifically, the MIMO variant of Mamba-3 achieves an average downstream accuracy improvement of 1.2 points over its SISO counterpart. This new model offers comparable pretraining perplexity to Mamba-2 while utilizing only half the state size. For instance, a Mamba-3 model with a state size of 64 matches the performance of a Mamba-2 model with a state size of 128.
SISO Mamba-3 kernels exhibit lower latency than the released Mamba-2 and GDN kernels under standard BF16 settings. This efficiency is attributed to the architectural adjustments that bridge the gap between theoretical sub-quadratic efficiency and practical modeling capability.
✨ Intelligent Curation Note
This article was processed by AI Universe’s Intelligent Curation system. We’ve decoded complex technical jargon and distilled dense data into this high-impact briefing.
Estimated time saved: ~2 minutes of reading.
Tools We Use for Working with AI:
