Building massive recommendation systems requires abandoning standard large language model infrastructure in favor of highly customized hardware-software co-design. To power ad recommendations on Facebook and Instagram, Meta Engineering revamped the training infrastructure behind GEM, its hybrid foundation model. Standard off-the-shelf frameworks fail to optimize this design, which uniquely merges billions of dense parameters alongside trillions of sparse embedding parameters.
Meta’s engineering blog notes that GEM achieved a 2x increase in end-to-end training efficiency, reaching a Model FLOPs Utilization (MFU) of 20–25%. This optimization occurred while scaling total training FLOPs fourfold over a 12-month period. The achievement highlights a growing divide between generic AI infrastructure and the bespoke systems required for multi-modal enterprise workloads.
Eliminating Compute Waste in Asymmetric Workloads
Standard training setups struggle with the uneven, variable-length sequence inputs typical of user engagement data. Meta Engineering mitigated this issue by developing Jagged Flash Attention (JFA), which successfully cut compute waste from padded inputs by up to 50%. This custom kernel bypasses traditional 2D masking, utilizing a “subtraction scheme” for jagged masking that removes approximately 28% of non-tensor-core instructions.
The performance gains scale up with newer iterations of the software. JFA v4, built using Triton Low-Level Extensions (TLX), achieves a 40-140% TFLOPS improvement over JFA v2. This upgrade contributes to an 18.5% relative local MFU gain and a 12% query-per-second (QPS) gain. The system handles a production jagged distribution sparsity of 0.5, demonstrating how tailoring kernels to actual data distributions prevents idle GPU cycles.
To refine JFA during its backward pass, developers adopted a split dQ computation within a non-seq-parallel scheme. Removing both redundant recomputation and atomic writes through this modification yields a 21-40% speedup. By integrating persistent kernel scheduling, explicit warp specialization, and Tensor Memory Accelerator (TMA) through TLX, Meta registered a TFLOPS gain ranging from 30-100%.
Custom Kernels and 5D Parallelism for Trillion-Parameter Scale
To manage massive user histories, Meta engineered custom kernels like BlockAttention, reducing attention complexity from O(L²) to O(L). Engineers also implemented Sliding Window Attention (SWA), keeping normalized entropy (NE) neutral while cutting long-sequence self-attention latency by as much as 68%. Self-attention layer MFU gains 30.6% compared to Triton block attention and 44% versus the SWA baseline after integrating TLX block attention with fused rotary embedding.
For asymmetric feature interactions, Meta designed Generalized Dot-Product Attention (GDPA). The GDPA kernel redesigns include a 6th-order Taylor expansion for ALU-only activation approximation, eliminating Special Function Unit (SFU) contention. Forward execution speeds up by 2x—reaching 1,145 BF16 TFLOPs with roughly 97% Tensor Core utilization—while backward execution shows a 1.6x speedup over the baseline using this optimized kernel. Under short key-value (K/V) production settings, GDPA achieves up to a 3.5x forward speedup over Flash Attention 4 (FA4).
These mathematical optimizations are paired with extreme hardware-level tuning. Meta resolved TMEM scale factor placement constraints in FA4 by overlapping scale factors with temporary registers or unused TMEM slots. Workload distribution shifted from a 6x imbalance to a near-balanced state after engineers introduced software-level tile scheduling within GDPA. Furthermore, warp savings from GDPA softmax correction stage elimination saved 4 warps (and their registers), while the inner-loop iteration count triggering GDPA outer-loop software pipelining was optimized at 1–2 iterations.
To scale these models across distributed clusters, Meta implemented a topology-aware 5D parallelism scheme. This framework combines 2D Fully Sharded Data Parallel (FSDP) and Expert Parallelism for dense parameters with Fully Sharded 2D Model Parallelism for sparse parameters. To measure success, Meta decomposes overall training performance into a formal framework where End-to-End MFU equals Local MFU (single-GPU compute efficiency) multiplied by Scaling Ratio (multi-GPU distributed efficiency), defining scaling efficiency as the ratio between local MFU and end-to-end MFU. Preventing additional BF16 global-memory traffic was achieved by fusing quantization straight into upstream projection and normalization kernels, establishing an MXFP8 ultra-low-precision training process across MLPs and attention modules.
📊 Key Numbers
- Model FLOPs Utilization (MFU): 20–25% end-to-end training efficiency achieved for GEM
- Training FLOPs Scaling: 4x increase in total training FLOPs over 12 months
- Compute Waste Reduction: Up to 50% of padding-related compute waste eliminated via Jagged Flash Attention
- JFA v4 Performance Gain: 40% to 140% TFLOPS improvement over JFA v2
- GDPA Forward Speedup: 2x faster than baseline, reaching 1,145 BF16 TFLOPs (~97% Tensor Core utilization)
- GDPA vs Flash Attention 4: Up to 3.5x forward speedup under short K/V production settings
- Sliding Window Attention Latency Reduction: Up to 68% reduction in long-sequence self-attention latency
- JFA Instruction Reduction: 28% reduction in non-tensor-core instructions via subtraction scheme
- Production Jagged Distribution Sparsity: 0.5
- GDPA Warp Savings: 4 warps and their registers saved by eliminating the softmax correction stage
🔍 Context
Meta Engineering conducted these infrastructure tests and optimizations to scale its proprietary recommendation models. The announcement addresses the severe inefficiency of using standard LLM training pipelines on highly asymmetric, sparse recommendation data, which previously resulted in massive compute waste from padding. This development accelerates a trend toward deep hardware-software co-design, challenging the assumption that general-purpose AI software stacks are sufficient for enterprise-scale workloads. Because this is first-party documentation detailing Meta’s proprietary GEM model, the alternative is not a commercial rival but rather relying on self-managed, unoptimized distributed training scripts and standard open-source attention kernels. The timing of these disclosures is tied directly to Meta successfully scaling its training FLOPs 4x over a 12-month period while doubling its end-to-end training efficiency.
💡 AIUniverse Analysis
Our reading: Meta’s engineering feat proves that maximizing hardware efficiency at the trillion-parameter scale requires breaking open the black box of standard GPU programming. By writing custom Triton kernels and implementing a highly complex 5D parallelism stack, Meta has successfully bypassed the memory and instruction bottlenecks that plague standard Transformer implementations on asymmetric data. The real advance here is the surgical elimination of padding waste through Jagged Flash Attention and the optimization of asymmetric feature shapes via GDPA, showing that mathematical and compiler-level co-design can extract massive performance gains without waiting for next-generation silicon.
However, this approach exposes a deeply brittle and fragmented reality for AI infrastructure. Standard open-source distributed frameworks and off-the-shelf GPU software stacks are clearly failing when applied to real-world, multi-modal enterprise data. The trade-off for Meta’s efficiency gains is extreme engineering complexity and maintenance overhead; writing bespoke low-level kernels tightly binds model code to specific GPU hardware generations. This creates a performance floor that only hyperscalers with massive, dedicated infrastructure teams can afford to reach, while leaving smaller enterprises locked out of similar efficiencies. Furthermore, these MFU metrics are vendor-reported and highly dependent on Meta’s specific sequence lengths and production K/V settings, meaning they may not generalize to standard LLM benchmarks.
In 12 months, the success of this strategy will be measured by whether these highly specialized kernels can easily adapt to evolving model architectures, or if they will become legacy technical debt as soon as the next generation of GPU hardware arrives.
⚖️ AIUniverse Verdict
👀 Watch this space. While the 20–25% Model FLOPs Utilization and 4x scaling are highly impressive, the extreme engineering complexity of maintaining custom Triton kernels and 5D parallelism makes this architecture viable only for hyperscalers with dedicated infrastructure teams.
🎯 What This Means For You
Founders & Startups: AI startups building specialized recommendation or ad-tech systems must recognize that standard LLM optimization playbooks will fail on asymmetric, sparse workloads.
Developers: Machine learning engineers working on non-standard Transformer workloads must adopt specialized memory and attention kernels like JFA to avoid wasting up to 50% of compute on zero-padding.
Enterprise & Mid-Market: High-volume enterprise recommendation platforms can dramatically cut data center training costs and carbon footprint by decoupling local compute optimizations from distributed network sharding strategies.
General Users: Facebook and Instagram users will experience faster, more accurately targeted ad recommendations as underlying foundation models update on fresher engagement data at higher scale.
⚡ TL;DR
- What happened: Meta doubled the training efficiency of its GEM ads foundation model to 20-25% MFU while scaling training FLOPs 4x over a year.
- Why it matters: Standard LLM infrastructure fails on asymmetric recommendation data, forcing hyperscalers to build highly complex, bespoke hardware-software co-designs.
- What to do: Machine learning teams working on non-standard workloads should explore custom jagged attention kernels to eliminate up to 50% of compute waste.
📖 Key Terms
- Jagged Flash Attention (JFA)
- A custom attention kernel designed by Meta to eliminate compute waste by processing variable-length sequences without traditional padding.
- Model FLOPs Utilization (MFU)
- A metric that measures the percentage of a hardware device’s theoretical maximum floating-point operations achieved during actual model training.
- Generalized Dot-Product Attention (GDPA)
- A specialized attention kernel engineered to optimize asymmetric feature interaction shapes in recommendation models.
- MXFP8
- An ultra-low-precision 8-bit floating-point format used to optimize training efficiency and reduce memory traffic for recommendation data.
- 5D Parallelism
- A highly advanced distributed training strategy that combines multiple sharding and parallelization techniques across dense and sparse model parameters.
Analysis based on reporting by Meta Engineering. Original article here.