Snap Enhances A/B Testing with NVIDIA GPUs on Google Cloud
Snapchat is accelerating its A/B testing capabilities by integrating open data processing libraries from NVIDIA onto Google Cloud services. This strategic move allows the company to efficiently scale experimentation for its 940 million monthly active users. Each new feature undergoes rigorous testing, analyzing approximately 6,000 metrics related to engagement, performance, and monetization. Snapchat runs thousands of these experiments monthly, processing over 10 petabytes of data within a three-hour window each morning using the Apache Spark distributed framework.
By adopting Apache Spark accelerated by NVIDIA cuDF, Snap is boosting its data processing workloads on NVIDIA GPUs. This enhancement achieves 4x speedups in runtime without increasing the number of machines, offering a cost-effective scaling solution. The company has already migrated its two largest pipelines and plans to expand the use of the Spark accelerator to other production workloads. This optimization is critical as Snapchat’s features, from AI-generated stickers to performance updates, evolve rapidly.
GPU Acceleration Drives Significant Cost Savings and Performance Gains
Snap’s team has optimized its data processing pipelines on Google Cloud‘s G2 virtual machines, powered by NVIDIA L4 GPUs. This migration has yielded substantial cost reductions. Based on internal data collected between January 1 and February 28, the company has realized a 76% daily cost saving by utilizing NVIDIA GPUs on Google Kubernetes Engine compared to CPU-only workflows. This efficiency gain is attributed to the adoption of NVIDIA cuDF, a GPU DataFrame library that scales for the Apache Spark framework.
The adoption of NVIDIA CUDA-X libraries, combined with Google’s infrastructure management services like Google Kubernetes Engine, provides Snap with a full-stack platform for large-scale data processing. The team utilized the cuDF suite of microservices to automate the qualification, testing, configuration, and optimization of Spark workloads for GPU acceleration. This technological shift was essential to support an ambitious roadmap for scaling experimentation, which would have significantly increased computing costs with their previous infrastructure.
Accelerated Data Processing with cuDF and Apache Spark
The integration of NVIDIA cuDF with Apache Spark is a key component of Snap’s enhanced data infrastructure. This open library for accelerated data processing allows for greater efficiency in handling the vast amounts of data generated by A/B testing. The results of initial experiments showed cost savings that exceeded expectations, with the Spark accelerator proving to be an ideal match for Snap’s specific workloads. This innovation allows for more extensive experimentation, leading to more innovative experiences for Snapchat users.
Snap’s commitment to experimentation is central to its product development cycle. The company has been actively migrating its data processing to GPU-accelerated pipelines. The success of the initial migration, covering two major pipelines, suggests significant future opportunities. This move is part of Snap’s ongoing effort to deliver cutting-edge features and behind-the-scenes improvements to its global user base. The company’s focus on leveraging advanced technologies from partners like NVIDIA and Google underscores its dedication to innovation and user experience.
✨ Intelligent Curation Note
This article was processed by AI Universe’s Intelligent Curation system. We’ve decoded complex technical jargon and distilled dense data into this high-impact briefing.
Estimated time saved: ~1 minutes of reading.
Tools We Use for Working with AI:
