Standard artificial intelligence benchmarks have long allowed language models to cheat by retrieving memorized text rather than executing code. According to Together AI’s blog post, prompt-only shortcut filtering in their framework demonstrates that models often solve benchmark queries without accessing underlying data files. To address this flaw, Together AI introduced DSGym, a unified evaluation and training framework operating inside real-time sandboxed execution containers.
When researchers subjected models to containerized execution, the actual source of agent failure became clear. According to Together AI’s report, manual trajectory analysis of failed runs revealed that 85% to 96% of model failures on domain-specific bioinformatics tasks (DSBio) stem from domain-grounding errors rather than general planning or statistical reasoning deficits. In contrast, deficits in statistical knowledge alongside flawed planning serve as the main impediments during general analysis workloads like QRData and DAEval.
According to the study’s conclusions, raw parameter count matters far less for data science capability than “skeptical” persistence involving continuous validation and iteration. Rather than actively resolving errors in code execution, AI agents frequently fall back on basic baselines or “safe” heuristics whenever faced with intricate datasets or technical hurdles.
Containerized Execution and Benchmark Realities
Evaluation within DSGym relies on two original benchmark suites: DSBio featuring 90 bioinformatics tasks from academic literature, and DSPredict featuring 92 Kaggle competitions split into Easy (38) and Hard (54) tasks. DSGym utilizes a CodeAct-like scaffold for DSPredict, imposing a 10-hour total time limit and a 2-hour limit per code execution.
For general evaluation benchmarks, model accuracy varies widely across tasks. On the DABStep-hard benchmark, Claude Sonnet 4.5 scored highest among tested models at 37.04%, while GPT-4o achieved 7.41% and Qwen3-4B-Instruct achieved 2.90%. Other models on DABStep-hard recorded: Claude Sonnet 4 at 31.75%, Kimi K2 Instruct at 28.84%, GPT-5.1 (medium) at 28.31%, Deepseek-v3.1 at 21.96%, Qwen3 235B Instruct at 17.46%, Qwen3-Coder 480B at 14.29%, GPT-5.1 (high) at 13.23%, GPT-5.1 (none) at 11.90%, GPT-OSS-120B at 7.94%, and Qwen2.5-7B-Instruct at 2.38%.
On DABStep-easy, models achieved higher accuracy: Claude Sonnet 4.5 reached 83.33%, Claude Sonnet 4 hit 81.94%, Kimi K2 Instruct scored 77.78%, Deepseek-v3.1 recorded 76.39%, Qwen3-Coder 480B reached 75.00%, GPT-5.1 (medium) hit 75.00%, GPT-5.1 (high), GPT-4o, and Qwen3 235B Instruct each scored 73.61%, GPT-5.1 (none) and GPT-OSS-120B logged 70.83%, Qwen3-4B-Instruct reached 58.33%, and Qwen2.5-7B-Instruct scored 47.22%.
For verified datasets, DAEval-Verified testing resulted in: Kimi K2 Instruct at 92.82%, GPT-4o at 92.26%, Claude Sonnet 4.5 at 91.71%, Claude Sonnet 4 at 90.91%, Qwen3-Coder 480B at 90.61%, GPT-5.1 (high) and GPT-5.1 (medium) both at 89.50%, GPT-5.1 (none) at 87.85%, Qwen3 235B Instruct at 85.08%, GPT-OSS-120B at 84.53%, Deepseek-v3.1 at 82.32%, Qwen3-4B-Instruct at 64.47%, and Qwen2.5-7B-Instruct at 50.56%. On QRData-Verified, scores were led by Kimi K2 Instruct at 63.68%, GPT-5.1 (medium) at 61.75%, Claude Sonnet 4.5 at 61.35%, GPT-4o at 60.24%, GPT-5.1 (high) at 60.16%, Claude Sonnet 4 at 59.06%, GPT-5.1 (none) at 58.96%, Deepseek-v3.1 at 57.37%, Qwen3-Coder 480B at 54.72%, Qwen3 235B Instruct at 54.18%, GPT-OSS-120B at 47.95%, Qwen3-4B-Instruct at 45.27%, and Qwen2.5-7B-Instruct at 35.04%.
| Model | DSPredict-Hard Medal Rate | Best For |
|---|---|---|
| GPT-5.1 (high) | 4.8% medal rate (14.3 percentile) | High-complexity competitive prediction |
| Claude Sonnet 4.5 | 0% medal rate (4.8 percentile) | Multi-step reasoning and DABStep-hard tasks |
| Qwen3 235B Instruct | 2.4% medal rate (2.4 percentile) | Open-weights agent execution |
Performance Limits on Competitive Data Science
Evaluating models on DSPredict tasks highlights the difficulty of autonomous end-to-end data processing. On DSPredict-Easy (Private), GPT-5.1 (high) achieved a 100% valid submission rate, a 60.4% percentile, and a 75 median. Under lower reasoning configs, GPT-5.1 (medium) achieved a 91.7% valid submission rate, a 55.7% percentile, and a 63.9 median, while GPT-5.1 (none) achieved a 97.2% valid submission rate, a 45.7% percentile, and a 41.7 median. Claude Sonnet 4.5 achieved a 100% valid submission rate, a 49% percentile, and a 52.8 median. Matching Kimi K2 Instruct (which reached a 97.2% rate of valid submissions, 43.9% percentile, and 41.7 median score), Qwen3 235B Instruct posted an identical 97.2% rate of valid submissions, a 42.9% percentile, and a 33.3 median score.
Performance declines sharply on DSPredict-Hard (Private). According to Together AI’s blog post, GPT-5.1 (high) achieved an 85.7% valid submission rate, a 4.8% medal rate, and a 14.3 percentile median leaderboard placement. Looking at variants with less reasoning, an 81.0% rate of valid submissions, 4.8% medal rate, and 7.1 median score were logged by GPT-5.1 (medium), compared to a 69.0% rate of valid submissions, 2.4% medal rate, and 10.3 median score for GPT-5.1 (none). Meanwhile, performance figures for Claude Sonnet 4.5 stood at a 71.4% rate of valid submissions, a 0% medal rate, along with a 4.8 median score. In comparison, Qwen3 235B Instruct registered a 2.4 median score, a 2.4% medal rate, and a 64.3% valid submission rate; meanwhile, Kimi K2 Instruct yielded a median score of 0, a 0% medal rate, and a 69% valid submission rate.
To perform manual error categorization, researchers pulled a uniform distribution of 50 failed trajectories for every pairing of task family and model. During trajectory generation, 3,700 synthetic queries were generated before quality filtering. Together AI claims that finetuning a 4B parameter model (Qwen3-4B-DSGym-SFT-2k) on 2,000 synthetic query-trajectory pairs generated by DSGym enabled competitive performance against much larger open and proprietary models.
📊 Key Numbers
- DSBio failure attribution: 85% to 96% of model failures on bioinformatics tasks stem from domain-grounding errors
- DABStep-hard highest score: 37.04% (Claude Sonnet 4.5)
- DABStep-hard baseline scores: 7.41% (GPT-4o) vs 2.90% (Qwen3-4B-Instruct)
- DSPredict code execution limits: 2 hours per execution (10 hours total time limit per competition)
- DSPredict-Hard top performance (GPT-5.1 high): 85.7% valid submission rate, 4.8% medal rate, 14.3 percentile median
- DSPredict-Easy top performance (GPT-5.1 high): 100% valid submission rate, 60.4% percentile, 75 median
- Synthetic generation volume: 3,700 synthetic queries generated before quality filtering down to 2,000 trajectory pairs
- Trajectory sample size: 50 failed trajectories sampled per model and task family for manual error analysis
🔍 Context
Together AI conducted benchmark tests on its DSGym evaluation framework using real-time sandboxed execution environments. Prior evaluations relied heavily on text-only question-answering formats, which allowed models to exploit memorized training data instead of actually running code against datasets. This framework shifts evaluation toward active code execution and environment feedback. Rather than comparing against proprietary commercial cloud suites, this approach contrasts against legacy uncontainerized evaluation scripts and static Q&A benchmark sets. Timeliness is driven by the formal release of DSGym’s containerized benchmark suites and its open-source trajectory generation pipeline.
💡 AIUniverse Analysis
Our reading: The primary advance in DSGym is its sandboxed container framework, which exposes data memorization shortcuts. By forcing models to execute Python code in isolated environments, the framework proves that high standard benchmark scores often disappear when agents face actual data files.
However, the framework relies heavily on synthetic query-trajectory pairs to train smaller models like Qwen3-4B-DSGym-SFT-2k. Training agents on synthetic distributions generated by the framework itself introduces potential evaluation bias and risks overfitting agents to standardized execution patterns rather than messy, real-world data science pipelines. Furthermore, requiring up to 10 hours of execution per task introduces substantial compute overhead that creates barriers for resource-constrained research teams.
For this approach to remain relevant in 12 months, sandboxed evaluation standards must demonstrate that agents trained on synthetic trajectories actually generalize to unscripted enterprise data problems.
⚖️ AIUniverse Verdict
👀 Watch this space. While containerized execution successfully eliminates text-based shortcut memorization, the high computational overhead and reliance on synthetic training trajectories leave long-term generalization unproven.
🎯 What This Means For You
Founders & Startups: Founders can leverage synthetic trajectory generation pipelines to finetune domain-specific 4B parameter agents that compete with frontier proprietary models at drastically lower inference overhead.
Developers: Developers must shift from testing data science agents on static Q&A sets to evaluating code execution within sandboxed runtime containers.
Enterprise & Mid-Market: Enterprises should re-evaluate their agent benchmarks to filter out dataset memorization flaws before trusting LLMs with autonomous data analysis.
General Users: Data scientists and researchers will gain access to more reliable automated analytics co-pilots, though domain-specific errors in fields like bioinformatics remain a high barrier.
⚡ TL;DR
- What happened: Together AI introduced DSGym, a containerized evaluation framework showing that 85% to 96% of domain-specific model failures stem from domain-grounding errors.
- Why it matters: Existing static benchmarks reward data memorization rather than actual code execution capabilities.
- What to do: Transition agent evaluation from static text prompts to sandboxed execution runtime environments with strict execution bounds.
📖 Key Terms
- DSGym
- A unified evaluation and training framework developed by Together AI that runs data science agents inside real-time sandboxed execution containers.
- DSBio
- Gathered from academic literature, this collection of 90 domain-specific bioinformatics tasks serves as a benchmark suite within DSGym.
- DSPredict
- A benchmark suite inside DSGym consisting of 92 Kaggle competitions split into Easy and Hard tasks to test predictive modeling agents.
- DAEval-Verified
- A verified general data analysis evaluation benchmark used to test statistical reasoning and planning in AI models.
- DABStep
- A benchmark split into easy and hard multi-step data science reasoning tasks designed to test agent persistence and code execution.
Editorial note: This article summarizes Together AI’s own product material, not independent reporting. Time-to-value, speed, and ROI statements reflect the publisher unless outside evidence is cited. Original post.
Analysis based on reporting by Together AI. Original article here.