MLCommons released MLPerf Training v6.1 on November 16, 2026, marking the benchmark suite's most significant expansion in years. The update adds two new workloads: DLRMv4-HSTU, a modernized recommender-system test, and Qwen3.5-397B, the first large-scale post-training benchmark built around Group Relative Policy Optimization. Early results show NVIDIA's next-generation Rubin GPUs completing the Qwen3.5-397B post-training task in 10.44 minutes, compared to 18.02 minutes for Google's Ironwood TPU and 22.78 minutes for a GB200 NVL72 system.
The timing is notable. As the industry's attention shifts from raw pretraining scale toward agentic systems, reinforcement learning, and specialized fine-tuning, MLPerf's benchmark committee is racing to keep pace with how AI labs actually build models today. The inclusion of a GRPO-based post-training test acknowledges that reasoning-heavy, RLHF-style optimization has become as commercially important as the pretraining runs MLPerf has measured since 2018, even as the new results arrive with caveats that complicate simple chip-versus-chip comparisons.
A Benchmark Suite Catches Up to Reality
For most of its existence, MLPerf Training has measured how fast systems can pretrain models from scratch on fixed datasets, a useful but increasingly incomplete picture of how modern AI labs spend their compute. The addition of Qwen3.5-397B as a post-training benchmark changes that calculus. By building the test around Group Relative Policy Optimization, the reinforcement-learning technique that underpins many of today's reasoning-focused model releases, MLCommons is effectively acknowledging that the industry's hardest computational problems have shifted from next-token prediction at scale toward iterative policy refinement.
The second new workload, DLRMv4-HSTU, replaces the long-serving DLRM-dcnv2 recommender benchmark and reflects a parallel shift in how recommendation systems are architected. HSTU-style models, which borrow sequential transformer techniques from language modeling, have become the de facto standard at companies running large-scale personalization systems, and the retirement of the older DLRM variant signals that MLPerf's steering committee is willing to retire benchmarks once they stop reflecting production workloads, not just add new ones indefinitely.
Rubin, Ironwood, and the Limits of a Stopwatch
The headline numbers are striking on their face: NVIDIA's Rubin-generation GPUs finished the Qwen3.5-397B post-training run in 10.44 minutes, roughly 42 percent faster than Google's Ironwood TPU at 18.02 minutes, and well ahead of a GB200 NVL72 system's 22.78 minutes. For a chip war that has increasingly been fought in press releases as much as in silicon, these are the kinds of figures that will circulate widely among investors and cloud buyers trying to decide where to allocate billions in capital expenditure.
But MLCommons and independent analysts are quick to caution against reading too much into the raw times. The submitted systems used different cluster sizes and distributed the workload across hardware in different ways, meaning the comparison captures system-level performance at a particular configuration rather than a controlled chip-to-chip measurement. That distinction matters enormously for enterprise buyers, who are less interested in which chip wins a benchmark in isolation and more interested in which combination of hardware, networking, and software stack delivers the best cost-per-token at the scale they actually need.
Why Post-Training Benchmarks Matter Now
The rise of reasoning-centric models, from OpenAI's GPT-6.2 family released just a day after the MLPerf update to Alibaba's Qwen4 Max, has made post-training compute a bottleneck in its own right. Techniques like GRPO allow labs to sharpen a model's reasoning and agentic capabilities after initial pretraining, often through thousands of iterative rollouts and reward evaluations that strain hardware differently than a straightforward forward-backward pretraining pass. Until now, there was no standardized way to measure how efficiently different hardware and software stacks handled that workload.
That gap has real financial consequences. As AI labs increasingly compete on post-training quality rather than raw parameter count, the infrastructure costs of reinforcement learning-style fine-tuning have become a larger share of total training budgets. A benchmark that quantifies those costs gives cloud providers, chipmakers, and enterprise buyers a shared reference point, much as the original MLPerf suite did for pretraining throughput nearly a decade ago.
A Crowded Week for Model Releases
The MLPerf update landed in the middle of one of the busier model-release weeks of the quarter. OpenAI shipped GPT-6.2 Astra and GPT-6.2 Sol on November 17, with independent leaderboard tracking placing Astra fourth overall with a score of 67.2 and Sol 24th at 62.4, alongside a specialized Astra for Law variant whose benchmark results are still pending. Alibaba's Qwen4 Max arrived a day earlier with an independent score of 57.3, while Microsoft introduced its MAI Code 2 and MAI Code 2 Flash coding models and NVIDIA released Nemotron 3.5 Bolt E4B.
Artificial Analysis also reported that Google's Gemini 4 Carbon had displaced OpenAI's GPT-6.1 Sol atop the MathArena rankings, a reminder that leaderboard positions remain volatile from week to week even as the underlying infrastructure used to train these models becomes the subject of its own standardized measurement. Taken together, the benchmark expansion and the wave of releases suggest an industry whose two central questions, how fast can you train and how well does the result perform, are becoming more tightly linked and more rigorously measured at the same time.
Comparing a single Rubin pod against a full-scale Ironwood deployment on a workload like this isn't apples to apples until you normalize for cluster size and parallelism strategy. The numbers are real, but the context around them matters just as much as the numbers themselves.
What Comes Next
MLCommons has signaled that future MLPerf releases will likely add further agentic and multimodal workloads, following the same logic that drove this week's additions: benchmarks should track where compute is actually being spent, not just where it was spent five years ago. Artificial Analysis's new multimodal-knowledge-agent evaluation, developed jointly with Google DeepMind, points in a similar direction, aiming to assess how well models handle complex visual files in agentic contexts rather than static question-answering tasks.
For now, the practical takeaway is that hardware comparisons in AI are entering a more complicated era. Raw throughput numbers for pretraining are no longer sufficient to describe a chip's real-world value, and the industry's benchmarking infrastructure is scrambling, not always successfully, to keep pace with a set of workloads that looks increasingly different from the one MLPerf was built to measure in 2018.
Sources
- https://www.nature.com/subjects/machine-learning?error=cookies_not_supported&code=fe14c225-a3a5-4a74-a7af-3505d9067043
- https://www.nature.com/subjects/machine-learning/ncomms?error=cookies_not_supported&code=337cf9af-56b1-4d83-ba89-c838320dc89b
- https://www.nature.com/subjects/artificial-intelligence?error=cookies_not_supported&code=ff9af51a-c80b-4449-92a2-d4121ff388a2
- https://link.springer.com/subjects/machine-learning
- https://jmlr.org/
- https://letsdatascience.com/news/topic/ai-research
- https://research.yandex.com/publications
- https://www.nature.com/subjects/machine-learning?error=cookies_not_supported&code=f77f55ca-0726-44c1-9fd0-ce812877f4cc
- https://www.nature.com/subjects/machine-learning/nature?error=cookies_not_supported&code=bc0a8f1f-8785-4898-8ba2-e03771659c11
- https://scipapermill.com/2026/10/17/machine-learnings-new-frontiers-from-quantum-realms-to-automated-science-and-robust-systems/












Leave a Comment