Nvidia’s Vera Rubin Platform Turns AI Infrastructure Into a Tokens-Per-Watt Contest

Last week, Nvidia invited a small group of journalists into one of its private data center labs, a working facility tucked into a residential pocket of Sunnyvale, California, to make a point that sounds obvious until you sit with it: AI infrastructure is physical. The company used the visit, reported by The Register, to detail its next-generation Vera Rubin platform. The message underneath the hardware tour was less about raw compute and more about economics: if an AI factory’s product is tokens, Nvidia argued, the metric that decides who wins isn’t how many chips you can buy. It’s how many tokens you can generate inside a fixed power budget.

The briefing doubles as Nvidia’s answer to a question every hyperscaler and AI lab is now asking. With power increasingly the hard limit on data center growth, how do you squeeze more revenue out of the same number of megawatts?

A Private Lab Tour Built to Make Infrastructure Feel Real

“Infrastructure is physical, it’s real,” said Ian Buck, general manager of Nvidia’s hyperscale and HPC computing business, according to The Register’s account of the briefing. “You can touch it, you can see it, and it’s what helps bring AI to life.”

The visit included a walk through a working mini data center, one of four such facilities Nvidia operates, though attendees were asked not to publish its exact location. There, Andrew Bell, senior vice president of hardware engineering, demonstrated how much faster the company’s newest rack hardware goes together compared with the prior generation. Using automated assembly, a Vera Rubin NVL72 compute tray can be built in about one minute, versus roughly 90 minutes for a comparable GB200 (Blackwell-generation) compute tray, a 90x reduction in assembly time, Bell said.

That’s a manufacturing detail, not a performance one, but it fits the larger point Nvidia was making: Vera Rubin isn’t being sold as just faster silicon. It’s being sold as a platform designed to be deployed and serviced with far less manual labor, which is its own kind of savings at data center scale.

What Is Actually Inside a Vera Rubin Rack

Vera Rubin was first announced at Computex 2024 and detailed further at CES this past January, but the Sunnyvale briefing was Nvidia’s most concrete public look yet at how the pieces fit together.

Six Purpose-Built Components

According to The Register, the platform is built from six purpose-designed chips: the Vera CPU, the Rubin GPU, an NVLink 6 switch, a ConnectX-9 network interface, a BlueField-4 data processing unit (DPU), and a Spectrum-6 Ethernet switch. Those parts ship inside several rack-mountable trays, including a Vera Rubin NVL72 compute tray, a matching NVLink switch tray, a Groq 3 LPX inference accelerator tray, a dedicated Vera CPU tray, a BlueField-4 storage tray, and a Spectrum-6 switch tray. Nvidia describes the whole set as co-designed, engineered together as one system rather than assembled from off-the-shelf parts.

The Official Rack Numbers

Nvidia’s own DGX Vera Rubin NVL72 product page fills in the specifics for a full rack.

Compute and Memory

• 72 Rubin GPUs and 36 Vera CPUs per rack
• 20.7 TB of total GPU memory
• Up to 1,580 TB/s of aggregate GPU memory bandwidth
• 3,600 petaflops of NVFP4 inference performance and 2,520 petaflops of dense NVFP4 training performance

Networking

• 144 single-port ConnectX-9 SuperNICs, each running at 800 Gb/s over InfiniBand or Ethernet
• 18 dual-port BlueField-4 DPUs, each at 400 Gb/s
• Nine NVLink switches, part of an NVLink 6 fabric Nvidia says moves data rack-wide at up to 260 TB/s

Why the Vera CPU Gets Equal Billing With the GPU

It would be easy to read a story about Nvidia’s newest AI hardware and assume the CPU is an afterthought next to the Rubin GPU. Nvidia’s pitch is the opposite. The Vera CPU uses a custom Arm-based core Nvidia calls Olympus, and the company argues that agentic AI workloads, meaning AI agents that plan, call tools, and reason across many steps rather than answering one prompt, put unusual pressure on the CPU: scheduling, data movement, storage, and networking all have to keep pace with a GPU fleet generating tokens continuously instead of in short bursts.

Nvidia told The Register that the Vera CPU’s Olympus core accelerates agentic workloads by roughly 2x, offers 3x the core-to-core bandwidth, and cuts latency by about 40 percent thanks to LPDDR5X memory. The bet is that making each core faster helps agentic workloads more than simply adding more cores would.

The Real Pitch: Tokens Per Watt, Not Tokens Per Second

The headline performance claim, repeated by nearly every outlet that covered the briefing, is that the Vera Rubin NVL72 generates roughly 10 times more tokens per megawatt than the Grace Blackwell (GB200) NVL72, based on an early benchmark CoreWeave ran on the DeepSeek-R1 model. Nvidia’s own blog post repeats that 10x figure and adds more: up to 10x higher token throughput per megawatt and one-tenth the cost per million tokens for early partners including Microsoft and Mistral, plus a 10x reduction in inference cost per token relative to a Google Cloud A5X comparison.

Why frame everything around watts instead of, say, FLOPS or GPU count? Buck laid out the logic directly: “Every AI factory is power-constrained,” he said, according to The Register. “And frankly the most important metric is your delivered performance in a fixed watt data center, your perf-per-watt. All that comes together in your AI factory revenue. You may rent infrastructure … for dollars per hour, but you generate revenue with profitable tokens now, in the hundreds of dollars per hour. Your AI Factory revenue is a function of how many tokens you can generate in a fixed watt data center.”

That framing only holds up if tokens keep being worth something. Nvidia’s bet, one shared by the frontier AI labs buying this hardware, is that even as more efficient chips and more competing providers push the supply of tokens up (which should push prices down), demand for tokens grows faster still, driven by agentic workloads that consume far more tokens per task than a single chat reply ever did.

Built for Agents, Not Just Chatbots

Nvidia frames the entire Vera Rubin pitch around agentic AI rather than simple one-shot inference. The company argues that AI agents need sustained inference with low latency across many reasoning steps, high decoding throughput, efficient handling of long context windows, large key-value cache capacity, and the ability to scale a single model across many linked GPUs. A benchmark built around answering one prompt doesn’t capture that workload shape, which is part of why Nvidia keeps returning to token economics instead of a plain throughput number.

Read the Benchmarks Carefully

It’s worth being direct about where these numbers come from before treating them as settled fact. The headline 10x tokens-per-megawatt figure is a benchmark that CoreWeave, an Nvidia cloud partner and customer, ran on Nvidia’s own hardware, then publicized through Nvidia’s blog and a press briefing Nvidia itself organized. That doesn’t make the number false. It does mean it’s a vendor and customer telling a story about hardware they have a direct commercial interest in promoting. Independent, reproducible benchmarking efforts such as MLPerf have not yet published third-party Vera Rubin results, so for now, “10x” is best read as Nvidia’s claim about Nvidia’s hardware, not an audited industry figure.

Separately, SiliconANGLE reported that Nvidia’s own testing spanned more than 250,000 configurations and 1.4 million GPU hours, and that software optimization alone lifted the existing GB200 NVL72’s tokens-per-megawatt by more than four times over three months. That’s a useful reminder that a meaningful share of Nvidia’s performance story now comes from firmware and scheduling software, not just new silicon, which also means today’s Vera Rubin numbers are a floor Nvidia will likely keep pushing upward rather than a fixed ceiling.

Why This Matters Beyond Nvidia’s Roadmap

Vera Rubin’s debut lands one day before rival AMD’s Advancing AI conference in San Francisco, and as the wider industry grapples with data centers hitting power ceilings faster than chip supply ceilings. For cloud providers and AI labs planning multi-year capacity, Nvidia’s tokens-per-watt framing is a deliberate attempt to shift the procurement conversation away from “how many GPUs can I buy” and toward “how much revenue can I generate per megawatt of grid capacity I can actually secure,” which in many markets is now the tighter constraint of the two.

That shift won’t be settled by one vendor briefing. It will be settled by what independent buyers measure once Vera Rubin racks actually ship in volume, and by whether the agentic workloads Nvidia is betting on generate enough real revenue to justify the assumption that token demand keeps outrunning token supply.