392 Tokens Per Second: How Databricks and the B300 Just Reset the AI Speed Limit

When Speed Becomes the Smartest Thing About Your Model

For a long time, picking an AI model felt like an old-school engineering trade-off. You could have intelligence, or you could have speed, but pretending you could have both at once usually meant something was about to break in production.

That trade-off is quietly dying. And in the enterprise, “smart but slow” is starting to feel like a luxury nobody can actually afford. A model that reasons brilliantly in 14 seconds is fine for a research demo. It’s useless behind a developer’s cursor, or inside an agent that has to think five times before it does anything at all.

The bottleneck has moved. It used to be the model. Now it’s the stack underneath it.

This is the backdrop against which Databricks just did something genuinely interesting.

392 tokens per second, and why that number matters

Databricks recently took the #1 spot on the Artificial Analysis leaderboard for GLM-5.2, clocking in at 392 tokens per second. For context, Fireworks AI sits at 328 t/s, and Databricks itself was hitting around 201 t/s on NVIDIA H200 nodes only a short while ago.

So what changed? Two things, mostly.

The first is hardware. They moved from H200 to NVIDIA’s Blackwell-based B300, and the jump is exactly the kind of “substantial additional boost” you’d hope for on paper but rarely see this cleanly in practice.

The second, and probably more impressive piece, is the software work. The team rebuilt a lot of the inference stack around the model’s Mixture-of-Experts (MoE) architecture — DeepGEMM’s FP8 grouped GEMM kernels, DeepEP for expert parallelism, custom kernels, speculative decoding tuned specifically for the way GLM-5.2 generates. None of these individually sound revolutionary. Stacked together, they’re the reason the number nearly doubled.

Matei Zaharia summed it up in his usual understated way:

“The Databricks inference team is 🔥”

The reason this matters more than a leaderboard slot is what it unlocks. Run GLM-5.2 in “High” reasoning mode — lighter chain-of-thought, faster outputs — and at ~400 t/s you finally have enough headroom for the multi-step agentic workflows everyone has been demoing but few have shipped. The agent gets to think, plan, retry, and act inside the same response budget that used to barely cover a single completion.

The cost story is, frankly, the more disruptive one

Speed is the headline. Cost is what actually changes purchasing decisions.

Databricks is serving GLM-5.2 at roughly 85% below what you’d pay for proprietary frontier models for comparable work, especially in coding. The Artificial Analysis numbers tell the story pretty plainly:

If you’ve ever had to explain a six-figure monthly AI bill to a CFO, you already know why this matters. “Near-Opus reasoning at one-eighth the cost” is the kind of line that turns AI from a budget item people fight over into something teams can actually deploy at scale without flinching.

A 1-million token context window (and what it actually costs to run)

Speed handles how fast. Context handles how much.

GLM-5.2 jumped from a 200K-token window in v5.1 to 1M tokens. That’s not an incremental bump; it’s a different way of working. Instead of retrieving snippets and praying the right ones made it into the prompt, you can hand the model the entire codebase and let it figure out the relevant parts itself.

In practice, this nudges agentic coding away from RAG and toward something closer to “just give it everything.” RAG isn’t going away, but a lot of the gymnastics around chunking, re-ranking, and hybrid search start to look optional rather than mandatory.

The catch nobody talks about is the KV cache. At 1M tokens, an FP16 KV cache balloons to around 160 GB — which is more than a single GPU node can comfortably hold. Databricks gets around this by running the KV cache in FP8, which cuts the footprint roughly in half to ~80 GB. That fits cleanly on 8x H200 or 8x B200 nodes, with enough room left over for the model weights themselves. Less elegant than it sounds in a blog post, but it’s the kind of detail that decides whether the feature actually works in production or just lives on the spec sheet.

“744 billion parameters” is not the number you should be looking at

There’s a common reflex when people hear “MoE model with 744B parameters” — they assume it must be slow. It isn’t, and the reason is worth understanding.

GLM-5.2 has 744B total parameters but only about 40B active per forward pass. The router picks which experts to consult on a per-token basis. You get the broad knowledge of a near-trillion-parameter model with the latency profile of a much smaller one.

DeepEP handles the expert routing across GPUs efficiently enough that the parallelism overhead doesn’t eat the gains. The net effect: deep knowledge, fast outputs, and a serving cost that doesn’t scale with the total parameter count.

The quieter strategic move: Mosaic AI Gateway

Benchmarks get the attention. The Gateway is the part I’d watch.

The Mosaic AI Gateway is an abstraction layer that lets you swap underlying models without touching application code. Today it’s GLM-5.2; tomorrow it’s whatever beats it. The application doesn’t care.

That’s a pretty pointed move at Snowflake, and it positions Databricks for the enterprise platform market rather than just the data-engineering crowd it grew up serving. It also gives buyers three things they actually need:

Governance — every call tracked and audited through Unity Catalog.
Rate limiting — real per-team controls on spend and token usage, instead of the usual “we’ll figure it out in next quarter’s invoice” approach.
PII filtering — automated scrubbing of sensitive data before anything leaves the secure perimeter.

The strategy here is to commoditize the model layer. If models become interchangeable, the platform underneath them — the governance, the security, the optimization — is what you’re really buying. That’s a comfortable position for Databricks to be in.

So where does this leave us?

The AI race is shifting under our feet. The “biggest brain” framing is starting to feel dated. The interesting competition now is about intelligence at speed and scale, served cheaply and governed well.

GLM-5.2 on Databricks isn’t perfect, and the leaderboard will move again — probably before this post is a month old. But the combination of 392 t/s, a 1M-token window, and an 85% cost advantage over the incumbents is hard to argue with. It’s the kind of stack that lets a team actually ship the agent they’ve been prototyping for the last year.

Which leaves the real question: does the future of AI belong to the labs building the biggest brains, or to the platforms building the fastest nerves?

In production, I think we already know the answer.