I read three unrelated announcements in the same week. Took me a day to realize they were all breaking the same thing.
TL;DR: Diffusion-based generation (Mercury), dynamic context management (CLM), and Kubernetes-native inference orchestration (llm-d) are rethinking three different layers of the LLM stack at the same time. The way we generate, feed, and serve LLMs is changing all at once.
I noticed a pattern
I was catching up on AI infrastructure news a few weeks ago. Three things landed in my feed almost back to back.
Inception released Mercury, an LLM that doesn't generate tokens the way every other LLM does. Researchers from UW and Meta published the Context Language Model, which lets the model manage its own context like a scratchpad instead of getting everything dumped in upfront. And Red Hat, Google Cloud, IBM, and NVIDIA shipped llm-d 1.0, a Kubernetes-native layer for routing and scaling LLM inference across GPU fleets.
Totally different teams. Different problems. Different parts of the stack. But they're all attacking the same assumption: that LLMs have to work the way they work today.
Layer 1: How tokens get generated
Every LLM you've used writes one token at a time. It predicts the next word, writes it down, then uses that word to predict the one after it. It literally can't write word five until word four is done. That's autoregressive generation, and it's been the only approach since GPT-2.
Mercury does something different. Instead of writing one word at a time, it starts with a rough draft of the entire response, all at once, and then sharpens it in passes. Each pass fixes multiple tokens simultaneously. After a few rounds, the blurry draft becomes a coherent answer.
If you've seen how image generators work (Midjourney, Stable Diffusion), it's the same idea. They start with static noise and refine it into a picture. Mercury does that, but with text. The technical name is discrete diffusion.
Why does this matter? Because writing in parallel is faster. A lot faster.
Mercury 3 (released October 2026) sustains over 1,000 tokens per second on H100s. GPT-4o Mini does about 59. Claude 3.5 Haiku does about 61. That's not a small improvement. That's a 15x gap.
And it's not trading quality for speed. Mercury 3 holds its own against GPT-6 Luna and Claude Haiku 5.5 on benchmarks. It handles a 1M token context window. TIME named it one of the best inventions of 2026. This isn't a research toy.
The real implication is cost. Higher throughput means more tokens per GPU-second. Mercury's pricing ($0.10 per million input tokens) reflects that. I wrote about how the language you generate affects token cost. Now the model architecture is moving that needle too.
Is diffusion the future of text generation? It's early. But the trajectory is steep, and the approach plugs into existing tooling (RAG, tool use, fine-tuning) without requiring anyone to rebuild their stack. That's the kind of thing that actually gets adopted.
Layer 2: How context gets managed
Here's a question. Why do we keep making context windows bigger?
Gemini supports 1M tokens. Claude supports 200K. GPT-4 went from 8K to 128K. The direction is clear: cram more stuff in. But there's a cost. Attention scales quadratically. Doubling the context window roughly quadruples the compute bill. And most of that context is wasted anyway. You pasted 50 pages of documentation so the model could use three paragraphs from page 37.
Here's how it works today. You have a question that requires context. Maybe it's a codebase, maybe it's a collection of documents. Your RAG pipeline retrieves a bunch of chunks, pastes them all into the prompt, and the model generates an answer. The model never asked for those chunks. It doesn't know if they're relevant. It just gets a wall of text and does its best.
The Context Language Model from researchers at UW and Meta changes who's in charge of the context.
CLM gives the model an editable file. Think of it as a text document the model can read, write to, and delete from between generation steps. When the model starts a task, the file might be empty. The model reads some source material, pulls out what matters, and writes a compressed summary to its file. Then it generates. If it hits a gap mid-answer, it pauses, searches for more information, updates the file, and continues.
The key difference: context isn't a static dump that happens once before generation. It's a living working memory the model curates as it thinks. The model decides what to keep, what to rewrite shorter, and what to throw away. Old information that stops being relevant gets deleted to make room for new information that is.
Compare the two approaches concretely. Say you ask "how does this repo handle authentication?"
With RAG: retrieve 20 code chunks that mention "auth", paste all of them into the prompt, hope the model picks the right ones. Most of those chunks are noise. The model attends to all of them anyway. Expensive.
With CLM: the model searches for "auth", reads a few files, writes to its context file "auth uses JWT middleware in server/auth.go, tokens stored in Redis, refresh flow in handlers/refresh.go." Then it searches for the refresh flow specifically, reads that file, adds the relevant details. The context file now has exactly what's needed. Nothing more.
The numbers reflect this. On BrowseComp-Plus, CLM hit 11.4% higher accuracy while using 21.5% fewer FLOPs. On a 12-hour benchmark, 5% better scores with 59% less compute. In serving, 35% less server-side compute compared to standard inference. Better answers, less work.
This is also where it connects to the agentic shift. An agent running for hours across multiple tasks can't afford to carry everything in context forever. It needs to forget what's no longer relevant and go looking for what is. CLM gives the model that ability natively instead of bolting it on from outside.
Layer 3: How inference gets served
You can have the best model in the world. Doesn't matter if you can't serve it in production.
Most LLM serving tools (vLLM, TGI, Ollama) handle single-instance serving well. One model, one GPU, requests in, tokens out. But production means fleet management. Routing across replicas. Autoscaling. Multi-model serving. Failover. Priority queues. That's a completely different problem.
llm-d 1.0 shipped October 2026, backed by Red Hat, Google Cloud, IBM Research, NVIDIA, and CoreWeave. It's a CNCF project with 60+ contributors. But it's not a model server. It's the layer that sits above model servers like vLLM and SGLang and orchestrates traffic across them. Routing, caching, scaling, scheduling.
The interesting part is how it handles agentic workloads.
When you ask an agent to do something, it doesn't make one LLM call. It might make 5 to 20. Tool calls, planning steps, validation checks, reflections. Without smart routing, each of those calls starts from scratch. The model has to re-read the entire conversation history every time, because there's no memory between calls.
LLMs actually do cache this work internally. It's called a KV-cache: a snapshot of everything the model has already processed, so it doesn't have to re-read it. But if each call in an agentic loop lands on a random server, that cache is useless. It's sitting on a different machine.
llm-d fixes this. It routes related calls to the same replica so they can reuse the cache from previous steps. In benchmarks with Llama 3.1 70B, that gave 3x throughput and 2x faster time-to-first-token. Not from a better model. Just from smarter routing.
It also splits the two phases of inference across different GPU pools. Processing your prompt (prefill) is compute-heavy. Generating the response (decode) is memory-heavy. Using the same GPU for both means one job is always waiting on the other. Splitting them and scaling each independently got up to 70% higher throughput in their benchmarks.
Why all three at once?
These aren't coincidences. They're responses to the same pressure: LLMs are moving from "call an API and get an answer" to "run a fleet that powers autonomous agents." That shift stresses every layer.
Generation needs to be faster because agentic loops multiply the number of inference calls per user action. Context management needs to be smarter because agents work across many documents and tools, not a single prompt. And infrastructure needs to understand these patterns because naive serving wastes GPUs on workloads it doesn't recognize.
The stack is splitting because each layer has its own bottleneck, and solving one doesn't solve the others. Faster generation doesn't help if your context management is still "paste everything into the prompt." Smarter context doesn't help if your serving layer treats every request like a stranger.
What this means if you're building
If you're running LLMs in production or planning to, here's what I'd pay attention to.
Watch the generation layer. Diffusion-based models are worth tracking. If Mercury's throughput holds up in independent testing, the cost math for inference changes significantly. Don't build around the assumption that autoregressive speeds are permanent.
Think about context differently. Instead of reaching for the biggest context window, think about what the model actually needs at each step. RAG was version one. Dynamic context management is where it's heading. Build retrieval that's queryable, not just stuffable.
Treat inference serving as real infrastructure. If you're running more than a couple models or doing any agentic work, you need routing, scheduling, and cache awareness. llm-d is one option. NVIDIA NIM and Ray Serve are others. But "vLLM behind a load balancer" stops working the moment you have a fleet.
The LLM stack is young enough that all three layers are getting rethought at the same time. That's rare. And it means the choices we make in the next year or two will shape how this stuff works for a long time.
Pay attention.
This is part 1 of a series on the changing LLM stack. Next up: a deep dive into diffusion-based generation and what it means for inference costs.