Batch inference. Broken. Agent traffic wins.
The first vLLM Conference, held alongside Ray Summit in San Francisco, delivered a seismic signal: the era of monolithic batch inference is over. Multiple independent teams—Intel, AMD, Prime Intellect, and the vLLM core contributors—converged on the same conclusion. Agentic workloads, with their multi-turn sessions, tool-call pauses, and persistent context, are incompatible with the old collocated prefill-decode model. The architecture must split. Prefill gets its own GPU pool. Decode gets its own. Cross-node KV cache transfer becomes the new lifeline.
Context: The Old Paradigm Crumbles
For years, batch inference ruled. One machine, one serving process, prefill and decode bundled together. It worked for high-throughput, short-query workloads. But the rise of AI agents—autonomous programs that call tools, maintain state, and interact over multiple rounds—has exposed the bottleneck. Prefill is compute-heavy. Decode is memory-bandwidth-heavy. When they share a GPU, one stage starves the other. Agent traffic, with its stop-and-go nature, amplifies the inefficiency. The result? Latency spikes, resource waste, and frustrated users.
vLLM, the open-source inference engine, is now the battleground. Its experimental disaggregated serving mode, still tagged as experimental, is the most aggressive response. Production users like Meta, LinkedIn, and Hugging Face still run collocated. But the infrastructure is pivoting fast.
Core: The Disaggregated Architecture Deep Dive
Here's the technical truth: split prefill and decode into separate vLLM instances, each with dedicated GPU resources. Prefill nodes crunch through prompt tokens at high compute density. Decode nodes focus on memory bandwidth, streaming tokens one by one. The two pools scale independently. Efficiency gains reported: AMD's MORI-IO connector achieved 2.5x higher goodput on 8x MI300X nodes compared to collocated.
But this isn't free. The glue is KV cache transfer. After prefill, the key-value tensors must be moved to the decode instance. vLLM's NixlConnector, default since v0.8, uses RDMA (Remote Direct Memory Access) for low-latency transport. MORI-IO is AMD's hardware-specific alternative. The network becomes the backbone. InfiniBand, RoCE, and Ultra Ethernet are no longer optional—they are mandatory.
Then there's the router. vLLM Router uses consistent hashing and sticky sessions to ensure that every request in a session lands on the same decode instance. Session context persists. No cache invalidation. This is a new layer of infrastructure: session-aware routing, KV cache management, and distributed state. Prime Intellect even stores KV cache on CPU memory and SSD for trillion-parameter MoE models.
I've seen this pattern before. During the 2021 NFT floor price verification sprint, I built Python scripts to detect wash trading. The lesson: when a new workload breaks the old architecture, the first to adapt wins. Today, agents are the new workload. The infrastructure is scrambling.
Contrarian: The Hype vs. The Reality
Not everyone should jump. The disaggregated architecture is still experimental. Production users—Meta, LinkedIn—haven't migrated. Why? Because for short, single-turn queries, collocated outperforms. The network overhead of KV transfer can erase gains. The 2.5x goodput number from AMD comes from carefully designed agent workloads. Blindly applying this to all traffic is a mistake.
Moreover, the complexity is real. Two GPU pools, RDMA network congestion, new failure modes (router goes down, sessions lost). The cost of additional GPUs and network gear may not be justified by agent traffic that is still a fraction of total inference. The analysis from the conference shows that vLLM's disaggregated mode is labeled experimental for a reason. It's not production-ready.
And the competition is watching. SGLang, TensorRT-LLM, and NVIDIA's Dynamo are likely developing similar approaches. If NVIDIA builds prefill-decode split into its NIM microservices, the vLLM ecosystem could lose its edge. The "multiple independent teams converging" narrative is also a sign that the idea is not unique—it's the obvious response to the same problem. The real differentiator will be execution, not invention.
Takeaway: The Next Watch for Blockchain AI
For blockchain-based GPU networks—think Akash, io.net, Render—this is a critical signal. Decentralized compute providers must support session-aware routing and disaggregated serving to capture agent workloads. If they don't, they'll be relegated to serving short, low-value queries. The architecture pivot is not a future trend. It's happening now. The question is: which networks will build the infrastructure to support it?
Trust bridge crossed. Agent traffic wins. Data checked. Community warned.