The Infrastructure Pivot: Why Agentic Traffic is Breaking the Batch Paradigm

Last week, at the inaugural vLLM Conference held alongside Ray Summit in San Francisco, a quiet but unmistakable narrative shift took root. Multiple teams—from Intel and Prime Intellect to the vLLM core maintainers—presented converging architectural conclusions: the era of monolithic batch inference is giving way to disaggregated prefill/decode serving. The hook is not a new theory; it's a collective engineering pivot. As one speaker put it, "We are no longer optimizing for throughput; we are optimizing for session persistence." To hunt the truth, one must first bury the hype—and here, the hype is that batch inference can handle the next wave of AI workloads. For context, batch inference has been the backbone of large language model serving since the rise of transformers. The model processes multiple requests together, maximizing GPU utilization by pipelining prefill (compute-intensive) and decode (memory-bandwidth-intensive) stages. This paradigm works well for short, isolated queries—think chatbot responses or code completions. But agentic workloads—multi-turn conversations, tool-calling loops, context retention across pauses—break the assumptions of continuous batch processing. The system must maintain session state, route requests to the same decoder instance, and handle variable-length pauses. The vLLM ecosystem, which powers many production deployments, recognized this friction and began designing a new architecture. The core insight is straightforward: prefill and decode have fundamentally different hardware requirements. Prefill saturates compute units (matrix multiply), while decode is bound by memory bandwidth. Running them on the same GPU creates resource contention and suboptimal utilization. Disaggregated serving separates these stages onto different GPU pools, each optimized for its workload. At the conference, Intel demonstrated a prefill/decode decoupling that improved throughput, and Prime Intellect applied the same principle to a trillion-parameter MoE model using distributed KV cache storage. The vLLM Router now uses consistent hashing and sticky routing to ensure session affinity—a requirement absent in batch inference. The evidence is compelling: multiple independent teams converging on the same solution signals a genuine technical necessity, not a marketing gimmick. But the contrarian angle is essential. Disaggregated serving introduces new complexities that are often glossed over. The most obvious is network dependency—KV cache transfer across nodes relies on RDMA (NixlConnector for general use, MORI-IO for AMD hardware). In a large cluster, this adds latency and bandwidth pressure that can negate the benefits for short queries. The vLLM separation of prefill and decode is still marked as experimental; production users like Meta and LinkedIn have not migrated. Furthermore, the architecture assumes that agentic traffic grows to dominate inference volume. If agents remain a niche, the overhead of disaggregated serving may prove unjustified. The narrative of "breaking batch inference" is partly a self-serving agenda by the vLLM ecosystem to attract attention and funding. Competition from frameworks like SGLang and TensorRT-LLM, or from cloud providers building their own solutions, could erode vLLM's first-mover advantage. The 2.5x goodput improvement claimed by AMD on 8x MI300X nodes is impressive, but it was likely measured under specific agentic workload profiles—not generalizable to all use cases. Takeaway: The infrastructure pivot from batch to disaggregated serving is a real trend, but it is a narrative in its early stages—a hypothesis backed by strong technical logic but unproven at scale. The next twelve months will reveal whether the promise of session-aware inference justifies the cost of building dedicated prefill and decode clusters. For now, the investors and developers should watch for three signals: the graduation of vLLM's experimental disaggregated feature to stable, the first public migration case from a major production user, and independent benchmarks that compare the architecture under diverse workloads. The truth of this narrative will be written in deployment logs, not conference slides.