Continuous Batching: The Scheduling Insight That Made LLM Serving Actually Work

Part I — Static Batching Is Broken by Design

To understand why continuous batching matters, you must first feel the pain of what preceded it.

In the naive serving paradigm — call it static batching — the inference server groups incoming requests into a fixed batch, launches a forward pass over all of them in lockstep, and releases results only when every sequence in the batch has finished generating. The logic seems reasonable: GPUs love wide matrix multiplications; batch together, amortize the fixed costs, ship the result.

The pathology is immediate once you look at real traffic.

Output length is not a fixed quantity. It is a random variable with a distribution that spans two to three orders of magnitude. A request asking “What is 2+2?” finishes in five tokens. A request asking for a 1,000-word literature review runs for eight minutes. When these two live in the same static batch, the arithmetic is brutal: the GPU sits idle waiting for the long tail. The short sequence finishes early — its KV cache slot is occupied, its compute budget exhausted — but the batch cannot be retired until the longest sequence is done. Every slot in the batch that belongs to a finished sequence is a dead weight the hardware must carry.

This creates what engineers call the padding problem at the token level, but the deeper issue is batch-level head-of-line blocking. The batch scheduling boundary is the request, not the iteration. You pay for the worst-case sequence in every batch, always.

Quantitatively: on a Llama-2-13B deployment running static batches of 16 on an A100, measured GPU utilization during inference hovers between 20% and 35% under realistic traffic distributions (where output lengths follow a roughly log-normal distribution with high variance). The GPU is not slow. The scheduler is blind.

Part II — The Continuous Batching Insight

The key paper is Orca (Yu et al., OSDI 2022). The insight is deceptively simple and, in retrospect, obvious — which is the hallmark of all genuinely important systems ideas.

Move the scheduling boundary from the request level to the iteration level.

In the autoregressive decode loop, each forward pass produces exactly one new token per sequence. This is the iteration. Orca’s observation: at the end of each iteration, check which sequences have emitted an <EOS> token. Those sequences are done. Immediately evict them from the batch and pull in new sequences — or new prefill chunks — from the waiting queue. The batch is no longer a monolithic unit that lives and dies together. It is a continuously evolving set of active sequences, with arrivals and departures happening at every decode step.

This is iteration-level scheduling, and it changes everything.

The GPU is now operating on a batch that is always as full as the memory allows. There is no dead time waiting for a long-tail sequence to finish before admitting the next wave of short requests. A sequence that generates five tokens and exits in five iterations frees its slot for a new request at iteration six. The hardware sees continuous utilization.

The mechanism requires careful state management. Each sequence in the batch maintains its own KV cache — the materialized key-value pairs for all previous tokens in that sequence’s context. When a sequence exits, its KV cache pages are deallocated. When a new sequence enters, a prefill pass over its prompt tokens is run (possibly injected as a chunk into the ongoing decode step — more on this shortly), and fresh KV cache is allocated. The paging abstraction introduced by vLLM (Kwon et al., SOSP 2023) makes this practical at scale: KV cache is managed in fixed-size blocks (pages), allocated and freed dynamically like virtual memory, eliminating the memory fragmentation that would otherwise make continuous batching unworkable.

The result: GPU utilization climbs from the 20–35% static-batching regime to sustained 70–85% under the same traffic, on the same hardware. No new silicon. Just a smarter scheduler.

Part III — The Throughput Mathematics

The decode phase of LLM inference is memory-bandwidth-bound, not compute-bound. This is a critical fact that governs everything about serving optimization.

During decode, each forward pass for a single token requires loading the entire model weight matrix from HBM (High Bandwidth Memory) into the streaming multiprocessors. For Llama-3-70B in BF16, model weights consume approximately 140 GB. On an H100 SXM, HBM bandwidth is 3.35 TB/s.

The time to load weights for a single decode step (ignoring KV cache, which scales with sequence length and batch size):

T_weights = 140 GB / 3.35 TB/s ≈ 41.8 ms

Now here is the crucial throughput insight. Loading those 140 GB takes ~41.8 ms regardless of whether you are decoding one token or one hundred tokens in parallel — because the weight matrices are the same; you are just multiplying against a wider activation matrix (batch dimension B instead of 1). The matrix multiplications scale as O(B), but the weight loading cost is amortized across all B sequences.

Therefore, throughput (tokens/sec) scales nearly linearly with batch size until compute or memory bandwidth saturates:

Throughput(B) ≈ B / T_weights  [in the memory-bandwidth-dominated regime]

At B=1: ~24 tokens/sec
At B=32: ~760 tokens/sec
At B=64: ~1,500 tokens/sec
At B=128: throughput begins to plateau

Where does saturation occur for Llama-3-70B on H100?

The H100 delivers ~989 TFLOPS (BF16 Tensor Core). Each decode forward pass for Llama-3-70B involves roughly 140B multiply-accumulate operations per token (2× parameters, accounting for the standard GEMM structure). At batch size B:

T_compute(B) = B × 140B MACs / (989 × 10^12 FLOPS) ≈ B × 0.141 ms

The transition from memory-bound to compute-bound occurs when T_compute ≥ T_weights:

B_sat = T_weights / (140B MACs / 989 TFLOPS)
B_sat = 41.8 ms / 0.141 ms ≈ 296

So the saturation batch size for Llama-3-70B on a single H100 is approximately B ≈ 300. Below this threshold, adding requests to the batch increases throughput nearly linearly. Above it, you are compute-bound and throughput plateaus. KV cache memory consumption becomes the practical ceiling before you ever hit B=300 under typical context lengths — at 4K context length, the KV cache for 300 sequences at BF16 consumes ~150 GB, which exceeds even H100 HBM capacity (80 GB per die) — so in practice the effective ceiling is hardware-memory-constrained at roughly B=64–128 for long contexts.

Continuous batching is what allows the system to actually reach these batch sizes under real mixed-length traffic, rather than being held hostage to the worst-case sequence in each static group.

Part IV — The Latency Cost and Chunked Prefill

Continuous batching is not free. There is a fundamental tension it introduces between two latency metrics: TTFT (Time to First Token) and TPOT (Time Per Output Token).

When a new request arrives and its prompt must be prefilled — that is, all input tokens processed in a single forward pass to populate the KV cache — this prefill step is compute-intensive. A 2,048-token prompt on Llama-3-70B requires processing 2,048 token positions simultaneously, which is a large matrix multiplication (compute-bound). If this prefill step is injected into an ongoing decode batch, it pauses the decode clock for all existing sequences in the batch. The sequences already generating tokens experience a sudden spike in TPOT — they have to wait for the newcomer’s prompt to be digested before they get their next decode step.

This is the prefill-decode interference problem, and it is the central engineering challenge in production LLM serving.

The solution, formalized in Sarathi-Serve (Agrawal et al., 2024 — arXiv:2403.02310), is chunked prefill. Instead of running the full prompt prefill in one giant step, the prefill is split into fixed-size chunks (e.g., 512 tokens per chunk). Each chunk is processed in a single iteration alongside the ongoing decode batch. The prefill of a 2,048-token prompt takes four iterations of 512 tokens each, instead of one large disruptive step.

The benefits are precise:

TPOT stability: The decode-step latency for existing sequences increases only by the incremental compute cost of one 512-token chunk, not the full 2,048-token prefill. The interference is bounded and predictable.

TTFT smoothing: Under static prefill injection, a long-prompt request arriving at an unlucky moment (when many sequences are mid-decode) might wait many decode iterations before it gets its prefill slot. Chunked prefill allows partial progress on the prefill every iteration, reducing worst-case TTFT jitter.

Arithmetic roofline alignment: Chunked prefill can be tuned to keep each mixed iteration (decode tokens + prefill chunk) landing in the compute-efficient regime — large enough GEMMs to exploit tensor cores, small enough to not starve the decode sequences of their decode budget.

Sarathi-Serve demonstrates that with chunk size C ≈ 512, TPOT degradation for existing sequences is held below 10% even under aggressive request arrival rates, while TTFT for new requests is reduced by 40–60% versus naive continuous batching (which either runs full prefills or queues new requests entirely). This is not a marginal tuning. It is an architectural redesign of the scheduling policy.

Part V — The Four Metrics That Actually Matter

Production LLM serving is not a benchmark. It is a contract. The metrics that determine whether a deployment is viable versus unusable are exactly four:

1. TTFT — Time to First Token
The latency from request submission to receipt of the first output token. This is dominated by prefill time and queue depth. For interactive applications (chatbots, copilots), TTFT is the perceived responsiveness. Users tolerate TTFT up to ~500 ms before cognitive friction sets in. Targets: < 200 ms for interactive, < 1,000 ms for batch-tolerant.

2. TPOT — Time Per Output Token
The per-token decode latency, averaged over the output sequence. This determines the streaming speed — how fast text appears on screen. Human reading speed is approximately 4–5 tokens/second. Below 50 ms/token (> 20 tok/s), the output feels instantaneous. Above 100 ms/token (< 10 tok/s), users perceive the model as “slow.” TPOT is memory-bandwidth-bound and largely fixed by hardware at a given batch size.

3. Throughput — Tokens/Second/GPU
The aggregate output token rate normalized to GPU count. This is the economic metric — it determines hardware cost per million tokens and directly sets pricing floors. For Llama-3-70B on 2× H100 (tensor parallel), a well-tuned continuous batching server should achieve 2,000–3,500 output tokens/sec/GPU at sustained load.

4. Goodput Under SLO
This is the metric most benchmark papers omit and most production teams care about most. Goodput is the fraction of requests completed within the SLO (Service Level Objective) — e.g., TTFT < 500 ms AND TPOT < 80 ms/token. A server can show high raw throughput while delivering catastrophic goodput if it is scheduling aggressively without respecting latency budgets. Goodput collapses at high load when the queue grows and TTFT spikes. The optimal operating point for a continuous batching server is not maximum throughput; it is maximum goodput, which typically occurs at 60–75% of the throughput ceiling.

Part VI — Concrete Numbers: vLLM vs TGI vs TensorRT-LLM

Benchmarks for Llama-3-70B at batch size 64, 2× H100 SXM (80 GB), TP=2, BF16, input 512 tokens, output 256 tokens:

These numbers reflect production-representative configurations, not cherry-picked peak-throughput settings. Several patterns emerge:

TensorRT-LLM leads on raw throughput and TPOT because it compiles kernel-fused execution graphs specifically for the target hardware, eliminating PyTorch dispatch overhead and enabling INT8/FP8 weight compression that reduces HBM bandwidth demand. Its TTFT advantage comes from aggressive chunked prefill scheduling with CUDA graph replay.

vLLM’s strength is operational flexibility — its PagedAttention memory manager handles a wider range of context lengths without OOM, and its Python-native scheduling makes it easier to extend. It is the industry default for good reason: the performance gap versus TensorRT-LLM is real but rarely the binding constraint in production.

TGI is the workhorse for teams already on HuggingFace infrastructure. Its continuous batching implementation (added in v1.0) closes most of the gap versus the static-batching baseline, but its memory manager is less sophisticated than vLLM’s paging, leading to higher memory fragmentation and thus lower effective batch sizes at long context lengths.

All three systems implement the Orca-style iteration-level scheduling. The differentiation is in the quality of the scheduler (chunked prefill, preemption policy, priority handling) and the efficiency of the kernel execution (CUDA graph capture, fused attention, quantization).

Conclusion — Why This Is Not Just an Engineering Detail

Continuous batching is not a performance optimization. It is a correctness fix for a broken abstraction. Static batching modeled LLM inference as request-level work, when the natural unit of work is the decode iteration. Orca’s contribution was recognizing this mismatch and redesigning the scheduler around the right abstraction boundary.

Every subsequent advance — PagedAttention’s virtual memory for KV cache, Sarathi’s chunked prefill, speculative decoding’s draft-verification pipeline — is built on top of the iteration-level scheduling foundation that continuous batching establishes. Without it, none of the throughput numbers that make large-scale LLM deployment economically viable would be achievable.

The saturation arithmetic is clear: at the memory-bandwidth-bound decode regime, batch size is the lever that converts idle silicon into output tokens, and continuous batching is the mechanism that keeps that lever pushed. The H100’s 3.35 TB/s HBM is only as useful as the scheduler’s ability to keep it fed with meaningful work.

The work of Orca, vLLM, and Sarathi-Serve collectively moved LLM serving from “impressive demo” to “production infrastructure.” Understanding why they work — at the level of HBM bandwidth equations, iteration scheduling theory, and goodput-under-SLO measurement — is what separates the engineers who operate these systems from the engineers who design the next generation of them.

References

1. Yu, G. et al. “Orca: A Distributed Serving System for Transformer-Based Generative Models.” OSDI 2022.
2. Kwon, W. et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP 2023. (vLLM)
3. Agrawal, A. et al. “Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.” arXiv:2403.02310, 2024.
4. NVIDIA H100 SXM5 Datasheet. HBM3 bandwidth: 3.35 TB/s. BF16 Tensor Core throughput: 989 TFLOPS.
5. Meta AI. Llama 3 Model Card. 70B parameter configuration, 8K context window.

@article{fp4-2606007,
  title   = {Continuous Batching: The Scheduling Insight That Made LLM Serving Actually Work},
  author  = {fp4 editorial desk},
  year    = {2026},
  url     = {https://fp4.dev/system/continuous-batching/},
  journal = {fp4}
}