Why does an LLM server need continuous batching, a paged KV cache and CUDA graphs? Drag the sliders and watch it happen. Numbers tagged measured come from my own engine, minivllm, benchmarked on a Tesla T4. Anything tagged simulated is a model fitted to those measurements.
Static vs continuous batching simulated
Same 64 requests, same arrival times, same GPU model. Each row is a batch slot. Hatched time is a slot that finished its request but cannot take a new one.
time
Static batching
wait for a batch, run it until the longest reply ends
Continuous batching
a freed slot is refilled on the very next step
prefilldecode, long replydecode, short replyslot idle, waiting on batchrequests waiting in queue (bottom strip)
What I actually measured measured
Stage 4 of minivllm, the same mixed workload, 64 requests, all present at t=0 (closed loop). Goodput in tokens per second.
staticcontinuous
How well does the simulator match? (calibration check)
The simulator has four fitted constants (decode step time and prefill cost). Fitted on these runs, so this checks the fit, not an independent prediction. The ratio between the policies is the part to trust. Absolute levels drift at large batch sizes, where the real engine slows down more than the model does.
Contiguous slots vs paged blocks simulated snapshot
One memory pool, the same requests. Contiguous reserves a full max-length slot per request. Paged hands out small blocks sized to what each request needs. Each row below is 732 tokens wide, the engine's slot size.
Contiguous
Paged
tokens stored (one colour per request)reserved, still emptyblock rounding wastefree
This is a snapshot of a full pool, so it shows the best case for paging. In the real runs below the pool was not always full (traffic arrived over time), so the throughput gain was smaller, 1.1x to 1.7x.
Goodput by pool size measured
Stage 5, mixed workload. Same memory, different layout.
contiguouspaged
Block size trade-off measured
Bigger blocks mean fewer table lookups but more rounding waste. Goodput barely moved, rounding waste exploded.
How fast can decode possibly go? published specs + formula
Each decode step reads every weight once plus every sequence's KV cache. Memory bandwidth sets the speed limit, memory capacity sets how many sequences fit.
My engine against vLLM and Hugging Face measured
Stage 12 and the vLLM comparison notebook. Tesla T4, same 64-request mixed trace replayed at each rate, over HTTP.
Single request latency measured
No queueing, 128 new tokens, mean over three workloads. Lower is better.
Capacity measured
Capacity here is the highest tested arrival rate where at least 90% of requests met the latency targets. The sweep stopped at 8 req/s, so "8+" means "did not break yet".