TL;DR
On one 8 GB laptop GPU, continuous batching cut p95 time-to-first-token from 503 ms to 119 ms at eight concurrent requests, while static batching kept slightly higher throughput. Neither wins outright. Moving dense KV tensors costs close to a third of batched compute, which makes a paged KV cache the clear next step.
- 3.2×token throughput from batching, 33.7 → 108.5 tokens/s
- 503 → 119 msp95 time-to-first-token, static → continuous batching
- 29–31%of batched runner time spent moving the KV cache
Abstract
Production inference engines such as vLLM are fast because of a handful of ideas: batched prefill, KV caching, continuous batching and careful request scheduling. Inside a large codebase those ideas are hard to see in isolation. AxiomServe rebuilds each moving part in a small, readable runtime and measures what each one buys on hardware a single person owns. Hugging Face Transformers supplies only the Qwen2 model definition and weights; AxiomServe owns the generation loop, batched prefill and decode, KV-cache handling, admission and scheduling, streaming and metrics.
On a seeded workload of 16 requests per case, continuous batching cut p95 time-to-first-token from 503 ms to 119 ms at eight concurrent requests compared with static batching, while static batching kept slightly higher throughput. Neither mode wins outright: the result is a latency/throughput trade-off. Profiling the runner shows that moving dense per-request KV tensors costs close to a third of batched compute time, which makes a paged KV cache the clear next step.
System overview
Requests can be cancelled or fail from any state before the last; cache and engine entries are released on finish.
Three batching modes behind one engine
| Mode | Admission | Trade-off |
|---|---|---|
| Sequential | One request at a time | Correctness baseline; everyone else waits |
| Static | Gathers a cohort over a short interval | Batched forwards, but finished requests hold their slot until the whole cohort is done |
| Continuous | Rechecks capacity between iterations | New requests prefill while older ones keep decoding; prefill briefly pauses decode |
What the runtime implements
- Prefill and decode: batched prompt prefill, then one-token decode that reuses the real KV tensors from earlier steps, with logical position IDs that preserve RoPE across rows of different length.
- Admission: FCFS or shortest-prompt-first, bounded queues, and reservations for context length, padded tokens and optionally estimated KV memory.
- Sampling: greedy, temperature, top-k and top-p with per-request seeds, EOS handling and output limits.
- Serving and measurement: HTTP completions with SSE streaming, cancellation and graceful shutdown; per-request timings, bounded metrics and seeded benchmark sweeps.
Results
Qwen2.5-0.5B-Instruct in float16 on an NVIDIA RTX 5070 Laptop GPU (8 GB, WSL2). Each case sent 16 requests in two bursts of eight, with prompts of 8–32 tokens and outputs of 8–24 tokens. All 192 requests across the 12 cases succeeded.
| Mode | Tokens/s ↑ | TTFT p50 (ms) ↓ | TTFT p95 (ms) ↓ | Latency p95 (ms) ↓ |
|---|---|---|---|---|
| Sequential | 33.7 | 2823 | 3478 | 4052 |
| Static | 108.5 | 57 | 503 | 1581 |
| Continuous | 100.5 | 62 | 119 | 1416 |
Table 1. Serving metrics at eight concurrent requests. Best value per column in bold.
- Batching tripled throughput, from 33.7 tokens/s with sequential serving to 100–108 tokens/s with either batching mode.
- Continuous batching kept tail latency low. Its p95 TTFT was 119 ms against 503 ms for static batching, because new requests join between decode steps instead of waiting for a cohort to finish.
- Static batching had slightly higher throughput in this run, so this is a latency/throughput trade-off rather than a universal win.
- Moving the KV cache around is expensive. At concurrency eight, preparing and splitting the cache took about 29–31% of batched runner time.
Correctness
Speed numbers mean little if the batched path generates different tokens. The test suite builds a tiny randomly initialised Qwen2 model so CPU checks need no download, and compares padded batched decode against full-sequence forwards. Manual greedy decoding is checked token-for-token against Transformers' generate(), which is used only as a test oracle; the serving engine never calls it.
- 25 tests pass, including pretrained CUDA greedy equivalence against a neutral
generate()oracle. - A GPU smoke test confirms all three modes produce identical token sequences, and that continuous mode admits a fresh request while an older one is still running.
- Strict mypy passes on 22 source files; Ruff lint and format checks pass. CPU checks run in CI on every push.
Limitations and next steps
Limitations
- Each request's KV cache is a dense tensor that is padded, packed and copied on every decode step; there is no paged cache.
- Prefilling a new prompt blocks decode for active requests; no chunked prefill or preemption.
- No custom kernels, CUDA graphs, quantization, prefix caching or speculative decoding.
- One process on one GPU, behind a minimal completions API that is not fully OpenAI-compatible. Full-attention Qwen2 models only.
- Shortest-prompt-first can starve long prompts.
What comes next
- A paged KV cache, to remove the per-step repacking.
- Chunked prefill, so new prompts stop stalling decode.
- Larger sweeps with longer contexts, repeated runs with confidence intervals and randomised case order.
- Prefix caching for shared prompts.
This is one small run with short contexts and no confidence intervals: a systems sanity check, not a statistical performance study.
Run it
Requires Python 3.11+ and uv. A CUDA GPU is optional; pass --device cpu without one.
git clone https://github.com/saidurpulok/axiomserve && cd axiomserve
uv sync --python 3.12 --group dev --extra bench
uv run axiom serve --config configs/serve.yaml
curl -N http://127.0.0.1:8000/v1/completions \
-H 'Content-Type: application/json' \
-d '{"prompt":"Explain a KV cache:","max_tokens":32,"temperature":0,"stream":true}'
Reproduce the benchmark and plots:
uv run axiom bench --config configs/bench_small.yaml --output results/gpu-small.json
uv run python scripts/plot_benchmarks.py results/gpu-small.json
Citation
@software{pulok2026axiomserve,
author = {Pulok, Saidur Rahman},
title = {AxiomServe: A From-Scratch LLM Serving Runtime for Studying
Scheduling, Batching and KV-Cache Trade-offs},
year = {2026},
url = {https://github.com/saidurpulok/axiomserve}
}