Lab notebook
A reviewer's question: how much of the speed-up is batching?
A batched Transformers baseline shows most of the earlier speed-up was batching; at identical work, vLLM was about 1.7 times faster.
What I ran
A reader pointed out that my first 100-request result compared batched vLLM with plain Transformers handling requests one by one. So I added a batched Transformers baseline and a matching vLLM script. Both use the same 100 questions, and each runs in two modes: answers stop naturally, and every answer forced to exactly 100 tokens. Three runs each, on my own Mac, with a 360-million-parameter model.
What I measured
(table, median of 3 runs with the range in brackets, 100 requests together)
| Measurement | Batched Transformers | vLLM |
|---|---|---|
| Tokens/sec, answers stop naturally | 2,454 (2,413 to 2,472) | 5,727 (4,464 to 6,170) |
| Tokens/sec, forced to 100 tokens each | 3,820 (3,669 to 4,496) | 6,619 (6,463 to 6,646) |
What I found
- Batching alone took plain Transformers from 181 to about 2,450 tokens/sec, roughly 13.6 times, with no vLLM involved.
- With identical work (10,000 tokens on both sides), vLLM was about 1.7 times faster than batched Transformers.
- The gap grows when answer lengths vary, which fits vLLM letting finished requests leave the batch early. This benchmark does not prove that.
What I haven’t separated yet
The backends differ: Transformers ran on PyTorch MPS, and vLLM ran through vllm-metal (MLX on Metal). Some of the gap may come from the backend and not from vLLM’s scheduling. Next, I want to run a batched MLX baseline to separate the two.