Lab notebook

Track A

A reviewer's question: how much of the speed-up is batching?

A batched Transformers baseline shows most of the earlier speed-up was batching; at identical work, vLLM was about 1.7 times faster.

Related link →

What I ran

A reader pointed out that my first 100-request result compared batched vLLM with plain Transformers handling requests one by one. So I added a batched Transformers baseline and a matching vLLM script. Both use the same 100 questions, and each runs in two modes: answers stop naturally, and every answer forced to exactly 100 tokens. Three runs each, on my own Mac, with a 360-million-parameter model.

What I measured

(table, median of 3 runs with the range in brackets, 100 requests together)

Measurement Batched Transformers vLLM
Tokens/sec, answers stop naturally 2,454 (2,413 to 2,472) 5,727 (4,464 to 6,170)
Tokens/sec, forced to 100 tokens each 3,820 (3,669 to 4,496) 6,619 (6,463 to 6,646)

What I found

  • Batching alone took plain Transformers from 181 to about 2,450 tokens/sec, roughly 13.6 times, with no vLLM involved.
  • With identical work (10,000 tokens on both sides), vLLM was about 1.7 times faster than batched Transformers.
  • The gap grows when answer lengths vary, which fits vLLM letting finished requests leave the batch early. This benchmark does not prove that.

What I haven’t separated yet

The backends differ: Transformers ran on PyTorch MPS, and vLLM ran through vllm-metal (MLX on Metal). Some of the gap may come from the backend and not from vLLM’s scheduling. Next, I want to run a batched MLX baseline to separate the two.

← All notebook entries