Introduction to vLLM: Optimizing Inference Performance

vLLM is an inference engine for LLMs designed to serve models at high speed and low cost. Where a Hugging Face Transformers deployment might generate 5-10 tokens/second on a GPU, vLLM can reach 50-100+ tokens/s on the same hardware thanks to techniques like PagedAttention (efficient KV cache management), continuous dynamic batching, and native quantization support. If you need to serve an LLM in production, vLLM is probably the first place you should look.

Pro Tip: vLLM is not a replacement for Ollama. Ollama is for local development; vLLM is for serving models in production with high concurrency and throughput.

Context

Serving an LLM in production has a memory problem: the KV cache (attention keys and values for each token) grows with prompt length and the number of concurrent requests. In naive implementations, each request reserves memory for the worst case, wasting VRAM. vLLM solves this with PagedAttention, inspired by operating system memory paging: it divides the KV cache into fixed-size blocks that are allocated on demand, eliminating fragmentation and allowing memory sharing between requests (useful for beam search and parallel sampling).

The result is that vLLM can serve 2-4x more requests than a traditional engine on the same GPU, with lower latency and higher throughput.

Note: vLLM supports Hugging Face, Tensorizer, and custom model formats. It does not train models — it only serves them.

Setup

1. Installation

pip install vllm

vLLM requires Python 3.9+ and a GPU with CUDA 11.8+ or 12.1+. It also supports AMD ROCm and Apple Silicon (experimental).

Warning: vLLM downloads and installs its own versions of torch and xformers. If you have pre-installed versions, use a fresh virtual environment to avoid conflicts.

2. Serve a model with one command

# Serves the model at http://localhost:8000
vllm serve mistralai/Mistral-7B-v0.1 \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --max-model-len 4096

This exposes an OpenAI-compatible API at /v1/chat/completions and /v1/completions.

3. Make requests from your code

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none"  # vLLM doesn't require an API key by default
)
 
response = client.chat.completions.create(
    model="mistralai/Mistral-7B-v0.1",
    messages=[{"role": "user", "content": "What is vLLM?"}]
)
 
print(response.choices[0].message.content)

Without changing anything besides base_url, your OpenAI code now points to your vLLM server.

4. Typical advanced configuration

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2 \       # use 2 GPUs
  --pipeline-parallel-size 1 \
  --gpu-memory-utilization 0.95 \  # use 95% of available VRAM
  --max-num-seqs 256 \             # max concurrent requests
  --max-model-len 8192 \           # max context length in tokens
  --enforce-eager \                # skip graph compilation (debug)
  --kv-cache-dtype fp8             # quantize KV cache (save VRAM)

Pro Tip: --gpu-memory-utilization 0.95 is a good default. If you hit OOM, lower it to 0.85. With quantization (AWQ or GPTQ), you can go up to 0.98.

Examples

Example 1: Basic REST service with streaming

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none"
)
 
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful and concise assistant."},
        {"role": "user", "content": "Explain what PagedAttention is in 3 lines"}
    ],
    temperature=0.7,
    max_tokens=512,
    stream=True
)
 
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Example 2: Load a quantized model (AWQ)

vLLM supports AWQ and GPTQ quantized models, which use less VRAM without much quality loss:

# Download the quantized model first
huggingface-cli download TheBloke/Llama-2-7B-Chat-AWQ

vllm serve TheBloke/Llama-2-7B-Chat-AWQ \
  --quantization awq \
  --dtype half

With AWQ quantization, a 7B model that normally takes ~14GB in FP16 drops to ~4GB, making it runnable on GPUs with 6-8GB VRAM.

Example 3: Using vLLM directly from Python (offline mode)

You don't need the REST server. Use vLLM directly from your script:

from vllm import LLM, SamplingParams
 
llm = LLM(
    model="mistralai/Mistral-7B-v0.1",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.90
)
 
params = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    max_tokens=256
)
 
outputs = llm.generate(
    ["What is fine-tuning?",
     "Explain the difference between RAG and fine-tuning"],
    params
)
 
for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Response: {output.outputs[0].text}")
    print("---")

Example 4: Benchmark a model served with vLLM

Since vLLM exposes an OpenAI-compatible API, you can run benchmarks directly:

from openai import OpenAI
 
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

lm_eval --model local-completions \
    --model_args model=default,base_url=http://localhost:8000/v1 \
    --tasks mmlu \
    --num_fewshot 5

Note: vLLM also has native integration with the evaluation framework, but the simplest approach is via its OpenAI-compatible API.

Particularities

Important limitations:

• Requires a GPU. vLLM does not run on CPU (unlike Ollama/llama.cpp).
• More complex setup than Ollama. Needs CUDA, drivers, and VRAM configuration.
• Not recommended for single requests. The server overhead is justified with multiple concurrent requests.
• New models: if a model uses an unsupported architecture, vLLM may not load it until support is added in a recent release.

Pro Tip: When starting with vLLM, use --enforce-eager the first time. It disables CUDA graph optimizations that can cause obscure errors. Once everything works, remove the flag for better performance.

History and evolution