LLM Evaluation: How to Measure Response Quality

Evaluating an LLM is not optional if you plan to use it in production. Without evaluation, you don't know if the model is accurate, hallucinating, safe, or whether an update made it worse. LLM evaluation is the process of systematically measuring response quality using objective criteria — accuracy, relevance, tone, safety, format — to decide whether a model is fit for your use case.

Pro Tip: Evaluation isn't a one-time step. It's an ongoing process: every time you change models, update a prompt, or add a new feature, you must re-evaluate.

What is it

Evaluating an LLM means comparing its output against a standard you define. Unlike traditional software — where a function returns true or false — an LLM returns natural language, which is inherently subjective. That's why evaluation has multiple dimensions:

1. Factual accuracy: Is the response correct against your ground truth?
2. Relevance: Does it answer the question or go off-topic?
3. Groundedness: Does the response rely on the context you provided or does it make things up?
4. Tone and style: Does it sound professional, friendly, on-brand?
5. Safety: Does it generate toxic, biased, or risky content?
6. Format: Does it respect the expected structure (JSON, XML, etc.)?

Each dimension requires a different evaluation method. There is no single metric that covers everything.

Note: Don't confuse evaluating an LLM with taking a test. You don't ask the model "how good are you?" — you give it a set of examples (test set) with expected answers and measure how close it was.

Mental model

Think of evaluation as a standardized exam for the model. You are the teacher preparing the exam: you define the questions (test set), decide what the correct answers are (ground truth), and then grade.

            ┌──────────────┐
Test set ──→│    LLM       │──→ Responses ──→ Score (metric)
            └──────────────┘
                  ↑
          Prompt / system

Unlike a human exam, grading can be automated: a larger LLM or a set of rules compares the model's response against the expected one and assigns a score.

How it's used

There are three main approaches, and in production you usually combine them.

1. Standardized benchmarks

Public test sets with expected answers that let you compare models against each other:

# lm-eval-harness: the de facto standard for benchmarks
pip install lm-eval
 
# Evaluate a model on MMLU
lm_eval --model hf \
    --model_args pretrained=mistralai/Mistral-7B-v0.1 \
    --tasks mmlu \
    --num_fewshot 5

Warning: Public benchmarks suffer from data contamination — the model may have seen the questions during training. Don't blindly trust a benchmark. Complement it with your own test set.

2. LLM-as-judge

Use a more powerful LLM (GPT-4, Claude 3.5 Opus) to evaluate another model's responses. This is the most practical method for teams without budget for human annotators:

from openai import OpenAI
 
client = OpenAI()
 
SYSTEM_PROMPT = """
You are an expert evaluator. Given a question, a response,
and an evaluation criterion, rate the response as:
- CORRECT: satisfies the criterion
- INCORRECT: does not satisfy it
- PARTIAL: partially satisfies it
 
Return only JSON with "score" and "reason".
"""
 
def evaluate_response(question, response, criterion):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"""
        Question: {question}
        Response: {response}
        Criterion: {criterion}
        """}
    ]
    result = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        response_format={"type": "json_object"}
    )
    return result.choices[0].message.content
 
# Example
eval = evaluate_response(
    question="What is the capital of Uruguay?",
    response="The capital of Uruguay is Montevideo.",
    criterion="The response must be factually correct."
)

The trick is the evaluator prompt: the more specific the criterion, the more consistent the scores.

Pro Tip: LLM-as-judge has biases. It tends to favor longer responses, ones with more formatting, or ones that match the evaluator's tone. Change the judge model periodically and calibrate against human evaluation every few hundred samples.

3. Custom test set with ground truth

Nothing beats having a set of examples from your domain with expected responses written by humans:

test_set = [
    {
        "question": "How do I reset my Huawei HG8145V5 router?",
        "expected": "Press the reset button on the back for 10 seconds.",
        "criteria": ["factual_accuracy", "clarity"]
    },
    {
        "question": "What are your business hours?",
        "expected": "Monday to Friday from 9:00 AM to 6:00 PM.",
        "criteria": ["factual_accuracy"]
    }
]
 
def evaluate_accuracy(response, expected):
    # Semantic comparison, not exact match
    # Use embeddings + cosine similarity or LLM-as-judge
    embedding = client.embeddings.create(
        model="text-embedding-3-small",
        input=[response, expected]
    )
    sim = cosine_similarity(
        embedding.data[0].embedding,
        embedding.data[1].embedding
    )
    return sim > 0.85  # empirically defined threshold

Full evaluation pipeline in production

import json
from openai import OpenAI
 
client = OpenAI()
 
def evaluate_response(question, response, expected, criteria):
    prompt = f"""
    Evaluate the following response and determine if it meets the criteria.
 
    Question: {question}
    Expected response: {expected}
    Model response: {response}
    Criteria: {json.dumps(criteria)}
 
    For each criterion, return "pass" or "fail".
    """
    result = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )
    return json.loads(result.choices[0].message.content)
 
# Test set of 50 questions from your domain
results = [evaluate_response(**item) for item in test_set]
pass_rate = sum(r.get("pass", False) for r in results) / len(results)
print(f"Pass rate: {pass_rate:.2%}")

When to use it / when not to

Pro Tip: Don't evaluate everything. Focus on the 10-20 questions that represent 80% of real usage. A small but representative test set beats a large but generic one.

History and evolution