What is Chain-of-Thought Prompting?

Chain-of-Thought (CoT) prompting is a technique where you ask the model to show its reasoning step by step before giving the final answer. Instead of asking "what is 24 × 37?" and expecting a number, you ask "let's go step by step: first compute 24 × 30, then 24 × 7, and add the results." The model not only answers better — it also lets you see how it got there, making errors detectable and fixable.

Pro Tip: CoT is the simplest prompting technique that delivers the biggest quality jumps. If you learn only one prompting technique, make it this one.

What is it

Chain-of-Thought prompting was introduced by Wei et al. in 2022 and has since become a cornerstone of advanced prompting. The idea is straightforward: LLMs predict tokens sequentially, and if you interleave intermediate reasoning steps between the question and the answer, the model has more "computational space" to arrive at the correct result.

There are two main variants:

1. Manual CoT (few-shot): you give the model 2-3 examples where you explicitly show step-by-step reasoning, then ask your question.
2. Zero-shot CoT: you simply append "Let's think step by step" to the prompt. It works surprisingly well without any examples.

Note: CoT doesn't modify the model or its architecture. It's purely a prompting technique. Any modern LLM can benefit from it.

Mental model

Imagine asking a mathematician what 47 × 83 is. If you ask for the immediate answer, they might rush and make a mistake. If you say "show your work," they go through intermediate calculations, verify them, and arrive at the correct result. CoT does exactly that with the LLM: it gives it room to "think out loud."

Without CoT:
  User: "A train travels at 120 km/h. How long to cover 360 km?"
  LLM: "3 hours."  ← could be right or wrong, we don't know how it got there
 
With CoT:
  User: "A train travels at 120 km/h. How long to cover 360 km? Let's think step by step."
  LLM: "Speed = distance / time, so time = distance / speed.
        time = 360 km / 120 km/h = 3 hours.
        Therefore, it takes 3 hours."
  ← we see the reasoning and can verify

How it's used

Zero-shot CoT (the simplest form)

Just add a phrase to the end of the prompt:

prompt = """
A customer bought a product that cost $450. They had a 15% discount coupon
and paid $30 for shipping. They paid in 3 interest-free installments.
How much did they pay per installment?
 
Let's think step by step.
"""
 
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}]
)
 
print(response.choices[0].message.content)

The model will typically respond with something like:

1. Calculate the discount: 15% of $450 = $67.50
2. Price after discount: $450 - $67.50 = $382.50
3. Add shipping: $382.50 + $30 = $412.50
4. Divide into 3 installments: $412.50 / 3 = $137.50
 
Answer: $137.50 per installment.

Warning: Zero-shot CoT doesn't work equally well on all models. On smaller models (<7B) the phrase "Let's think step by step" may not produce genuine reasoning. On GPT-4 and Claude 3.5+ it works very well.

Few-shot CoT (with examples)

For tasks where the model tends to fail even with zero-shot CoT, give it examples:

reasoning_examples = """
Q: A pencil costs $15 and an eraser costs $8. If I buy 3 pencils and 2 erasers,
how much do I spend in total?
A: 3 pencils × $15 = $45. 2 erasers × $8 = $16. Total = $45 + $16 = $61.
The answer is $61.
 
Q: John is 24 years old. His sister is 3 years younger, and his cousin
is twice his sister's age. How old is the cousin?
A: Sister's age = 24 - 3 = 21 years. Cousin's age = 21 × 2 = 42 years.
The answer is 42 years.
 
Q: An item costs $200. If it has a 25% discount and then an 18% tax
is applied, what is the final price?
A:"""

Self-consistency CoT

Run the same CoT prompt multiple times with temperature > 0, and take the most frequent answer:

import json
from collections import Counter
 
responses = []
for _ in range(5):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user",
                   "content": "What is 156 + 239? Let's think step by step."}],
        temperature=0.7
    )
    text = response.choices[0].message.content
 
    import re
    numbers = re.findall(r"\b(\d+)\b", text.split("Answer")[-1])
    if numbers:
        responses.append(numbers[-1])
 
majority_vote = Counter(responses).most_common(1)[0][0]
print(f"Final answer (consensus): {majority_vote}")

Pro Tip: Self-consistency improves accuracy on math problems by 5-15 percentage points over simple CoT, at the cost of 3-5x more API calls. Use it only when accuracy matters more than cost.

CoT in system prompts for assistants

You can bake CoT into a permanent instruction:

system_prompt = """
You are a technical support assistant. Before giving an answer,
always break down your reasoning step by step internally.
Your final response should be clear and direct for the user,
but you must have done the complete reasoning beforehand.
"""

Some models allow inserting an invisible "reasoning block" using tags like [REASONING] that you filter out before showing the final response to the user.

When to use it / when not to

Pro Tip: Don't use CoT for tasks where the model is already near-perfect without it. Each extra step costs tokens and latency. If a model scores well on a benchmark without CoT, don't add it.

History and evolution