How to Run Local Models with Ollama in Your Development Environment

Ollama is the most straightforward way to run an LLM on your machine without depending on OpenAI, Anthropic, or any external API. Download the binary, run a command, and in minutes you have a local model ready for development, prototyping, or integration — no data sent to third parties, no rate limits, no per-token cost.

Pro Tip: Use Ollama in development to save API costs while prototyping. When you go to production, just swap the endpoint to a cloud provider with minimal code changes.

Context

Before Ollama, running an LLM locally meant cloning Hugging Face repos, installing Python dependencies, configuring CUDA, and writing boilerplate inference code. Ollama solves all that with a streamlined interface: download the binary, run ollama pull <model>, and you get a REST API at localhost:11434 ready to use.

Ollama is a wrapper that bundles the model runtime (llama.cpp) and exposes:

• A CLI to download, run, and manage models.
• A REST API compatible with OpenAI's format (so migration is trivial).
• Bindings for Python, JavaScript, Go, and more.
• Support for NVIDIA GPUs (CUDA), AMD (ROCm), and CPU.

Models range from lightweight options like Llama 3.2 (1B and 3B) to full 70B quantized weights, including Mistral, Gemma, Phi, Qwen, DeepSeek, and hundreds more in the official Ollama library.

Note: Ollama does not train models. It only runs them locally. Training is still the domain of frameworks like Axolotl or Unsloth.

Setup

1. Installation

Linux / macOS:

curl -fsSL https://ollama.com/install.sh | sh

Windows:

Download the installer from ollama.com/download and run it.

Warning: On Windows, Ollama runs on WSL2. Make sure WSL2 is installed and configured before installing.

2. Verify installation

ollama --version
# ollama version 0.6.2

3. Pull a model

# Lightweight model for development
ollama pull llama3.2:3b
 
# More capable model (~8GB RAM needed)
ollama pull mistral
 
# Multimodal model (text + images)
ollama pull llava

Each model downloads once and stays cached in ~/.ollama/models/ ready to run.

Pro Tip: Start with llama3.2:3b or phi4:latest. These are lightweight models that run on almost any modern machine without a GPU.

4. Run the model

# Interactive terminal mode
ollama run llama3.2:3b
 
>>> What is the capital of Chile?
The capital of Chile is Santiago.

5. Use the REST API

Ollama exposes a server at http://localhost:11434 automatically when you run a model. You can make direct HTTP requests:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "What is an LLM?",
  "stream": false
}'

6. OpenAI SDK compatibility

Ollama exposes an endpoint compatible with the OpenAI API, so you can point your existing code to localhost with a single change:

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:11434/v1",  # ← only this changes
    api_key="ollama"  # not used, but the SDK requires it
)
 
response = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)
 
print(response.choices[0].message.content)

That's it. Your production code that targets OpenAI now points to your local model.

Warning: Not all local models respond like GPT-4. Don't expect the same reasoning level, especially with 3B or 7B models. Use them for development, prototyping, and testing.

Examples

Example 1: Interactive chat with streaming

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)
 
stream = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Explain what a transformer is in 3 lines"}],
    stream=True
)
 
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Example 2: Using Ollama with LangChain

from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage
 
llm = ChatOllama(model="llama3.2:3b", temperature=0)
 
response = llm.invoke([
    HumanMessage(content="Give me 3 project ideas with AI")
])
 
print(response.content)

Example 3: Ollama as a function calling backend

ollama pull llama3.1:8b  # supports tool calls

from openai import OpenAI
 
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
 
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"}
                },
                "required": ["location"]
            }
        }
    }
]
 
response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "What's the weather in Buenos Aires?"}],
    tools=tools
)
 
print(response.choices[0].message.tool_calls)

Note: Not all models on Ollama support tool calls (function calling). Check the model description on the Ollama library to see if it's included.

Example 4: Using Ollama as a remote server

Run the Ollama server on a remote machine or container and connect from your code:

# On the remote machine or server
ollama serve

# From your local machine
client = OpenAI(
    base_url="http://192.168.1.100:11434/v1",
    api_key="ollama"
)

This lets you have a "GPU box" on your local network that all developers can connect to.

Particularities

Important limitations:

• Speed: On CPU, 7B models may generate 2-10 tokens/second. On a decent GPU, 20-50 tokens/s. Far from cloud API speeds.
• Quality: Larger local models (70B quantized) approach GPT-3.5, but no open model consistently matches GPT-4 or Claude 3.5 Opus today.
• Tool calling: Experimental in Ollama and not all models implement it correctly.

Pro Tip: If you have a GPU with at least 8GB VRAM, use Q4_K_M quantized models: they offer the best balance between quality and performance. On CPU, prioritize 3B models or smaller.

History and evolution