What is Function Calling in LLMs?

Function calling is an LLM's ability to request the execution of an external function mid-conversation. The model doesn't run code — you do. It returns a structured object (typically JSON) describing which function to call and with what arguments. Your application executes it, gets the result, and feeds it back to the model for the final answer. This turns the LLM from a static text generator into an orchestrator that decides which tools to use to answer with up-to-date data.

Pro Tip: Function calling does not mean the model can execute code on your server. The model only suggests calls. You decide whether to run them. Security stays on your side.

What is it

Function calling is a mechanism baked into modern LLMs (GPT-4, Claude 3.5, Gemini, Mistral, Llama 3). Here's the flow:

You define functions with their name, description, and parameter schema (typically JSON Schema).
You include those definitions in the API call to the LLM.
The model analyzes the prompt and, if appropriate, returns a tool_call with the function name and arguments.
Your code runs the function and sends the result back to the model.
The model generates the final natural-language answer using that result.

The LLM never executes anything. It only recognizes when it needs external data and structures the request so you can fetch it.

Note: Don't confuse function calling with "the model can call APIs." The model only generates JSON. Everything else — authentication, HTTP requests, business logic — is on you.

Mental model

Think of function calling as an assistant who knows when to ask for help. The assistant (LLM) detects when something is beyond its reach — real-time data, user accounts, exact computation — and instead of making things up, it fills out a form (JSON) requesting the information it needs.

User: "How much is my portfolio worth?"
LLM:   [returns tool_call → get_portfolio_value(user_id: "123")]
App:   [executes the function, gets $45,230]
App → LLM: "The current value is $45,230"
LLM:   "Your portfolio is worth $45,230 as of today's close."

Without function calling, the model would either hallucinate a number or say "I can't access that information." With function calling, it knows exactly what to ask for and in what format.

How it's used

Most LLM providers support function calling. The most common example uses OpenAI's API.

1. Define the function

Pass the model a function definition using JSON Schema:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and country, e.g. Bogotá, Colombia"
                    },
                    "units": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"]
                    }
                },
                "required": ["location"]
            }
        }
    }
]

Warning: Descriptions matter. A poor description will cause the model to skip the function or pass wrong arguments. Be explicit.

2. Make the API call

Include tools in the request along with the user message:

from openai import OpenAI
 
client = OpenAI()
 
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What's the weather in Bogotá?"}],
    tools=tools,
    tool_choice="auto"  # the model decides whether to call or not
)

3. Handle the tool_call

If the model decides to call a function, the response includes tool_calls:

message = response.choices[0].message
 
if message.tool_calls:
    for tool_call in message.tool_calls:
        if tool_call.function.name == "get_weather":
            args = json.loads(tool_call.function.arguments)
            # args → {"location": "Bogotá, Colombia", "units": "celsius"}
            result = get_weather(args["location"], args["units"])
            # Send the result back to the model

4. Return the result to the model

Add the result as a message with role: "tool":

messages.append(message)
messages.append({
    "role": "tool",
    "tool_call_id": tool_call.id,
    "content": json.dumps(result)
})
 
second_response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools
)

The model now generates its final answer in natural language based on the real function result.

Pro Tip: Use tool_choice: "required" when you want to force a function call on every turn. Useful for assistants that must always query a database before replying.

Minimal complete example

import json
from openai import OpenAI
 
client = OpenAI()
 
def get_weather(location, units="celsius"):
    # In production this would call a real weather API
    return {"temperature": 19, "condition": "cloudy", "units": units}
 
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }
]
 
messages = [{"role": "user", "content": "What's the weather in Bogotá?"}]
 
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools
)
 
message = response.choices[0].message
messages.append(message)
 
if message.tool_calls:
    for tc in message.tool_calls:
        args = json.loads(tc.function.arguments)
        result = get_weather(**args)
        messages.append({
            "role": "tool",
            "tool_call_id": tc.id,
            "content": json.dumps(result)
        })
 
    final = client.chat.completions.create(
        model="gpt-4o", messages=messages, tools=tools
    )
    print(final.choices[0].message.content)

When to use it / when not to

✅ Use it when	❌ Don't use it when
You need real-time data (weather, prices, APIs)	The information is already in the model's context
You want to trigger actions (send email, create a ticket)	The task is purely generative (summarize, translate, create)
You need to validate or structure user input	A direct prompt with `response_format` suffices
You're building an autonomous agent that picks tools	The round-trip latency cost isn't justified
You want to reduce hallucinations by providing verified sources	You only need structured JSON output without external logic

Pro Tip: If you just need structured JSON output (no execution), use response_format: { "type": "json_object" } without function calling. It's simpler, cheaper, and faster.

History and evolution

View history

Function calling as we know it today debuted with GPT-4 in June 2023. Before that, developers hacked JSON outputs through prompt engineering and hoped the format would be valid — which it often wasn't.

OpenAI introduced functions in the Chat Completions API. Months later, others followed: Anthropic launched tool use in Claude, Google added it to Gemini, and open-source models like Llama 3 and Mistral baked it in natively.

The standard has evolved toward tools (an array of tool definitions, not just functions) and is expected to converge on interoperable protocols like the increasingly adopted MCP (Model Context Protocol), proposed by Anthropic, which lets developers wire tools to models without custom integration code.