How to Build a Customer Support AI Agent With Confidence-Based Fallback Routing

Customer support is one of the most common places teams put a language model into production, and one of the fastest places a cheap model choice turns into a bad customer experience. A small, fast model answers routine questions instantly and for very little money. The same small model will also confidently mangle a refund dispute or a legal threat if you let it. A recent DigitalOcean tutorial lays out the pattern most teams reach for to solve this: fallback routing, where routine messages stay on a fast, cheap model and anything uncertain, sensitive, or high-stakes gets escalated to a stronger one.

This tutorial builds that pattern yourself, end to end, in Python, using Ollama to run two local models for free with no API key and no cloud bill. By the end, you will have a working support_router.py that answers simple tickets with a small model, asks that model to honestly rate its own confidence, and automatically escalates to a stronger model whenever the confidence is low, the model flags the topic itself, or the message matches a deterministic list of high-risk keywords. Every command and API detail below is cross-checked against Ollama’s own download documentation, the Ollama API reference, and the official Ollama Python library.

What fallback routing is, and why a single model is not enough

Picture a support inbox. Most incoming messages are routine: “where is my order,” “how do I reset my password,” “what are your support hours.” A handful are not: a customer threatening a chargeback, someone asking you to delete their account and all their data, a message that reads as furious rather than confused. Answering all of these with one model forces a bad trade-off. Use a small, fast model everywhere and it will occasionally give a confident, wrong answer on the messages that matter most. Use a large, expensive model everywhere and you pay far more, and wait longer, for the 90% of messages a small model would have handled just as well.

Fallback routing resolves that trade-off by treating “which model should answer this” as its own decision, made per message. A fast, inexpensive model handles a request first. If it reports that it is confident and the topic is routine, its answer goes out. If it reports low confidence, flags the topic as sensitive, or the message matches a known high-risk pattern, the request is escalated to a stronger, slower model instead. The DigitalOcean tutorial this pattern is drawn from calls the deciding component a router, and frames the whole system as “not just a chatbot,” but an intelligent workflow that combines routing, retrieval, evaluation, and human handoff.

The piece that makes routing possible is confidence scoring: some signal the router can check to decide whether the primary model’s answer is safe to send as is. There is more than one way to produce that signal. This tutorial uses the simplest one to build yourself: asking the model to self-report a confidence score as part of a structured response, using a feature called structured outputs that forces the model to return valid, schema-shaped JSON instead of free-form text. You will also add a second, deterministic layer on top of that self-reported score, for reasons Step 6 explains in detail: a language model rating its own confidence is a useful signal, not a guarantee.

Prerequisites

• Operating system: commands below are written for Ubuntu 24.04 LTS or newer (Debian-based Linux in general). Ollama also ships macOS and Windows builds; only the install command in Step 1 changes on those platforms.
• Hardware: a machine with at least 8 GB of RAM. Both models used here are deliberately small (1.3 GB and 2.0 GB on disk), so this runs comfortably on a laptop with no GPU required.
• Python: Python 3.10 or newer, with python3 -m venv available (it ships with the standard library on any normal Python install). This tutorial uses the X | None type hint syntax, which requires 3.10+.
• Internet access: required once, to install Ollama and download the two models. Once downloaded, everything in this tutorial runs fully offline.
• Prior knowledge assumed: comfort running commands in a terminal and reading basic Python. No prior experience with Ollama, structured outputs, or Pydantic is assumed; each concept is explained the first time it appears.

Step 1: Install Ollama

Ollama is a tool for downloading and running open-weight language models directly on your own machine, so the model weights and every prompt you send stay local. Install it with the official script:

# Context: Ubuntu 24.04 LTS or newer, sudo access.
# Purpose: download and install Ollama, and set it up as a background service.
curl -fsSL https://ollama.com/install.sh | sh

Expected output: the script detects your OS, downloads the Ollama binary, installs it, and configures it to run as a systemd service that starts automatically. Confirm it installed correctly:

# Context: same machine, right after installation.
# Purpose: confirm the Ollama binary is installed and print its version.
ollama --version

Expected output: a line similar to ollama version is 0.x.x. If you instead see “command not found,” open a new terminal (some shells cache command locations from before the install) or run hash -r to clear that cache.

Step 2: Pull the primary and fallback models

This tutorial uses two sizes of the same model family, Meta’s Llama 3.2, so the only variable that changes between routes is model capability, not model family. The 1B tag is your primary, fast route:

# Context: Ollama installed and running.
# Purpose: download the small, fast model used for the primary route.
ollama pull llama3.2:1b

Expected output: a progress bar downloading roughly 1.3 GB, ending with success. Now pull the larger model for the fallback route:

# Context: same machine.
# Purpose: download the stronger model used for escalated tickets.
ollama pull llama3.2:3b

Expected output: a second progress bar, roughly 2.0 GB, also ending with success. Confirm both are available locally:

# Context: same machine.
# Purpose: list every model Ollama has downloaded.
ollama list

Expected output: a table listing both llama3.2:1b and llama3.2:3b with their size and the date they were pulled. If a tag is missing here, the command in Step 4 that references it will fail with a “model not found” error, so it is worth confirming both rows are present before moving on.

Step 3: Set up a Python project

Create a project folder and an isolated virtual environment so these packages do not collide with anything else on your system:

# Context: wherever you keep coding projects.
# Purpose: create and activate an isolated Python environment for this tutorial.
mkdir support-agent-tutorial && cd support-agent-tutorial
python3 -m venv venv
source venv/bin/activate

Expected output: your shell prompt gains a (venv) prefix, showing the virtual environment is active. Every pip and python3 command from here on should be run from this same activated terminal. Now install the two libraries this tutorial depends on:

# Context: venv activated, inside support-agent-tutorial/.
# Purpose: install the Ollama client library and Pydantic for structured responses.
pip install ollama pydantic

Expected output: pip resolves and installs both packages (and their small set of dependencies) with no errors. Ollama’s official Python library talks to the local Ollama service over HTTP on your behalf; Pydantic is what defines the exact shape of JSON you want a model to return, which the next step uses directly.

Step 4: Define the structured response every model must return

Create a file named support_router.py in the project folder. Start with the schema that every model call in this tutorial will be forced to follow:

# Context: support-agent-tutorial/support_router.py, new file.
# Purpose: define the structured JSON shape both models must return, and the router's static config.

from pydantic import BaseModel
from ollama import chat

PRIMARY_MODEL = "llama3.2:1b"
FALLBACK_MODEL = "llama3.2:3b"
CONFIDENCE_THRESHOLD = 0.75

ESCALATION_KEYWORDS = [
    "refund", "chargeback", "cancel my account", "delete my data",
    "lawyer", "lawsuit", "legal action", "data breach", "unacceptable",
]

SYSTEM_PROMPT = (
    "You are a first-line customer support assistant for a small SaaS company. "
    "Answer routine questions directly and rate your own confidence honestly, "
    "from 0.0 (guessing) to 1.0 (certain). Set needs_escalation to true for "
    "anything involving billing disputes, refunds, legal threats, account "
    "deletion, security incidents, or a customer who sounds upset."
)


class RouteDecision(BaseModel):
    answer: str
    confidence: float
    needs_escalation: bool
    reason: str


def contains_escalation_keyword(message: str) -> bool:
    lowered = message.lower()
    return any(keyword in lowered for keyword in ESCALATION_KEYWORDS)

What this does and why it matters: RouteDecision is a Pydantic model, a Python class that describes exactly which fields a piece of JSON must have and what type each one is. Ollama supports structured outputs: pass a JSON schema in the format parameter of a chat request, and Ollama constrains the model’s output so it matches that schema, field for field. Instead of parsing free-form text and hoping the model mentioned a confidence number somewhere, you get back answer, confidence, needs_escalation, and reason as real, typed values every single time. contains_escalation_keyword is the deterministic safety net Step 6 wires in, checked with plain Python string matching, not a model call.

Step 5: Write the primary route

Add this function below what you already have. It is the one place both the primary and fallback routes actually call a model, so you only write the structured-output logic once:

# Context: support_router.py, appended below Step 4's code.
# Purpose: call a given model and return its response as a validated RouteDecision.

def ask_model(model: str, message: str) -> RouteDecision:
    response = chat(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": message},
        ],
        format=RouteDecision.model_json_schema(),
        options={"temperature": 0.2},
    )
    return RouteDecision.model_validate_json(response.message.content)

What this does and why it matters: RouteDecision.model_json_schema() generates a JSON schema straight from the Pydantic class you wrote in Step 4, and passing it as format is exactly the pattern shown in Ollama’s own structured-outputs example. options={"temperature": 0.2} keeps the model’s output close to deterministic; a high temperature makes both the wording and the self-reported confidence number noisier and less trustworthy, and a router needs that number to be as stable as possible. RouteDecision.model_validate_json(...) parses the model’s JSON text back into a real Python object, so the rest of your code works with primary.confidence or primary.needs_escalation as normal attributes instead of dictionary lookups.

Step 6: Do not trust self-reported confidence alone

It is tempting to stop here and route purely on primary.confidence. Do not. Small models are frequently overconfident: a 1-billion-parameter model asked to rate its own certainty will often report a high score on an answer that is subtly wrong, because self-assessment is itself a reasoning task, and it is exactly the kind of nuanced judgment small models are weakest at. Relying on a self-reported number by itself means the riskiest category of message, a confidently wrong answer, is the one your router is least equipped to catch.

That is why contains_escalation_keyword from Step 4 exists as a second, independent layer. It does not ask the model anything; it is a plain, deterministic check against a list of terms you control directly. A message containing “refund” or “lawyer” gets escalated regardless of how confident the primary model claims to be. Step 7 combines both signals: the model’s self-reported score and flag, and this deterministic keyword check, so neither one is a single point of failure. In a real deployment, expand ESCALATION_KEYWORDS to match the specific high-risk categories your own support inbox actually sees.

Step 7: Write the router and the fallback route

Add the function that ties everything together. This is the actual “router” the whole tutorial is building toward:

# Context: support_router.py, appended below Step 5's code.
# Purpose: decide whether a message stays on the primary model or escalates to the fallback model.

def handle_ticket(message: str) -> dict:
    keyword_hit = contains_escalation_keyword(message)

    try:
        primary = ask_model(PRIMARY_MODEL, message)
    except Exception as exc:
        return route_to_fallback(message, f"primary route failed to return a usable answer: {exc}")

    if keyword_hit or primary.needs_escalation or primary.confidence  dict:
    fallback = ask_model(FALLBACK_MODEL, message)
    return {
        "route": "fallback",
        "model": FALLBACK_MODEL,
        "answer": fallback.answer,
        "confidence": fallback.confidence,
        "escalation_reason": reason,
        "primary_confidence": primary_confidence,
    }

What this does and why it matters: handle_ticket checks the deterministic keyword match first, then the model’s own needs_escalation flag, then the numeric confidence threshold, escalating if any single one trips. Notice the try/except around the primary call: if the primary model errors, times out, or (rarely) returns text that fails schema validation, the router treats that failure itself as a reason to escalate, rather than crashing or silently returning nothing. That is a deliberate fail-safe design choice: when something goes wrong, fail toward the safer, stronger model, not toward no answer at all. route_to_fallback reuses the exact same ask_model function from Step 5, just pointed at FALLBACK_MODEL, which is the entire payoff of writing that function once in Step 5 instead of duplicating the chat-call logic for each route.

Step 8: Run it end to end

Add a small runnable block at the bottom of the file so you can try it directly:

# Context: support_router.py, appended at the very end of the file.
# Purpose: send one routine ticket and one high-risk ticket through the router and print the result.

if __name__ == "__main__":
    tickets = [
        "What time zone are your support hours listed in?",
        "I was charged twice this month and I want a refund right now or I'm calling my lawyer.",
    ]
    for ticket in tickets:
        result = handle_ticket(ticket)
        print(f"nTicket: {ticket}")
        print(f"Routed to: {result['route']} ({result['model']})")
        print(f"Confidence: {result['confidence']:.2f}")
        if result["route"] == "fallback":
            print(f"Escalation reason: {result['escalation_reason']}")
        print(f"Answer: {result['answer']}")

Run the script:

# Context: venv activated, both models pulled.
# Purpose: execute the full router against two example tickets.
python3 support_router.py

Expected output: the exact wording will vary between runs, since both models are generative, but the routing behavior should not. The first ticket is a plain factual question with no risk signal, so it should print Routed to: primary (llama3.2:1b) with a confidence at or above 0.75. The second ticket contains both “refund” and “lawyer,” so contains_escalation_keyword catches it before the confidence number is even checked, and it should print Routed to: fallback (llama3.2:3b) along with an Escalation reason line. If you see the opposite, routine questions escalating and risky ones staying on the primary model, re-read Step 6: it usually means ESCALATION_KEYWORDS was edited to no longer match your test message.

Common mistakes and how to catch them

• Referencing a model tag you never pulled. If you skip ahead and call ask_model("llama3.2:8b", ...) without running ollama pull llama3.2:8b first, the request fails with a 404-style error. Ollama’s own Python README documents catching this directly: wrap the call in try/except ollama.ResponseError as e, check e.status_code == 404, and call ollama.pull(model) on the spot. Run ollama list any time a model error looks unfamiliar.
• Raising the temperature “to make answers more interesting.” A higher temperature makes the self-reported confidence value noisier from one run to the next, which directly undermines the router’s main signal. Keep it low (this tutorial uses 0.2) for the routing decision itself; if you want more varied phrasing in the final answer shown to a customer, apply that separately, not to the model making the escalation call.
• Deleting the keyword safety net because “the model is confident enough.” Step 6 exists specifically because small models overrate their own certainty. Skipping contains_escalation_keyword and trusting primary.confidence alone reintroduces the exact failure mode fallback routing is meant to prevent: a confidently wrong answer on your highest-risk message.
• Swallowing JSON validation errors instead of escalating on them. If you replace Step 7’s try/except with a bare except: pass, a malformed response silently disappears instead of triggering a fallback. Always fail toward the safer route, never toward no response.
• Forgetting the Ollama service is not running. If chat() raises a connection error, the Ollama service itself is not reachable. The Linux installer sets it up to start automatically; systemctl status ollama confirms it is active, and ollama serve starts it manually in the foreground if you ever need to run it that way.
• Judging the first response as slow and assuming something is broken. Ollama loads a model into memory on its first use and keeps it loaded for a default of five minutes afterward (the keep_alive setting). The first call to a model you have not used yet will noticeably pause while it loads; repeated calls right after that are much faster. This is normal, not a bug.

Verifying it works end to end

1. Run ollama list and confirm both llama3.2:1b and llama3.2:3b are present.
2. Run python3 support_router.py and confirm it completes with no traceback.
3. Confirm the first ticket (the time zone question) prints Routed to: primary with a confidence at or above your CONFIDENCE_THRESHOLD.
4. Confirm the second ticket (the refund and lawyer message) prints Routed to: fallback with a non-empty Escalation reason.
5. As a negative test, edit the second ticket to remove “refund” and “lawyer” and replace them with a neutral rephrasing, then rerun. Whether it now stays on the primary route depends entirely on that run’s self-reported confidence, which is a useful, honest demonstration that the keyword layer and the model’s own judgment are genuinely two separate signals, not one signal in disguise.
6. Try a message that is emotionally upset but uses none of your listed keywords (for example, several exclamation points and short, frustrated sentences with no refund or legal language). Whether this escalates depends entirely on whether primary.needs_escalation catches it, since no keyword will. This is the clearest illustration of why both layers matter: expand ESCALATION_KEYWORDS as you learn what your own keyword layer misses.

Next steps

This router is a deliberately small, readable starting point, not a production system. From here, the natural extensions are the same ones the DigitalOcean article this tutorial is based on discusses at the architecture level: track per-message latency and cost so you can see the actual savings fallback routing produces, add real observability (structured logs of every routing decision, not just print statements), and connect the fallback route to an actual human handoff path for the cases even llama3.2:3b should not answer alone. It is also worth building a small labeled test set of past tickets with a known correct route, so you can tune CONFIDENCE_THRESHOLD against real data instead of the starting guess of 0.75 used here. Finally, consider wrapping handle_ticket in a small web API (FastAPI is a common choice) so it can sit behind a real support widget instead of only running from the command line.

Sources

• DigitalOcean Community Tutorials: Real-Time Customer Support Agent with Fallback Routing
• Ollama: Download Ollama on Linux
• Ollama API Reference (structured outputs and JSON mode)
• Ollama Python Library (official GitHub repository)
• Ollama Python: structured-outputs.py example
• Ollama Library: llama3.2
• Featured image source: Bell System telephone switchboard photograph, Wikimedia Commons

Image credit: Photograph of women working at a Bell System telephone switchboard, U.S. National Archives and Records Administration, No known copyright restrictions (U.S. government work) via Wikimedia Commons; cropped, resized, and converted to WebP for sxz.io.