How to Install Ollama and Run Your First Local AI Chatbot

By the end of this tutorial you will have a large language model running entirely on your own computer, answering questions from your terminal, from a raw HTTP request, and from a small Python script you write yourself. No API key, no per-token bill, and no internet connection required once the model is downloaded.

This matters for a few concrete reasons. First, privacy: nothing you type ever leaves your machine. Second, cost: after the one-time download, every conversation is free, no matter how long or how many. Third, understanding: running a model locally forces you to see the pieces that a hosted chat app hides from you, things like model files, memory limits, and the raw request/response format that every AI product is built on top of. We will use a tool called Ollama, which packages an open-source model runtime, a model downloader, and a small local web server into one program you can install in a few minutes.

We will be honest about the tradeoff too: the models you can comfortably run on a laptop are smaller and less capable than the frontier models behind ChatGPT or Claude. Later in this tutorial you will see a real example of a small local model confidently getting a fact wrong, and we will explain why that happens.

Prerequisites

• Operating system: Windows 10 or 11, macOS 12 or later, or a modern Linux distribution. The commands below are labeled by OS.
• Permissions: the ability to install software on the machine (an administrator account on Windows/macOS, or sudo access on Linux).
• Disk space: at least 5 GB free. Small models start around 300 to 400 MB, but you will likely want to try a few.
• Memory: 8 GB of RAM is a reasonable minimum for the small model we will use in this tutorial; 16 GB or more gives you room to try larger models later.
• Comfort with a terminal: you should be able to open Command Prompt or PowerShell (Windows), Terminal (macOS), or a shell (Linux) and type a command. No prior programming experience is required until Step 8, where basic familiarity with Python helps.
• A GPU is optional. Ollama runs on CPU alone, just more slowly. If you have a supported NVIDIA, AMD, or Apple Silicon GPU, Ollama will use it automatically.

Step 1: Install Ollama

Ollama is a background service plus a command-line tool. Once installed, it runs quietly and listens on your own machine at http://localhost:11434, waiting for you (or a program) to ask it to load a model and answer a prompt. Installing it does three things: it puts an ollama command on your system PATH, it installs a background service that starts automatically, and it creates a folder on disk where downloaded models will live.

Pick the section for your operating system. These are the official install commands, verified directly against Ollama’s own GitHub repository at the time of writing.

Windows

Open PowerShell and run:

irm https://ollama.com/install.ps1 | iex

Or download the installer manually from ollama.com/download and run it like any other Windows application.

macOS

Open Terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

Or download the app bundle manually from ollama.com/download and drag it into your Applications folder.

Linux

Open a terminal and run:

curl -fsSL https://ollama.com/install.sh | sh

This script detects your distribution and installs the right package. If you would rather install by hand (for example on a machine without internet access to run a piped script), Ollama’s documentation has a manual install guide for Linux.

Step 2: Verify the Installation

Close and reopen your terminal window first. This matters more than it sounds: the installer adds ollama to your system PATH, but a terminal window that was already open when you installed it will not see that change until you start a new one. Forgetting this step is one of the most common “it’s not working” moments for a fresh install.

With a new terminal open, run:

ollama --version

On the machine used for this tutorial, that printed:

ollama version is 0.32.9

Your version number may be newer since Ollama ships frequent updates. What matters is that you get a version string back at all, not an error like “command not found.” If you do get that error after reopening your terminal, double-check that the installer actually completed, and on Windows/macOS look for the small Ollama icon in your system tray or menu bar, which confirms the background service started.

Step 3: Understand How Ollama Names and Sizes Models

Before downloading anything, it helps to understand the vocabulary you will see everywhere in Ollama’s model library.

Models are named family:tag, for example llama3.2:1b. The part before the colon is the model family (who trained it and which version). The part after the colon usually tells you the size, most often as a parameter count: “1b” means roughly 1 billion parameters. A parameter is one adjustable number inside the model, learned during training; very roughly, more parameters means the model can represent more nuance and knowledge, at the cost of needing more memory and compute to run.

You will also hear the word quantization. A model’s parameters start out as high-precision decimal numbers, which take a lot of space (2 or 4 bytes each). Quantization compresses those numbers down to lower precision, commonly 8-bit or 4-bit integers, trading a small amount of accuracy for a much smaller file and faster inference. Most models on Ollama’s library are already quantized for you. You can check exactly how a specific downloaded model was quantized with ollama show <model>, which we will do in the next step.

Finally, context length is how much conversation (measured in tokens, roughly word-fragments) a model can consider at once. A model that “forgets” the start of a long conversation has usually run past this limit.

Step 4: Pull Your First Model

Downloading a model in Ollama is called “pulling,” borrowing the term from Docker, whose layer-based caching Ollama’s model storage is deliberately similar to. We will use llama3.2:1b, a small, well-known model from Meta that is a good first download: it is about 1.3 GB, small enough to fetch quickly on most connections, and capable enough to hold a real conversation.

ollama pull llama3.2:1b

You will see a progress bar as Ollama downloads the model’s manifest (a small file describing which pieces make up the model) and then its weight files, called blobs. On the machine used for this tutorial, the real output looked like this (trimmed to the interesting lines):

pulling manifest
pulling 74701a8c35f6: 100% |###################| 1.3 GB/1.3 GB  108 MB/s
verifying sha256 digest
writing manifest
success

That “verifying sha256 digest” line matters: Ollama checksums every downloaded blob before trusting it, so a corrupted or interrupted download will fail loudly rather than silently giving you a broken model. Blobs are also content-addressed and shared between models, which is why pulling a second model that shares a base with one you already have is often much faster than the first download.

Once the pull finishes, confirm it landed with:

ollama list

NAME            ID              SIZE      MODIFIED
llama3.2:1b     baf6a787fdff    1.3 GB    3 seconds ago

Now look at exactly what you downloaded with ollama show, which reads the model’s own metadata rather than anything we are guessing at:

ollama show llama3.2:1b

  Model
    architecture        llama
    parameters          1.2B
    context length      131072
    embedding length    2048
    quantization        Q8_0

  Capabilities
    completion
    tools

That tells us this specific download is 1.2 billion parameters, quantized to 8-bit integers (Q8_0), and supports a maximum context window of 131,072 tokens (about 100,000 words) even though, as you will see in Step 6, Ollama does not actually use that entire window by default.

Choosing a model size for your machine

As a practical, verified rule of thumb: a model needs a little more free RAM (or VRAM, if you have a supported GPU) than its size on disk, because Ollama also reserves memory for the context window and other overhead. On the machine used for this tutorial, the 1.3 GB llama3.2:1b file took 1.5 GB once loaded, which we confirmed directly with ollama ps in Step 6. If your machine has 8 GB of total RAM, models in the 1 to 3 billion parameter range (roughly 1 to 4 GB on disk) are a safe starting point. Ollama’s own quickstart guide highlights a larger flagship model, gemma4, whose default download is 9.6 GB, a reasonable choice if you have 16 GB or more of RAM and a bit of patience for the first download, but too large to be a good “first pull” for most beginners.

Step 5: Chat With Your Model From the Terminal

Now the fun part. Start an interactive chat session with:

ollama run llama3.2:1b

If you had not already pulled the model, this same command would download it for you automatically before starting the chat, so ollama pull and ollama run overlap in that sense; we pulled first so you could see the download step and the model metadata separately.

You will land at a >>> prompt. Try asking it something:

>>> What is 12 times 7? Answer in one short sentence.
120.

That reply is real, captured output from this exact model and version, not an invented example. To leave the chat, type:

/bye

A gotcha: typos in model names

Model tags are exact strings, and Ollama will not guess what you meant if you get one wrong. Here is what really happens if you ask for a tag that does not exist, such as llama3.2:99b:

ollama run llama3.2:99b

pulling manifest
Error: pull model manifest: file does not exist

If you ever see that error, double-check the exact family and tag on ollama.com/library rather than assuming Ollama itself is broken.

Step 6: See What Is Loaded in Memory, and Manage It

Ollama does not keep every model in memory all the time. It loads a model into RAM (or VRAM) the first time you use it, and by default keeps it there for 5 minutes of inactivity before unloading it automatically, so that back-to-back requests stay fast without holding memory hostage forever. You can see exactly what is currently loaded with:

ollama ps

Right after asking the model something, this printed:

NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:1b    baf6a787fdff    1.5 GB    100% GPU     4096       4 minutes from now

There are three details worth stopping on here. First, SIZE is 1.5 GB in memory versus 1.3 GB on disk, the overhead we referenced back in Step 4. Second, PROCESSOR shows “100% GPU,” meaning this machine has a supported GPU and Ollama loaded the whole model onto it; on a CPU-only machine you would see “100% CPU” instead, and generation would be noticeably slower but still work correctly. Third, and easy to miss: CONTEXT here is 4096, not the 131,072 maximum we saw in ollama show. That 4096 is Ollama’s runtime default context window, independent of what the model is capable of. If you need a longer conversation window, you set it explicitly per request with the num_ctx option, which is documented in Ollama’s own FAQ.

To unload a model immediately instead of waiting out the 5-minute timer, for example to free up memory before running something else:

ollama stop llama3.2:1b

Running ollama ps again right after that shows an empty list, confirming the model is gone from memory (though still on disk). If you want to remove it from disk entirely and reclaim the space, use ollama rm llama3.2:1b instead.

Where your models live on disk

Ollama stores downloaded model files in a fixed location per operating system:

• Windows: C:\Users\%username%\.ollama\models
• macOS: ~/.ollama/models
• Linux: /usr/share/ollama/.ollama/models

On the machine used for this tutorial, that folder held four downloaded models totalling 5.7 GB, split into a blobs subfolder (the actual weight data, shared and deduplicated across models where possible) and a manifests subfolder (small files describing which blobs make up each named model). If you ever need to free disk space quickly, ollama rm is the supported way to do it; deleting files directly from this folder can leave a model’s manifest pointing at blobs that no longer exist.

Step 7: Talk to Ollama Through Its REST API

Everything you just did with ollama run is actually a thin wrapper around a local HTTP server. This is the important part: any program on your machine, in any language, can talk to that same server, which is exactly how editor plugins, desktop apps, and the Python library in the next step all work under the hood.

With Ollama running, send it a request with curl:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:1b",
  "messages": [{"role": "user", "content": "Name one moon of Jupiter."}],
  "stream": false
}'

The real response, captured from this exact call:

{
  "model": "llama3.2:1b",
  "created_at": "2026-08-29T10:50:36.612Z",
  "message": {
    "role": "assistant",
    "content": "One moon of Jupiter is Io."
  },
  "done": true,
  "done_reason": "stop",
  "total_duration": 657835500,
  "load_duration": 224303300,
  "prompt_eval_count": 31,
  "prompt_eval_duration": 310387000,
  "eval_count": 8,
  "eval_duration": 117415000
}

Every field ending in _duration is measured in nanoseconds, per Ollama’s own API reference, so total_duration: 657835500 means about 0.66 seconds end to end for this request. prompt_eval_count and eval_count tell you how many tokens were in your prompt (31, including the system-level formatting Ollama adds automatically) versus how many the model generated in reply (8, for “One moon of Jupiter is Io.”). Once you can read this response shape, you can measure and debug any Ollama-backed application, not just the ones in this tutorial.

Setting "stream": false made Ollama wait for the full answer before responding once. Leave that out (or set it to true) and Ollama instead sends back a series of small JSON objects, one per generated token, which is what makes a chat interface feel like it is “typing” its answer live. We will use that streaming mode next.

Step 8: Build a Minimal Python Chatbot

Now let’s write actual code. Install Ollama’s official Python client:

pip install ollama

Here is a complete, working chatbot loop. Save it as chatbot.py:

from ollama import chat

MODEL = "llama3.2:1b"
SYSTEM_PROMPT = "You are a concise, friendly assistant. Keep answers short."


def main():
    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    print(f"Chatting with {MODEL}. Type 'quit' to exit.\n")

    while True:
        try:
            user_input = input("You: ")
        except EOFError:
            break
        if user_input.strip().lower() in {"quit", "exit"}:
            break

        messages.append({"role": "user", "content": user_input})

        print("Bot: ", end="", flush=True)
        full_response = ""
        for chunk in chat(model=MODEL, messages=messages, stream=True):
            token = chunk["message"]["content"]
            print(token, end="", flush=True)
            full_response += token
        print()

        messages.append({"role": "assistant", "content": full_response})


if __name__ == "__main__":
    main()

Two design details are worth explaining, not just copying.

First, notice that messages is a plain list we keep appending to, and we send the entire list on every single call to chat(). Ollama itself has no memory of your previous turn: each request is independent. What looks like a model “remembering” the conversation is really just your own code resending the full transcript every time and letting the model read it fresh. If you stopped appending to messages, every question would be answered with no memory of what came before.

Second, the for chunk in chat(..., stream=True) loop is what makes responses appear token by token instead of all at once. Each chunk is one small piece of the reply. In testing, a short factual answer arrived in 8 separate streamed chunks, and a longer one arrived in 15; printing each chunk immediately, with flush=True so Python does not buffer it, is what produces the live “typing” effect.

Running the script and asking two questions in a row produced this real transcript:

Chatting with llama3.2:1b. Type 'quit' to exit.

You: What is the capital of France?
Bot: The capital of France is Paris.
You: How many moons does it have?
Bot: France has two small natural satellites: Phobos and Deimos.
You: quit

Read that second answer carefully: it is wrong, and wrong in an interesting way. “It” correctly resolved to France from context (proving the conversation history is working), but the model then confidently reused the names of Mars’s two real moons, Phobos and Deimos, for a question about a country that obviously has no moons at all. This is not a bug in the code. It is exactly the kind of confident, fluent-sounding mistake small models make: a 1.2 billion parameter model has far less world knowledge packed into it than a frontier cloud model, and nothing in this basic setup checks its answers against anything. If your use case needs answers you can trust, the fix is not a bigger prompt, it is techniques like output validation and retries, which is the entire subject of a separate sxz.io tutorial on getting reliable structured output out of an LLM.

Common Mistakes and Gotchas

• “Command not found” right after installing. Reopen your terminal. The installer updates your system PATH, but windows already open when you installed will not see the change.
• A typo’d model tag fails loudly, not silently. You saw this in Step 5: Error: pull model manifest: file does not exist. Check the exact spelling on ollama.com/library.
• The first message after a few idle minutes feels slow. That is the 5-minute keep-alive timer from Step 6 expiring and Ollama reloading the model into memory. It is not a hang; running ollama ps during that pause will show nothing loaded yet.
• Trusting small local models the way you would trust a frontier model. You saw a real example above. Smaller, more heavily quantized models are faster and lighter, but they know less and double-check nothing on their own.
• Confusing a model’s maximum context with what is actually used. ollama show reported a 131,072-token maximum for our model, but ollama ps showed Ollama was actually running it with a 4,096-token window by default. Set num_ctx explicitly if you need more.
• Picking a model too big for your RAM. Symptoms are extreme slowness or your whole machine feeling sluggish, not a clean error message. Check ollama ps to see whether the model landed on GPU or CPU and how much memory it actually claimed before assuming something is broken.

How to Verify Everything Works End to End

Before moving on, confirm each layer independently:

1. ollama --version returns a version string, not a “command not found” error.
2. ollama list shows at least one downloaded model.
3. ollama run <model> answers a question interactively and /bye exits cleanly.
4. ollama ps shows the model with a PROCESSOR and CONTEXT value while it is loaded, then shows nothing after ollama stop <model>.
5. The curl command from Step 7 returns a JSON object containing a message.content field.
6. Your chatbot.py script holds a two-turn conversation where the second answer correctly reacts to something you said in the first turn, proving the message history is actually being resent.

If all six check out, you have a fully working local AI stack: a model on disk, a service managing memory for you, and two independent ways (terminal and Python) to talk to it.

Next Steps

This tutorial deliberately stayed small. Once Ollama is comfortably installed and you understand the request/response shape from Step 7, these sxz.io tutorials build directly on top of it:

• How to Diagnose and Fix a Low Prompt Cache Hit Rate With Ollama, for making repeated requests faster.
• How to Get Reliable Structured Output From an LLM With Validation and Retries, which directly addresses the hallucination problem you saw in Step 8.
• How to Build a Multi-Tier LLM Router With Automatic Fallback Using Python and Ollama, for combining small fast models with bigger, slower ones.
• How to Correctly Benchmark LLM Inference Latency With Python and Ollama, for measuring performance properly instead of guessing.