How to Serve a Multi-User AI Agent With FastAPI and Streamlit

A local AI agent that only runs in one terminal window is a demo, not a service. The moment someone else wants to use it, or you want two conversations happening at once without them bleeding into each other, a Python script that reads from input() stops being enough. You need an interface other programs and other people can call, and you need each caller’s conversation kept separate from everyone else’s.

This tutorial teaches you how to do both. You will take a small local AI agent built with LangChain and Ollama, wrap it in a FastAPI REST API so it can be called over HTTP, and put a Streamlit chat interface on top so it is usable from a browser. Along the way you will learn what each of those three tools actually does, why a chat app needs the concept of a “session,” and how to keep one user’s conversation from leaking into another user’s.

Every command and every line of code in this guide was written into real files and run on a live machine while writing this tutorial, backend and frontend both, including the full multi-user isolation test in Step 5. The exact model responses shown are real output from that run, not invented examples, so if your local model gives slightly different wording, that is normal; language models are not perfectly deterministic even at low temperature. This guide targets Python 3.11 on Linux; the same code also runs unmodified on macOS and Windows, the only difference being how you activate a virtual environment (noted in Step 2).

What You Will Build

Before writing any code, it helps to understand the shape of the finished system and a few terms this tutorial leans on repeatedly.

An API (Application Programming Interface) is simply a defined way for one piece of software to ask another for something. A REST API is a very common style of API built on plain HTTP requests, the same protocol your browser uses to load web pages, where a client sends a request to a URL and gets a response back. Building a REST API in front of the agent means any program that can make an HTTP request, a browser, a phone app, another script, can now use it, instead of only whoever has the terminal open.

An AI agent, in the LangChain sense used here, is a language model wired up to a set of tools (ordinary Python functions) it can choose to call. Instead of only generating text, the model can decide “I need the current time for this” and call a current_time() function to get a real answer, then use that result to finish its reply.

A session is the state tied to one particular conversation: the back-and-forth messages that give a chat its memory. Multi-user support means the system can hold many sessions open at once, one per user, without any of them seeing or overwriting each other’s messages. Without it, every visitor to your app would appear to share one giant, constantly overwritten conversation, which is confusing at best and a privacy problem at worst.

The finished architecture has three pieces working together:

• Ollama runs a language model directly on your machine and exposes it locally, so there is no per-request API bill and no data leaving your computer.
• FastAPI exposes one HTTP endpoint, /chat/stream. It validates incoming requests, hands the message to the LangChain agent, and streams the agent’s reply back token by token instead of making the caller wait for the full answer.
• Streamlit is a thin browser-based frontend that calls that endpoint and renders the conversation as a chat interface, giving each browser tab its own random session ID so the backend can tell users apart.

Prerequisites

• Python 3.10 or newer. This guide was written and tested against Python 3.11.15 on Ubuntu 26.04 LTS; the original tutorial this is adapted from was tested on macOS. Any of Linux, macOS, or Windows works.
• Ollama installed on your machine, covered in Step 1.
• At least 4 GB of free RAM for the qwen3.5:4b model used in this guide. If your machine has less, Step 1 shows a smaller model you can swap in instead.
• Basic comfort with the command line and with reading Python. No prior FastAPI, Streamlit, or LangChain experience is assumed; every new concept is explained the first time it comes up.
• No API keys and no cloud account are required. Everything in this tutorial runs entirely on your own machine.

Step 1: Install Ollama and Pull a Model

Ollama is a tool that downloads and runs open-weight language models locally, and exposes them through a simple local API that libraries like LangChain know how to talk to. Install it for your platform from the official download page, then pull the chat model this tutorial uses:

# Context: any machine with Ollama installed (Linux, macOS, or Windows).
# Purpose: download the qwen3.5:4b model so ChatOllama can use it locally.
ollama pull qwen3.5:4b

Expected output: a series of progress bars for the model’s manifest and layers, ending in a line similar to success. This tutorial was verified against Ollama 0.31.2 and a 3.4 GB download for qwen3.5:4b. If your machine has less than 4 GB of free RAM, pull the smaller qwen3.5:0.8b instead and use that model name everywhere qwen3.5:4b appears in this guide; the code does not change, only the model tag.

Confirm the model is actually available before moving on:

ollama list

Expected output: a table including a row for qwen3.5:4b with its ID and size. If the table is empty or the row is missing, the pull above did not finish; rerun it before continuing.

Step 2: Create a Project and Install Python Dependencies

Create a project folder and an isolated Python environment, so this tutorial’s packages do not collide with anything else on your machine:

# Context: a fresh project directory, Python 3.10+ already installed.
# Purpose: isolate this project's dependencies, then install every package the backend and frontend need.
mkdir agent-service && cd agent-service
python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn streamlit requests langchain langchain-core langchain-ollama langgraph

On Windows, activate the environment with venv\Scripts\activate instead of the source venv/bin/activate line above; everything else is identical.

Expected output: pip resolving and installing each package, ending in a Successfully installed ... line. This tutorial was tested against fastapi 0.139.2, uvicorn 0.51.0, streamlit 1.59.2, requests 2.34.2, langchain 1.3.14, langchain-core 1.5.0, langchain-ollama 1.1.0, and langgraph 1.2.9. You need langchain>=1.0.0 at minimum: create_agent, used in the next step, is a LangChain v1 API and does not exist in older 0.x releases.

Confirm the install worked before writing any code:

python3 -c "from langchain.agents import create_agent; from langgraph.checkpoint.memory import InMemorySaver; print('ok')"

Expected output: a single line reading ok. A ModuleNotFoundError here almost always means the virtual environment was not activated in the current shell; rerun the source venv/bin/activate (or venv\Scripts\activate on Windows) command and try again.

Step 3: Build the Agent and API Layer With FastAPI

This file has three jobs. FastAPI exposes an HTTP endpoint, Pydantic validates whatever arrives at that endpoint, and LangChain runs the actual agent loop, including deciding when to call a tool and remembering each user’s own conversation. Save the following as app.py:

from datetime import datetime
from uuid import UUID

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_ollama import ChatOllama
from langgraph.checkpoint.memory import InMemorySaver

CHAT_MODEL = "qwen3.5:4b"
SYSTEM_PROMPT = (
    "You are a helpful assistant with access to tools for getting the current time "
    "and counting words in text. "
    "Use tools when needed. If the question does not need a tool, answer directly."
)


class ChatRequest(BaseModel):
    user_id: UUID
    message: str


@tool
def current_time() -> str:
    """Return the current local date and time."""
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")


@tool
def word_count(text: str) -> int:
    """Count the number of words in a piece of text."""
    return len(text.split())


checkpointer = InMemorySaver()


def build_agent():
    model = ChatOllama(model=CHAT_MODEL, temperature=0)
    return create_agent(
        model=model,
        tools=[current_time, word_count],
        system_prompt=SYSTEM_PROMPT,
        checkpointer=checkpointer,
    )


agent = build_agent()

app = FastAPI()


@app.post("/chat/stream")
def chat_stream(req: ChatRequest):
    def generate():
        run = agent.stream_events(
            {
                "messages": [{"role": "user", "content": req.message}],
            },
            config={
                "configurable": {
                    "thread_id": str(req.user_id),
                }
            },
            version="v3",
        )
        for message in run.messages:
            for token in message.text:
                yield token

    return StreamingResponse(generate(), media_type="text/plain")

A few pieces are worth understanding line by line, since most of them reappear in any LangChain agent you build later:

The @tool decorator, imported from langchain_core.tools, turns an ordinary Python function into something the model is allowed to call. The docstring on each tool is not a comment for humans; LangChain sends it to the model as that tool’s description, so the model uses it to decide when the tool is relevant. That is why current_time and word_count both have short, literal docstrings instead of no docstring at all.

ChatRequest, a Pydantic model, defines the shape of every incoming request: a user_id that must be a valid UUID, and a plain string message. FastAPI reads this class and automatically rejects any request missing either field, or sending a user_id that is not a real UUID, before your own code ever runs.

InMemorySaver, imported from langgraph.checkpoint.memory, is a checkpointer: an object LangChain uses to save and reload conversation state, keyed by a thread_id string. This code passes str(req.user_id) as that thread_id, which is the entire mechanism behind multi-user support here. Every request from the same user_id shares one thread’s history; a different user_id gets a completely separate thread, with no shared state at all. As the name implies, InMemorySaver keeps everything in the Python process’s memory, so restarting the backend erases every conversation; the “What to Improve Before Production” section later in this guide covers swapping in something durable.

The agent is built once, at import time, with agent = build_agent(), not inside the request handler. Rebuilding the model and tool list on every single request would add unnecessary latency; one shared agent instance, backed by a checkpointer that already separates state by thread, safely serves every user.

Inside chat_stream, agent.stream_events(..., version="v3") is the part most likely to look unfamiliar. It returns an object your code iterates to drive the run, rather than a plain list of strings. Iterating run.messages yields one item per chat-model turn in the agent’s run, and each item’s .text attribute is itself iterable, yielding the model’s output piece by piece as it is generated. That inner for token in message.text: loop is what turns a normal, wait-for-the-whole-answer model call into a live stream. StreamingResponse, from fastapi.responses, then wraps that generator so FastAPI sends each token to the client as soon as it is produced, instead of buffering the entire reply first. This is LangGraph’s newer streaming interface (the version="v3" argument selects it over the older "v1"/"v2" event formats); as of langgraph 1.2.9 it is documented as experimental and may still change in a future release, so pin your langgraph version if you depend on this exact shape in production.

Step 4: Build the Streamlit Frontend

With a working API, the frontend’s only job is to call it and display the result as a chat. Save the following as streamlit_app.py, in the same project folder as app.py:

import uuid

import requests
import streamlit as st

API_URL = "http://127.0.0.1:8001/chat/stream"

st.title("Local AI Agent")

if "user_id" not in st.session_state:
    st.session_state.user_id = str(uuid.uuid4())

if "chat_history" not in st.session_state:
    st.session_state.chat_history = []

for item in st.session_state.chat_history:
    with st.chat_message(item["role"]):
        st.markdown(item["content"])

message = st.chat_input("Enter a message")

if message:
    st.session_state.chat_history.append({"role": "user", "content": message})
    with st.chat_message("user"):
        st.markdown(message)

    full_answer = ""
    with st.chat_message("assistant"):
        placeholder = st.empty()

        with requests.post(
            API_URL,
            json={
                "message": message,
                "user_id": st.session_state.user_id,
            },
            stream=True,
        ) as response:
            response.raise_for_status()
            for chunk in response.iter_content(chunk_size=None, decode_unicode=True):
                if chunk:
                    full_answer += chunk
                    placeholder.markdown(full_answer)

    st.session_state.chat_history.append(
        {"role": "assistant", "content": full_answer}
    )

Streamlit reruns this entire script top to bottom on every user interaction, so it needs somewhere to keep state between reruns; that place is st.session_state, a dictionary-like object Streamlit persists for the lifetime of one browser tab. The first time the script runs for a given tab, it generates a random UUID into st.session_state.user_id and an empty list into st.session_state.chat_history; on every later rerun in that same tab, the if ... not in st.session_state checks are false, so the existing values are reused instead of overwritten. That per-tab user_id is exactly the value sent as user_id to the FastAPI backend, which is what lets one browser tab’s conversation stay separate from another’s.

st.chat_input and st.chat_message are Streamlit’s built-in chat widgets: the first renders a message box pinned to the bottom of the page, the second renders one message bubble styled for either the "user" or "assistant" role. The loop near the top replays chat_history on every rerun so earlier messages stay visible instead of disappearing each time Streamlit reruns the script.

The requests.post(..., stream=True) call, combined with response.iter_content(...), is the frontend half of streaming: instead of blocking until the backend’s full reply is ready, it reads the HTTP response as chunks arrive and appends each one to full_answer, calling placeholder.markdown(full_answer) after every chunk so the growing answer redraws live on the page.

Step 5: Run the Backend and Test It

Start the API with Uvicorn, an ASGI server that runs FastAPI applications:

# Context: agent-service/, venv activated, app.py saved from Step 3.
# Purpose: start the FastAPI backend on port 8001 with auto-reload for development.
uvicorn app:app --reload --port 8001

Expected output:

INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
INFO:     Started server process [...]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Open http://127.0.0.1:8001/docs in a browser. FastAPI generates this interactive documentation page automatically from the ChatRequest model and the /chat/stream route, and it confirmed 200 OK in testing for this guide; you can send a test request directly from that page without writing any client code. Visiting the bare root address, http://127.0.0.1:8001/, is not useful here and will return a plain {"detail":"Not Found"} with a 404 status; that is expected, since this app never defines a route at /, only at /chat/stream and the auto-generated /docs.

With the server running, open a second terminal (leave Uvicorn running in the first one) and send a real request:

# Context: second terminal, backend from above already running on port 8001.
# Purpose: exercise the tool-calling path by asking a question the agent must use a tool to answer.
curl -X POST http://127.0.0.1:8001/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"message":"What time is it?","user_id":"123e4567-e89b-12d3-a456-426614174000"}'

Expected output: here is the real, unedited response this exact command produced while writing this tutorial:

The current time is July 21, 2026 at 02:42:03 AM local time.

Your own output will show your machine’s actual time instead, but it should read as a similar sentence built around a real clock reading, confirming the agent actually called the current_time tool rather than guessing. Try the word-count tool the same way:

curl -X POST http://127.0.0.1:8001/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"message":"How many words are in: LangChain makes tool calling easier","user_id":"123e4567-e89b-12d3-a456-426614174000"}'

Expected output (also a real, unedited capture):

There are **5** words in: "LangChain makes tool calling easier"

And a plain question that needs no tool at all:

curl -X POST "http://127.0.0.1:8001/chat/stream" \
  -H "Content-Type: application/json" \
  -d '{"message":"What is the capital of France?","user_id":"123e4567-e89b-12d3-a456-426614174000"}'

Paris is the capital of France. It's located in north-central France and has been the country's political, cultural, economic, and educational center for centuries.

Now confirm the part that actually matters for a multi-user service: that two different user_id values get two completely separate memories. Send these four requests in order, swapping in two different UUIDs for the two “users”:

# Context: same running backend; USER_A and USER_B are two different UUIDs representing two separate people.
# Purpose: prove conversation memory is isolated per user_id, not shared globally.
curl -X POST http://127.0.0.1:8001/chat/stream -H "Content-Type: application/json" \
  -d '{"message":"My name is Alice. Just remember that.","user_id":"11111111-1111-1111-1111-111111111111"}'

curl -X POST http://127.0.0.1:8001/chat/stream -H "Content-Type: application/json" \
  -d '{"message":"My name is Bob. Just remember that.","user_id":"22222222-2222-2222-2222-222222222222"}'

curl -X POST http://127.0.0.1:8001/chat/stream -H "Content-Type: application/json" \
  -d '{"message":"Who am I?","user_id":"11111111-1111-1111-1111-111111111111"}'

curl -X POST http://127.0.0.1:8001/chat/stream -H "Content-Type: application/json" \
  -d '{"message":"Who am I?","user_id":"22222222-2222-2222-2222-222222222222"}'

Expected output for the two “Who am I?” calls, again a real captured run:

You are Alice! That's who you told me your name was in our conversation.

You are Bob! That's who you told me your name is at the beginning of our conversation, and I've remembered that for us.

If both answers instead mention the same name, or an error, the most likely cause is reusing one UUID across every request by copy-paste mistake; double-check each command uses the UUID for the intended “user.”

Step 6: Run the Frontend and Try It in a Browser

Leave the backend running. In a new terminal, activate the same virtual environment and start Streamlit:

# Context: a new terminal, same agent-service/ project directory, backend still running from Step 5.
# Purpose: start the Streamlit chat UI on its default port.
source venv/bin/activate
streamlit run streamlit_app.py

Expected output: Streamlit prints a local URL, normally http://localhost:8501, and opens it in your default browser automatically. In testing for this guide, the app’s root page returned 200 OK with no startup errors.

Type a message like “What is the capital of France?” into the chat box at the bottom of the page. You should see your message appear, followed by the assistant’s reply streaming in progressively rather than popping in all at once, the visible effect of the token-by-token streaming built in Steps 3 and 4. To see multi-user isolation from inside the UI itself rather than curl, open the same localhost:8501 address in a second browser (or a private/incognito window, which gets a fresh st.session_state) and have a different conversation there; each tab keeps its own user_id and therefore its own memory, exactly like the two curl “users” in Step 5. To stop either process, press Ctrl+C in its terminal.

Common Mistakes and Gotchas

• Forgetting to activate the virtual environment in the second or third terminal. Each new terminal starts fresh; if a command that worked a moment ago suddenly reports ModuleNotFoundError, run source venv/bin/activate (or venv\Scripts\activate on Windows) again in that shell before anything else.
• Starting the backend before pulling the model. ChatOllama only contacts Ollama when a request actually arrives, so the server itself starts fine either way; the first chat request then fails with a model-not-found style error from Ollama’s API. Run ollama list to confirm qwen3.5:4b is present, and ollama pull qwen3.5:4b again if not.
• Reusing one user_id for every request. This is the single easiest way to make multi-user support silently fail: if every client sends the same UUID (or none at all), every conversation lands in the same checkpointer thread and every “user” appears to share one memory. Each genuinely separate user needs its own UUID.
• Restarting the backend and expecting history to survive. InMemorySaver keeps state only in the running process’s memory. A restart, whether manual or triggered by --reload picking up a code change, silently clears every conversation. That is expected behavior for this tutorial’s scope, not a bug; see the next section for what to use instead in production.
• Port already in use. If uvicorn or streamlit refuses to start with an “address already in use” style error, another process (possibly a previous run you forgot to stop) is already bound to port 8001 or 8501. Stop that process, or pass a different --port to either command and update API_URL in streamlit_app.py to match.

What to Improve Before Production

The application in this tutorial is fully functional but intentionally minimal, meant to teach the moving parts clearly rather than to be deployed as-is. Before putting something like it in front of real users, consider adding:

• Authentication. Nothing in this tutorial’s API checks who is calling it; anyone who can reach /chat/stream can supply any user_id and read or continue that thread’s conversation. A real deployment needs to verify identity and tie user_id to an authenticated session, not accept it as a free-form client-supplied value.
• Persistent, durable memory. Swap InMemorySaver for one of LangGraph’s persistent checkpointer implementations (backed by SQLite, Postgres, or Redis, depending on your infrastructure) so conversations survive a restart or a deploy.
• Structured logging and monitoring, so you can see request volume, latency, and error rates instead of only Uvicorn’s default access log lines.
• A production-grade deployment setup: running Uvicorn behind a process manager and reverse proxy, without --reload, with TLS terminated in front of it, rather than the bare development server used here.

It is also worth knowing when not to build this yourself. If your actual goal is a polished, full-featured self-hosted chat UI rather than understanding how the pieces fit together, projects like LibreChat and Open WebUI already provide one, with more features than this tutorial’s Streamlit frontend. Building it by hand, as done here, trades that off for a clear view of exactly how the request, the agent, and the streamed response fit together.

Verifying Everything Works End to End

1. With the backend running, http://127.0.0.1:8001/docs loads and returns 200 OK.
2. The three single-user curl requests from Step 5 each return a coherent answer: a real clock time for the time question, the correct count of 5 for the word-count question, and a Paris-based answer for the geography question.
3. The four-request Alice/Bob sequence from Step 5 returns two different names, each user correctly recalling only their own introduction.
4. With both the backend and streamlit run streamlit_app.py running, http://localhost:8501 loads, a typed message appears in the chat, and the assistant’s reply visibly streams in rather than appearing all at once.
5. Two separate browser windows (or one normal window plus one private/incognito window) pointed at the same Streamlit URL can hold two different conversations without either one seeing the other’s messages.

If all five checks pass, you have a working, session-isolated, streaming AI agent service, callable by anything that can make an HTTP request and usable from a browser.

Next Steps

• Replace InMemorySaver with a persistent LangGraph checkpointer and confirm a conversation survives a backend restart.
• Add a third tool to the agent, beyond current_time and word_count, and watch how the model decides when to call it based only on your new docstring.
• Put a lightweight API key check in front of /chat/stream as a first step toward real authentication, before deciding on a full auth system.
• Read LangGraph’s own documentation on the version="v3" streaming protocol used in Step 3, since it is explicitly marked experimental and worth tracking as it stabilizes.

Sources: freeCodeCamp: How to Serve a Multi-User AI Agent with FastAPI and Streamlit (Darsh Shah); FastAPI documentation; Streamlit documentation; LangChain documentation: Agents overview and create_agent reference; Ollama Library: qwen3.5. Featured image: photograph by Bench Accounting, CC0 1.0 Universal (public domain), via Wikimedia Commons.