How to Fix Vague AI Agent Answers With Prompt and Context Engineering

Notice what changed and what did not. The question is now specific (it names a real service), and the system prompt gives the model a role, a required format, and explicit permission to say “I don’t know” instead of inventing an answer. What still has not changed is the information available: the model has no logs, no error message, nothing beyond a service name. Real output from this exact code:

Root Cause: Insufficient context is provided to determine the specific failure reason without access to service logs or configuration details.
Fix: Retrieve detailed diagnostics using journalctl -u demo-broken-app.service, inspect unit files for syntax errors with systemd-analyze verify, and check system resources via top.
Verification Command: journalctl -xe --unit=demo-broken-app.service

This is a meaningfully better answer than the baseline, and it is worth being precise about why. The model followed the format exactly, gave concrete and correct next steps for someone with no logs in hand yet, and, notably, did not hallucinate a specific cause it could not actually know. That last part is the real payoff of prompt engineering here: a well-engineered prompt does not make the model psychic, but it does make an honest “I don’t know, here’s how to find out” far more likely than a confident-sounding guess. What it still cannot do is name the actual root cause, because that information has never been given to it. That is context engineering’s job, not prompt engineering’s.

Step 5: Add Context Engineering

To demonstrate context engineering properly, you need a real failure to diagnose, not a hypothetical one. Create a small Python script that intentionally crashes because it is missing a required environment variable, a common and realistic misconfiguration:

import os

def main():
    print("Starting invoice-worker...", flush=True)
    database_url = os.environ["DATABASE_URL"]
    print(f"Connecting to {database_url}", flush=True)

if __name__ == "__main__":
    main()

Save this as /opt/demo-broken-app/app.py. Now wire it up as a systemd service, deliberately without setting DATABASE_URL:

sudo mkdir -p /opt/demo-broken-app
sudo tee /etc/systemd/system/demo-broken-app.service > /dev/null << 'EOF'
[Unit]
Description=Invoice worker demo app (intentionally broken for tutorial)

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/demo-broken-app/app.py
Restart=no

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start demo-broken-app.service

systemctl daemon-reload tells systemd to re-read unit files from disk before you try to use a new or changed one; skipping it is a common source of “why isn’t my change taking effect” confusion. Now check what happened:

systemctl status demo-broken-app.service --no-pager -l

Real output from this exact sequence:

× demo-broken-app.service - Invoice worker demo app (intentionally broken for tutorial)
     Loaded: loaded (/etc/systemd/system/demo-broken-app.service; disabled; preset: enabled)
     Active: failed (Result: exit-code) since Sun 2026-07-26 14:48:17 EDT; 5s ago
   Duration: 424ms
    Process: 533507 ExecStart=/usr/bin/python3 /opt/demo-broken-app/app.py (code=exited, status=1/FAILURE)
   Main PID: 533507 (code=exited, status=1/FAILURE)

That tells you the service failed, but not why. For the actual reason, pull the service’s journal:

journalctl -u demo-broken-app.service --no-pager -n 50

Jul 26 14:48:16 vixn systemd[1]: Started demo-broken-app.service - Invoice worker demo app (intentionally broken for tutorial).
Jul 26 14:48:16 vixn python3[533507]: Starting invoice-worker...
Jul 26 14:48:17 vixn python3[533507]: Traceback (most recent call last):
Jul 26 14:48:17 vixn python3[533507]:   File "/opt/demo-broken-app/app.py", line 10, in <module>
Jul 26 14:48:17 vixn python3[533507]:     main()
Jul 26 14:48:17 vixn python3[533507]:   File "/opt/demo-broken-app/app.py", line 6, in main
Jul 26 14:48:17 vixn python3[533507]:     database_url = os.environ["DATABASE_URL"]
Jul 26 14:48:17 vixn python3[533507]:                    ~~~~~~~~~~^^^^^^^^^^^^^^^^
Jul 26 14:48:17 vixn python3[533507]: KeyError: 'DATABASE_URL'
Jul 26 14:48:17 vixn systemd[1]: demo-broken-app.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 14:48:17 vixn systemd[1]: demo-broken-app.service: Failed with result 'exit-code'.

Now you have real context: an actual Python traceback showing exactly which environment variable is missing. Feed both of these real outputs into the same prompt-engineered agent from Step 4, changed only by adding this data to the user message:

context_message = f"""My systemd service demo-broken-app.service failed to start. Why?

Here is the output of `systemctl status demo-broken-app.service`:
{status_output}

Here is the output of `journalctl -u demo-broken-app.service --no-pager -n 50`:
{journalctl_output}"""

context_agent = build_agent(system_prompt=PROMPT_ENGINEERED_SYSTEM)
result = context_agent.invoke({"messages": [{"role": "user", "content": context_message}]})
print(result["messages"][-1].content)

Here is real, unedited output from the first attempt at running that exact code:

Root Cause: The application crashed because the required environment variable DATABASE_URL was missing from the process context when Python executed under systemd.
Fix: Add an [Environment] section to /etc/systemd/system/demo-broken-app.service containing DATABASE_URL=your_connection_string, then run systemctl daemon-reload && systemctl restart demo-brown-app.service. (Wait, typo in thought again). Correct name is broken.

Let’s fix the typos and ensure strict adherence.
Fix: Add an [Environment] section […] then run systemctl daemon-reload && systemctl restart demo-brown-app.service. (Wait, I keep typing brown instead of broken). Okay, stop that mental slip.

[…continues second-guessing itself over “broken” versus “brown” for several more paragraphs without ever completing a clean final answer…]

Look closely at the first sentence, and this is worth sitting with for a second: the root cause is exactly correct, identified immediately, in the first sentence, with the specific missing variable named. Context engineering did its job. But the response then falls apart into a rambling, self-correcting loop about whether the service is named “broken” or “brown”, something neither the logs nor the question ever actually said. This is a real gotcha, not a fabricated one, and it is worth understanding rather than just retrying until it goes away.

Diagnosing the Gotcha: Context Window Size

The model itself was never confused about the file name; every quoted command above correctly says demo-broken-app.service. What happened is that the response got cut off mid-generation and looped back on itself, which is a symptom of running out of room, not a reasoning failure. Check how much context Ollama actually allocated for this model:

curl -s http://localhost:11434/api/show -d '{"model":"qwen3.5:4b"}' | python3 -c "
import json, sys
d = json.load(sys.stdin)
print('max supported:', d['model_info']['qwen35.context_length'])
"

max supported: 262144

The model architecture supports up to 262,144 tokens of context, but that is not what actually gets allocated. The num_ctx parameter on ChatOllama defaults to None, meaning it does not request a specific context size at all; when that happens, Ollama’s server picks a default based on available VRAM. Per Ollama’s own documentation, machines with less than 24 GiB of VRAM default to a 4k (4096 token) context window, and a CPU-only machine with no dedicated GPU, like the one this tutorial was built on, falls into that same bucket. You can confirm exactly what a running model is using with ollama ps, which shows a CONTEXT column.

Qwen3.5 is also a “thinking” model: by default it spends tokens on hidden reasoning before it writes the visible answer you actually see. Add a system prompt, a full journalctl dump, and a systemctl status block to that hidden reasoning process, and a 4096-token window can fill up before the model ever finishes writing its final answer, which is exactly what produces this kind of derailed, looping output.

There are two independent ways to fix this. You can give the reasoning more room by raising num_ctx (8192 or higher). Or, since this task only needs a short, formatted final answer and does not benefit from visible step-by-step reasoning, you can turn thinking off entirely with Ollama’s documented think parameter, exposed in ChatOllama as reasoning:

context_agent = build_agent(
    system_prompt=PROMPT_ENGINEERED_SYSTEM,
    num_ctx=4096,
    reasoning=False,
)

This is the cleaner fix for a task like this one: it removes the hidden reasoning tokens that were consuming the context budget in the first place, rather than just giving them more room to expand into. Re-running the exact same prompt and the exact same context, with only that change, produces a fast, clean, complete answer:

Root Cause: The application crashed because the required environment variable DATABASE_URL was not defined in its execution context.
Fix: Export or set the DATABASE_URL environment variable before running the service (e.g., via .env, systemd drop-in, or manual export) and reload/restart the unit with systemctl daemon-reload && systemctl restart demo-broken-app.service.
Verification Command: journalctl -u demo-broken-app.service --no-pager | grep "Starting invoice-worker"

This is the complete picture. Prompt engineering got the model to follow a format and admit uncertainty honestly instead of guessing. Context engineering, in the form of real log data, got the model to the correct root cause immediately, in every single run, regardless of whether the final formatting came out clean. And a third, easy-to-miss detail, how much of a small local model’s limited context window is quietly consumed by hidden reasoning, determined whether that correct answer arrived as a clean final response or got lost in a loop. All three matter, and none of them are visible from the outside until you go looking.

Step 6: Verify the Diagnosis Was Actually Correct

An AI agent’s suggested fix is only a hypothesis until you have actually tested it. Never apply a suggested fix to a real system without verifying it worked. Apply the fix the agent recommended, setting DATABASE_URL in the unit file:

sudo tee /etc/systemd/system/demo-broken-app.service > /dev/null << 'EOF'
[Unit]
Description=Invoice worker demo app (intentionally broken for tutorial)

[Service]
Type=simple
Environment=DATABASE_URL=postgresql://invoice_worker@localhost:5432/invoices
ExecStart=/usr/bin/python3 /opt/demo-broken-app/app.py
Restart=no

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start demo-broken-app.service

The Environment= directive is the standard systemd way to pass an environment variable to a service’s process; it is documented in the official systemd.exec manual page. Check the result:

systemctl status demo-broken-app.service --no-pager -l

○ demo-broken-app.service - Invoice worker demo app (intentionally broken for tutorial)
     Loaded: loaded (/etc/systemd/system/demo-broken-app.service; disabled; preset: enabled)
     Active: inactive (dead)

journalctl -u demo-broken-app.service --no-pager -n 5

Jul 26 14:48:32 vixn systemd[1]: Started demo-broken-app.service - Invoice worker demo app (intentionally broken for tutorial).
Jul 26 14:48:32 vixn python3[533661]: Starting invoice-worker...
Jul 26 14:48:32 vixn python3[533661]: Connecting to postgresql://invoice_worker@localhost:5432/invoices
Jul 26 14:48:32 vixn systemd[1]: demo-broken-app.service: Deactivated successfully.

The service now starts, connects using the environment variable you supplied, and exits cleanly (this demo script does not loop or daemonize, so “inactive (dead)” after a successful run is expected, not a new failure). The diagnosis was correct. Clean up the demo service when you are done experimenting:

sudo systemctl stop demo-broken-app.service
sudo systemctl disable demo-broken-app.service
sudo rm -f /etc/systemd/system/demo-broken-app.service
sudo systemctl daemon-reload
sudo systemctl reset-failed
sudo rm -rf /opt/demo-broken-app

Common Mistakes and Gotchas

Treating a Silent Truncation as a Reasoning Failure

As Step 5 demonstrated directly, a response that trails off, repeats itself, or loops is often a context window problem, not evidence that the model “can’t reason.” Before concluding a model is unreliable, check the actual allocated context with ollama ps, compare it against how much text you are sending (system prompt, injected context, and hidden reasoning for “thinking” models all count), and either raise num_ctx or turn thinking off for tasks that only need a short final answer.

Confusing the Two Techniques

It is tempting to treat “write a better prompt” and “give the model more information” as the same lever turned up or down. They are not. A beautifully worded prompt cannot substitute for missing facts, and dumping in more raw data will not fix a vague, unstructured question. Diagnose which one you are actually short on before reaching for either.

Context Pollution

More context is not automatically better context. Pasting an entire log file, an entire codebase, or an entire ticket history at a model can bury the one relevant line under thousands of irrelevant ones, and every extra token also eats into the context budget described above. Include what is relevant to the task, not everything you have.

Trusting a Confident-Sounding Answer Without Verification

Step 6 was not optional filler. An AI agent can state a wrong root cause with exactly the same confident tone as a correct one. Always verify a suggested fix against the real system, the same way you would verify a suggestion from a colleague.

How to Verify It All Worked

You have completed this tutorial successfully if:

• ollama list shows qwen3.5:4b (or your chosen model) installed locally
• The baseline agent (Step 3) produced a vague, clarifying-question style answer
• The prompt-engineered agent (Step 4) produced a correctly formatted answer that honestly admitted it lacked enough information, rather than guessing
• The context-engineered agent (Step 5) named DATABASE_URL as the specific missing variable in every run, and produced a clean, complete, correctly formatted answer once thinking was disabled (or given a larger context window)
• Applying the suggested fix (Step 6) made the demo service start and exit cleanly instead of failing

If your context-engineered output still looks truncated or repetitive after setting reasoning=False, double check you are running langchain-ollama>=1.1.0 so the parameter is actually supported, or fall back to raising num_ctx further (16384 is a safe next step for most local hardware).

Next Steps

From here, a few natural directions to keep building on what you have learned: