AI Governance for Agentic Apps: A Practical Checklist for Builders

AI agents stop being a lab experiment the moment they can read production data, call tools, open tickets, change code, or make decisions that affect users. Governance is the practical layer that decides what those systems may do, who owns the risk, and what evidence proves they behaved within bounds.

Docker frames AI governance as the frameworks, policies, and controls used to build, deploy, and oversee AI systems responsibly, and its own article ties the topic to the current jump in agent adoption and the security/compliance concerns that follow it. Docker NIST’s AI Risk Management Framework makes the same point in more neutral language: the framework is meant to help organizations manage AI risks and promote trustworthy and responsible AI development and use. NIST AI RMF 1.0

Start with ownership, not paperwork

The first governance mistake is treating AI oversight as a document someone signs after the build is done. NIST describes GOVERN as a cross-cutting function that should be infused throughout AI risk management, not a final approval step. NIST AI RMF 1.0

For a builder, that means every agent or AI-assisted workflow needs a named owner before it receives real access. The owner should be accountable for the use case, model choice, data exposure, tool permissions, failure mode, monitoring plan, and retirement criteria.

The minimum ownership record

• Business owner: who benefits from the system and accepts operational risk.
• Technical owner: who can change prompts, tools, model settings, and deployment code.
• Data owner: who approves the datasets, logs, files, or systems the agent may touch.
• Security owner: who signs off on tool permissions, isolation, monitoring, and incident response.
• Review cadence: when the use case is re-evaluated after model, data, tool, or business changes.

Map the agent before you govern it

NIST’s MAP function is about establishing the context needed to frame AI-system risk. NIST AI RMF 1.0 For agentic apps, the context is not just “which model are we using?” It is the full chain of inputs, retrieved data, tools, permissions, logs, human approvals, and downstream systems.

Draw the action graph

A useful governance review starts with an action graph: what can the agent read, what can it write, what can it execute, and what can it trigger indirectly through other tools. This matters because OWASP defines prompt injection as a case where user prompts alter an LLM’s behavior or output in unintended ways, including inputs that may not be human-visible but are still parsed by the model. OWASP LLM01 Prompt Injection

If an agent can read email, summarize a document, and then call a deployment tool, the governance boundary is not the chat box. It is the combination of email content, document retrieval, tool invocation, network access, and deployment authority.

Classify data by consequence

Data labels should be operational, not decorative. “Public,” “internal,” “customer confidential,” “regulated,” and “secret” are only useful if they directly control retrieval, logging, retention, tool access, and human approval requirements.

• Public data can be used for low-risk drafting and summarization.
• Internal data should require workspace-level authorization and retention rules.
• Customer or regulated data should require explicit purpose, minimization, access logging, and approval gates.
• Secrets should not enter prompts, retrieved context, traces, screenshots, or generated output.

Control what the agent can do

OpenAI’s safety guidance emphasizes measures such as adversarial testing, human oversight, and careful deployment controls for responsible AI applications. OpenAI safety best practices Anthropic’s tool-use documentation also makes clear that tools connect a model to external APIs and systems, which means tool design is part of the security boundary. Anthropic tool-use docs

Use least privilege for tools

Every tool should have a narrow purpose, a narrow permission set, and a narrow input schema. A “run_shell_command” tool is rarely governable in production. A “create_read_only_jira_summary” tool or “open_low_priority_ticket_with_template” tool is much easier to inspect, test, and revoke.

Separate read, suggest, and act

The safest rollout path is usually read-only first, suggestion-only second, and action-taking last. The agent can observe and recommend before it is allowed to mutate data or trigger external systems. High-impact actions should remain behind human approval until the team has enough logs, tests, and incident drills to justify automation.

Put policy near execution

Do not rely only on a system prompt that says “be careful.” Enforce policy close to the tool call: schema validation, allowlists, deny lists, rate limits, spend limits, sandboxing, approval gates, and separate credentials for each capability. Prompt instructions are useful, but runtime controls are what make governance auditable.

Measure behavior continuously

NIST’s MEASURE function calls for quantitative, qualitative, or mixed-method tools to analyze, assess, benchmark, and monitor AI risk and impact. NIST AI RMF 1.0 For agentic apps, that means the test suite cannot stop at “the answer looked good.”

What to measure

• Task success rate: did the agent complete the intended job without manual rescue?
• Policy violation rate: did it request forbidden data, tools, or actions?
• Tool-call accuracy: did it call the right tool with the right arguments?
• Data exposure: did sensitive content enter prompts, traces, logs, or outputs?
• Human override rate: how often did reviewers reject or modify the agent’s recommendation?
• Cost and latency: did the workflow stay within expected token, runtime, and infrastructure budgets?

These metrics should be reviewed after model changes, prompt changes, retrieval changes, and tool changes. A model upgrade is a production change, not a background detail.

Keep evidence for audits and incidents

Docker’s governance article highlights auditability, observability, access control, and lifecycle oversight as practical governance concerns for teams shipping AI systems. Docker Evidence is what turns a governance program from a slide deck into an operating system.

Evidence to keep

• Approved use case and risk tier.
• Model, prompt, tool, and retrieval configuration versions.
• Data classes the agent may access.
• Tool permissions and credential scopes.
• Evaluation results and red-team findings.
• Human approvals for high-risk actions.
• Incident response contacts and rollback steps.
• Retention and deletion rules for prompts, outputs, traces, and logs.

A practical pre-production checklist

Govern

• Name the business, technical, data, and security owners.
• Define the use case, risk tier, success criteria, and stop conditions.
• Approve the data classes, tools, models, and deployment environment.

Map

• Diagram all inputs, retrieval sources, tools, outputs, and downstream systems.
• Identify prompt-injection paths through documents, webpages, emails, tickets, and tool output.
• Record what the agent can read, suggest, write, delete, buy, deploy, or message.

Measure

• Create golden-task evaluations and adversarial tests before launch.
• Test tool-call arguments, not just final natural-language answers.
• Track policy violations, overrides, cost, latency, and sensitive-data exposure.

Manage

• Start read-only, then suggestion-only, then limited action-taking.
• Use scoped credentials, rate limits, network limits, and approval gates.
• Publish rollback and incident-response steps before the agent touches production.

The goal is not to slow every AI project into a committee. The goal is to make agentic work repeatable: the same kind of ownership, controls, evidence, and review that mature teams already expect from production software.

python -m pytest tests/agent_evals tests/policy_gates -q
python -m pytest tests/redteam/test_prompt_injection.py -q

docker build --pull --tag agentic-app:governance-review .
docker image inspect agentic-app:governance-review --format '{{json .Config.Env}}'
docker history --no-trunc agentic-app:governance-review