Red Hat’s 6 AM Incident Shows Why Agent Frameworks Need a Platform Layer

Red Hat says an AI agent deployment failed three different ways in a single overnight shift, and none of the three failures were really the agent’s fault. According to a companion post on Red Hat’s blog, a support-queue agent built on LangChain called a ticketing API to log a customer case. The API created the ticket, but the response back to the agent timed out. From the agent’s point of view the call had failed, so the framework did what a well-built framework is supposed to do: it retried. The retry created a second ticket, and the same pattern repeated until the queue held 43 duplicate tickets by morning. In the same shift, the agent charged $4,000 to the wrong billing account because it was running with credentials broad enough to reach an account it should never have touched, and it told a customer the return window was 90 days instead of the real 30, a hallucinated policy that cost the company an unapproved $280 refund.

Red Hat’s point, laid out in a follow-up post titled “Why your AI agent framework isn’t enough: 7 platform capabilities missing from production,” is that all three failures trace back to the same gap: the agent worked exactly as designed, and the infrastructure around it did not exist to catch what that design never anticipated. Red Hat’s AI business depends on selling that missing infrastructure through Red Hat AI and OpenShift AI, so the company has an obvious commercial interest in this argument. That doesn’t make the underlying diagnosis wrong. The post, written by Richard Naszcyniec, AI Technical Product Marketing Director, and Joshua Wilson, catalogs seven specific capabilities it says are absent from every popular agent framework on the market today, specific enough to double as an audit checklist for any team about to put an agent in front of paying customers or real money.

A Retry Storm, a Wrong Account, and a Policy No One Wrote

Each of the three failures points at a different missing safeguard, and Red Hat’s account is specific about the mechanics of each. The duplicate tickets happened because nothing in the stack tracked whether a “failed” tool call had actually succeeded on the other end: LangChain retried a timed-out request exactly as its retry logic is supposed to, but no layer above it recognized that the first call had already created a ticket, so the retry created a second one, and the same blind spot kept firing every time the pattern repeated.

The wrong-account charge happened because the agent’s credentials were scoped for development convenience, not production risk. Nothing in the infrastructure constrained which billing accounts the agent’s identity was allowed to touch, so when the underlying model picked the wrong account identifier, the request simply went through. The hallucinated refund policy happened because no evaluation or safety layer checked the agent’s output against the company’s actual return policy before it reached the customer; a static test suite run before launch would not have caught it, because nobody had anticipated that specific hallucination in advance.

Seven Capabilities No Framework Ships

Red Hat says it sees the same seven gaps recur “every time” it maps agent failures like this one across customer deployments. The list doubles as a description of what a production-grade agent platform has to provide underneath whatever framework a team already chose.

Cryptographic Identity

Cryptographic identity gives verifiable proof of which specific workload is making a request, with identity attributes that determine which downstream services it is allowed to reach. Applied to the wrong-account incident, an agent with a properly scoped identity could still have picked the wrong account internally, but the platform would have constrained which services and accounts it could reach at all, and correctly configured downstream APIs would have rejected parameters outside that scope. The model’s mistake would have been contained instead of billed.

Execution Sandboxing

Agents operate autonomously, calling APIs and executing tool actions at machine speed, which is exactly why Red Hat argues they need isolation a typical application doesn’t. Without sandboxing, one workload’s failure can become every workload’s failure on shared infrastructure, and an agent able to write to the filesystem or open network connections outside its intended scope is, in Red Hat’s framing, a liability no security team should approve for production.

Tool Governance

Tool governance is infrastructure-level policy that decides which tools an agent can call, enforced at the network layer rather than in the prompt. Red Hat is blunt about why the distinction matters: teams that try to restrict tool access through prompt engineering alone find that a prompt injection, malicious input designed to trick an agent into unintended actions, or simply “a sufficiently creative model” can route around it. As the post puts it, “Governance belongs in the infrastructure, not the prompt.”

Observability and Tracing

Observability and tracing means capturing full execution traces, every prompt, every tool call, every intermediate result, so a failure is visible before a customer or an invoice surfaces it. Red Hat notes that all three failures in its incident went undetected until the damage was already done. With full tracing, the retry storm, the wrong-account call, and the hallucinated policy would each have shown up in a trace a developer or auditor could inspect, the same way a distributed microservice call gets debugged today.

Continuous Evaluation

Continuous evaluation checks an agent’s live output against ground truth on an ongoing basis, not just against a pre-launch test suite. Red Hat’s framing is direct: static test suites catch the failure modes engineers anticipated in advance, while continuous evaluation is built to catch the ones they didn’t, which is precisely the category the hallucinated refund policy fell into.

Safety Enforcement

Safety enforcement puts guardrails at the inference boundary that intercept an agent’s output before it reaches a customer, a database, or another agent downstream. In the incident, the framework routed the model’s fabricated refund policy straight to the customer with no check in between. Red Hat describes the fix as turning that boundary into “a checkpoint, not a pass-through.”

Lifecycle Management

Lifecycle management covers deploying, updating, scaling, and retiring agents across a fleet under one consistent operational and security posture, rather than each agent accumulating its own ad hoc deployment process over time. Red Hat treats it as the capability that keeps the other six from decaying into inconsistency as the number of agents in production grows.

Why the Usual Workarounds Don’t Scale

Red Hat’s post walks through the major agent frameworks on the market and credits each with real strengths: LangChain and LangGraph offer chains, tool calling, structured output, graph-based orchestration, and retrieval integration with, in the post’s words, “genuinely strong” composability; CrewAI’s multi-agent coordination and role-based task delegation are “well thought out”; Google’s Agent Development Kit integrates tightly with Google’s own ecosystem; Anthropic’s Claude Agents bring reasoning depth; and AWS’s Strands offers AWS-native workflow integration. Red Hat’s argument isn’t that any of these frameworks are poorly built. It’s that not one of them ships cryptographic identity, execution sandboxing, or network-layer tool governance, because that was never the problem a framework is designed to solve.

Teams that notice the gap tend to try one of three fixes, and Red Hat says each one runs into a different wall. Building the platform layer in-house works fine for a single agent, and Red Hat concedes it can even feel satisfying since the team understands every piece of what they built. It stops working once an organization reaches something like 10 agents spread across 3 teams, each with its own security model, its own tracing format, and its own deployment process. Buying a point solution for one capability, LangSmith for tracing is the example Red Hat names, genuinely solves that one piece, but an organization running LangChain, CrewAI, and custom agents side by side now needs three separate production stories: three tracing formats, three security models, three sets of tooling for engineers to learn, which Red Hat frames as trading sovereignty for convenience rather than actually closing the gap. A third pattern, wiring up framework-specific middleware for each agent runtime individually, avoids locking into a single vendor but, in the post’s words, still “fragments the production story across framework boundaries.”

Red Hat’s Answer, and Why to Read It Skeptically

Red Hat’s proposed fix is what it calls BYOA, bring your own agent: an operating principle under Red Hat AI and OpenShift AI meant to inject identity, safety, governance, and observability underneath an existing agent runtime, LangChain, CrewAI, LangGraph, or custom code, without requiring the agent itself to be rewritten. The post points readers toward a free OpenShift AI developer sandbox, starter kits pre-configured for several of the frameworks it just finished critiquing, and a no-cost Red Hat AI Foundations course, which is a reasonable sales funnel for a company whose AI revenue depends on being the platform layer teams buy instead of build.

That commercial angle is worth keeping in view while reading the rest of the argument. Red Hat is not a neutral research lab publishing an independent study of agent failure rates, it is a vendor diagnosing a problem it also sells the cure for, and the specific incident numbers, 43 tickets, $4,000, $280, are Red Hat’s own retelling rather than an independently audited case. None of that makes the diagnosis worthless. Whether a team ultimately adopts Red Hat’s platform, a competitor’s, or builds its own, the seven-capability list is a legitimate starting checklist for anyone about to move an agent from a demo into a system that touches real customer accounts or real money. Red Hat says its next post in the series will focus specifically on the security layer.

The Takeaway

The framing Red Hat lands on is that “which framework should I use” and “who provides the production infrastructure my framework doesn’t ship” are two separate decisions, and teams that only make the first one are the ones who get paged at 6 AM. That split holds regardless of how much of Red Hat’s own pitch a reader accepts: a framework choice governs how an agent reasons and calls tools, while a production-infrastructure choice governs what happens when that agent inevitably does something its designers didn’t anticipate. Skipping the second decision doesn’t make it disappear, it just moves the discovery process from a design review to an incident report.