Microsoft’s Echoverse Turns Computer-Use Agent Training Into a Depth-Over-Scale Problem

Microsoft Research published a detailed account on July 30, 2026 of how it trains AI agents to operate real software, and the headline finding cuts against the usual instinct to just generate more training environments. In a post titled Echoverse: Deep, evolving environments for computer-use agents, a Microsoft Research team that includes principal researcher Akshay Nambi describes building twelve synthetic training worlds, most of them modeled closely enough on real applications to reproduce actual backend state, and found that a 9B-parameter model trained on all twelve nearly doubled its task-completion score, from 36.5 percent to 67.1 percent, landing within fourteen points of GPT-5.4.

The result the team keeps returning to is not the score itself but what drove it. Training on shallow versions of the same applications made the model worse, not better. Training on deep, stateful clones made it generalize to software it had never seen. That distinction, plus a companion technique for isolating the specific UI controls agents tend to get stuck on, is the real subject of the post.

Why Computer-Use Agents Can’t Train on the Live Web

A computer-use agent, in Microsoft’s framing, only learns from actions that have real consequences: a click that changes saved state, a message that reaches someone, a page that refuses to move because the last action did nothing. A screenshot shows what an interface looks like; only a working system shows what an action actually caused. The problem is that most of the software worth automating, email and chat, banking, health records, the internal consoles that run cloud and machine learning infrastructure, sits behind a login, and none of it can be trained against directly. Every attempt would write to a real account, there is no reset between tries, and the true state of the system stays hidden behind the screen.

Microsoft’s answer is to rebuild the target application as a synthetic system where the researchers own the database underneath: the state is real and changes for real, but it is safe to break, quick to reset, and graded from the data itself rather than from a screenshot.

What Microsoft Means by a “World”

The post defines a “world” as three things bound together: an environment (the application, its state, and the actions that change it), a set of tasks that assign goals inside it, and a verifier that grades whether the agent actually reached the goal, checked against ground truth. Pipelines that generate all three automatically are no longer rare in agent research, and Microsoft’s team credits that progress directly, noting that the field is now good at yielding hundreds of environments and thousands of checkable tasks. Echoverse builds on that base rather than replacing it.

What the team argues is that once worlds are plentiful, raw count stops being the bottleneck. What matters instead is internal structure: whether state stays coherent across different users and screens, whether multi-step workflows preserve their real dependencies, and whether a skill an agent is weak at shows up in enough different forms to actually be learned rather than memorized.

Inside the Echoverse Factory

A Two-Phase Pipeline: Build, Then Grow

Echoverse is described as a single pipeline that produces two kinds of output: full domain worlds that preserve an application’s entire workflow depth, and capability worlds that isolate one specific interaction. Both lean on the same underlying advantage: because Microsoft’s team owns the database, success can be measured as a property of the application’s own state rather than a model’s interpretation of a screenshot.

Building a world starts by expanding a handful of seed scenarios into a specification, which is then compiled into machine-checkable claims about the application’s routes, state, and expected behavior. Only after that does the pipeline generate the actual app: a FastAPI and SQLite backend behind a React interface, confirmed directly in the project’s own GitHub repository. A freshly generated app is treated as a hypothesis rather than a finished world: a builder process runs every claim against the live environment and repairs the database, backend, or frontend until each one passes, then writes a readiness record that separates hard blockers from lower-priority risks. A world with open blockers does not advance to training.

Growing the task corpus is a separate but tightly coupled phase. Every task goal is re-grounded against the live database, drawn from entities that actually exist in it, then passed through a panel of automated analyzers checking whether its entities are real, whether the goal is plausible, and whether its difficulty matches the work involved. The sharpest of these checks runs an agent against the real interface in a browser to confirm the task can actually be completed before any training model ever sees it. Microsoft describes building and growing as one loop rather than two separate stages: most defects trace back to the world itself, so the environment gets re-versioned with every iteration, and harder tasks that expose gaps in the world in turn produce a sturdier world that can carry still-harder tasks.

The trajectories collected this way become supervised fine-tuning (SFT) data through a specific filter: GPT-5.4 attempts to solve each task, and only the trajectories a verifier confirms against ground truth are kept.

Grading Against the Database, Not a Screenshot

Every task in Echoverse carries its own answer key: a value or state change minted directly from the real database by a SQL query at the moment the task is generated, true by construction and re-checked once the agent finishes. A read-type task is graded on semantic equivalence to the stored value (Microsoft’s own example: an agent answering “$288” passes against a stored value of $287.62). A write-type task is graded on an actual before-and-after database diff, so an agent that merely reports a ticket was closed fails unless the underlying row actually flipped. A combined read_write task takes the lower of the two scores. Because grading is grounded in real state rather than a human label, the team argues it is considerably harder to game than a system that relies on a human-written rubric.

Two Kinds of Worlds: Deep Domains and Isolated Capabilities

Ten Deep Domain Worlds Carry the Full Workflow

Ten of the twelve worlds are full domain simulations, each a self-contained, fully interactive clone of a real kind of application, complete with its own backend, seeded database, and feature surface. Tasks inside these worlds run five to twenty actions deep and only count as finished once the underlying state has actually changed, not when the agent merely claims it has.

Two Capability Worlds Isolate a Single Skill

The other two worlds take a narrower approach aimed at a specific failure mode: an agent that can navigate an entire application competently except for one recurring control. Microsoft’s own example is an agent booking a trip that searches correctly, filters correctly, opens the right listing, then stalls at a date picker, unable to translate “the second week of March” into the right sequence of clicks on an unfamiliar calendar widget. Building another booking site would not fix that gap, since the missing skill is not booking a trip in general, it is operating a date picker specifically, and it only gets learned once that control shows up in enough different visual forms. Echoverse’s two capability worlds drill exactly that: date pickers and nested filter-and-search controls, each tested across many different layouts, since Microsoft’s team found agents repeatedly stalling on the same handful of UI elements across otherwise unrelated applications.

The Result: Depth Beat Volume

The team ran a direct comparison to isolate the effect of depth: training the same model on shallow and deep builds of the same sites. The model got worse on the shallow versions and better on the deep ones, summed up in the post as “shallow worlds backfire; deep worlds transfer.” Trained across all twelve worlds together, the 9B model’s task-completion score nearly doubled, climbing from a 36.5 percent baseline to 67.1 percent and closing to within fourteen points of GPT-5.4 on the same evaluation. Just as notably, the skill drilled inside the narrow capability worlds carried over to full domains the model had never trained on, evidence that isolating one control in many forms teaches something more general than memorizing a single site.

Reinforcement Learning Pushes Past Imitation

Every result described so far comes from imitation learning: the 9B model is trained by copying the trajectories GPT-5.4 produced that a verifier confirmed as correct. Microsoft’s team is explicit that imitation has a ceiling, since a clean demonstration never shows an agent how to recover from a mistake or when to stop, and those are exactly the failure modes that break agents once they leave a training set.

Reinforcement learning, by contrast, optimizes directly against the graded outcome and lets the model learn from its own trajectories rather than a teacher’s. Doing that at scale requires an RL environment (RLE) that can be reset to a known state on every rollout, something the live web fundamentally cannot offer: there is no world to repair mid-task on a real website, so when an action lands on nothing, correctness depends entirely on the agent noticing and correcting course itself. In a five-world, two-epoch RL experiment, Microsoft reports the held-out judge score, based on 25 held-out tasks per world, climbing from 58 percent to 69 percent. The reward signal combined a trajectory-level reward from the same database-grounded verifier, scored by an LLM judge (GPT-4.1), with a dense, per-step reward from a separate multimodal judge (GPT-4.1 with vision).

What Microsoft Is Releasing

Microsoft is open-sourcing four of the twelve worlds, complete with code, data, and the grounded graders that score them, “to support research on high-fidelity computer-use worlds.” The release spans two full domain worlds, EchoForge (project and entity management over a large application database) and EchoStay (lodging and stay-management workflows), plus the two capability worlds covering date pickers and nested filters. The two capability worlds each ship with both an in-distribution and an out-of-distribution test split, while EchoForge and EchoStay each ship as a single configuration, for six dataset configurations and 722 total tasks in all, confirmed directly on the project’s Hugging Face dataset page. Both the harness code on GitHub and the dataset on Hugging Face are released under the MIT license. The grounding database for each environment is distributed separately through the Hugging Face CLI rather than bundled into the git repository, since the SQLite files back real, queryable application state rather than static text.

Why This Matters

The broader argument Microsoft is making is a shift in how a training environment should be thought of. As the post puts it, “a world is no longer a fixed benchmark you score against; it is a training surface you keep improving,” where the same graded run that measures a model’s progress also surfaces the defects that make the world itself sharper for the next round. Any team trying to build agents that operate real internal software, procurement tools, admin consoles, ticketing systems, runs into the same problem Echoverse targets: there is no live, resettable copy of the target application to train against safely. Microsoft’s own numbers suggest a handful of deep, faithful environments will teach a model more than a much larger pile of shallow ones.