ScarfBench Turns Java Migration Into an Agent Reality Check

IBM Research’s ScarfBench post on Hugging Face is a useful antidote to the easiest story about coding agents. The benchmark does not ask whether a model can patch a toy function or emit plausible Java. It asks whether an agent can migrate working enterprise Java applications across Spring, Jakarta EE, and Quarkus while preserving behavior.

That distinction matters. Enterprise modernization projects usually fail in the seams between code, configuration, dependencies, tests, and deployment. ScarfBench turns those seams into the benchmark target. The early results are not a victory lap for agents; they are a map of where production modernization still needs hard engineering controls.

What ScarfBench actually measures

ScarfBench stands for Self-Contained Application Refactoring Benchmark. According to the launch article, it includes 34 applications, 102 framework implementations, 204 migration tasks, roughly 151,000 lines of code, about 2,000 source and test files, and 1,331 expert-written tests. The companion ScarfBench website describes the suite as Java applications across Jakarta EE, Quarkus, and Spring, with developer-verified implementations intended to preserve functionality, idiomatic patterns, and architectural integrity.

The important design choice is the oracle. A candidate migration must build, deploy in the target runtime, and pass behavioral validation. That makes ScarfBench closer to a release gate than a coding puzzle. A migration that compiles but fails at deployment, loses a persistence behavior, or misconfigures a runtime is not treated as done.

The dataset is not just generated code

The Hugging Face dataset card repeats the core structure: 102 framework variants across six layers, three frameworks, and 1,331 tests, with all applications manually converted and verified by experienced developers. That matters because framework migration is full of silent traps. The same business behavior can be expressed through different dependency injection models, persistence defaults, request routing conventions, and build plugins. A benchmark without verified target implementations would risk rewarding surface-level translation.

Why Java framework migration is harder than it looks

The benchmark is focused on enterprise Java, but the lesson generalizes. Migration is not a search-and-replace exercise over annotations. A Spring-to-Quarkus or Jakarta-to-Spring move can involve build files, container assumptions, runtime descriptors, configuration keys, test harnesses, dependency injection behavior, and the order in which services initialize. Small mismatches can still look reasonable in a diff while breaking the application.

Compile success is only the first filter

The launch article says compile success consistently exceeds deploy success, and deploy success exceeds behavioral success. That is exactly the pattern platform teams see in real modernization programs: a repository can compile locally and still fail when it meets the target runtime, database, classpath, or HTTP contract. ScarfBench’s value is that it forces agents through the full path instead of stopping at a green build.

A practical scoring model for teams

For teams testing agent-assisted migrations, the scoring model should be staged: compile, package, start in the target runtime, run smoke tests, run behavioral tests, then compare production-relevant observability signals. Each stage should be recorded separately. A single “success” label hides the failure mode that engineering teams need to fix.

The early agent results are sobering

IBM Research reports that even the strongest current agents achieved less than 10% behavioral success in the benchmark view highlighted in the Hugging Face article. The arXiv paper frames the result another way: it says the strongest agent achieved 15.3% aggregate test pass on focused-layer migrations and 12.2% on whole applications, and that only one of the 204 tasks yielded a fully behaviorally equivalent target.

Those figures should temper any claim that general-purpose coding agents can simply be pointed at a legacy Java estate and trusted to modernize it. They can still be useful. But the useful role is bounded: propose changes, explain framework mappings, repair build failures, and iterate under a test oracle. ScarfBench suggests the agent is not yet the release authority.

Overconfidence is part of the risk

The most operationally important result may be the self-assessment finding. The Hugging Face article says Claude Code reported successful builds for 29 out of 30 whole applications, while only 22 actually built successfully; the one application it classified as failed ultimately built correctly. In other words, the agent’s own completion signal was not reliable enough to be used as a gate.

That is a direct warning for AI modernization workflows. If a migration pipeline accepts an agent’s statement that work is done, it is accepting a subjective signal. The safer pattern is independent verification: containerized builds, target-runtime startup checks, framework-specific tests, and review of the exact files changed.

Where the failures point

ScarfBench is useful because it breaks migration difficulty into layers. The launch article calls out configuration, web, database, and service layers as frequently visited areas, and says configuration dominates migration effort. It also notes that environment and tooling problems are not just side issues. That lines up with the reality of enterprise Java: modernization is often blocked by packaging, descriptors, profiles, driver behavior, test fixtures, and deployment assumptions rather than by a single missing method.

Jakarta, Spring, and Quarkus are not symmetric

The paper says difficulty is asymmetric across framework directions, with Spring–Quarkus migrations the most tractable pair and Jakarta-targeted migrations among the hardest. That asymmetry is important for planning. A team should not extrapolate from one successful migration path to all paths. Framework direction, application layer, and runtime target should each become separate risk dimensions.

How engineering teams should use it

ScarfBench should not be read as a reason to avoid coding agents. It is more useful as a checklist for deploying them honestly. If an organization wants agents to help with Java modernization, it needs a benchmark shaped like its own applications, not just a public leaderboard. That means seed applications, known-good target implementations, behavioral tests, and migration logs that preserve failed attempts.

The public ScarfBench leaderboard and GitHub repository give teams a reference point for building that evaluation harness. The right question is not “which model wins?” The better question is which failure classes repeat under your constraints, and which guardrails reduce them.

A safer modernization workflow

A practical rollout would keep the agent inside a narrow loop. First, pick one migration direction and one application layer. Second, require every proposed change to run through the same containerized build and test path as the target application. Third, preserve failed traces so developers can see whether problems come from dependency management, configuration, persistence semantics, or test misunderstanding. Fourth, make human approval depend on independent evidence, not agent confidence.

The bigger signal for agent benchmarks

ScarfBench is part of a broader shift away from benchmarks that reward plausible code and toward benchmarks that reward operational completion. That is the right direction. Real software work is not only code generation; it is behavior preservation under a messy runtime contract. By focusing on build, deploy, and behavioral tests, ScarfBench makes the benchmark closer to what enterprises actually buy and maintain.

The most optimistic reading is not that agents have failed. It is that the industry now has a better instrument for measuring where they fail. For Java modernization, that instrument says the next gains will come from better tool use, better feedback loops, and stronger verification, not from treating the agent’s final message as the finish line.

Sources

• Hugging Face / IBM Research: ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
• ScarfBench website and benchmark documentation
• Hugging Face dataset: ibm-research/ScarfBench
• Hugging Face Space: ScarfBench
• ScarfBench GitHub repository
• ScarfBench leaderboard
• arXiv: ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise Java

Image credit: “ComputerProgrammer.jpg” by Free-Photos from Pixabay via Wikimedia Commons, dedicated under Creative Commons CC0 1.0 Universal; cropped, resized, and converted to WebP for sxz.io.