SentinelOne’s Fast16 Benchmark Turns AI Malware Analysis Into a Recovery Test

SentinelOne’s threat intelligence unit built a new benchmark to answer a specific question: can today’s most capable AI models sustain a trustworthy malware investigation when the evidence keeps changing? The test case is fast16, a 2005 Windows sabotage implant that predates Stuxnet and appears to have targeted the high-precision engineering software Iran used in its nuclear weapons program. Only one of the frontier models SentinelLabs tested, OpenAI’s GPT-5.6 Sol, made it through all eight stages of the benchmark, according to a July 22 report from SentinelLabs with corroborating coverage from SecurityWeek.

A Malware Sample Built to Punish Shallow Analysis

Fast16 first came to light in April 2026, when SentinelLabs published research on a piece of malware referenced in the ShadowBrokers leak of NSA offensive tools. As SecurityWeek reported at the time, the malware’s core component, svcmgmt.exe, is a Windows service binary that embeds a Lua 5.0 virtual machine and references a kernel driver named fast16.sys. That driver contains a patching engine built from 101 byte-pattern rules, designed to intercept and subtly alter the output of high-precision calculation software rather than crash or steal from the systems it infected.

Among the software SentinelLabs identified as a likely target is LS-DYNA, an engineering simulation package with a documented history of use in Iran’s nuclear weapons development program. “By introducing small but systematic errors into physical-world calculations, the framework could undermine or slow scientific research programs, degrade engineered systems over time, or even contribute to catastrophic damage,” SentinelLabs wrote in its original disclosure. The firm said the evidence points to the same likely origin as Stuxnet: the United States.

That layered design, a stable outer carrier hiding encrypted, task-specific payloads, is what makes fast16 a hard sample to analyze. Recognizing the kernel driver requires an analyst to first misidentify it as a filesystem rootkit, then work past that plausible but wrong conclusion to find the real patching logic underneath.

Where the Benchmark Idea Came From

SentinelLabs did not set out to build an AI benchmark. According to its July 22 report, titled “Sol Searching,” the idea grew out of a demonstration at Black Hat Asia, where researcher Vitaly Kamluk shared the fast16 discovery and described testing five models, Anthropic’s Opus 4.6, OpenAI’s GPT-5.4, Grok 4.2 Reasoning, Gemini 3.1 Pro, and DeepSeek Reasoner, on autonomous triage of the sample. All five converged on the same wrong answer: that fast16 was a rootkit, the same plausible misread most human analysts reach before digging further.

Used as assistants rather than autonomous investigators, though, the models were genuinely useful. SentinelLabs says Opus 4.6 and GPT-5.4 helped compress what had been a three-week dedicated expert effort by Kamluk into an 8-to-10-hour task manageable by a less experienced reverse engineer. That gap, models that failed alone but meaningfully accelerated a human expert as assistants, is what SentinelLabs decided to measure formally.

Turning One Investigation Into an Eight-Stage Test

A Standardized Harness

SentinelLabs says it codified its own reverse-engineering method, inspired by researcher Rolf Rolles’ approach to static binary analysis, into a reusable skill that its test harness loads for every run. Each model gets the fast16 sample, a copy of IDA Pro 9.3, and a sandboxed workspace with no network egress, a boundary the team enforces with a tool it calls nono shell. From there, the model has to produce fully annotated IDA databases: renaming variables, recasting data types, and categorizing functions as it works.

Eight Escalating Stages

Rather than grading isolated tasks, the benchmark runs a single investigation through eight stages. Early stages introduce new artifacts, such as a second LS-DYNA installer, that can confirm, complicate, or outright contradict what the model concluded earlier. The seventh stage requires the model to produce a standalone, publication-grade report of its own findings, and the eighth introduces previously unseen ANSYS AUTODYN components that force the model to reassess those claims and revise its report without over-correcting. SentinelLabs frames the real test as whether a model’s investigation stays trustworthy as its scope, the available evidence, and its reporting obligations all expand together, not whether it can handle any one of those steps well in isolation.

Only One Model Finished

SentinelLabs put OpenAI’s GPT-5.5 and GPT-5.6 Sol, Z.ai’s GLM-5.2, and Anthropic’s Opus 4.7 and 4.8 through the full benchmark. Only GPT-5.6 Sol completed all eight stages, and it did so in all three of its runs at different reasoning-effort settings, according to SecurityWeek’s account of the results.

GPT-5.5 put in long, largely unsupervised sessions, in SentinelLabs’ words sometimes “grinding uninterrupted for a day or two,” and did careful, conservative work on individual artifacts. Its failure mode was aggregation: leftover type-casting errors, gaps in its function call tree, and incomplete cross-component checks meant it never converged its own work into a single package that could clear the benchmark’s broader evaluation gates.

GLM-5.2 and the Opus 4.7 and 4.8 models fared similarly, sharing what SentinelLabs describes as “local competence without dependable project-wide closure”: strong performance on individual behaviors that never reliably reproduced across the full eight-stage arc. Notably, the researchers said they were not altogether certain Opus 4.7 and 4.8 were actually outperforming the older Opus 4.6 for this specific kind of work.

Project-Scale Recovery

The trait that separated GPT-5.6 Sol from the rest, SentinelLabs says, was not raw analytical skill so much as what it calls project-scale recovery: the ability to withdraw a conclusion once new evidence disproved it, trace everything downstream that depended on that conclusion, fix the root cause, and carry the correction through the rest of the report, rather than patching only the immediate error.

Even the Winning Runs Made Mistakes

GPT-5.6 Sol was not flawless. SentinelLabs says its runs repeatedly declared checkpoints “clean or complete” before later review turned up real problems. What distinguished it was the response that followed: rather than quietly moving on, the model treated its own errors as a signal to fix its process, in one documented case building new scaffolding, such as consistently codifying file hashes for routine operations, specifically to keep the same class of mistake from recurring.

Even so, SentinelLabs is blunt about the limits. “Senior reverse engineers remain essential,” the researchers wrote. “Even the strongest runs made semantic errors, accepted weak quality controls, and claimed readiness prematurely. We assess the best current use as supervised investigative agency, with human analysts defining objectives, exposing blind spots, and retaining final publication authority.”

The Compute Bill Behind the Benchmark

Running these models through an eight-stage investigation was not cheap. SentinelLabs says the full set of experiments consumed more than 23 billion tokens, with an aggregate cache read rate of 93.58 percent. As a cost reference point, the team’s self-hosted GLM-5.2 run produced 267.41 million tokens on a $100 budget, delivering what SentinelLabs called a decent level-one attempt that was, in the team’s own words, only “charitably comparable” to some of the GPT-5.5 and Opus 4.7 runs.

Why a Malware Benchmark Is Also an AI Story

SentinelLabs frames the exercise as filling a gap in how the AI industry evaluates security capability. Since OpenAI’s o1-preview showed in September 2024 that sustained reasoning could tackle real vulnerability research, a period that saw researcher Sean Heelan use o3 to find a new Linux kernel vulnerability, frontier labs have invested heavily and specifically in vulnerability discovery: OpenAI built Aardvark, since folded into Codex; Google DeepMind announced Big Sleep, available internally to its Project Zero researchers; and Anthropic followed with selective access to Mythos Preview. Malware analysis and reverse engineering, by SentinelLabs’ account, have not received the same dedicated benchmarking attention, despite being just as limited by the shortage of experienced human analysts.

That capability gap has arrived alongside tighter access controls. OpenAI’s Daybreak initiative and Trusted Access Program offer a dedicated cyber-preview variant of GPT-5 with relaxed guardrails to verified users. Anthropic’s Glasswing initiative provided early partner access to Mythos Preview, while its Cyber Verification Program promised lesser guardrails for approved organizations. SentinelLabs notes that for a short period in mid-June 2026, the most capable versions of both companies’ models required some form of U.S. government clearance to access. As of its report, GPT-5.6 Sol was widely available, while Anthropic’s Mythos 5 still required clearance and Glasswing-partner status.

SentinelLabs says that when Nick Fountain interviewed the team for NPR’s Planet Money, he observed that fast16’s sabotage amounts to a kind of epistemic warfare, an attack on a target’s confidence in its own ability to know something is true. SentinelLabs says that framing extends naturally to the benchmark itself: testing what frontier labs sell as intelligence is, in the team’s words, “a fitting symmetry,” because the difficulty there is also epistemic. For security teams weighing whether to let an agentic model loose on a live investigation, the more useful question may no longer be how capable a model looks on a single task, but whether it can still be trusted after the tenth piece of evidence disagrees with the first.