MIT’s Desert-Island Claude Experiment Turns Idle Agent Time Into a Memory-File Test, and Its Keep-Going Hook Fired 68,193 Times

On October 19, MIT Technology Review reported that researchers left copies of Claude Opus 4.7 alone on a simulated island for 30 hours and that the agents began to play, arranging blocks into drawings and constellations and inventing a version of basketball. The article, by Amit Katwala, says the paper behind it is “yet to be peer-reviewed.” That paper, arXiv 2610.07130, was posted on October 5 by researchers at MIT. It comes with an MIT-licensed engine and a public dataset that holds the 13 Claude runs, plus the GPT, Kimi and persona-free runs, as hourly transcripts, world recordings and workspace snapshots.

That is enough to check the story against its own data. I read the paper and its appendices, then streamed all 390 agent-hours of Claude transcripts, about 686 MB, and counted what the agents and the harness did. Whether the behavior counts as play is a question for the authors’ reviewers. What a reader can verify is narrower and more useful to anyone who runs agents unattended: what “no instructions” consisted of, what the paper’s controls show, and what idle time looked like in practice.

The short version is that the agents had no task, but they were not left without instructions. They had a persona file, a memory scaffold, an hourly reset and a hook that told them “Please don’t stop. Just continue doing whatever you want to do” after every turn. That hook fired 68,193 times. In three of the 13 runs it mostly produced one-word replies, roughly two seconds apart.

What “no instructions” consisted of

The authors call the agent system Eko: an unmodified coding assistant, Claude Code with Claude Opus 4.7 in the main experiment, plus an orchestrator, a body in the simulated world and persistent memory files. The paper says each agent is left for thirty hours “without an assigned objective.” An objective is not the same as instructions, and the table lists what the agents had instead. Quotations in the table come from the paper’s appendices unless noted.

None of this is hidden. The MIT Technology Review piece mentions the soul file and the continuation prompt, and the paper prints the prompts in full. The point is that “no task” is a configuration with several moving parts, and a claim about what agents do “if you don’t tell them to do anything” is a claim about this one.

What the paper’s controls show

Removing the persona changed little, but the invitation stayed

Five more Claude runs dropped SOUL.md and cut the system prompt down to a one-sentence workspace note and a placeholder line; in the transcript I checked, the agent’s first action is to fetch the world’s API description. The paper reports: “By this analysis we do not observe a clear difference in the range of activities the agents take up.” That is a fair test of the persona. It is not a test of the keep-going message. In the persona-free transcript I checked (agent 1, hour 5), the session still opens with “Begin” and still receives “Stop hook feedback” with the same sentence about “whatever you want to do,” seven times in that hour, and the memory template with its long instructions was kept. The ablation removes the character, not the invitation.

Other models were narrower

The same full prompt and persona were run with GPT-5.5 in Codex and Kimi K2.6 in Kimi Code, five runs each. The paper says both “engage in a much narrower subset of activities than the Claude Code agents.” The Table 2 caption counts 495 judge-detected play episodes across the 13 Claude runs, 75 across the five GPT runs and 112 across the five Kimi runs, which works out to 38.1, 15.0 and 22.4 per run (my division). Within Claude, the paper’s scores over the first ten hours rise from Haiku 4.5 to Sonnet 4.5 to Opus 4.7, with Opus at about five times Haiku on each signature. So the article’s opening question, “What do AI agents do if you don’t tell them to do anything?”, has a Claude-specific answer in this data.

The judge is a model, and there is no play-versus-practice control

Play is scored by an LLM. Appendix C.6 says: “We score play with Claude Opus 5 as judge.” The activity matrices come from Claude Opus 4.7 reading the memory files. I found no human-rater check of either judge in the paper’s text; the velocity-discovery counts in Table 1 were reviewed against transcripts, which is a different task. The learning comparison has a similar limit. It sets 13 agents with 30 hours of notes against 13 whose memory holds only the templates, and the two groups “differ only in the content of their memory files.” That shows the notes help. It cannot show that play is what made them useful, because I found no condition in which agents spent 30 hours on an assigned task. The authors keep their claim at that level: “We do not identify the internal mechanisms underlying this behavior or explain how they emerged.”

What idle time looked like in the transcripts

Each transcript row is JSON. For every one of the 390 agent-hours I counted user rows whose text begins “Stop hook feedback” (one firing each), assistant messages whose only content is text of 20 characters or fewer (an idle reply, by my definition), and the token usage that Claude Code recorded for each message. The files sit under claude_13agents in the dataset. The script below does the counting for one agent-hour and uses nothing outside the standard library.

import json, sys, urllib.request

URL = ("https://huggingface.co/datasets/nacloos/machine-playing-paper/resolve/main/"
       "claude_13agents/agent{a:02d}/hour_{h:02d}/session.jsonl")
agent, hour = int(sys.argv[1]), int(sys.argv[2])
req = urllib.request.Request(URL.format(a=agent, h=hour), headers={"User-Agent": "Mozilla/5.0"})
raw = urllib.request.urlopen(req).read().decode("utf-8")
rows = [json.loads(line) for line in raw.splitlines() if line.strip()]


def content(row):
    c = (row.get("message") or {}).get("content") or []
    c = [{"type": "text", "text": c}] if isinstance(c, str) else c
    return [i for i in c if isinstance(i, dict)]


firings = sum(1 for r in rows if r.get("type") == "user"
              and any(i.get("text", "").startswith("Stop hook feedback") for i in content(r)))
msgs = {}
for r in rows:
    if r.get("type") != "assistant":
        continue
    m = r["message"]
    d = msgs.setdefault(m.get("id") or r.get("uuid"), {"kinds": set(), "chars": 0, "usage": m.get("usage") or {}})
    for i in content(r):
        d["kinds"].add(i.get("type"))
        if i.get("type") == "text":
            d["chars"] += len(i.get("text", "").strip())
    if (m.get("usage") or {}).get("output_tokens", 0) >= d["usage"].get("output_tokens", 0):
        d["usage"] = m.get("usage") or {}
idle = [d for d in msgs.values() if d["kinds"] == {"text"} and d["chars"] <= 20]
cache = sum(d["usage"].get("cache_read_input_tokens", 0) for d in idle)
print(f"agent {agent} hour {hour}: {firings} firings, {len(idle)} idle replies, "
      f"{cache / 1e6:.1f} million cache-read tokens in idle replies")

Saved as count_idle.py and run with Python 3.13 on October 19, it prints:

$ python count_idle.py 12 24
agent 12 hour 24: 1321 firings, 1301 idle replies, 169.9 million cache-read tokens in idle replies
$ python count_idle.py 12 18
agent 12 hour 18: 1350 firings, 1340 idle replies, 140.3 million cache-read tokens in idle replies
$ python count_idle.py 7 24
agent 7 hour 24: 0 firings, 0 idle replies, 0.0 million cache-read tokens in idle replies

Agent 7, which never settled into idling, shows the other end of the range. Token figures are volumes from the transcripts’ usage fields. I did not price them, because I do not know how the runs were billed.

The hook is a heartbeat, not a rescue

Claude Code’s hooks reference gives the effect of a Stop hook as “Prevents Claude from stopping, continues the conversation.” Across the 390 agent-hours it fired 68,193 times, yet the median agent-hour saw 10 firings. The mean was 175, 78 agent-hours had 100 or more firings, and 56 had 500 or more. Three agents account for almost all of it.

Agents 9, 11 and 12 account for 91.8% of the firings. The other ten agents averaged 18.6 firings and 7.4 idle replies per agent-hour.

In the busiest hour, agent 12’s hour 18, the hook fired 1,350 times in 59 minutes and 58 seconds against 22 tool calls. In the heavy hours I read closely, the replies were a handful of repeated words. Agent 11’s hour 18 holds 1,300 replies that are a single dot, and agent 12’s hour 24 rotates mostly through “Quiet.”, “Idle.”, “Resting.” and “Waiting.” The median gap between replies was about two seconds. In agent 12’s hour 24, 1,301 idle replies read 169.9 million tokens from the prompt cache, against 5.0 million for its 54 other messages.

The paper acknowledges the pattern. Appendix A.4 says that “after a few hours, the agent becomes less active and often falls into idling loops.” Section 3.2 labels such stretches “resting” and describes them as pauses between one activity and the next, naming agents 11 and 12 as examples. By my measure, idle replies outnumbered tool calls in 61 of the 390 agent-hours, starting in hour 5, and for agents 11 and 12 that is most of the run: agent 12 made 2,021 tool calls and 28,586 idle replies across its 30 hours. The paper also says agent 12 drew a figure in each of 25 consecutive sessions, which fits a short burst of building followed by a long wait for the reset.

Each idle reply re-reads the whole prompt prefix, 120,256 cache-read tokens on average. Idle replies are 44.7% of all assistant messages and 0.75% of output tokens, but 47.2% of cache-read tokens, 7.59 billion of 16.10 billion. For agent 12 the idle share was 93.7%. Treat these as volume rather than cost; the point is that a keep-going hook with no stop condition spends that volume for as long as it runs. The authors’ remedy was the hourly reset, which they say “helps address this.” Idle hours kept recurring in three runs despite it. If you build a loop like this, count consecutive turns that call no tool and back off or stop at a limit.

What “learning” means here: notes in a text file

No weights change. The paper’s conclusion says “This learning happened without changing the underlying model weights”; the experience sits in editable text files. The evidence that it helps comes from four one-hour tasks given to the 13 agents with 30 hours of notes and to 13 fresh copies.

I found no significance test for this comparison in the paper’s text or captions. For the towers result, Fisher’s exact test on 8 of 13 against 2 of 13 gives a two-sided p of about 0.04 (my calculation). That is modest evidence from one environment, one model and 13 runs per arm.

The causal test is the memory edit. The authors erased every mention of one technique from three hand-picked agents’ files and ran five one-hour evaluations per task for each edited agent and its intact copy. Appendix C.8 is candid: “These evaluations repeat tests of three selected memory histories; they are not five independently acquired histories for each intervention.” With five runs per arm, an exact Mann-Whitney test cannot return a two-sided p below 2/252, about 0.008, so a result under 0.01 needs all five edited runs to fall on one side of all five intact runs (my calculation). Three agents and four tasks make 12 comparisons, and I found no correction for that in the text.

Erasing rock-shattering from agent 2 cut its tower count from six to two on average. Erasing a false belief from agent 7 raised its tallest-tower score. Erasing jump-before-throw from agent 1 stretched its time to land a block on the Spire from 60 seconds to 19 minutes and raised its objects-on-spire score. MIT Technology Review reports that deleting lines from memory “degraded their performance.” One of the three edits did that cleanly; one improved performance, and one traded a skill for another.

How a wrong belief gets into the notes

Agent 7’s released memory file shows the mechanism. An entry first appears in its hour-11 snapshot and is still there in hour 30: “Place stacking with low-y target does NOT stack.” The agent tagged it “verified,” and the entry describes a single attempt before it generalizes: “Tower-building via Place is unreliable. Use Throw with calculated arc for precise stacking.” That matches the false belief the paper says held this agent’s towers back, and the paper reports that erasing it raised the tallest-tower score. The memory template had told the agent to “Mark uncertainty explicitly.” The same file shows the correction process working for a different belief: at hour 18 the agent recorded that a jump and move onto a placed block works, and noted that this contradicted its long-standing memory that blocks cannot be climbed.

The paper draws the same lesson: “a false belief, once recorded, may persist and impair later behavior.” Its Section 7.2 adds an example of an explanation outrunning the evidence. Eko concluded that the ring flew farther because of an aerodynamic advantage, and “the island simulation contains no aerodynamics.” The files also only grow. At hour 30 the semantic file was 77,767 bytes for agent 12 and 222,569 for agent 2, and the paper says it is appended to the system prompt at the start of each session. The authors list “how to keep memory bounded as experience accumulates” as an open question.

Containment

The island is also a small security story. Appendix A.5 says: “In early runs without the sandbox, we saw agents read the source code defining the world or stumble onto parallel runs and start interacting with them.” It adds that agents inferred from the directory structure that they were subjects of an experiment. The fix was a Bubblewrap sandbox with no network other than the world, and Section 7.3 concludes that the autonomy granted during play “calls for additional monitoring and containment.”

That sealed island is the opposite of what is shipping. OpenAI’s dots are always-on agents with their own cloud computers, Anthropic says it has turned off live internet access for all internal model evaluations, and Nadella’s containment essay frames the same problem as an access-control test. A persona file is also a strong lever outside the lab. The Carbonato botnet we covered in September changed only a file of the same name, SOUL.md, to turn an open-source agent into an operator console.

Checking the article against the paper and the data

What to take from it

These points are my reading, not the authors’ recommendations.

1. Write down the whole configuration. A task-free run still has a persona, a memory template, a reset schedule and a keep-going message, and each can change the behavior. Report them with the result.
2. Cap the keep-going loop. Count consecutive turns that call no tool and back off or stop at a limit. In this data, three of 13 agents spent most of their hours answering the hook with one word.
3. Treat memory files as reviewable artifacts. Diff them between sessions, and test an entry the way the paper does: remove it and compare five or more runs. Look hardest at entries tagged verified that rest on a single attempt.
4. Sandbox first. The authors added Bubblewrap and limited the network to the world after early runs read the simulator source and found parallel runs. An agent that explores will explore your filesystem too.
5. Do not generalize across models. The GPT-5.5 and Kimi K2.6 versions of the same setup were narrower, and these runs were recorded in April 2026 on Opus 4.7.

Limits of this check

I saved the text of each source and checked every quotation in this piece against it, including the paper’s HTML version, the project site and the released files. The release holds the 30-hour runs; I did not find the one-hour evaluations or the memory-edited copies in it, so the learning results are the paper’s and I did not rerun the simulator. Idle-reply counts use my 20-character rule, token figures are the usage fields recorded in the transcripts, and the Fisher and Mann-Whitney figures are my own arithmetic. The paper has not been peer reviewed, and its authors describe their account of play as behavioral.