Microsoft’s TokenEconomizer Turns Coding-Agent Cost Into a Learned Policy, but Its 18.4% to 30.9% Was Measured on the Paper’s Controller, Not the Plugin’s Defaults: Micro-Skills Ship Off, the Plugin Never Stops a Turn, and No Pass@1 Gain Clears a Three-Seed Test
TokenEconomizer’s 18.4% to 30.9% cost cut reproduces from its project page’s data, but it was measured on the paper’s controller, not the plugin’s defaults, and none of the Pass@1 gains clears a...
On November 16, Microsoft Research published a blog post on TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research and Microsoft Security that learns from a coding agent’s earlier sessions in a repository and uses what it learned to cut the cost of later ones. The post’s headline is that, across 277 SWE-Bench tasks, the plugin lowered mean cost by 18.4% to 30.9% over five agent and model combinations, and that “Costs fell in every repository tested, while task success rates remained broadly stable.” The numbers come from a paper (arXiv 2610.22732, submitted October 15), a project page that ships the data behind its charts, and a public GitHub repository that was created on October 13 and showed 7,369 stars on November 17.
Table Of Content
- What each source covers
- What reproduces
- The cost figures
- The 30 cells and the sixth task
- The profiling figures
- Where the headline reads stronger than the evidence
- The Pass@1 gains are inside the seed noise
- “Saved the most” in all 30 pairs
- What was tuned on what, and how the streams were built
- The plugin is not the paper’s controller
- Side by side
- Where the measured saving sits
- What a default install does on synthetic sessions
- Tool pruning
- Micro-skills
- Smaller things worth knowing
- How to try it without fooling yourself
- Run the checks yourself
- Recompute the paper’s numbers
- Drive the shipped hooks
- What would settle it
- Bottom line
- Sources
I checked the blog against all three, recomputed what could be recomputed, and drove the plugin’s own hooks with synthetic Claude Code sessions. The short version:
- The cost numbers reproduce. Every cost figure I could recompute from the project page’s data matches the paper, including 30 of 30 repository-by-setting cells cheaper, from 5.2% to 40.1%.
- No Pass@1 gain clears a three-seed test. The four gains give Welch p-values of 0.27, 0.33, 0.06 and 0.09, and the largest is 7.2 tasks out of 277. The paper’s abstract says “comparable task success”; the README and the project page headline Pass@1 as up in 4 of 5 settings.
- The paper’s numbers belong to the experimental controller, not the plugin’s defaults. The plugin’s own design document lists the differences: micro-skills are off by default, tool pruning has no per-task judge and protects tools a person uses, and the loop guard never stops a turn, where the paper’s guard aborted about two trajectories per stream. The blog mentions only the last one.
- One blog sentence overreaches. It says the plugin “saved the most in all 30 repository-combination pairs”. The paper says it saved something in all 30, and lowering reasoning effort saved more in 4 of 5 settings (medium) and 5 of 5 (low).
Two caveats up front. I did not run a coding agent with the plugin installed, so nothing here measures what the plugin saves in real use. And the repository’s documentation is more open about the differences than the blog or the project page, which is the only reason I can list them.
What each source covers
The blog summarizes the paper, and the project page’s data.js begins with the comment “Generated by scripts/build_page_assets.py from the paper sources; do not edit.” The repository holds two things: the plugin (tokeneconomizer/ and plugin/) and the benchmark harness behind the paper (experiments/). Its architecture note says the harness holds “benchmark-only controllers”, and its CLAUDE.md says “No run outputs in the repository”, so the code and task lists are public but per-task results are not. I read the repository at commit b57d03f, the merge of pull request 246 at 07:56 UTC on November 17.
What reproduces
The cost figures
Table 1 of the paper gives the change in mean cost against default execution, with the standard deviation across three seeds. The project page’s data has the same figures for the plugin’s arm:
| Setting | Cost change | Seed SD (points) | Change in seed SDs |
|---|---|---|---|
| Claude Code, Sonnet 5 | −23.6% | 1.8 | 13.1 |
| Claude Code, Opus 4.8 | −30.9% | 2.1 | 14.7 |
| Codex, GPT-5.6 Terra | −24.7% | 1.2 | 20.6 |
| Codex, GPT-5.6 Sol | −26.1% | 2.1 | 12.4 |
| Copilot CLI, Sonnet 5 | −18.4% | 1.5 | 12.3 |
The mean is 24.7%, and the blog’s range is simply the smallest and largest of these five. Each figure sits 12 to 21 seed standard deviations from zero, so inside this benchmark the saving is not run-to-run noise.
The 30 cells and the sixth task
The claim that cost fell in every repository under every setting also reproduces. All 30 cells in the project page’s per-repository data are negative, from −5.2% (Sonnet 5 on Django) to −40.1% (Opus 4.8 on Sphinx), and three are under 10%: Django with Sonnet 5, and SymPy and Django with Copilot CLI at −8.3% and −9.3%. The cells below are listed in the order Claude Code with Sonnet 5, Claude Code with Opus 4.8, Codex with Terra, Codex with Sol, and Copilot CLI with Sonnet 5.
| Repository | Tasks | Cost change in the five settings (%) | Mean |
|---|---|---|---|
| scikit-learn | 60 | −28.6, −24.4, −26.4, −27.8, −23.2 | −26.1 |
| SymPy | 60 | −25.8, −33.9, −23.7, −25.1, −8.3 | −23.4 |
| Django | 40 | −5.2, −28.5, −15.3, −16.7, −9.3 | −15.0 |
| Sphinx | 44 | −20.8, −40.1, −27.9, −29.3, −19.9 | −27.6 |
| Matplotlib | 34 | −27.0, −37.0, −27.9, −29.3, −16.6 | −27.6 |
| xarray | 39 | −27.0, −21.2, −27.1, −28.5, −30.1 | −26.8 |
Django is the weakest stream by a wide margin. It is also the stream the authors built to have related tasks close together: its 40 Verified tasks come “all from the start of its Verified history (November 2016 to September 2019), so that consecutive tasks are typically days apart”. That the densest stream saved the least counts in the paper’s favor, but it shows how much the choice of repository and stream moves the result.
The blog’s Figure 4 caption says “every repository becomes cheaper by the sixth task”, and that holds too. The cumulative cost change, averaged over the two Claude Code settings, is below zero for good from task 1 on xarray, task 2 on Matplotlib and SymPy, task 4 on scikit-learn and Sphinx, and task 6 on Django. It ends between −16.9% and −32.0%, the paper’s 16.9% to 32.0%. In five of the six streams the curve starts above zero, at +6.8% to +38.1% after one task, as the paper says.
The profiling figures
The numbers that motivate the design match the paper as well: 23 tool schemas loaded on every turn with 16 never called, failed runs at 17% of tasks but 32% of spend, and 30% of tool calls repeating an earlier action in the same repository against 7% for another repository’s history. They come from 60 profiled tasks, ten per repository, and at that size the three fixed strategies (RTK, Serena and RepoMap) swing from −20.2% to +30.8% per repository. The larger run backs the paper’s “no fixed strategy reduces cost consistently” on its own: RTK alone raised cost in 12 of the 30 cells and RepoMap alone in 15.
Where the headline reads stronger than the evidence
The Pass@1 gains are inside the seed noise
The abstract promises “comparable task success”. The introduction goes further, “improving Pass@1 over default execution in 4 of 5” settings, and the project page lists “Pass@1 also improves” as one of its three headline tiles. Table 1 gives each Pass@1 as a mean with a standard deviation across three seeds, which is enough for a Welch test of each pair of arms:
| Setting | Default | TokenEconomizer | Change (points) | Tasks of 277 | Welch p | Gap needed for p = 0.05 |
|---|---|---|---|---|---|---|
| Claude Code, Sonnet 5 | 74.7 ± 1.3 | 76.0 ± 1.2 | +1.3 | +3.6 | 0.27 | 2.8 |
| Claude Code, Opus 4.8 | 74.0 ± 0.6 | 73.0 ± 0.6 | −1.0 | −2.8 | 0.11 | 1.4 |
| Codex, GPT-5.6 Terra | 73.8 ± 1.0 | 74.7 ± 1.0 | +0.9 | +2.5 | 0.33 | 2.3 |
| Codex, GPT-5.6 Sol | 80.1 ± 0.8 | 81.7 ± 0.7 | +1.6 | +4.4 | 0.06 | 1.7 |
| Copilot CLI, Sonnet 5 | 75.0 ± 1.5 | 77.6 ± 1.3 | +2.6 | +7.2 | 0.09 | 3.2 |
None of the four gains reaches p = 0.05, and the nearest, Sol, needs 1.7 points and shows 1.6. The one loss, Opus 4.8 at −1.0 point, is a stronger signal in standard-error terms (t = −2.04) than the Sonnet 5 and Terra gains (t = 1.27 and 1.10). The largest gain is 7.2 tasks out of 277, while the default arm’s own seed-to-seed standard deviation is 1.7 to 4.2 tasks. Reading the reported deviations as population rather than sample deviations raises every p-value, to between 0.10 and 0.42.
These tests are generous. The 277 tasks are held fixed, so they ask only whether a gap exceeds the agent’s run-to-run noise on those tasks, not whether the tasks stand for anything else. A paired per-task test would be sharper, but the per-task outcomes are not published. The data support the abstract’s wording, comparable success at lower cost. They do not support reading “4 of 5” as an improvement.
“Saved the most” in all 30 pairs
The blog says TokenEconomizer “saved the most in all 30 repository-combination pairs without performance loss”. The paper’s statement is narrower: it reduced cost in all 30 cells, by 5.2% to 40.1%. The blog’s next sentence, that lowering reasoning effort “saved similar or larger amounts”, is the accurate one. Medium effort saved more than TokenEconomizer in 4 of the 5 settings and low effort in all 5, at a price of 1.8 to 4.7 points of Pass@1 and 3.6 to 10.3 points respectively. Read literally, the first sentence is not what the tables show, and it is the one a reader is likely to quote.
What was tuned on what, and how the streams were built
- Constants. Appendix C says “All values were fixed before the main runs, in earlier development runs on the same six SWE-Bench repositories.” No SWE-Bench repository was held out. The Web-Bench and security results reuse the constants on other data, with two changes the paper states: a pre-registered 30-turn budget for the short Web-Bench streams, and micro-skills switched off on the security benchmarks, where every task is a new repository.
- Task mix. “Of the 277 tasks, 199 come from SWE-Bench Verified and 78 from the full SWE-Bench test split.” The paper says the extra scikit-learn and SymPy tasks were added to raise difficulty, that 30 of xarray’s 39 tasks come from the full split too, and that the authors applied each added task’s gold patch in the official image to check that it passes. The blog’s “277 real bug-fixing tasks” does not mention the split.
- Streams. Each repository’s tasks run in order, so the controller only learns from earlier work, but the paper’s limitations say that “our SWE-Bench streams consist of related but independent fixes rather than actual user sessions”.
- Other benchmarks. Costs fell 6.4% to 19.6% on the two security benchmarks and 16.0% to 21.2% on Web-Bench, with quality “remaining comparable to default execution”. ReposVul’s F1 sits between 17.5% and 22.5% in Table 2(b), so that comparison is between two weak detectors, and Copilot CLI’s score fell slightly on CredData (68.2 to 67.7) and on Web-Bench Pass@2 (60.8 to 59.3).
The plugin is not the paper’s controller
The repository is unusually candid about this. docs/PLUGIN.md has a section titled “What changes relative to the experimental setting”, and CLAUDE.md says the plugin’s own text “makes no claims of effect”. The blog’s only mention of a difference is the loop guard: “In the open-source plugin, if the agent is still stuck after the note, TokenEconomizer alerts the developer rather than stopping the work itself.” The other differences matter more.
Side by side
| Mechanism | Measured in the paper | Plugin as shipped, on Claude Code |
|---|---|---|
| Micro-skills | A model proposes facts after task 4 and every 5 tasks, from digests of the 8 latest tasks, in a Python and git or a Node and web vocabulary. Alone it saved 11.8% and 14.3%. | Off by default, “because its proposer adds model calls and how much it saves depends on the repository and the work you do there”. When on, the proposer runs every 3 sessions with a generic vocabulary; a free layer, envfacts, learns from repeated failures without a model call. |
| Tool pruning | A tool is masked after 3 consecutive tasks without a call, or when a per-task LLM judge that reads the issue text finds it irrelevant. The full catalog returns every 5 tasks for a re-audit, and the mask is set at launch. Alone it saved 13.2% and 17.7%. | A tool is proposed after going unused over 10 prompts across at least 2 sessions, or over 5 sessions. The plugin builds its pruning stage with no judge and no re-audit, and protects AskUserQuestion, plan mode, sub-agents, the Task tools and the web. A deny list is written only after the repository’s sessions have been idle (60 minutes by default) and applies from the next session. |
| Stuck turn | One redirect, in which the agent is killed and resumed with a continuation prompt, then a terminal abort, plus a forced stop at 2.5 times the repository’s turn budget when a strict stall rule holds. Alone it saved 7.1% and 5.4%, with 1.4 and 1.7 aborts per stream (2.0 and 2.2 in the full stack). | A mid-turn reminder attached to a tool result, then a one-line notice to the person: “the plugin never stops a turn”. The docs call it a “Different intervention, so its effect is not inherited from the experiments”. |
| Success signal | The benchmark’s grader. | A turn that “ended green”: the agent stopped on its own, the guard did not stop it, and no verification after its last edit failed. |
| Model for overhead | The agent’s own model, “so overhead could not look cheaper than its arm”. | The cheapest capable model on the user’s own agent, haiku on Claude Code. This difference favors the plugin. |
Some code is shared. The plugin’s loop guard is the benchmark guard’s control flow applied one tool call at a time, and PLUGIN.md says “None of the thresholds were re-tuned.” What the table shows is that the plugin’s defaults enable a more conservative controller than the one that produced the headline.
Where the measured saving sits
The ablation in Table 2(a) splits the Claude Code results. Alone, tool pruning saved 13.2% (Sonnet 5) and 17.7% (Opus 4.8), micro-skills 11.8% and 14.3%, and the failure-loop guard 7.1% and 5.4%; the full stack saved 23.6% and 30.9%. By Table 6, the full stack had about 15 tool schemas masked at the end of a stream, against the 23 the profile found loaded, and its guard stopped 2.0 and 2.2 trajectories per stream. The two largest shares are the two that the shipped defaults change most: micro-skills is off, and pruning runs without the per-task judge, with more tools protected, and only after the repository has been idle. The appendix calls the guard’s aborts rare, “fewer than one in twenty” tasks, and the plugin’s limitations section says that “whether a nudge changes a looping model’s course is not established”.
What a default install does on synthetic sessions
To see the defaults in action I wrote a script that calls tokeneconomizer.plugin.hook.main, the entry point every hook executes, with synthetic Claude Code transcripts, the way the repository’s own tests/test_live_hooks.py does. Everything is local, folds run inline, and no model call is made, because the micro-skills proposer is off by default and the loop guard is never woken. The sessions are made up, so the results show the plugin’s gating, not its savings. The full script is further down; this is what it found.
Tool pruning
| Scenario | Sessions | Result |
|---|---|---|
| A: fresh install, no catalog snapshot | 12 sessions (36 prompts) that use only Read, Grep, Edit and Bash | 0 tools proposed after every session and 0 deny rules written; the catalog held only the 4 tools it had seen used |
B: default settings after refresh-catalog (the repository’s own list of 24 Claude Code tool names) |
6 sessions | Nothing until session 4, then 12 tools waiting; 0 deny rules written after any session |
| B0: as B with the idle wait set to 0 minutes | 6 sessions | 12 deny rules written after session 4 and announced at the start of session 5 |
The 12 are CronCreate, CronDelete, CronList, DesignSync, EnterWorktree, ExitWorktree, ListAgents, PushNotification, ReportFindings, ScheduleWakeup, SendMessage and Workflow. Of the other 12 names in the snapshot, three were used in the sessions, eight are on the protected list (Agent, Monitor, Skill, TaskOutput, TaskStop, WebFetch, WebSearch and Write), and NotebookEdit, which the adapter marks high-risk, was not proposed in six sessions. This matches the README’s settings table, which sets the evidence bar at 10 prompts across at least 2 sessions or at 5 sessions and the idle wait at 60 minutes. It also shows why the Quick start asks for the one-time refresh: without it the catalog is only what sessions have used, and the adapter’s own docstring says that without a snapshot “only deferred tools can be proposed”, which it calls “honest but small”. My synthetic transcripts list no deferred tools, so scenario A is the plain case.
Micro-skills
Eight sessions that all hit the same trap, bare pytest missing while python -m pytest works, left a default install silent: 0 of 8 session starts carried a fact. With TE_MICROSKILLS=envfacts, the free layer that needs no model call, the fact first appeared at the start of session 5 and then in 4 of the 8 session starts. The mechanism works as described. It is simply not on by default, and the docs give a reason that is also a security point: “what it injects is derived from command output a repository controls: turning it on is the person’s call”.
Smaller things worth knowing
- License. The README badge and
pyproject.tomlsay Apache-2.0, while theLICENSEfile is the MIT license (“Copyright (c) 2026 Microsoft”) and GitHub’s detector reports MIT. Anyone adopting the plugin should ask which one governs. - Agents. The project page says the plugin “Works with Claude Code, Codex, GitHub Copilot CLI, and Pi”. The paper evaluated three of those; its limitations say its experiments cover “three widely used coding agents but not more minimal harnesses such as Pi”, and the plugin’s docs say that on Pi “tool pruning is not planned there”. The README lists six more hosts as experimental.
- Tests. On Windows 11 with Python 3.13.14 and pytest 9.2.0,
pytest testsran 1,198 tests: 1,163 passed, 7 failed and 28 were skipped. The failures are five launcher, bundle, symlink and install-copy tests intest_plugin_package.py, one dead-code script test and one session-id path test.pytest experiments/testsran 397: 381 passed, 10 failed and 6 were skipped, five of the failures in a file whose test names mention cgroups. The repository’s CI runstestson Ubuntu and Windows runners, so these failures may come from my machine; I did not investigate. The lab script below does not use the shell launcher. - What status reports.
te statusprints sessions folded, effective tokens and the “plugin overhead” as a share. It does not estimate what the sessions would have cost without the plugin. Ashadowsetting exists that will “decide and log everything, act on nothing”.
How to try it without fooling yourself
- Install from a commit you have read. The README’s commands clone
main. After the clone, rungit -C TokenEconomizer checkout b57d03f81585a18b936a0eaecbd1eb51cdcecf71, or a later commit you reviewed, because a plugin runs code at every hook; our report on a tampered Claude Code extension shows why that matters. - Start in shadow mode. The plugin README describes the
shadowsetting as “decide and log everything, act on nothing”, and the README says each setting’s environment variable isTE_plus its name in capitals, soTE_SHADOW=1should switch it on for one session;plugin/config.pylists the variable. I did not run Claude Code with it. - Run the one-time catalog step the Quick start asks for,
/tokeneconomizer:toolprune refresh-catalog, and run it again after Claude Code updates. - Decide about micro-skills on purpose with
/tokeneconomizer:config set microskills=on, knowing that it feeds text derived from command output into your later sessions. - Keep a control.
TE_DISABLE=1 claudeturns the plugin off for one session. Compare effective tokens per session over weeks, not days: the paper’s default arm had a seed-to-seed standard deviation of 1.7 to 4.2 tasks on a fixed 277-task set, and your own work is far less uniform than a fixed task set. - Check what it wrote. Tool pruning writes deny rules to
.claude/settings.local.json, which the README says the plugin keeps out of git, and/tokeneconomizer:toolprune restoregives the tools back.
Run the checks yourself
Recompute the paper’s numbers
This script needs only Python 3.10 or newer and network access. It downloads the project page’s data file, implements the Student t distribution without a library, and prints the figures quoted above. I checked its p-values against SciPy’s. The page can be updated after this article, so the script prints a hash and byte count of the file it read.
# te_checks.py
"""Recompute TokenEconomizer's headline checks from the numbers its own project page ships.
Python 3.10 or newer, standard library only. The page's assets/js/data.js is generated from the paper's sources;
this script downloads it, so it needs network access. Output is the numbers the article quotes.
"""
import hashlib
import json
import math
import urllib.request
URL = "https://tokeneconomizer.github.io/assets/js/data.js"
N_TASKS, SEEDS = 277, 3
def load():
req = urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"})
raw = urllib.request.urlopen(req, timeout=60).read()
text = raw.decode("utf-8")
return json.loads(text[text.index("{"): text.rindex("}") + 1]), raw
def betacf(a, b, x): # continued fraction for the incomplete beta function (Lentz's method)
tiny = 1e-300
qab, qap, qam = a + b, a + 1.0, a - 1.0
c, d = 1.0, 1.0 - qab * x / qap
d = 1.0 / (d if abs(d) > tiny else tiny)
h = d
for m in range(1, 300):
m2 = 2 * m
aa = m * (b - m) * x / ((qam + m2) * (a + m2))
d = 1.0 / (1.0 + aa * d if abs(1.0 + aa * d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
h *= d * c
aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
d = 1.0 / (1.0 + aa * d if abs(1.0 + aa * d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
delta = d * c
h *= delta
if abs(delta - 1.0) < 1e-14:
break
return h
def betainc(a, b, x):
if x <= 0.0:
return 0.0
if x >= 1.0:
return 1.0
front = math.exp(math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b) + a * math.log(x) + b * math.log(1.0 - x))
if x < (a + 1.0) / (a + b + 2.0):
return front * betacf(a, b, x) / a
return 1.0 - front * betacf(b, a, 1.0 - x) / b
def p_two_sided(t, df): # Student t distribution
return betainc(df / 2.0, 0.5, df / (df + t * t))
def t_crit(df, alpha=0.05):
lo, hi = 0.0, 60.0
for _ in range(80):
mid = (lo + hi) / 2.0
lo, hi = (mid, hi) if p_two_sided(mid, df) > alpha else (lo, mid)
return (lo + hi) / 2.0
D, raw = load()
S = [s["id"] for s in D["settings"]]
NAME = {s["id"]: f'{s["agent"]} {s["model"]}' for s in D["settings"]}
T1 = D["table1"]
print("data.js sha256", hashlib.sha256(raw).hexdigest()[:16], "bytes", len(raw))
print("\n1. Cost, TokenEconomizer against default (percent change, SWE-Bench, 277 tasks)")
costs = [T1["te"][s]["cost"]["v"] for s in S]
print(" per setting", costs, "| range", max(costs), "to", min(costs), "| mean", round(sum(costs) / len(costs), 2))
print(" as multiples of the seed SD:", [round(T1["te"][s]["cost"]["v"] / T1["te"][s]["cost"]["sd"], 1) for s in S])
print("\n2. Pass@1, Welch test with three seeds per arm (reported SD read as a sample SD)")
print(f' {"setting":22} {"default":>10} {"TE":>10} {"diff":>6} {"tasks":>6} {"t":>6} {"df":>4} {"p":>6} {"needed":>7}')
for s in S:
d, e = T1["default"][s]["pass"], T1["te"][s]["pass"]
diff = round(e["v"] - d["v"], 1)
v1, v2 = d["sd"] ** 2 / SEEDS, e["sd"] ** 2 / SEEDS
se = math.sqrt(v1 + v2)
df = (v1 + v2) ** 2 / (v1 ** 2 / (SEEDS - 1) + v2 ** 2 / (SEEDS - 1))
t = diff / se
print(f' {NAME[s]:22} {d["v"]:5.1f}+/-{d["sd"]:.1f} {e["v"]:5.1f}+/-{e["sd"]:.1f} {diff:+6.1f} {diff * N_TASKS / 100:+6.1f} '
f'{t:6.2f} {df:4.1f} {p_two_sided(t, df):6.3f} {t_crit(df) * se:7.2f}')
print(" (diff in points; tasks = diff x 277 / 100; needed = the gap that would give p = 0.05)")
print("\n3. Effort arms against TokenEconomizer: which saves more")
print(" medium effort saves more in", sum(1 for s in S if T1["medium"][s]["cost"]["v"] < T1["te"][s]["cost"]["v"]), "of 5 settings;",
"low effort in", sum(1 for s in S if T1["low"][s]["cost"]["v"] < T1["te"][s]["cost"]["v"]), "of 5")
print("\n4. The 30 repository-by-setting cost cells")
cells = {(r["setting"], r["repo"]): r["cost"] for r in D["perRepo"] if r["arm"] == "te"}
vals = sorted(cells.values())
print(" cells", len(vals), "| all cheaper:", all(v < 0 for v in vals), "| smallest saving", vals[-1], "| largest", vals[0])
print(" under 10 percent:", [(NAME[s], r, v) for (s, r), v in sorted(cells.items(), key=lambda kv: -kv[1]) if v > -10])
for repo in D["repos"]:
row = [cells[(s, repo)] for s in S]
print(f" {repo:13} {row} mean {sum(row) / 5:6.1f}")
print("\n5. Cumulative cost change over each stream (two Claude Code settings averaged)")
for repo, pts in D["amortization"].items():
xs, ys = [p[0] for p in pts], [p[1] for p in pts]
first = next(x for i, x in enumerate(xs) if all(y < 0 for y in ys[i:]))
print(f" {repo:13} tasks {len(xs):2} first task {ys[0]:+6.1f} cheaper for good from task {first} final {ys[-1]:+6.1f}")
finals = [pts[-1][1] for pts in D["amortization"].values()]
print(" final range", max(finals), "to", min(finals))
Output on November 17, 2026:
data.js sha256 2b6a02437bea4853 bytes 37410
1. Cost, TokenEconomizer against default (percent change, SWE-Bench, 277 tasks)
per setting [-23.6, -30.9, -24.7, -26.1, -18.4] | range -18.4 to -30.9 | mean -24.74
as multiples of the seed SD: [-13.1, -14.7, -20.6, -12.4, -12.3]
2. Pass@1, Welch test with three seeds per arm (reported SD read as a sample SD)
setting default TE diff tasks t df p needed
Claude Code Sonnet 5 74.7+/-1.3 76.0+/-1.2 +1.3 +3.6 1.27 4.0 0.272 2.84
Claude Code Opus 4.8 74.0+/-0.6 73.0+/-0.6 -1.0 -2.8 -2.04 4.0 0.111 1.36
Codex GPT-5.6 Terra 73.8+/-1.0 74.7+/-1.0 +0.9 +2.5 1.10 4.0 0.332 2.27
Codex GPT-5.6 Sol 80.1+/-0.8 81.7+/-0.7 +1.6 +4.4 2.61 3.9 0.061 1.72
Copilot CLI Sonnet 5 75.0+/-1.5 77.6+/-1.3 +2.6 +7.2 2.27 3.9 0.087 3.21
(diff in points; tasks = diff x 277 / 100; needed = the gap that would give p = 0.05)
3. Effort arms against TokenEconomizer: which saves more
medium effort saves more in 4 of 5 settings; low effort in 5 of 5
4. The 30 repository-by-setting cost cells
cells 30 | all cheaper: True | smallest saving -5.2 | largest -40.1
under 10 percent: [('Claude Code Sonnet 5', 'django', -5.2), ('Copilot CLI Sonnet 5', 'sympy', -8.3), ('Copilot CLI Sonnet 5', 'django', -9.3)]
scikit-learn [-28.6, -24.4, -26.4, -27.8, -23.2] mean -26.1
sympy [-25.8, -33.9, -23.7, -25.1, -8.3] mean -23.4
django [-5.2, -28.5, -15.3, -16.7, -9.3] mean -15.0
sphinx [-20.8, -40.1, -27.9, -29.3, -19.9] mean -27.6
matplotlib [-27.0, -37.0, -27.9, -29.3, -16.6] mean -27.6
xarray [-27.0, -21.2, -27.1, -28.5, -30.1] mean -26.8
5. Cumulative cost change over each stream (two Claude Code settings averaged)
django tasks 40 first task +6.8 cheaper for good from task 6 final -16.9
matplotlib tasks 34 first task +22.5 cheaper for good from task 2 final -32.0
scikit-learn tasks 60 first task +15.6 cheaper for good from task 4 final -26.5
sphinx tasks 44 first task +38.1 cheaper for good from task 4 final -30.4
sympy tasks 60 first task +11.3 cheaper for good from task 2 final -29.9
xarray tasks 39 first task -8.5 cheaper for good from task 1 final -24.1
final range -16.9 to -32.0
Drive the shipped hooks
Clone the repository, check out the commit named above, and pass the clone’s path to the script. It imports the plugin from the clone, strips any environment variable that looks like a key or token before doing so, and writes only to a temporary folder.
git clone https://github.com/microsoft/TokenEconomizer.git
git -C TokenEconomizer checkout b57d03f81585a18b936a0eaecbd1eb51cdcecf71
python default_install_lab.py TokenEconomizer
# default_install_lab.py
"""Drive the shipped TokenEconomizer hooks with synthetic Claude Code sessions and report what a default install does.
Usage: python default_install_lab.py PATH_TO_CLONE (a clone of microsoft/TokenEconomizer at the commit named in the article)
Python 3.10 or newer, standard library only. The plugin package is imported from the clone. Everything is local: a
temporary repository, a temporary state folder, folds run inline (TE_FOLD_SYNC=1), and no model call is made (the
micro-skills proposer is off by default and the loop guard is never woken). The driving pattern is the one the repository's
own tests/test_live_hooks.py uses.
"""
import io
import json
import os
import sys
import tempfile
CLONE = os.path.abspath(sys.argv[1])
sys.path.insert(0, CLONE)
for name in list(os.environ): # nothing secret should be near code we did not write
if any(word in name.upper() for word in ("KEY", "TOKEN", "SECRET", "PASSWORD")):
os.environ.pop(name)
WORK = tempfile.mkdtemp(prefix="te_lab_")
os.environ["HOME"] = os.environ["USERPROFILE"] = os.path.join(WORK, "userhome")
from tokeneconomizer.plugin import denylist, hook # noqa: E402
from tokeneconomizer.plugin.paths import repo_identity, repo_state_dir # noqa: E402
from tokeneconomizer.plugin.store import Store # noqa: E402
DOTS = chr(0x2026)
def ascii_line(text, n=170):
return text.replace(DOTS, "...").replace("\n", " ").encode("ascii", "replace").decode()[:n]
def make_repo(name):
repo = os.path.join(WORK, name, "repo")
os.makedirs(os.path.join(repo, ".git", "info"))
with open(os.path.join(repo, ".git", "HEAD"), "w") as fh:
fh.write("ref: refs/heads/main\n")
with open(os.path.join(repo, ".git", "config"), "w") as fh:
fh.write('[remote "origin"]\n\turl = https://github.com/acme/widget.git\n')
with open(os.path.join(repo, "pyproject.toml"), "w") as fh:
fh.write("[project]\nname='w'\n")
env = {"TOKENECONOMIZER_HOME": os.path.join(WORK, name, "home"), "TE_FOLD_SYNC": "1",
"GIT_CEILING_DIRECTORIES": WORK, "PATH": os.environ.get("PATH", "")}
return repo, env
def write_session(path, prompts):
"""An interactive Claude Code transcript: a user entry per prompt, then a tool_use and a tool_result per call."""
lines, n = [], 0
for text, calls in prompts:
lines.append({"type": "user", "message": {"role": "user", "content": text}})
for tool, tool_input, result, is_error in calls:
n += 1
lines.append({"type": "assistant", "message": {
"id": f"msg_{os.path.basename(path)}_{n}", "model": "claude-opus-5-5", "role": "assistant",
"usage": {"input_tokens": 10, "cache_read_input_tokens": 1000, "cache_creation_input_tokens": 100, "output_tokens": 50},
"content": [{"type": "tool_use", "id": f"tu{n}", "name": tool, "input": tool_input}]}})
lines.append({"type": "user", "message": {"role": "user", "content": [
{"type": "tool_result", "tool_use_id": f"tu{n}", "content": result, "is_error": is_error}]}})
with open(path, "w") as fh:
fh.write("\n".join(json.dumps(x) for x in lines) + "\n")
return path
def ordinary_session(path, repo, i):
return write_session(path, [
(f"look at module {i}", [("Read", {"file_path": f"{repo}/a.py"}, "x", False), ("Grep", {"pattern": "foo"}, "a.py:1", False)]),
("edit it", [("Edit", {"file_path": f"{repo}/a.py", "old_string": "a", "new_string": "b"}, "ok", False)]),
("run the checks", [("Bash", {"command": "ls"}, "a.py", False)])])
def trap_session(path, repo, i):
"""Every session falls into the same trap: bare pytest is missing here, python -m pytest works."""
return write_session(path, [(f"fix bug {i}", [
("Edit", {"file_path": f"{repo}/pkg/mod{i}.py", "old_string": "a", "new_string": "b"}, "ok", False),
("Bash", {"command": "pytest -q tests/test_a.py"}, "bash: pytest: command not found", True),
("Bash", {"command": "python -m pytest -q tests/test_a.py"}, "1 passed in 0.1s", False)])])
def call(env, payload):
out = io.StringIO()
hook.main([], io.StringIO(json.dumps(payload)), out, env)
return json.loads(out.getvalue().splitlines()[-1])
def run_session(env, repo, sid, transcript):
base = {"session_id": sid, "transcript_path": transcript, "cwd": repo}
start = call(env, dict(base, hook_event_name="SessionStart", source="startup"))
call(env, dict(base, hook_event_name="SessionEnd", reason="other"))
return start # what the NEXT-session hook said when this session began
def untool_state(env, repo):
return Store(repo_state_dir(repo_identity(repo, env), "claude", env)).read("untool.json") or {}
def deny_count(repo):
path = denylist.settings_path(repo)
return 0 if not os.path.exists(path) else len(json.load(open(path)).get("permissions", {}).get("deny", []))
def write_catalog(env, tools): # what `toolprune refresh-catalog` stores for Claude Code
folder = os.path.join(env["TOKENECONOMIZER_HOME"], "hosts", "claude")
os.makedirs(folder, exist_ok=True)
with open(os.path.join(folder, "catalog.json"), "w") as fh:
json.dump({"tools": tools}, fh)
def pruning(label, extra_env, snapshot, sessions):
repo, env = make_repo(label)
env.update(extra_env)
if snapshot:
write_catalog(env, snapshot)
proposed, denied, notices = [], [], []
for i in range(1, sessions + 1):
start = run_session(env, repo, f"s{i}", ordinary_session(os.path.join(WORK, f"{label}_{i}.jsonl"), repo, i))
state = untool_state(env, repo)
proposed.append(len(state.get("proposal") or (state.get("auto_pending") or {}).get("items") or []))
denied.append(deny_count(repo))
if start.get("systemMessage"):
notices.append((i, start["systemMessage"]))
state = untool_state(env, repo)
waiting = sorted((state.get("auto_pending") or {}).get("items") or [])
return proposed, denied, notices, waiting, state.get("catalog_source")
snapshot = list(json.load(open(os.path.join(CLONE, "tokeneconomizer", "trace", "claude_tool_catalog.json"), encoding="utf-8")))
print("snapshot:", len(snapshot), "Claude Code tool names from the repository's own catalog file")
p, d, n, w, src = pruning("A", {}, None, 12)
print("\nA fresh install, no refresh-catalog step, 12 sessions that only use Read, Grep, Edit and Bash")
print(" tools proposed after each session:", p, "| deny rules written:", d[-1], "| catalog source:", src)
p, d, n, w, src = pruning("B", {}, snapshot, 6)
print("\nB default settings after refresh-catalog (24 names), 6 sessions")
print(" tools waiting after each session:", p, "| deny rules written after each:", d)
print(" waiting:", ", ".join(w))
p, d, n, w, src = pruning("B0", {"TE_TOOLPRUNE_QUIET_MINUTES": "0"}, snapshot, 6)
print("\nB0 the same with the idle wait set to 0 minutes")
print(" deny rules written after each session:", d)
for i, text in n:
print(f" notice at the start of session {i}:", ascii_line(text))
print("\nC micro-skills, 8 sessions that all fall into the missing-pytest trap")
for label, extra in (("default", {}), ("TE_MICROSKILLS=envfacts", {"TE_MICROSKILLS": "envfacts"})):
repo, env = make_repo("C" + str(len(label)))
env.update(extra)
got = []
for i in range(8):
start = run_session(env, repo, f"c{i}", trap_session(os.path.join(WORK, f"C{len(label)}_{i}.jsonl"), repo, i))
got.append((start.get("hookSpecificOutput") or {}).get("additionalContext", ""))
first = next((i + 1 for i, text in enumerate(got) if text), None)
print(f" {label:24} session starts that carried a fact: {sum(1 for t in got if t)} of 8; first at session {first}")
if first:
print(" ", ascii_line(got[first - 1], 200))
Output:
snapshot: 24 Claude Code tool names from the repository's own catalog file
A fresh install, no refresh-catalog step, 12 sessions that only use Read, Grep, Edit and Bash
tools proposed after each session: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | deny rules written: 0 | catalog source: observed
B default settings after refresh-catalog (24 names), 6 sessions
tools waiting after each session: [0, 0, 0, 12, 12, 12] | deny rules written after each: [0, 0, 0, 0, 0, 0]
waiting: CronCreate, CronDelete, CronList, DesignSync, EnterWorktree, ExitWorktree, ListAgents, PushNotification, ReportFindings, ScheduleWakeup, SendMessage, Workflow
B0 the same with the idle wait set to 0 minutes
deny rules written after each session: [0, 0, 0, 12, 12, 12]
notice at the start of session 5: TokenEconomizer removed 12 tool(s) this repository's sessions never used (CronCreate, CronDelete, CronList, DesignSync...), from this session on. Review with `/tokenecono
C micro-skills, 8 sessions that all fall into the missing-pytest trap
default session starts that carried a fact: 0 of 8; first at session None
TE_MICROSKILLS=envfacts session starts that carried a fact: 4 of 8; first at session 5
# environment facts learned from earlier sessions in this repository - `pytest ...` fails here with command not found; `python -m pytest ...` works.
What would settle it
- A run of the plugin as shipped. The harness’s controllers differ from the plugin’s, so the 18.4% to 30.9% cannot stand in for it. The same 277 tasks run through the plugin’s hooks, with micro-skills on and off, would give the number a user needs.
- Per-task, per-seed outcomes. With them, Pass@1 could be tested pair by pair instead of through three seed means, and the claim could be stated as a confidence interval.
- Held-out repositories for the constants. Appendix C’s values were fixed on the same six repositories the SWE-Bench results use. Web-Bench and the security sets are the only evidence outside them.
Bottom line
TokenEconomizer’s cost result is solid for what was measured: a learned controller, run in a sealed benchmark harness, lowered cost by roughly a fifth to a third without a detectable loss in success. What the evidence does not show is an improvement in Pass@1, or what a default install of the plugin saves, and the repository’s own documentation says as much. The blog and the project page would be more accurate if they carried the paper’s abstract wording and the one table from PLUGIN.md that lists what changes.
Sources
- Microsoft Research blog post (November 16, 2026)
- TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents, arXiv 2610.22732, with the HTML version
- Project page and its data file
- Repository at commit b57d03f: README, docs/PLUGIN.md, plugin/README.md, docs/ARCHITECTURE.md, CLAUDE.md, tokeneconomizer/plugin/adapter.py, tokeneconomizer/plugin/fold.py, tokeneconomizer/plugin/config.py, experiments/tests/test_tooling.py, LICENSE and pyproject.toml








No Comment! Be the first one.