TRENDING
A person sweeping a metal detector across a wet sandy beach with a sea fort on the horizon
November 22, 2026
How to Write YARA Rules With YARA-X to Hunt Malicious Python .pth Startup Files and Measure False Positives Before You Trust Them
A peeling black-and-white metal door standing ajar, with a dark empty hall and bright windows beyond
November 22, 2026
AWS’s “Found a Zero-Day on Their Own” Post Never Names the Product; the Advisory Behind Its Link Shows a Server That Ships With No Login, a README Now Saying Keep It Off the Internet, and a Fix Merged October 19 That PyPI and Docker Hub Still Do Not Carry
A hand paging through index cards in an open wooden library card-catalog drawer labeled Agriculture, with a second drawer open below to the right
November 22, 2026
Microsoft Tells Customers Unauthorized Parties Read Admin Email Addresses and Tenant Metadata Through an Entra Connect Health Flaw Introduced July 23, but Gives No Tenant Count or Start Date, and Its Own Docs Send Tenant Security Notices to the Same Kinds of Contacts
Swimmers in separate lanes of a blue swimming pool, seen from above the water, with lane lines running across the picture
November 22, 2026
How to Size a PostgreSQL Connection Pool in Python: Find Where Throughput Stops Growing, Split max_connections Across Workers, and Stop Slow Handlers From Hoarding Connections
Workers in caps lean over a 1913 Ford assembly line, each assembling parts as they pass on the moving line
November 22, 2026
OpenAI’s Performance Factory Turns Regression Response Into a Task Queue for Agents, but Its Numbers Stop at 600 Root-Caused Cases in 1,700 Investigations and a 70% Grade on 100 Suggestions, and in a Toy World Each of Its Reviewer’s Two Counterfactual Questions Can Return a Confident Wrong Answer
22 Nov 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
The sun setting over the sea behind a pyramid-shaped stone monument at Cape Soya, the northernmost point of Japan
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
November 22, 2026
Brass sextant in an open wooden case inside a museum display, an instrument for measuring the angle to the horizon
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
November 22, 2026
Brass telescope on a wooden tripod inside a white observatory dome with the roof slit open to the sky
Rackspace Says Leaked Credentials Let an Intruder Act as Legitimate Users in Some Spot Customer Organizations and Reach Their Kubernetes Secrets, a Platform9 Co-Founder Names a Breached Observability System as the Root Cause, and the Status Page Says Only “System Maintenance Operations”
November 22, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 310 Posts
News 316 Posts
Learning Hub 282 Posts
Home/Articles/Microsoft’s TokenEconomizer Turns Coding-Agent Cost Into a Learned Policy, but Its 18.4% to 30.9% Was Measured on the Paper’s Controller, Not the Plugin’s Defaults: Micro-Skills Ship Off, the Plugin Never Stops a Turn, and No Pass@1 Gain Clears a Three-Seed Test
Articles

Microsoft’s TokenEconomizer Turns Coding-Agent Cost Into a Learned Policy, but Its 18.4% to 30.9% Was Measured on the Paper’s Controller, Not the Plugin’s Defaults: Micro-Skills Ship Off, the Plugin Never Stops a Turn, and No Pass@1 Gain Clears a Three-Seed Test

TokenEconomizer’s 18.4% to 30.9% cost cut reproduces from its project page’s data, but it was measured on the paper’s controller, not the plugin’s defaults, and none of the Pass@1 gains clears a...

November 17, 2026 23 Min Read
69

On November 16, Microsoft Research published a blog post on TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research and Microsoft Security that learns from a coding agent’s earlier sessions in a repository and uses what it learned to cut the cost of later ones. The post’s headline is that, across 277 SWE-Bench tasks, the plugin lowered mean cost by 18.4% to 30.9% over five agent and model combinations, and that “Costs fell in every repository tested, while task success rates remained broadly stable.” The numbers come from a paper (arXiv 2610.22732, submitted October 15), a project page that ships the data behind its charts, and a public GitHub repository that was created on October 13 and showed 7,369 stars on November 17.

Table Of Content

  • What each source covers
  • What reproduces
  • The cost figures
  • The 30 cells and the sixth task
  • The profiling figures
  • Where the headline reads stronger than the evidence
  • The Pass@1 gains are inside the seed noise
  • “Saved the most” in all 30 pairs
  • What was tuned on what, and how the streams were built
  • The plugin is not the paper’s controller
  • Side by side
  • Where the measured saving sits
  • What a default install does on synthetic sessions
  • Tool pruning
  • Micro-skills
  • Smaller things worth knowing
  • How to try it without fooling yourself
  • Run the checks yourself
  • Recompute the paper’s numbers
  • Drive the shipped hooks
  • What would settle it
  • Bottom line
  • Sources

I checked the blog against all three, recomputed what could be recomputed, and drove the plugin’s own hooks with synthetic Claude Code sessions. The short version:

  • The cost numbers reproduce. Every cost figure I could recompute from the project page’s data matches the paper, including 30 of 30 repository-by-setting cells cheaper, from 5.2% to 40.1%.
  • No Pass@1 gain clears a three-seed test. The four gains give Welch p-values of 0.27, 0.33, 0.06 and 0.09, and the largest is 7.2 tasks out of 277. The paper’s abstract says “comparable task success”; the README and the project page headline Pass@1 as up in 4 of 5 settings.
  • The paper’s numbers belong to the experimental controller, not the plugin’s defaults. The plugin’s own design document lists the differences: micro-skills are off by default, tool pruning has no per-task judge and protects tools a person uses, and the loop guard never stops a turn, where the paper’s guard aborted about two trajectories per stream. The blog mentions only the last one.
  • One blog sentence overreaches. It says the plugin “saved the most in all 30 repository-combination pairs”. The paper says it saved something in all 30, and lowering reasoning effort saved more in 4 of 5 settings (medium) and 5 of 5 (low).

Two caveats up front. I did not run a coding agent with the plugin installed, so nothing here measures what the plugin saves in real use. And the repository’s documentation is more open about the differences than the blog or the project page, which is the only reason I can list them.

What each source covers

The blog summarizes the paper, and the project page’s data.js begins with the comment “Generated by scripts/build_page_assets.py from the paper sources; do not edit.” The repository holds two things: the plugin (tokeneconomizer/ and plugin/) and the benchmark harness behind the paper (experiments/). Its architecture note says the harness holds “benchmark-only controllers”, and its CLAUDE.md says “No run outputs in the repository”, so the code and task lists are public but per-task results are not. I read the repository at commit b57d03f, the merge of pull request 246 at 07:56 UTC on November 17.

What reproduces

The cost figures

Table 1 of the paper gives the change in mean cost against default execution, with the standard deviation across three seeds. The project page’s data has the same figures for the plugin’s arm:

Setting Cost change Seed SD (points) Change in seed SDs
Claude Code, Sonnet 5 −23.6% 1.8 13.1
Claude Code, Opus 4.8 −30.9% 2.1 14.7
Codex, GPT-5.6 Terra −24.7% 1.2 20.6
Codex, GPT-5.6 Sol −26.1% 2.1 12.4
Copilot CLI, Sonnet 5 −18.4% 1.5 12.3

The mean is 24.7%, and the blog’s range is simply the smallest and largest of these five. Each figure sits 12 to 21 seed standard deviations from zero, so inside this benchmark the saving is not run-to-run noise.

The 30 cells and the sixth task

The claim that cost fell in every repository under every setting also reproduces. All 30 cells in the project page’s per-repository data are negative, from −5.2% (Sonnet 5 on Django) to −40.1% (Opus 4.8 on Sphinx), and three are under 10%: Django with Sonnet 5, and SymPy and Django with Copilot CLI at −8.3% and −9.3%. The cells below are listed in the order Claude Code with Sonnet 5, Claude Code with Opus 4.8, Codex with Terra, Codex with Sol, and Copilot CLI with Sonnet 5.

Repository Tasks Cost change in the five settings (%) Mean
scikit-learn 60 −28.6, −24.4, −26.4, −27.8, −23.2 −26.1
SymPy 60 −25.8, −33.9, −23.7, −25.1, −8.3 −23.4
Django 40 −5.2, −28.5, −15.3, −16.7, −9.3 −15.0
Sphinx 44 −20.8, −40.1, −27.9, −29.3, −19.9 −27.6
Matplotlib 34 −27.0, −37.0, −27.9, −29.3, −16.6 −27.6
xarray 39 −27.0, −21.2, −27.1, −28.5, −30.1 −26.8

Django is the weakest stream by a wide margin. It is also the stream the authors built to have related tasks close together: its 40 Verified tasks come “all from the start of its Verified history (November 2016 to September 2019), so that consecutive tasks are typically days apart”. That the densest stream saved the least counts in the paper’s favor, but it shows how much the choice of repository and stream moves the result.

The blog’s Figure 4 caption says “every repository becomes cheaper by the sixth task”, and that holds too. The cumulative cost change, averaged over the two Claude Code settings, is below zero for good from task 1 on xarray, task 2 on Matplotlib and SymPy, task 4 on scikit-learn and Sphinx, and task 6 on Django. It ends between −16.9% and −32.0%, the paper’s 16.9% to 32.0%. In five of the six streams the curve starts above zero, at +6.8% to +38.1% after one task, as the paper says.

The profiling figures

The numbers that motivate the design match the paper as well: 23 tool schemas loaded on every turn with 16 never called, failed runs at 17% of tasks but 32% of spend, and 30% of tool calls repeating an earlier action in the same repository against 7% for another repository’s history. They come from 60 profiled tasks, ten per repository, and at that size the three fixed strategies (RTK, Serena and RepoMap) swing from −20.2% to +30.8% per repository. The larger run backs the paper’s “no fixed strategy reduces cost consistently” on its own: RTK alone raised cost in 12 of the 30 cells and RepoMap alone in 15.

Where the headline reads stronger than the evidence

The Pass@1 gains are inside the seed noise

The abstract promises “comparable task success”. The introduction goes further, “improving Pass@1 over default execution in 4 of 5” settings, and the project page lists “Pass@1 also improves” as one of its three headline tiles. Table 1 gives each Pass@1 as a mean with a standard deviation across three seeds, which is enough for a Welch test of each pair of arms:

Setting Default TokenEconomizer Change (points) Tasks of 277 Welch p Gap needed for p = 0.05
Claude Code, Sonnet 5 74.7 ± 1.3 76.0 ± 1.2 +1.3 +3.6 0.27 2.8
Claude Code, Opus 4.8 74.0 ± 0.6 73.0 ± 0.6 −1.0 −2.8 0.11 1.4
Codex, GPT-5.6 Terra 73.8 ± 1.0 74.7 ± 1.0 +0.9 +2.5 0.33 2.3
Codex, GPT-5.6 Sol 80.1 ± 0.8 81.7 ± 0.7 +1.6 +4.4 0.06 1.7
Copilot CLI, Sonnet 5 75.0 ± 1.5 77.6 ± 1.3 +2.6 +7.2 0.09 3.2

None of the four gains reaches p = 0.05, and the nearest, Sol, needs 1.7 points and shows 1.6. The one loss, Opus 4.8 at −1.0 point, is a stronger signal in standard-error terms (t = −2.04) than the Sonnet 5 and Terra gains (t = 1.27 and 1.10). The largest gain is 7.2 tasks out of 277, while the default arm’s own seed-to-seed standard deviation is 1.7 to 4.2 tasks. Reading the reported deviations as population rather than sample deviations raises every p-value, to between 0.10 and 0.42.

These tests are generous. The 277 tasks are held fixed, so they ask only whether a gap exceeds the agent’s run-to-run noise on those tasks, not whether the tasks stand for anything else. A paired per-task test would be sharper, but the per-task outcomes are not published. The data support the abstract’s wording, comparable success at lower cost. They do not support reading “4 of 5” as an improvement.

“Saved the most” in all 30 pairs

The blog says TokenEconomizer “saved the most in all 30 repository-combination pairs without performance loss”. The paper’s statement is narrower: it reduced cost in all 30 cells, by 5.2% to 40.1%. The blog’s next sentence, that lowering reasoning effort “saved similar or larger amounts”, is the accurate one. Medium effort saved more than TokenEconomizer in 4 of the 5 settings and low effort in all 5, at a price of 1.8 to 4.7 points of Pass@1 and 3.6 to 10.3 points respectively. Read literally, the first sentence is not what the tables show, and it is the one a reader is likely to quote.

What was tuned on what, and how the streams were built

  • Constants. Appendix C says “All values were fixed before the main runs, in earlier development runs on the same six SWE-Bench repositories.” No SWE-Bench repository was held out. The Web-Bench and security results reuse the constants on other data, with two changes the paper states: a pre-registered 30-turn budget for the short Web-Bench streams, and micro-skills switched off on the security benchmarks, where every task is a new repository.
  • Task mix. “Of the 277 tasks, 199 come from SWE-Bench Verified and 78 from the full SWE-Bench test split.” The paper says the extra scikit-learn and SymPy tasks were added to raise difficulty, that 30 of xarray’s 39 tasks come from the full split too, and that the authors applied each added task’s gold patch in the official image to check that it passes. The blog’s “277 real bug-fixing tasks” does not mention the split.
  • Streams. Each repository’s tasks run in order, so the controller only learns from earlier work, but the paper’s limitations say that “our SWE-Bench streams consist of related but independent fixes rather than actual user sessions”.
  • Other benchmarks. Costs fell 6.4% to 19.6% on the two security benchmarks and 16.0% to 21.2% on Web-Bench, with quality “remaining comparable to default execution”. ReposVul’s F1 sits between 17.5% and 22.5% in Table 2(b), so that comparison is between two weak detectors, and Copilot CLI’s score fell slightly on CredData (68.2 to 67.7) and on Web-Bench Pass@2 (60.8 to 59.3).

The plugin is not the paper’s controller

The repository is unusually candid about this. docs/PLUGIN.md has a section titled “What changes relative to the experimental setting”, and CLAUDE.md says the plugin’s own text “makes no claims of effect”. The blog’s only mention of a difference is the loop guard: “In the open-source plugin, if the agent is still stuck after the note, TokenEconomizer alerts the developer rather than stopping the work itself.” The other differences matter more.

Side by side

Mechanism Measured in the paper Plugin as shipped, on Claude Code
Micro-skills A model proposes facts after task 4 and every 5 tasks, from digests of the 8 latest tasks, in a Python and git or a Node and web vocabulary. Alone it saved 11.8% and 14.3%. Off by default, “because its proposer adds model calls and how much it saves depends on the repository and the work you do there”. When on, the proposer runs every 3 sessions with a generic vocabulary; a free layer, envfacts, learns from repeated failures without a model call.
Tool pruning A tool is masked after 3 consecutive tasks without a call, or when a per-task LLM judge that reads the issue text finds it irrelevant. The full catalog returns every 5 tasks for a re-audit, and the mask is set at launch. Alone it saved 13.2% and 17.7%. A tool is proposed after going unused over 10 prompts across at least 2 sessions, or over 5 sessions. The plugin builds its pruning stage with no judge and no re-audit, and protects AskUserQuestion, plan mode, sub-agents, the Task tools and the web. A deny list is written only after the repository’s sessions have been idle (60 minutes by default) and applies from the next session.
Stuck turn One redirect, in which the agent is killed and resumed with a continuation prompt, then a terminal abort, plus a forced stop at 2.5 times the repository’s turn budget when a strict stall rule holds. Alone it saved 7.1% and 5.4%, with 1.4 and 1.7 aborts per stream (2.0 and 2.2 in the full stack). A mid-turn reminder attached to a tool result, then a one-line notice to the person: “the plugin never stops a turn”. The docs call it a “Different intervention, so its effect is not inherited from the experiments”.
Success signal The benchmark’s grader. A turn that “ended green”: the agent stopped on its own, the guard did not stop it, and no verification after its last edit failed.
Model for overhead The agent’s own model, “so overhead could not look cheaper than its arm”. The cheapest capable model on the user’s own agent, haiku on Claude Code. This difference favors the plugin.

Some code is shared. The plugin’s loop guard is the benchmark guard’s control flow applied one tool call at a time, and PLUGIN.md says “None of the thresholds were re-tuned.” What the table shows is that the plugin’s defaults enable a more conservative controller than the one that produced the headline.

Where the measured saving sits

The ablation in Table 2(a) splits the Claude Code results. Alone, tool pruning saved 13.2% (Sonnet 5) and 17.7% (Opus 4.8), micro-skills 11.8% and 14.3%, and the failure-loop guard 7.1% and 5.4%; the full stack saved 23.6% and 30.9%. By Table 6, the full stack had about 15 tool schemas masked at the end of a stream, against the 23 the profile found loaded, and its guard stopped 2.0 and 2.2 trajectories per stream. The two largest shares are the two that the shipped defaults change most: micro-skills is off, and pruning runs without the per-task judge, with more tools protected, and only after the repository has been idle. The appendix calls the guard’s aborts rare, “fewer than one in twenty” tasks, and the plugin’s limitations section says that “whether a nudge changes a looping model’s course is not established”.

What a default install does on synthetic sessions

To see the defaults in action I wrote a script that calls tokeneconomizer.plugin.hook.main, the entry point every hook executes, with synthetic Claude Code transcripts, the way the repository’s own tests/test_live_hooks.py does. Everything is local, folds run inline, and no model call is made, because the micro-skills proposer is off by default and the loop guard is never woken. The sessions are made up, so the results show the plugin’s gating, not its savings. The full script is further down; this is what it found.

Tool pruning

Scenario Sessions Result
A: fresh install, no catalog snapshot 12 sessions (36 prompts) that use only Read, Grep, Edit and Bash 0 tools proposed after every session and 0 deny rules written; the catalog held only the 4 tools it had seen used
B: default settings after refresh-catalog (the repository’s own list of 24 Claude Code tool names) 6 sessions Nothing until session 4, then 12 tools waiting; 0 deny rules written after any session
B0: as B with the idle wait set to 0 minutes 6 sessions 12 deny rules written after session 4 and announced at the start of session 5

The 12 are CronCreate, CronDelete, CronList, DesignSync, EnterWorktree, ExitWorktree, ListAgents, PushNotification, ReportFindings, ScheduleWakeup, SendMessage and Workflow. Of the other 12 names in the snapshot, three were used in the sessions, eight are on the protected list (Agent, Monitor, Skill, TaskOutput, TaskStop, WebFetch, WebSearch and Write), and NotebookEdit, which the adapter marks high-risk, was not proposed in six sessions. This matches the README’s settings table, which sets the evidence bar at 10 prompts across at least 2 sessions or at 5 sessions and the idle wait at 60 minutes. It also shows why the Quick start asks for the one-time refresh: without it the catalog is only what sessions have used, and the adapter’s own docstring says that without a snapshot “only deferred tools can be proposed”, which it calls “honest but small”. My synthetic transcripts list no deferred tools, so scenario A is the plain case.

Micro-skills

Eight sessions that all hit the same trap, bare pytest missing while python -m pytest works, left a default install silent: 0 of 8 session starts carried a fact. With TE_MICROSKILLS=envfacts, the free layer that needs no model call, the fact first appeared at the start of session 5 and then in 4 of the 8 session starts. The mechanism works as described. It is simply not on by default, and the docs give a reason that is also a security point: “what it injects is derived from command output a repository controls: turning it on is the person’s call”.

Smaller things worth knowing

  • License. The README badge and pyproject.toml say Apache-2.0, while the LICENSE file is the MIT license (“Copyright (c) 2026 Microsoft”) and GitHub’s detector reports MIT. Anyone adopting the plugin should ask which one governs.
  • Agents. The project page says the plugin “Works with Claude Code, Codex, GitHub Copilot CLI, and Pi”. The paper evaluated three of those; its limitations say its experiments cover “three widely used coding agents but not more minimal harnesses such as Pi”, and the plugin’s docs say that on Pi “tool pruning is not planned there”. The README lists six more hosts as experimental.
  • Tests. On Windows 11 with Python 3.13.14 and pytest 9.2.0, pytest tests ran 1,198 tests: 1,163 passed, 7 failed and 28 were skipped. The failures are five launcher, bundle, symlink and install-copy tests in test_plugin_package.py, one dead-code script test and one session-id path test. pytest experiments/tests ran 397: 381 passed, 10 failed and 6 were skipped, five of the failures in a file whose test names mention cgroups. The repository’s CI runs tests on Ubuntu and Windows runners, so these failures may come from my machine; I did not investigate. The lab script below does not use the shell launcher.
  • What status reports. te status prints sessions folded, effective tokens and the “plugin overhead” as a share. It does not estimate what the sessions would have cost without the plugin. A shadow setting exists that will “decide and log everything, act on nothing”.

How to try it without fooling yourself

  1. Install from a commit you have read. The README’s commands clone main. After the clone, run git -C TokenEconomizer checkout b57d03f81585a18b936a0eaecbd1eb51cdcecf71, or a later commit you reviewed, because a plugin runs code at every hook; our report on a tampered Claude Code extension shows why that matters.
  2. Start in shadow mode. The plugin README describes the shadow setting as “decide and log everything, act on nothing”, and the README says each setting’s environment variable is TE_ plus its name in capitals, so TE_SHADOW=1 should switch it on for one session; plugin/config.py lists the variable. I did not run Claude Code with it.
  3. Run the one-time catalog step the Quick start asks for, /tokeneconomizer:toolprune refresh-catalog, and run it again after Claude Code updates.
  4. Decide about micro-skills on purpose with /tokeneconomizer:config set microskills=on, knowing that it feeds text derived from command output into your later sessions.
  5. Keep a control. TE_DISABLE=1 claude turns the plugin off for one session. Compare effective tokens per session over weeks, not days: the paper’s default arm had a seed-to-seed standard deviation of 1.7 to 4.2 tasks on a fixed 277-task set, and your own work is far less uniform than a fixed task set.
  6. Check what it wrote. Tool pruning writes deny rules to .claude/settings.local.json, which the README says the plugin keeps out of git, and /tokeneconomizer:toolprune restore gives the tools back.

Run the checks yourself

Recompute the paper’s numbers

This script needs only Python 3.10 or newer and network access. It downloads the project page’s data file, implements the Student t distribution without a library, and prints the figures quoted above. I checked its p-values against SciPy’s. The page can be updated after this article, so the script prints a hash and byte count of the file it read.

# te_checks.py
"""Recompute TokenEconomizer's headline checks from the numbers its own project page ships.

Python 3.10 or newer, standard library only. The page's assets/js/data.js is generated from the paper's sources;
this script downloads it, so it needs network access. Output is the numbers the article quotes.
"""
import hashlib
import json
import math
import urllib.request

URL = "https://tokeneconomizer.github.io/assets/js/data.js"
N_TASKS, SEEDS = 277, 3


def load():
    req = urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"})
    raw = urllib.request.urlopen(req, timeout=60).read()
    text = raw.decode("utf-8")
    return json.loads(text[text.index("{"): text.rindex("}") + 1]), raw


def betacf(a, b, x):  # continued fraction for the incomplete beta function (Lentz's method)
    tiny = 1e-300
    qab, qap, qam = a + b, a + 1.0, a - 1.0
    c, d = 1.0, 1.0 - qab * x / qap
    d = 1.0 / (d if abs(d) > tiny else tiny)
    h = d
    for m in range(1, 300):
        m2 = 2 * m
        aa = m * (b - m) * x / ((qam + m2) * (a + m2))
        d = 1.0 / (1.0 + aa * d if abs(1.0 + aa * d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        h *= d * c
        aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
        d = 1.0 / (1.0 + aa * d if abs(1.0 + aa * d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        delta = d * c
        h *= delta
        if abs(delta - 1.0) < 1e-14:
            break
    return h


def betainc(a, b, x):
    if x <= 0.0:
        return 0.0
    if x >= 1.0:
        return 1.0
    front = math.exp(math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b) + a * math.log(x) + b * math.log(1.0 - x))
    if x < (a + 1.0) / (a + b + 2.0):
        return front * betacf(a, b, x) / a
    return 1.0 - front * betacf(b, a, 1.0 - x) / b


def p_two_sided(t, df):  # Student t distribution
    return betainc(df / 2.0, 0.5, df / (df + t * t))


def t_crit(df, alpha=0.05):
    lo, hi = 0.0, 60.0
    for _ in range(80):
        mid = (lo + hi) / 2.0
        lo, hi = (mid, hi) if p_two_sided(mid, df) > alpha else (lo, mid)
    return (lo + hi) / 2.0


D, raw = load()
S = [s["id"] for s in D["settings"]]
NAME = {s["id"]: f'{s["agent"]} {s["model"]}' for s in D["settings"]}
T1 = D["table1"]
print("data.js sha256", hashlib.sha256(raw).hexdigest()[:16], "bytes", len(raw))

print("\n1. Cost, TokenEconomizer against default (percent change, SWE-Bench, 277 tasks)")
costs = [T1["te"][s]["cost"]["v"] for s in S]
print("   per setting", costs, "| range", max(costs), "to", min(costs), "| mean", round(sum(costs) / len(costs), 2))
print("   as multiples of the seed SD:", [round(T1["te"][s]["cost"]["v"] / T1["te"][s]["cost"]["sd"], 1) for s in S])

print("\n2. Pass@1, Welch test with three seeds per arm (reported SD read as a sample SD)")
print(f'   {"setting":22} {"default":>10} {"TE":>10} {"diff":>6} {"tasks":>6} {"t":>6} {"df":>4} {"p":>6} {"needed":>7}')
for s in S:
    d, e = T1["default"][s]["pass"], T1["te"][s]["pass"]
    diff = round(e["v"] - d["v"], 1)
    v1, v2 = d["sd"] ** 2 / SEEDS, e["sd"] ** 2 / SEEDS
    se = math.sqrt(v1 + v2)
    df = (v1 + v2) ** 2 / (v1 ** 2 / (SEEDS - 1) + v2 ** 2 / (SEEDS - 1))
    t = diff / se
    print(f'   {NAME[s]:22} {d["v"]:5.1f}+/-{d["sd"]:.1f} {e["v"]:5.1f}+/-{e["sd"]:.1f} {diff:+6.1f} {diff * N_TASKS / 100:+6.1f} '
          f'{t:6.2f} {df:4.1f} {p_two_sided(t, df):6.3f} {t_crit(df) * se:7.2f}')
print("   (diff in points; tasks = diff x 277 / 100; needed = the gap that would give p = 0.05)")

print("\n3. Effort arms against TokenEconomizer: which saves more")
print("   medium effort saves more in", sum(1 for s in S if T1["medium"][s]["cost"]["v"] < T1["te"][s]["cost"]["v"]), "of 5 settings;",
      "low effort in", sum(1 for s in S if T1["low"][s]["cost"]["v"] < T1["te"][s]["cost"]["v"]), "of 5")

print("\n4. The 30 repository-by-setting cost cells")
cells = {(r["setting"], r["repo"]): r["cost"] for r in D["perRepo"] if r["arm"] == "te"}
vals = sorted(cells.values())
print("   cells", len(vals), "| all cheaper:", all(v < 0 for v in vals), "| smallest saving", vals[-1], "| largest", vals[0])
print("   under 10 percent:", [(NAME[s], r, v) for (s, r), v in sorted(cells.items(), key=lambda kv: -kv[1]) if v > -10])
for repo in D["repos"]:
    row = [cells[(s, repo)] for s in S]
    print(f"   {repo:13} {row}  mean {sum(row) / 5:6.1f}")

print("\n5. Cumulative cost change over each stream (two Claude Code settings averaged)")
for repo, pts in D["amortization"].items():
    xs, ys = [p[0] for p in pts], [p[1] for p in pts]
    first = next(x for i, x in enumerate(xs) if all(y < 0 for y in ys[i:]))
    print(f"   {repo:13} tasks {len(xs):2}  first task {ys[0]:+6.1f}  cheaper for good from task {first}  final {ys[-1]:+6.1f}")
finals = [pts[-1][1] for pts in D["amortization"].values()]
print("   final range", max(finals), "to", min(finals))

Output on November 17, 2026:

data.js sha256 2b6a02437bea4853 bytes 37410

1. Cost, TokenEconomizer against default (percent change, SWE-Bench, 277 tasks)
   per setting [-23.6, -30.9, -24.7, -26.1, -18.4] | range -18.4 to -30.9 | mean -24.74
   as multiples of the seed SD: [-13.1, -14.7, -20.6, -12.4, -12.3]

2. Pass@1, Welch test with three seeds per arm (reported SD read as a sample SD)
   setting                   default         TE   diff  tasks      t   df      p  needed
   Claude Code Sonnet 5    74.7+/-1.3  76.0+/-1.2   +1.3   +3.6   1.27  4.0  0.272    2.84
   Claude Code Opus 4.8    74.0+/-0.6  73.0+/-0.6   -1.0   -2.8  -2.04  4.0  0.111    1.36
   Codex GPT-5.6 Terra     73.8+/-1.0  74.7+/-1.0   +0.9   +2.5   1.10  4.0  0.332    2.27
   Codex GPT-5.6 Sol       80.1+/-0.8  81.7+/-0.7   +1.6   +4.4   2.61  3.9  0.061    1.72
   Copilot CLI Sonnet 5    75.0+/-1.5  77.6+/-1.3   +2.6   +7.2   2.27  3.9  0.087    3.21
   (diff in points; tasks = diff x 277 / 100; needed = the gap that would give p = 0.05)

3. Effort arms against TokenEconomizer: which saves more
   medium effort saves more in 4 of 5 settings; low effort in 5 of 5

4. The 30 repository-by-setting cost cells
   cells 30 | all cheaper: True | smallest saving -5.2 | largest -40.1
   under 10 percent: [('Claude Code Sonnet 5', 'django', -5.2), ('Copilot CLI Sonnet 5', 'sympy', -8.3), ('Copilot CLI Sonnet 5', 'django', -9.3)]
   scikit-learn  [-28.6, -24.4, -26.4, -27.8, -23.2]  mean  -26.1
   sympy         [-25.8, -33.9, -23.7, -25.1, -8.3]  mean  -23.4
   django        [-5.2, -28.5, -15.3, -16.7, -9.3]  mean  -15.0
   sphinx        [-20.8, -40.1, -27.9, -29.3, -19.9]  mean  -27.6
   matplotlib    [-27.0, -37.0, -27.9, -29.3, -16.6]  mean  -27.6
   xarray        [-27.0, -21.2, -27.1, -28.5, -30.1]  mean  -26.8

5. Cumulative cost change over each stream (two Claude Code settings averaged)
   django        tasks 40  first task   +6.8  cheaper for good from task 6  final  -16.9
   matplotlib    tasks 34  first task  +22.5  cheaper for good from task 2  final  -32.0
   scikit-learn  tasks 60  first task  +15.6  cheaper for good from task 4  final  -26.5
   sphinx        tasks 44  first task  +38.1  cheaper for good from task 4  final  -30.4
   sympy         tasks 60  first task  +11.3  cheaper for good from task 2  final  -29.9
   xarray        tasks 39  first task   -8.5  cheaper for good from task 1  final  -24.1
   final range -16.9 to -32.0

Drive the shipped hooks

Clone the repository, check out the commit named above, and pass the clone’s path to the script. It imports the plugin from the clone, strips any environment variable that looks like a key or token before doing so, and writes only to a temporary folder.

git clone https://github.com/microsoft/TokenEconomizer.git
git -C TokenEconomizer checkout b57d03f81585a18b936a0eaecbd1eb51cdcecf71
python default_install_lab.py TokenEconomizer
# default_install_lab.py
"""Drive the shipped TokenEconomizer hooks with synthetic Claude Code sessions and report what a default install does.

Usage: python default_install_lab.py PATH_TO_CLONE      (a clone of microsoft/TokenEconomizer at the commit named in the article)

Python 3.10 or newer, standard library only. The plugin package is imported from the clone. Everything is local: a
temporary repository, a temporary state folder, folds run inline (TE_FOLD_SYNC=1), and no model call is made (the
micro-skills proposer is off by default and the loop guard is never woken). The driving pattern is the one the repository's
own tests/test_live_hooks.py uses.
"""
import io
import json
import os
import sys
import tempfile

CLONE = os.path.abspath(sys.argv[1])
sys.path.insert(0, CLONE)
for name in list(os.environ):  # nothing secret should be near code we did not write
    if any(word in name.upper() for word in ("KEY", "TOKEN", "SECRET", "PASSWORD")):
        os.environ.pop(name)
WORK = tempfile.mkdtemp(prefix="te_lab_")
os.environ["HOME"] = os.environ["USERPROFILE"] = os.path.join(WORK, "userhome")

from tokeneconomizer.plugin import denylist, hook  # noqa: E402
from tokeneconomizer.plugin.paths import repo_identity, repo_state_dir  # noqa: E402
from tokeneconomizer.plugin.store import Store  # noqa: E402

DOTS = chr(0x2026)


def ascii_line(text, n=170):
    return text.replace(DOTS, "...").replace("\n", " ").encode("ascii", "replace").decode()[:n]


def make_repo(name):
    repo = os.path.join(WORK, name, "repo")
    os.makedirs(os.path.join(repo, ".git", "info"))
    with open(os.path.join(repo, ".git", "HEAD"), "w") as fh:
        fh.write("ref: refs/heads/main\n")
    with open(os.path.join(repo, ".git", "config"), "w") as fh:
        fh.write('[remote "origin"]\n\turl = https://github.com/acme/widget.git\n')
    with open(os.path.join(repo, "pyproject.toml"), "w") as fh:
        fh.write("[project]\nname='w'\n")
    env = {"TOKENECONOMIZER_HOME": os.path.join(WORK, name, "home"), "TE_FOLD_SYNC": "1",
           "GIT_CEILING_DIRECTORIES": WORK, "PATH": os.environ.get("PATH", "")}
    return repo, env


def write_session(path, prompts):
    """An interactive Claude Code transcript: a user entry per prompt, then a tool_use and a tool_result per call."""
    lines, n = [], 0
    for text, calls in prompts:
        lines.append({"type": "user", "message": {"role": "user", "content": text}})
        for tool, tool_input, result, is_error in calls:
            n += 1
            lines.append({"type": "assistant", "message": {
                "id": f"msg_{os.path.basename(path)}_{n}", "model": "claude-opus-5-5", "role": "assistant",
                "usage": {"input_tokens": 10, "cache_read_input_tokens": 1000, "cache_creation_input_tokens": 100, "output_tokens": 50},
                "content": [{"type": "tool_use", "id": f"tu{n}", "name": tool, "input": tool_input}]}})
            lines.append({"type": "user", "message": {"role": "user", "content": [
                {"type": "tool_result", "tool_use_id": f"tu{n}", "content": result, "is_error": is_error}]}})
    with open(path, "w") as fh:
        fh.write("\n".join(json.dumps(x) for x in lines) + "\n")
    return path


def ordinary_session(path, repo, i):
    return write_session(path, [
        (f"look at module {i}", [("Read", {"file_path": f"{repo}/a.py"}, "x", False), ("Grep", {"pattern": "foo"}, "a.py:1", False)]),
        ("edit it", [("Edit", {"file_path": f"{repo}/a.py", "old_string": "a", "new_string": "b"}, "ok", False)]),
        ("run the checks", [("Bash", {"command": "ls"}, "a.py", False)])])


def trap_session(path, repo, i):
    """Every session falls into the same trap: bare pytest is missing here, python -m pytest works."""
    return write_session(path, [(f"fix bug {i}", [
        ("Edit", {"file_path": f"{repo}/pkg/mod{i}.py", "old_string": "a", "new_string": "b"}, "ok", False),
        ("Bash", {"command": "pytest -q tests/test_a.py"}, "bash: pytest: command not found", True),
        ("Bash", {"command": "python -m pytest -q tests/test_a.py"}, "1 passed in 0.1s", False)])])


def call(env, payload):
    out = io.StringIO()
    hook.main([], io.StringIO(json.dumps(payload)), out, env)
    return json.loads(out.getvalue().splitlines()[-1])


def run_session(env, repo, sid, transcript):
    base = {"session_id": sid, "transcript_path": transcript, "cwd": repo}
    start = call(env, dict(base, hook_event_name="SessionStart", source="startup"))
    call(env, dict(base, hook_event_name="SessionEnd", reason="other"))
    return start  # what the NEXT-session hook said when this session began


def untool_state(env, repo):
    return Store(repo_state_dir(repo_identity(repo, env), "claude", env)).read("untool.json") or {}


def deny_count(repo):
    path = denylist.settings_path(repo)
    return 0 if not os.path.exists(path) else len(json.load(open(path)).get("permissions", {}).get("deny", []))


def write_catalog(env, tools):  # what `toolprune refresh-catalog` stores for Claude Code
    folder = os.path.join(env["TOKENECONOMIZER_HOME"], "hosts", "claude")
    os.makedirs(folder, exist_ok=True)
    with open(os.path.join(folder, "catalog.json"), "w") as fh:
        json.dump({"tools": tools}, fh)


def pruning(label, extra_env, snapshot, sessions):
    repo, env = make_repo(label)
    env.update(extra_env)
    if snapshot:
        write_catalog(env, snapshot)
    proposed, denied, notices = [], [], []
    for i in range(1, sessions + 1):
        start = run_session(env, repo, f"s{i}", ordinary_session(os.path.join(WORK, f"{label}_{i}.jsonl"), repo, i))
        state = untool_state(env, repo)
        proposed.append(len(state.get("proposal") or (state.get("auto_pending") or {}).get("items") or []))
        denied.append(deny_count(repo))
        if start.get("systemMessage"):
            notices.append((i, start["systemMessage"]))
    state = untool_state(env, repo)
    waiting = sorted((state.get("auto_pending") or {}).get("items") or [])
    return proposed, denied, notices, waiting, state.get("catalog_source")


snapshot = list(json.load(open(os.path.join(CLONE, "tokeneconomizer", "trace", "claude_tool_catalog.json"), encoding="utf-8")))
print("snapshot:", len(snapshot), "Claude Code tool names from the repository's own catalog file")

p, d, n, w, src = pruning("A", {}, None, 12)
print("\nA  fresh install, no refresh-catalog step, 12 sessions that only use Read, Grep, Edit and Bash")
print("   tools proposed after each session:", p, "| deny rules written:", d[-1], "| catalog source:", src)

p, d, n, w, src = pruning("B", {}, snapshot, 6)
print("\nB  default settings after refresh-catalog (24 names), 6 sessions")
print("   tools waiting after each session:", p, "| deny rules written after each:", d)
print("   waiting:", ", ".join(w))

p, d, n, w, src = pruning("B0", {"TE_TOOLPRUNE_QUIET_MINUTES": "0"}, snapshot, 6)
print("\nB0 the same with the idle wait set to 0 minutes")
print("   deny rules written after each session:", d)
for i, text in n:
    print(f"   notice at the start of session {i}:", ascii_line(text))

print("\nC  micro-skills, 8 sessions that all fall into the missing-pytest trap")
for label, extra in (("default", {}), ("TE_MICROSKILLS=envfacts", {"TE_MICROSKILLS": "envfacts"})):
    repo, env = make_repo("C" + str(len(label)))
    env.update(extra)
    got = []
    for i in range(8):
        start = run_session(env, repo, f"c{i}", trap_session(os.path.join(WORK, f"C{len(label)}_{i}.jsonl"), repo, i))
        got.append((start.get("hookSpecificOutput") or {}).get("additionalContext", ""))
    first = next((i + 1 for i, text in enumerate(got) if text), None)
    print(f"   {label:24} session starts that carried a fact: {sum(1 for t in got if t)} of 8; first at session {first}")
    if first:
        print("      ", ascii_line(got[first - 1], 200))

Output:

snapshot: 24 Claude Code tool names from the repository's own catalog file

A  fresh install, no refresh-catalog step, 12 sessions that only use Read, Grep, Edit and Bash
   tools proposed after each session: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | deny rules written: 0 | catalog source: observed

B  default settings after refresh-catalog (24 names), 6 sessions
   tools waiting after each session: [0, 0, 0, 12, 12, 12] | deny rules written after each: [0, 0, 0, 0, 0, 0]
   waiting: CronCreate, CronDelete, CronList, DesignSync, EnterWorktree, ExitWorktree, ListAgents, PushNotification, ReportFindings, ScheduleWakeup, SendMessage, Workflow

B0 the same with the idle wait set to 0 minutes
   deny rules written after each session: [0, 0, 0, 12, 12, 12]
   notice at the start of session 5: TokenEconomizer removed 12 tool(s) this repository's sessions never used (CronCreate, CronDelete, CronList, DesignSync...), from this session on. Review with `/tokenecono

C  micro-skills, 8 sessions that all fall into the missing-pytest trap
   default                  session starts that carried a fact: 0 of 8; first at session None
   TE_MICROSKILLS=envfacts  session starts that carried a fact: 4 of 8; first at session 5
       # environment facts learned from earlier sessions in this repository - `pytest ...` fails here with command not found; `python -m pytest ...` works. 

What would settle it

  • A run of the plugin as shipped. The harness’s controllers differ from the plugin’s, so the 18.4% to 30.9% cannot stand in for it. The same 277 tasks run through the plugin’s hooks, with micro-skills on and off, would give the number a user needs.
  • Per-task, per-seed outcomes. With them, Pass@1 could be tested pair by pair instead of through three seed means, and the claim could be stated as a confidence interval.
  • Held-out repositories for the constants. Appendix C’s values were fixed on the same six repositories the SWE-Bench results use. Web-Bench and the security sets are the only evidence outside them.

Bottom line

TokenEconomizer’s cost result is solid for what was measured: a learned controller, run in a sealed benchmark harness, lowered cost by roughly a fifth to a third without a detectable loss in success. What the evidence does not show is an improvement in Pass@1, or what a default install of the plugin saves, and the repository’s own documentation says as much. The blog and the project page would be more accurate if they carried the paper’s abstract wording and the one table from PLUGIN.md that lists what changes.

Sources

  • Microsoft Research blog post (November 16, 2026)
  • TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents, arXiv 2610.22732, with the HTML version
  • Project page and its data file
  • Repository at commit b57d03f: README, docs/PLUGIN.md, plugin/README.md, docs/ARCHITECTURE.md, CLAUDE.md, tokeneconomizer/plugin/adapter.py, tokeneconomizer/plugin/fold.py, tokeneconomizer/plugin/config.py, experiments/tests/test_tooling.py, LICENSE and pyproject.toml

Tags:

AI BenchmarksAI Coding AgentsClaude CodeMicrosoft ResearchSWE-BenchTokenEconomizer

Share

The Hall of Justice in San Francisco, a concrete courthouse and police building, seen from the street under streaks of cirrus cloud
Previous Post

California Sues OpenAI Over Its Escaped AI Agents, Citing Over 80 Incidents Against 27 Public Reports, and OpenAI’s Own Index Shows a Median 114 Days Between an Incident and Its Posting

Crowd of people waiting in front of a row of ticket counters at a metro station
Next Post

How to Serve Several Users From One Ollama Server Without Surprise Waits: Parallel Slots, the Queue Limit, keep_alive, and Model Swaps

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
22 Nov
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
22 Nov
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
Trending
November 22, 2026
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
November 22, 2026
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
November 22, 2026
Rackspace Says Leaked Credentials Let an Intruder Act as Legitimate Users in Some Spot Customer Organizations and Reach Their Kubernetes Secrets, a Platform9 Co-Founder Names a Breached Observability System as the Root Cause, and the Status Page Says Only “System Maintenance Operations”
November 21, 2026
How to Write YARA Rules With YARA-X to Hunt Malicious Python .pth Startup Files and Measure False Positives Before You Trust Them
November 21, 2026
AWS’s “Found a Zero-Day on Their Own” Post Never Names the Product; the Advisory Behind Its Link Shows a Server That Ships With No Login, a README Now Saying Keep It Off the Internet, and a Fix Merged October 19 That PyPI and Docker Hub Still Do Not Carry
November 21, 2026
Microsoft Tells Customers Unauthorized Parties Read Admin Email Addresses and Tenant Metadata Through an Entra Connect Health Flaw Introduced July 23, but Gives No Tenant Count or Start Date, and Its Own Docs Send Tenant Security Notices to the Same Kinds of Contacts

Related Posts

Blue-lit server racks in a modern data center, illustrating the compute infrastructure behind the AI boom.
Articles

The AI Boom Is Spending Real Money Before Proving Real Returns

June 7, 2026
Technician working with a laptop beside server racks, representing enterprise AI retrieval infrastructure
Articles

Google’s Agentic RAG Push Makes Enterprise AI Less of a One-Shot Guess

June 7, 2026
A person with a laptop and smartphone, representing digital attention and AI-assisted work
Articles

AI Chatbots Are Making Attention a Design Problem

June 7, 2026
A customer-support representative wearing a headset against a dark studio background.
Articles

The Meta AI Support Hack Was a Plain Old Authorization Failure

June 7, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026