Ai2’s GPU Scheduler Turns Priority Into a Budget, but Leaves the Preemption Bill Unreported

Ai2, the Allen Institute for AI, has published an account of how its AI Infrastructure team replaced a priority-based GPU scheduler with GPU-time budgets, hierarchical fair-share ordering and a protected runtime of up to 8 hours. The post, written by Jeremy Tryba and dated October 9, 2026, covers clusters of 88 to 1,024 GPUs that serve about 150 researchers whose requests run at two to three times supply. The reported results are favorable: over a 30-day test, teams received 98 percent of the GPU hours they were owed, cluster occupancy stayed at 98 percent, and the 90th-percentile (p90) queue wait for short debug jobs fell from 2 hours to 30 seconds.

I read the post alongside the documentation it leans on, checked its claims about prior work against that documentation, and wrote a few short Python scripts to test its reasoning. The scripts are mine, not Ai2’s, and I flag the assumptions behind each one where it appears. What stands out is something the post does not contain. The design makes preemption routine, and the post reports who got GPU time and how long they waited, but never how many workloads were preempted or how much work they lost.

What Ai2 replaced

Ai2’s team describes its job as a pyramid of four metrics. Availability is how often the hardware is healthy and ready for work. Occupancy is “the fraction of available time assigned to a specific workload.” Impact is how often the most valuable workloads are chosen to receive resources. Utilization is the fraction of GPU capacity a workload actually uses over its lifetime. This post is about impact. It starts from a priority scheduler in which workloads could opt out of preemption and each team had a limit on how many concurrent GPUs its protected workloads could hold.

That arrangement produced three problems, in the post’s account. Researchers who could not launch a debugging job with low enough latency parked no-op workloads they could connect to later, which the post calls GPU squatting. Priority inflated until “100% of scheduled workloads used HIGH priority,” which starved the lower levels. And on-call engineers spent “a majority of their ticket response time” negotiating the shutdown of non-preemptable workloads on hosts with known maintenance problems.

Ai2 reads this as a tragedy of the commons and points to the 2011 paper that introduced Dominant Resource Fairness, “Dominant Resource Fairness: Fair Allocation of Multiple Resource Types” by Ali Ghodsi and colleagues at UC Berkeley. I checked the anecdote the post borrows from it. The paper does say that “another big search company provided dedicated machines for jobs only if the users could guarantee high utilization” and that “users would sprinkle their code with infinite loops to artificially inflate utilization levels.” It is the same incentive problem in a different decade: a signal that earns protection gets gamed.

How the new scheduler works

Budgets in GPU time, set by managers

Ai2 stopped handing teams GPUs and started handing them shares of GPU time. Its reasoning is that fixed GPU monopolies “caused GPUs to sit idle due to the seasonality of research,” and that demand cannot be forecast because it depends on the results of novel experiments, whereas “priority across research efforts, however, is a question of strategy” that can be argued out in advance. So, in the post’s words, “we enabled leadership to think like investors.” Each level of the tree is set by the person with the most context: a lead researcher within a project, a principal investigator within a program, and a lead program manager or the CEO across programs.

The rule that makes this work is that protection costs budget. “In this system, every request for GPU time must be funded by a budget, or it is not protected from preemption.” In the old system, the post says, HIGH priority “carried no cost,” so everyone used it. Ai2’s stated strategy is “to make gaming the scheduler more expensive than honestly engaging in the debate for a larger budget.”

Fair-share over a seven-day window

On top of the budgets, the scheduler orders work by fair-share. It “tracks occupancy over a sliding lookback window (we default to 7 days) and sorts workloads from under-utilized allocations above those from over-utilized allocations.” In the workload lifecycle the post lists, a workload is scheduled “weighted by a ratio of actual occupancy to allocated time in the lookback window.” The post gives no formula, so the script below is my reading of it: one number per team, the GPU-hours charged to its budget in the window divided by the GPU-hours its share is worth, lowest first. The teams and numbers are invented to show the rule.

# fairshare.py
# My reading of the ordering rule the Ai2 post describes in words. It is not Ai2's code.
CLUSTER_GPUS = 100
WINDOW_H = 168  # the 7-day lookback the post names as its default


def allocated_hours(share):
    """GPU-hours that a budget share is worth over one lookback window."""
    return share * CLUSTER_GPUS * WINDOW_H


teams = {
    # name: (budget share, GPU-hours charged to that budget during the last 7 days)
    "A": (0.35, 5880.0),  # used its whole share
    "B": (0.35, 2000.0),  # experiments were not ready, so most of its share went unused
    "C": (0.30, 5040.0),  # used its whole share
    "D": (0.10, 1344.0),  # parked a no-op job on 8 GPUs for the whole week
}

rows = []
for name, (share, charged) in teams.items():
    alloc = allocated_hours(share)
    rows.append((charged / alloc, name, share, alloc, charged))

print("queue order: the lowest usage ratio is served first")
print("rank team share allocated charged ratio headroom")
for rank, (ratio, name, share, alloc, charged) in enumerate(sorted(rows), start=1):
    print(f"{rank:>4} {name:>4} {share:>5.2f} {alloc:>9.0f} {charged:>7.0f} {ratio:>5.2f} {alloc - charged:>8.0f}")

b_headroom = allocated_hours(0.35) - 2000.0
print()
print(f"B has {b_headroom:.0f} GPU-hours of headroom: 40 GPUs for {b_headroom / 40:.0f} hours before its ratio reaches 1.00")

d_alloc = allocated_hours(0.10)
d_squat = 8 * WINDOW_H
d_left = d_alloc - d_squat
print(f"D owns {d_alloc:.0f} GPU-hours a week and spent {d_squat} of them on a no-op job")
print(f"D has {d_left:.0f} GPU-hours left for real work: 8 GPUs for {d_left / 8:.0f} hours")

queue order: the lowest usage ratio is served first
rank team share allocated charged ratio headroom
   1    B  0.35      5880    2000  0.34     3880
   2    D  0.10      1680    1344  0.80      336
   3    A  0.35      5880    5880  1.00        0
   4    C  0.30      5040    5040  1.00        0

B has 3880 GPU-hours of headroom: 40 GPUs for 97 hours before its ratio reaches 1.00
D owns 1680 GPU-hours a week and spent 1344 of them on a no-op job
D has 336 GPU-hours left for real work: 8 GPUs for 42 hours

Team B left most of its share unused last week, so it sorts first and has 3,880 GPU-hours of headroom, enough for 40 GPUs for 97 hours before its ratio reaches 1.00. That is the mechanism behind a remark from a user quoted in the post, Chris Clark, that the scheduler “makes it feel like we have an extra 30% compute”: unused time earlier in the window becomes a claim on bursts later. Team D parked a no-op job on 8 GPUs for the week and spent 1,344 of its 1,680 GPU-hours on nothing. It still sorts ahead of A and C, whose ratios are 1.00, but it has only 336 GPU-hours left for real work, or 8 GPUs for 42 hours. In the post’s phrase, “a squatting workload is spending team budget on nothing.”

A protected runtime of up to eight hours

The “scheduling contract” asks each workload to declare a minimum runtime, “the shortest amount of occupancy required to make meaningful progress,” and whether it is resumable. During that window the workload is protected from preemption and the time is charged to its budget. Afterward the workload may keep running “as long as the associated allocations continue to prioritize it over others,” and it may be preempted and requeued. Setting the minimum runtime to zero marks the time as unallocated, which is free and always preemptible. Ai2 chose 8 hours as the largest minimum runtime it allows.

The contract has a second use. Hosts with maintenance problems can drain as their workloads reach their minimum runtimes, which the post credits for a 74 percent drop in repairs that needed a human in the loop. And unallocated work, about 18 percent of delivered GPU time, has no protected window at all: it “may be preempted by any allocated request.”

What the numbers show

Every figure in the table is Ai2’s own measurement of its own clusters, and I could not check any of it against data. What I can do is check the figures against each other and against the definitions the post gives, and the short script below does that arithmetic.

# checks.py
# Arithmetic on the numbers the Ai2 post reports. Nothing here is measured by me.
sim_before, sim_after = 6 * 3600, 5 * 60  # simulated p90 debug wait, seconds
real_before, real_after = 2 * 3600, 30  # measured p90 debug wait, seconds
print(f"debug p90 wait, simulated: 6 h to 5 min is {sim_before / sim_after:.0f}x faster")
print(f"debug p90 wait, measured: 2 h to 30 s is {real_before / real_after:.0f}x faster")

print(f"median wait, largest H100 cluster: 5 min to 24 s is {300 / 24:.1f}x faster")
print(f"p90 wait, largest H100 cluster: 2.8 h to 1.8 h is {(2.8 - 1.8) / 2.8:.1%} lower")

occupancy, unallocated_share = 0.98, 0.18
print(f"unallocated work: {unallocated_share:.0%} of delivered time is {occupancy * unallocated_share:.1%} of all cluster hours")
print(f"allocated work: {occupancy * (1 - unallocated_share):.1%} of all cluster hours")
print(f"hours not delivered at 98% occupancy: {1 - occupancy:.0%} of all cluster hours")

debug p90 wait, simulated: 6 h to 5 min is 72x faster
debug p90 wait, measured: 2 h to 30 s is 240x faster
median wait, largest H100 cluster: 5 min to 24 s is 12.5x faster
p90 wait, largest H100 cluster: 2.8 h to 1.8 h is 35.7% lower
unallocated work: 18% of delivered time is 17.6% of all cluster hours
allocated work: 80.4% of all cluster hours
hours not delivered at 98% occupancy: 2% of all cluster hours

Occupancy did not change. It held at 98 percent before and after, with demand at two to three times capacity in both periods. The gain is distributional: who gets the time and how long they wait. That fits the post’s own framing, which targets impact, the third level of its pyramid, and not occupancy or utilization.

“Owed” is defined narrowly. The post counts a team’s owed time “as its allocation capped hour by hour at its actual demand.” That is a fair test of whether funded work ran when it was ready. It cannot register work that was never submitted because researchers expected to wait, and it says nothing about whether the funded work was the right work, which is the budget meeting’s job.

The free tier is large. If 18 percent of delivered time was unallocated and occupancy was 98 percent, then 17.6 percent of all cluster hours ran outside any budget and 80.4 percent were charged to one (my arithmetic). Budgets and demand did not line up, and the free, preemptible tier absorbed the gap, as the post intends.

The simulator was directionally right. Ai2 predicted from hand-built test cases that the p90 debug wait would fall from 6 hours to 5 minutes, a 72-fold improvement. The measured change, 2 hours to 30 seconds, is 240-fold. The post calls its simulator results “directionally accurate” and cautions that the baseline had a smaller sample of debug workloads and so higher variance, so the match is best read as direction, not size.

Queue latency improved on the largest H100 cluster. The median wait fell 12.5-fold and the p90 fell 35.7 percent, which matches the post’s “about a third.” The post presents this as a side effect of time-slicing and gives the figures for that one cluster.

The preemption bill nobody has priced

Time-slicing works because the scheduler can take GPUs back, and taking them back costs something. A preempted job restarts, and whatever it did since its last checkpoint is gone. The post lists “bootstrapping, checkpointing” among the things it still has to make efficient and calls utilization the capstone of its pyramid, so the bill is on Ai2’s roadmap. It is not in the results. By the post’s own definition, occupancy is “the fraction of available time assigned to a specific workload,” so GPUs that are redoing lost work still count as occupied.

Here is the arithmetic. If a preemption lands at a random point in the checkpoint cycle, a job loses half a checkpoint interval of work on average, plus the time to get back to the same point: container start, data loader warm-up and loading the checkpoint. The script prices that for three settings. The inputs are my assumptions, not Ai2 data, and a job that saves on a preemption signal would lose less work and pay for the save instead. This site’s tutorial on checkpointing and resuming a training job shows how to catch that signal and write the checkpoint atomically.

# preempt_bill.py
# Back-of-envelope cost of time-slicing: what one preemption throws away.
SLICE_MIN = 8 * 60  # the 8-hour cap on protected runtime that the post names


def lost_minutes(restart_min, ckpt_min):
    """Expected minutes discarded per preemption.

    Assumes the preemption lands at a random point in the checkpoint cycle (so half an
    interval of work is lost on average) and that getting back to the same point costs
    restart_min (container start, data loader warm-up, checkpoint load). A job that saves
    on a preemption signal would lose less work and pay for the save instead.
    """
    return restart_min + ckpt_min / 2


if __name__ == "__main__":
    cases = [
        ("fast", 5, 15),
        ("typical", 10, 30),
        ("heavy", 20, 60),
    ]
    gpu_counts = [64, 256, 1024]

    for label, restart, ckpt in cases:
        per = lost_minutes(restart, ckpt)
        print(f"{label:<8} restart {restart:>2} min, checkpoint every {ckpt:>2} min: {per:>4.1f} min lost per preemption, {per / SLICE_MIN:>5.1%} of an 8 h slice")
        for gpus in gpu_counts:
            print(f"           {gpus:>4} GPUs discard {gpus * per / 60:>6.1f} GPU-hours")

fast     restart  5 min, checkpoint every 15 min: 12.5 min lost per preemption,  2.6% of an 8 h slice
             64 GPUs discard   13.3 GPU-hours
            256 GPUs discard   53.3 GPU-hours
           1024 GPUs discard  213.3 GPU-hours
typical  restart 10 min, checkpoint every 30 min: 25.0 min lost per preemption,  5.2% of an 8 h slice
             64 GPUs discard   26.7 GPU-hours
            256 GPUs discard  106.7 GPU-hours
           1024 GPUs discard  426.7 GPU-hours
heavy    restart 20 min, checkpoint every 60 min: 50.0 min lost per preemption, 10.4% of an 8 h slice
             64 GPUs discard   53.3 GPU-hours
            256 GPUs discard  213.3 GPU-hours
           1024 GPUs discard  853.3 GPU-hours

With the middle setting, a 256-GPU job discards about 107 GPU-hours each time it is preempted, and if every protected slice ended in a preemption the bill would be 5.2 percent of the slice. That is an upper bound. The post says a workload may keep running after its protected runtime while its allocations still prioritize it, so not every slice ends in a preemption, and the share shrinks as checkpoints get shorter or the cap gets longer. But it is not zero, and it lands unevenly. Unallocated work has no protected window, so the roughly 18 percent of delivered time in that tier is where I would look first for discarded work.

What I would want Ai2 to publish is preemptions per workload-week, GPU-hours discarded per week as a share of delivered hours, and median restart time, split by allocated and unallocated work. The post also does not say what happens at the end of the protected window to a workload that was submitted as not resumable, and a preempted job that cannot resume has the most to lose.

Why large jobs may wait longer

Ai2 names one problem it is still investigating: capacity fragmentation, which “may result in queue wait times increasing for the largest workloads.” Its intuition is that jobs which used to be preemptible “to exceed their team concurrent GPU limit” are now protected by minimum runtimes, so the scheduler “may have fewer opportunities to interrupt many jobs at once to place a large pending workload.” The team says it is using its simulator to reproduce the effect while measuring the ground truth in production.

A toy bound shows why the cap matters so much. Take a cluster of 128 nodes (1,024 GPUs as 8-GPU nodes) in which some fraction of nodes are held by protected jobs whose remaining protection is spread evenly over the 8-hour cap, while every other node can be preempted at once. A pending job that needs more nodes than the preemptible ones can supply must wait for protected jobs to age out. The wait is an order statistic with a simple mean: the cap times the shortfall, divided by the number of protected nodes plus one. The script checks that formula against a simulation with a fixed seed.

# slice_tradeoff.py
# Toy bound on how long a large job waits when protected runtimes block preemption,
# and how the slice length trades that wait against the preemption bill.
import random

from preempt_bill import lost_minutes

NODES = 128  # 1,024 GPUs as 8-GPU nodes
SLICE_H = 8.0  # the protected-runtime cap the post names


def short_nodes(protected_frac, need_nodes, nodes=NODES):
    """Nodes the pending job needs that only free up when a protected slice runs out."""
    protected = round(nodes * protected_frac)
    return protected, need_nodes - (nodes - protected)


def formula_mean(protected_frac, need_nodes, slice_h=SLICE_H):
    protected, short = short_nodes(protected_frac, need_nodes)
    if short <= 0:
        return 0.0
    # mean of the short-th smallest of `protected` uniform draws on [0, slice_h]
    return slice_h * short / (protected + 1)


def simulate(protected_frac, need_nodes, trials=4000, seed=11):
    rng = random.Random(seed)
    protected, short = short_nodes(protected_frac, need_nodes)
    waits = []
    for _ in range(trials):
        if short <= 0:
            waits.append(0.0)
            continue
        left = sorted(rng.uniform(0, SLICE_H) for _ in range(protected))
        waits.append(left[short - 1])
    waits.sort()
    return sum(waits) / trials, waits[trials // 2], waits[int(trials * 0.9)]


print("wait for a large job when protected slices have random time left (8 h cap, 128 nodes)")
print("job size  protected  formula  simulated mean  median  p90   (hours)")
for need_nodes in (16, 64):
    for frac in (0.5, 0.7, 0.9, 1.0):
        mean, median, p90 = simulate(frac, need_nodes)
        print(f"{need_nodes * 8:>5} GPU {frac:>8.0%} {formula_mean(frac, need_nodes):>8.2f} {mean:>14.2f} {median:>7.2f} {p90:>5.2f}")

print()
print("slice length trade-off for a 512-GPU job, 90% of nodes protected, 25 min lost per preemption")
print("slice_h  bill_if_preempted  mean_wait_h")
per = lost_minutes(10, 30)
for slice_h in (1, 2, 4, 8, 16, 24):
    print(f"{slice_h:>7} {per / (slice_h * 60):>18.1%} {formula_mean(0.9, 64, slice_h):>12.2f}")

wait for a large job when protected slices have random time left (8 h cap, 128 nodes)
job size  protected  formula  simulated mean  median  p90   (hours)
  128 GPU      50%     0.00           0.00    0.00  0.00
  128 GPU      70%     0.00           0.00    0.00  0.00
  128 GPU      90%     0.21           0.21    0.19  0.37
  128 GPU     100%     0.99           0.99    0.98  1.30
  512 GPU      50%     0.00           0.00    0.00  0.00
  512 GPU      70%     2.29           2.29    2.28  2.80
  512 GPU      90%     3.52           3.52    3.52  4.00
  512 GPU     100%     3.97           3.97    3.96  4.41

slice length trade-off for a 512-GPU job, 90% of nodes protected, 25 min lost per preemption
slice_h  bill_if_preempted  mean_wait_h
      1              41.7%         0.44
      2              20.8%         0.88
      4              10.4%         1.76
      8               5.2%         3.52
     16               2.6%         7.03
     24               1.7%        10.55

In this toy, a 512-GPU job waits about 2.3 hours on average when 70 percent of nodes are protected, and about 3.5 hours (p90 of 4 hours) at 90 percent, while a 128-GPU job barely notices until nearly everything is protected. The second table is the point. With the same assumptions, cutting the cap from 8 hours to 4 halves the large job’s wait, from 3.52 to 1.76 hours, and doubles the worst-case bill, from 5.2 to 10.4 percent of each slice. The cap Ai2 chose is a position on that curve, and the post gives figures for neither side of it.

This is a toy, not a model of Ai2’s clusters. It ignores backfill, reservations, other pending jobs and drain time, and it assumes protected jobs are spread evenly across the cap. It is useful for direction and scaling, not for magnitudes.

How it compares with Slurm, Kueue and YARN

Ai2 says plainly that “the algorithm here is not new.” The post says what is new is the inputs: the tree mirrors the research program structure, and “the weights are budgets set by managers rather than static quotas.” It names Slurm’s Fair Tree and YARN’s Fair Scheduler as current uses and links the Hadoop Fair Scheduler ticket for the lineage. I checked each against its documentation.

Slurm. The Fair Tree page says that starting with Slurm 19.05 the fair-share factor “will default to the Fair Tree algorithm.” Two settings in the slurm.conf reference match the other pieces of Ai2’s design. PriorityDecayHalfLife controls how long prior resource use counts, decays usage so that “half of the original value” is cleared at that age, and defaults to 7 days. That is a decaying average where Ai2 describes a sliding window, with the same time scale. PreemptExemptTime is a “global option for minimum run time for all jobs before they can be considered for preemption,” and a QOS value takes precedence over the global one. The minimum-runtime idea is not new either. This site’s checklist for running Slurm on Kubernetes with Slinky covers the Slurm side for teams on that stack.

Kueue. Kubernetes’ queueing project documents Admission Fair Sharing, “a mechanism for ordering workloads based on the historical resource usage of their source LocalQueues, giving preference to those that have consumed fewer resources over time.” It is a beta feature that is enabled by default, and the documentation’s example sets usageHalfLifeTime to 168h, the same week. Kueue also carries a lesson the post leaves open. It added an entry penalty in v0.13.0 “to prevent exploitation where tenants could submit many workloads quickly before usage statistics are updated.” The post says a workload’s minimum runtime “is charged to its allocations” but not whether the charge lands up front or as the time accrues, and that decides whether a burst of submissions can sort ahead on stale usage. For preemption on the Kubernetes side more broadly, see this site’s piece on Kubernetes v1.37 scheduler preemption.

YARN. The Fair Scheduler documentation (I read the stable-branch page, which is Hadoop 3.3.5) describes fair scheduling as “assigning resources to applications such that all apps get, on average, an equal share of resources over time,” supports hierarchical queues, and ships with preemption off: yarn.scheduler.fair.preemption “defaults to false.” The DRF paper is the source of the multiple-resource fairness idea the YARN page names.

The Hadoop date. The post puts the Hadoop Fair Scheduler lineage in 2009. The ticket it links, HADOOP-3746, was opened on July 11, 2008 and fixed in Hadoop 0.19.0, released November 20, 2008, according to Apache’s JIRA record. It is a small point, and the lineage claim holds.

What a team running a shared GPU cluster can do with this

Price protection, not priority. A label anyone can set for free will inflate, as Ai2’s HIGH priority did. If you cannot charge for protection, expect squatting to come back in some form.

Keep a free, preemptible tier. Ai2’s unallocated work let it hold occupancy at 98 percent even when allocations did not match demand. It also means that tier needs jobs that tolerate being cut off at any moment, which is a checkpointing requirement.

Set the protected-runtime cap from your checkpoint economics. Divide the discard per preemption (restart time plus half a checkpoint interval) by the cap to get the worst-case bill, then compare it with how long your largest jobs can afford to wait. The two tables above are a starting point. Replace my assumptions with your own restart and checkpoint times.

Report the bill and the tail. Add preemptions per workload-week, discarded GPU-hours and queue wait by job size to the dashboard. An overall median and p90, which is what the post reports for its largest cluster, can improve while the largest jobs wait longer.

Give interactive sessions their own pool. Ai2’s own surprise was that “we did not appreciate the extent to which researchers were dependent on the volatile state of these sessions.” The 8-hour cap made week-long sessions preemptible, and the team is now planning a CPU-only cluster for dev sessions and restorable sessions.

Budget the humans. The post spends as much space on explanation sessions and visualizations as on algorithms. Documentation “alone did not resolve the confusion,” and old interface terms such as priority kept their names after their meanings changed. If the scheduler is a budget, the budget process needs an owner.

The bottom line

Ai2 has described a clean design: charge for protection, order work by usage against budget, and let a bounded protected runtime make preemption routine. The reported results support the claim that funded teams get their time and that short jobs start quickly, at the same occupancy. They do not yet support a claim about cost, because the post never says how much work the scheduler throws away. Ai2 says it is moving up to utilization next. The first number I would look for is how many GPU-hours a week the new scheduler discards.

Sources and method

I read Ai2’s post on allenai.org and its copy on Hugging Face, the text of the Dominant Resource Fairness paper, the HADOOP-3746 record through Apache’s JIRA REST interface, the Slurm Fair Tree and slurm.conf pages, the YARN Fair Scheduler page and Kueue’s Admission Fair Sharing page. Every quotation was checked against the fetched text. I searched the post for every mention of preemption, restarts, checkpoints and discarded work to confirm that it reports no counts for them. The four scripts use Python 3.13.14 and only the standard library, and the simulation uses a fixed random seed. They are my own toy models and my own arithmetic on the post’s figures; I did not have Ai2’s data or code, and I did not test any scheduler.