OpenAI’s Pando Turns Git Scale Into a Per-Branch Consistency Trade, but Two Client Features the Post Describes Are Not in Upstream Git
OpenAI says its Pando backend cut median push latency nearly 5x by giving each branch its own consistency, so we measured the Git protocol limits behind it and checked its public patch tracker...
On November 2 OpenAI published “Scaling Git at OpenAI,” an engineering post by Aaron Friel, Taylor Blau and Ted Nyman about Pando, which the authors call “a new backend for Git repositories.” The headline claims are that Pando “now powers all development in OpenAI’s monorepo,” that it “cut median push latency by nearly 5x,” and that it delivers “nearly a petabyte of Git data per day.”
Table Of Content
- What Pando does, in the post’s own terms
- Two stores and one contended key
- Causal consistency, stated plainly
- Why only trunk is advertised
- Why packfile URIs, and what the covers are
- The numbers, checked against each other
- Test 1: the ref advertisement is real, and protocol v2 does not reach push
- Test 2: what trunk-only costs a branch you keep pushing
- Test 3: packfile URIs are opt-in on stock Git
- The client half: what OpenAI’s fork and tracker show
- Two capabilities the post describes
- What upstream Git has
- How big the fork is
- The simulation, and what it can show
- What the post leaves open
- What an unmodified client sees
- Atomic multi-ref pushes
- Reading more than one ref
- Storage and tail behavior
- What to check on your own Git hosting
- Sources
Most of the post is not a speed claim, though. It describes a trade. Pando gives up one repository-wide order of updates so that pushes to different branches do not have to wait on each other, and it gives up the per-client packfile so that one stored pack can be served to many readers. The post also links two public artifacts, OpenAI’s Git fork and a patch tracker, which let us check part of the story from the outside. We read the post against those, ran three small experiments on Git 2.55.0 for the protocol limits it leans on, and compared the tracker with upstream git/git on November 3. We could not test Pando itself, so everything below about its behavior comes from the post.
What we found:
- The numbers agree with each other but have no baseline. A 5x cut and an 80 percent cut are the same thing, and 57 terabytes in one hour fits inside a petabyte-a-day service. The post gives no absolute latency, no percentile above the median, and no repository size, branch count, push rate or client count.
- The protocol limit behind the design is real. In our test a pushing client received 84 bytes per ref, 8.4 MB for 100,001 refs, and asking for protocol v2 changed nothing because receive-pack ignores the request. A v2 fetch of one branch received one ref.
- The trunk-only pack rule has a price for a branch you keep pushing. In our toy model the pack for push number k carries all k commits, and 40 pushes cost 19.7 times the bytes of also excluding the branch’s previous tip.
- Two client features the post describes are not in upstream Git. Parallel indexing of independent packs and “explicit haves” rest on the
no-ref-deltaandexplicit-havescapabilities, which appear in OpenAI’s fork and in none of the nine upstream source and documentation files we searched on master, next and seen. By subject match, 15 of the 50 patches the tracker lists for upstream are in master. - The simulation is a teaching model. Its writer picker lists 100 to 1,000 writers, while the text says Pando supports “many thousands of clients.”
- Left open: what an unmodified Git client sees on Pando, whether atomic multi-ref pushes work, and how much storage a design that keeps packs at every level needs.
What Pando does, in the post’s own terms
Two stores and one contended key
When a client pushes, Pando “stores the incoming packfile as an immutable blob in a blob store” and, each time the branch moves, records a ref version in a metadata store: the new tip, the pack that supplied it and the earlier ref versions that pack depends on. The push path “validates the incoming packfile while streaming it to blob storage, without unpacking it into a local repository,” so the servers “primarily perform negotiation and packfile selection” and can run on commodity hardware.
Updates to unrelated refs do not coordinate. “The only contended key is the ref pointer,” which the post says it guards with “an atomic compare-and-swap-like operation.” That is the same rule a stock Git server applies when two pushes race for one branch. In our piece on GitHub’s Git rebuild, a local race between two pushes to one branch ended with the second push refused because its expected old value no longer matched the branch.
Causal consistency, stated plainly
The consistency choice is the center of the design. Pando provides “a relaxed consistency model (causal consistency) to scale writes,” which the post describes as “a weaker form of consistency across the entire repository, while providing serializability for updates to individual refs.” In practice “Clients only conflict when pushing to the same ref simultaneously,” and a reader gets no repository-wide snapshot. The post’s example is a full clone while branches foo and bar are both updated: “one client may observe foo at version 1 and bar at version 2, while another client observes foo at version 2 and bar at version 1.”
The reasoning offered is that “users and agents are agnostic to the global ordering of ref updates, and instead depend on the order of updates to a single reference.” That is a statement about workloads, and the post gives no data for it. It is plausible for developers and agents who each work on one branch. It is also the first place to look for anything that reads several refs and assumes they belong together, such as a job that builds trunk plus a topic branch or a mirror that copies every ref.
Why only trunk is advertised
Before a push, a Git server lists its refs so the client can leave out objects the server already has. The post argues that listing every ref is the expensive part: “With millions of branches, the ref advertisement alone can exceed 100 MiB,” and a consistent view of all refs needs either coordination or a query that scales with the ref count. So Pando advertises “only a designated trunk branch as a base” by default. The price is stated plainly: “This causes successive branch pushes to be larger, containing potentially-redundant objects, and uses additional client bandwidth.” The measured side of that price is that “the median packfile produced this way was under 20 KiB, and the 99th percentile was under 3 MiB.”
Why packfile URIs, and what the covers are
On the read side Pando serves stored packs instead of building one per client. The post leans on “an under-appreciated feature of the Git protocol, packfile-uri,” which lets a server return links to packs instead of sending them inline, and says “Pando uses this exclusively, and transmits an empty packfile inline.” Because “each day, a client would need to download tens of thousands of packfiles to catch up their trunk,” background workers merge the one-pack-per-push “L0” packs into larger covers, in a design the authors compare to log-structured merge-trees. “Today, our monorepo uses up to L4 packfile covers,” and, unlike many Git servers, “packfiles are kept at every level and not discarded or garbage collected.”
The numbers, checked against each other
The post’s figures are internally consistent, and the arithmetic below is ours. Where a row says “check,” it is a calculation from the post’s own numbers, not new data.
| Claim in the post | Check | Result |
|---|---|---|
| “cut median push latency by nearly 5x” in the introduction, and “reduced the latency of the median push by nearly 80%” in the deployment section | 1 minus 1/5 is 0.80 | One claim stated twice. No previous system is named, and there is no absolute time and no percentile above the median. |
| “nearly a petabyte of Git data per day,” later “a petabyte of Git data per day with just a handful of CPU cores” | 1015 bytes over 86,400 seconds is 92.6 Gbit/s on average | The wording moves from nearly to exactly. The post does not say whether the handful of cores includes the separate Express cache servers that deliver the bytes. |
| “more than 57 terabytes of Git packs in a single hour” | 57 × 1012 bytes × 8 over 3,600 seconds is 126.7 Gbit/s, about 13 clients saturating 10 Gbit/s links for the full hour | If the day totals no more than a petabyte, the busiest hour is at least 1.37 times the average hour (57 against 41.7 terabytes). A flat profile like that fits automated fetching better than office hours, which is our inference. |
| “cache hit rates of over 99%” for Express, and “more than 99% of those bytes from local disk cache” | The first is a hit rate whose unit the post does not state, the second a share of bytes | No conflict, though the denominators may differ. |
| “With millions of branches, the ref advertisement alone can exceed 100 MiB.” | Test 1 below: 84 bytes per ref for 38-character names | 100 MiB arrives at about 1.25 million refs, so “millions” is the right order of magnitude and slightly generous. |
| Median trunk-only pack “under 20 KiB,” 99th percentile “under 3 MiB” | Both are upper bounds, and 3 MiB is about 150 times 20 KiB | The tail is where trunk-only packs cost most (Test 2). The post gives no count of pushes per branch. |
| “reliability at times falling below 90%” before Pando, “improved merge reliability” after | Nothing to compute | No figure for the after state. |
| “tens of thousands of packfiles” a day to catch up trunk | 86,400 seconds divided by 10,000 to 99,999 | A trunk push every 0.9 to 8.6 seconds, if “tens of thousands” means 10,000 to 99,999. |
What the post does not give: the system Pando replaced, absolute push and fetch times, any percentile above the median, the repository’s size, the number of branches and clients, the push rate, a failure rate after migration, or how the “nearly 5x” was measured. The section titled “Compaction and time to first byte” contains no time-to-first-byte figure. None of these omissions is unusual for a vendor engineering post, but together they mean the 5x can be read as a direction and not as a size.
Test 1: the ref advertisement is real, and protocol v2 does not reach push
A Git push begins with the server sending its ref list, called the advertisement, so the client can leave out objects the server already has. We wanted to know how large that list gets and whether the newer protocol version, which lets a fetching client ask for only the refs it needs, helps pushes too. The script builds a bare repository with 100,001 refs of the kind a busy monorepo accumulates, with names like refs/heads/dev/user0042/feature-000042, and measures both directions. It ran on Git 2.55.0 (Windows build) with Python 3.13 and takes under a minute.
# ref_advertisement.py
"""ref_advertisement.py: how much must a push client download before it sends anything?
Git 2.55.0, Python 3.13, standard library only. Builds a bare repository with
100,001 refs, then measures what receive-pack advertises to a pushing client
and what a protocol v2 fetch of one branch receives.
"""
import os
import re
import shutil
import subprocess
import tempfile
N = 100_000
root = tempfile.mkdtemp(prefix="refadv_")
bare = os.path.join(root, "refs.git")
client = os.path.join(root, "client")
trace = os.path.join(root, "trace.txt")
ENV = dict(os.environ, GIT_AUTHOR_NAME="Lab", GIT_AUTHOR_EMAIL="[email protected]",
GIT_COMMITTER_NAME="Lab", GIT_COMMITTER_EMAIL="[email protected]")
def git(*args, env=None, stdin=None, cwd=None):
result = subprocess.run(["git", *args], cwd=cwd, env=dict(ENV, **(env or {})),
input=stdin, capture_output=True, check=True)
return result.stdout
# One empty-tree commit, then 100,000 branch names written straight into packed-refs.
git("init", "--bare", "-q", bare)
tree = git("hash-object", "-t", "tree", "-w", "--stdin", cwd=bare, stdin=b"").decode().strip()
commit = git("commit-tree", tree, "-m", "root", cwd=bare).decode().strip()
names = [f"refs/heads/dev/user{i % 2000:04d}/feature-{i:06d}" for i in range(N)] + ["refs/heads/main"]
names.sort(key=str.encode)
with open(os.path.join(bare, "packed-refs"), "w", newline="\n") as f:
f.write("# pack-refs with: peeled fully-peeled sorted \n")
f.writelines(f"{commit} {name}\n" for name in names)
count = len(git("for-each-ref", cwd=bare).decode().splitlines())
print(f"refs: {count}, mean ref name length: {sum(map(len, names)) / len(names):.1f}")
# 1. What a pushing client receives before it sends a single object.
default = git("receive-pack", "--advertise-refs", bare)
asked_v2 = git("receive-pack", "--advertise-refs", bare, env={"GIT_PROTOCOL": "version=2"})
per_ref = len(default) / count
print(f"push advertisement: {len(default)} bytes ({per_ref:.1f} bytes per ref)")
print(f"push advertisement with version=2 requested: {len(asked_v2)} bytes, identical: {default == asked_v2}")
print(f"refs needed to reach 100 MiB at this size: {round(100 * 1024 * 1024 / per_ref)}")
# 2. A protocol v2 fetch of one branch only asks for, and receives, matching refs.
git("init", "-q", client)
def traced(*args):
if os.path.exists(trace):
os.remove(trace)
git(*args, cwd=client, env={"GIT_TRACE_PACKET": trace})
return open(trace, encoding="utf-8", errors="replace").read().splitlines()
lines = traced("-c", "protocol.version=2", "fetch", "-q", bare, "main")
prefixes = [line for line in lines if "fetch> ref-prefix" in line]
returned = [line for line in lines if re.search(r"fetch<\s+[0-9a-f]{40} refs/", line)]
print(f"v2 fetch of one branch: {len(prefixes)} ref-prefix lines sent, {len(returned)} ref line returned")
# 3. A push that asks for protocol v2 still receives every ref.
lines = traced("-c", "protocol.version=2", "push", "--dry-run", bare, "FETCH_HEAD:refs/heads/zz-new")
seen = [line for line in lines if re.search(r"push<\s+[0-9a-f]{40} refs/", line)]
print(f"push with protocol.version=2 requested: {len(seen)} ref lines received by the client")
shutil.rmtree(root, ignore_errors=True)
refs: 100001, mean ref name length: 38.0
push advertisement: 8400199 bytes (84.0 bytes per ref)
push advertisement with version=2 requested: 8400199 bytes, identical: True
refs needed to reach 100 MiB at this size: 1248288
v2 fetch of one branch: 7 ref-prefix lines sent, 1 ref line returned
push with protocol.version=2 requested: 100001 ref lines received by the client
Each advertised ref is one line: a four-character length prefix, a 40-digit object ID, a space, the ref name and a newline, which is 4 + 40 + 1 + 38 + 1 = 84 bytes for our 38-character names. That is why 100,001 refs cost 8,400,199 bytes. With SHA-256 object names every line grows by 24 bytes, so 100 MiB would arrive at about 970,000 refs; with our SHA-1 names it takes about 1.25 million. “Millions of branches” is the right order of magnitude for a 100 MiB advertisement, and a repository with shorter names would need more refs to get there.
The second result matters more for the design. A fetching client can narrow the list: our v2 fetch of main sent seven ref-prefix lines and received one ref. A push cannot. With version=2 requested, receive-pack returned byte for byte the same advertisement, and the push client still received all 100,001 refs. A Git service that lets branches pile up therefore has to decide what a pusher sees, and advertising a single trunk, as Pando does, is one answer to a limit that exists in stock Git 2.55.0. We tested Git here, not Pando.
Test 2: what trunk-only costs a branch you keep pushing
A pushing client builds a pack of the objects the server lacks. Stock send-pack excludes everything reachable from every advertised ref: in upstream’s send-pack.c each advertised object ID is added as a negative object for pack-objects. When only trunk is advertised, the pack for a topic branch holds every commit on the branch since trunk, again and again. The script below builds that pack for 40 successive pushes of one branch, each adding a commit with 20 KiB of new random text, and compares it with the pack that would also exclude the branch’s previous tip.
# trunk_only_pack.py
"""trunk_only_pack.py: what does it cost to push one branch again and again if
each pack may exclude only trunk?
Git 2.55.0, Python 3.13, standard library only. Every push adds one commit with
20 KiB of new random text. We build the pack a client would send twice: once
excluding only trunk, once excluding trunk and the branch's previous tip.
"""
import os
import random
import shutil
import subprocess
import tempfile
ENV = dict(os.environ, GIT_AUTHOR_NAME="Lab", GIT_AUTHOR_EMAIL="[email protected]",
GIT_COMMITTER_NAME="Lab", GIT_COMMITTER_EMAIL="[email protected]")
repo = tempfile.mkdtemp(prefix="trunk_only_")
def git(*args, stdin=None):
result = subprocess.run(["git", *args], cwd=repo, env=ENV, input=stdin, capture_output=True, check=True)
return result.stdout
def pack_bytes(spec):
return len(git("pack-objects", "--revs", "--stdout", stdin=spec.encode()))
rnd = random.Random(7)
git("init", "-q", "-b", "trunk")
os.makedirs(os.path.join(repo, "base"))
for i in range(200): # trunk: 200 files of 5,000 random letters
with open(os.path.join(repo, "base", f"f{i:03d}.txt"), "w") as f:
f.write("".join(rnd.choice("abcdefghijklmnopqrstuvwxyz \n") for _ in range(5000)))
git("add", ".")
git("commit", "-q", "-m", "trunk")
git("checkout", "-q", "-b", "topic")
PUSHES, PAYLOAD = 40, 20 * 1024
previous, total_trunk, total_tip = None, 0, 0
print("push | trunk only | trunk + previous tip | running total, trunk only | running total, with tip")
for k in range(1, PUSHES + 1):
with open(os.path.join(repo, f"commit{k:03d}.txt"), "w") as f:
f.write("".join(rnd.choice("abcdefghijklmnopqrstuvwxyz0123456789 \n") for _ in range(PAYLOAD)))
git("add", ".")
git("commit", "-q", "-m", f"c{k}")
tip = git("rev-parse", "HEAD").decode().strip()
only_trunk = pack_bytes(f"{tip}\n^trunk\n")
with_tip = pack_bytes(f"{tip}\n^trunk\n" + (f"^{previous}\n" if previous else ""))
total_trunk, total_tip, previous = total_trunk + only_trunk, total_tip + with_tip, tip
if k in (1, 2, 5, 10, 20, 40):
print(f"{k:>4} | {only_trunk / 1024:>7.1f} KiB | {with_tip / 1024:>11.1f} KiB | "
f"{total_trunk / 1024:>16.1f} KiB | {total_tip / 1024:>13.1f} KiB")
print(f"after {PUSHES} pushes the trunk-only total is {total_trunk / total_tip:.1f} times the other")
shutil.rmtree(repo, ignore_errors=True)
push | trunk only | trunk + previous tip | running total, trunk only | running total, with tip
1 | 13.9 KiB | 13.9 KiB | 13.9 KiB | 13.9 KiB
2 | 27.7 KiB | 13.9 KiB | 41.6 KiB | 27.8 KiB
5 | 69.2 KiB | 14.0 KiB | 207.7 KiB | 69.7 KiB
10 | 138.3 KiB | 14.1 KiB | 761.0 KiB | 139.9 KiB
20 | 276.9 KiB | 14.4 KiB | 2906.2 KiB | 282.4 KiB
40 | 554.2 KiB | 14.9 KiB | 11349.4 KiB | 574.8 KiB
after 40 pushes the trunk-only total is 19.7 times the other
Excluding only trunk makes the pack for push k carry k commits: 13.9 KiB at push 1, 276.9 KiB at push 20 and 554.2 KiB at push 40, while the previous-tip pack stays near 14 KiB. The running total grows fastest because it adds up every pack sent so far: 11,349 KiB against 575 KiB after 40 pushes, a factor of 19.7. Pack sizes varied by under 0.1 percent between runs on one machine, so the last digit is noise.
This is a toy. Random text does not compress or delta the way source code does, and real pushes add files and edits of very different sizes. It shows the shape: linear growth per push and quadratic growth in total. The post’s figures, a median under 20 KiB and a 99th percentile under 3 MiB, say that most pushes carry a commit or two. They do not say how many pushes a typical branch gets, and that number decides whether the tail matters for agents that push the same branch dozens of times. The authors call the rule “a sensible trade-off” because a reader that already has trunk can fetch the branch with that one pack, and the same property lets Pando serve a pack unchanged to every reader.
The post describes the client feature that goes with this rule. OpenAI added the idea of explicit haves “so a server can advertise a branch’s current tip separately from the history a pushing client may leave out of its pack.” On Pando, the post says, “this enables clients to use --force-with-lease to check that a feature branch hasn’t moved and build its pack against trunk alone.” That is the trunk-only column above with a safety check added.
Test 3: packfile URIs are opt-in on stock Git
Pando serves reads through packfile URIs “exclusively.” That is a feature of Git’s protocol version 2, and it works only when the client asks. Upstream’s technical note says the client’s fetch.uriprotocols setting “determines which protocols the end user is willing to use. By default, this is empty.” The protocol document describes the request as telling the server that “the client is willing to receive URIs of any of the given protocols in place of objects in the sent packfile.” In fetch-pack.c the client sends a packfile-uris line only for http and https entries it finds in that setting. The script below configures a bare repository to offer a packfile URI for one blob, clones it twice over protocol v2 and prints what the client sent.
# uri_request_trace.py
"""uri_request_trace.py: does a stock Git client ask for packfile URIs?
Git 2.55.0, Python 3.13, standard library only. A bare repository is configured
to offer packfile URIs for one blob. We clone it over protocol v2 twice and
read the fetch request the client sends, with and without fetch.uriprotocols.
"""
import os
import re
import shutil
import subprocess
import tempfile
root = tempfile.mkdtemp(prefix="uri_trace_")
src, server = os.path.join(root, "src"), os.path.join(root, "server.git")
trace = os.path.join(root, "trace.txt")
ENV = dict(os.environ, GIT_AUTHOR_NAME="Lab", GIT_AUTHOR_EMAIL="[email protected]",
GIT_COMMITTER_NAME="Lab", GIT_COMMITTER_EMAIL="[email protected]")
def git(*args, cwd=None, env=None):
return subprocess.run(["git", *args], cwd=cwd, env=dict(ENV, **(env or {})), capture_output=True, check=True).stdout.decode()
git("init", "-q", src)
with open(os.path.join(src, "big.bin"), "wb") as f:
f.write(os.urandom(4096))
git("add", ".", cwd=src)
git("commit", "-q", "-m", "one", cwd=src)
blob = git("rev-parse", "HEAD:big.bin", cwd=src).strip()
git("clone", "-q", "--bare", src, server)
git("config", "uploadpack.blobPackfileUri", f"{blob} {'0' * 40} http://127.0.0.1:9/never.pack", cwd=server)
for label, extra in [("default configuration", []), ("fetch.uriprotocols=http", ["-c", "fetch.uriprotocols=http"])]:
dest = os.path.join(root, "clone-" + str(len(label)))
if os.path.exists(trace):
os.remove(trace)
git("-c", "protocol.version=2", *extra, "clone", "-q", "--no-local", server, dest, env={"GIT_TRACE_PACKET": trace})
lines = [re.sub(r"^.*?packet:\s+", "", line) for line in open(trace, encoding="utf-8", errors="replace").read().splitlines()]
advertised = [line for line in lines if line.startswith("clone< fetch=")]
start = next(i for i, line in enumerate(lines) if line == "clone> command=fetch")
request = [line for line in lines[start:] if line.startswith("clone> ")]
uri_lines = [line for line in request if "packfile-uris" in line]
print(f"[{label}]")
print(" server advertises:", advertised[0].removeprefix("clone< "))
print(" fetch request lines mentioning packfile-uris:", uri_lines if uri_lines else "none")
shutil.rmtree(root, ignore_errors=True)
[default configuration]
server advertises: fetch=shallow wait-for-done packfile-uris
fetch request lines mentioning packfile-uris: none
[fetch.uriprotocols=http]
server advertises: fetch=shallow wait-for-done packfile-uris
fetch request lines mentioning packfile-uris: ['clone> packfile-uris http']
The server advertised the feature both times, and the client asked for URIs only when configured. We did not examine what the server does with a client that does not ask. The stock server’s own mechanism is also narrower than Pando’s use: the upstream note describes configured entries where “all such blobs are excluded, replaced with URIs,” and says clients “should not expect that packfiles downloaded in this way only contain single blobs.” Pando uses URIs for whole packs. The post says OpenAI began “shipping our public distribution of Git to developer machines and within the ChatGPT desktop app,” and migrated clients with “an internal wrapper around the Git CLI to migrate clients in phases.” It does not say what a client that never asks for URIs receives.
The client half: what OpenAI’s fork and tracker show
The post points to its own code: “Several of these improvements are already in upstream Git, and many others in our public repository.” The tracker headline counts “92 Commits carried on top of upstream,” 50 patches proposed upstream and 23 topics in the published build. It is stamped October 31 and names the upstream commits it was verified against: master c51408889c3b, next 68944adf515b and seen b75530aec9ab. In our clone the master and next commits are dated October 16 and 17, and master has taken 163 commits since, from “The 6th batch” to “The 14th batch,” so the tracker’s upstream statuses may describe an older upstream than its date suggests. We checked the same things on November 3 with a tree-less clone of git/git and GitHub’s compare view.
Two capabilities the post describes
Both client features in the post are defined by capabilities in the fork’s protocol document. For explicit-haves the document says a receive-pack server can advertise it “to indicate that the object IDs of its named refs are not necessarily pack prerequisites,” and adds: “A server that relies on this behavior MUST verify that the client requested the capability before accepting the pack.” The commit that adds it, by Taylor Blau on September 14, gives the reason: “Today send-pack treats every advertised ref as a negative revision, so replacing such a ref can omit objects that the receiver loses with the old pack.” It says the change “lets whole-pack replacement servers advertise the current ref OID for --force-with-lease without making the replacement depend on the pack it supersedes.”
For parallel indexing the capability is no-ref-delta. The fork’s document says that when a server advertises it, “the client MUST NOT send OBJ_REF_DELTA (aka type 7) in a packfile,” and that for protocol v2 fetch “the promise covers both the inline pack and any packs named by a packfile-uris section.” The reason is in the commit that teaches the client to index URI packs concurrently: “Packfile URI responses are indexed in advertised order because a later URI pack can contain a REF_DELTA whose base is installed by an earlier response pack.” An earlier commit adds the option for servers that retain a pushed pack “without building an index of object IDs,” which is consistent with the post’s description of a push path that never unpacks. The tracker lists the topic as “Negotiate packs without REF_DELTA objects so independent URI packs can be indexed concurrently.”
The tracker also shows work in progress. A development branch named for Pando, listed under “Not in this build,” adds a third capability: “Add --require-exact-old-oid and the exact-old-oid-v1 capability.” In the tracker’s words it would “Require both exact old-OID support and explicit-haves before sending updates or a pack.”
What upstream Git has
Neither capability is in upstream Git’s current branches. On November 3 we fetched send-pack.c, builtin/receive-pack.c, builtin/pack-objects.c, builtin/index-pack.c, upload-pack.c, fetch-pack.c, remote-curl.c and the two protocol documents from the master, next and seen branches of git/git, and searched them for explicit-haves, no-ref-delta and exact-old-oid. Not one of the 27 fetches contained any of the three strings. Neither of the first two appears among the 50 patches the tracker lists for upstream either.
For the 50 patches themselves we matched each tracker subject against commit subjects in upstream master, next, seen and jch since April 1, first by exact subject and failing that by a close one:
| Group of patches | Count | Where upstream on November 3 (date of the commit) |
|---|---|---|
| Packfile-URI download fixes: concurrent appends, HTTP 416, double-closed index-pack input, Windows partial-pack unlink, “pack” output, a documentation fix | 6 | master (July 26) |
| Trace2 patches: packfile-URI downloads, pack bytes written | 2 | master (July 26 and August 19) |
| Sparse-index checks, path-walk with bitmaps and delta islands, a perf-test fix | 6 | master (June 21 and July 20) |
| CI workaround for Debian 12 | 1 | master (September 24) |
| MIDX: skip writes with no object entries | 1 | next (October 29) |
| Repack and MIDX cruft series | 8 | seen (September 30) |
| Skipping negotiator: parse parents before queueing | 1 | seen (merge commit of November 3, patch dated October 26) |
| Bitmap selection (5), incremental MIDX chains (11), parallel checkout (2), promisor maintenance (2), unchanged-index checkout, branch Coccinelle fix, fetch pack locks, a curl test fix, an RFC for duplicate objects in index-pack | 25 | no subject match |
Master does not change retroactively, so its 15 are firm. They include all six packfile-URI download fixes the tracker describes with “These six patches are now in the upstream base.” Next and seen are rebuilt often and maintainers reword subjects, so read the other rows as approximate. The tracker says three custom-base MIDX patches are in seen; our matcher did not find them by subject, so the 25 is an upper bound on what is missing. One row moved after the tracker’s check: on November 3 seen contained a merge of tb/negotiator-skipping-parse-parents, Taylor Blau’s October 26 submission of the skipping-negotiator fix under the subject “negotiator/skipping.c: parse parents before queueing,” while the tracker lists its version as “prepared for upstream; not in master, next, or seen.”
The split is informative. What OpenAI has already upstreamed makes packfile-URI downloads safe and observable. What defines Pando’s push and parallel-index behavior is not there yet, and the place we found it is the fork.
How big the fork is
GitHub’s compare view of git/git master against openai:codex on November 3 listed 115 commits ahead, 23 of them merge commits and 92 not, and 522 behind. The 92 matches the tracker headline. The merge base is upstream’s “23rd batch” from September 7, so the fork’s base lags master by 522 commits. The tracker’s published build, v2.55.0-openai.921.g9fe64f46bdef from October 19, has archives for macOS, Ubuntu and Windows on arm64 and x64 with checksum files, and a release eight minutes later carries codex-unstable in its name. The fork was created on July 10 and its earliest carried commit is dated June 12, while the post says Pando “began serving production traffic in April of this year,” so the public record starts after production did.
The simulation, and what it can show
The post embeds an interactive comparison of three designs: three-phase commit, an object-store write-ahead log and Pando. The page offers three infrastructure profiles (Standard, Fast, Ultrafast), writer counts of 100, 200, 400, 600 and 1,000, and toggles labeled “Simulate merge” and “Batch trunk merges.” The caption is blunt: “Synthetic teaching models, not production benchmarks.” The setup is “1,000 clients to each select a distinct branch and then repeatedly push unique commits,” and “we’ve relaxed repository-wide consistency for three-phase commit.”
Two things limit what it can show. First, distinct branches are the workload per-ref coordination was built for. The modeled write-ahead log funnels every push through one shared manifest by construction, so it has the most to lose when every writer has its own branch. Trunk merges, where writers share one ref, appear only as an option: “With merges enabled, writer completion includes the required fetches and trunk merge.” Second, the text claims more than the page offers: “In both real world testing on our monorepo and in this simulation, Pando supports many thousands of clients able to push concurrently without impacting performance under load.” The writer picker stops at 1,000, and the post gives no client count for the real-world side.
The three-phase-commit design the post models resembles the one GitHub describes for Spokes, with full copies on several servers and a quorum for each reference update, which we covered in our piece on GitHub’s Git rebuild. The post does not say which systems it modeled. For a different answer to the same agent-scale problem, see Cloudflare’s Artifacts, which gives each task its own repository, and for OpenAI’s other recent infrastructure write-up, the Habitat storage sequel.
What the post leaves open
What an unmodified client sees
The post says it kept “compatibility with the Git protocol and client.” The fork’s own protocol text says a server that relies on explicit haves “MUST verify that the client requested the capability before accepting the pack,” stock Git 2.55.0 asks for packfile URIs only when configured, and the no-ref-delta commit describes servers that keep pushed packs without indexing them. Read together, they leave unclear what a stock client gets on Pando for a push or a clone: a slower path, an error or something else. The post says OpenAI migrated clients “in phases” through its own wrapper and builds, which suggests the optimized path assumes them.
Atomic multi-ref pushes
Pando “coordinates updates to each branch independently.” Git’s push documentation says --atomic will “use an atomic transaction on the remote side if available. Either all refs are updated, or on error, no refs are updated. If the server does not support atomic pushes the push will fail.” Whether a branch and a tag, or two branches, can move together on Pando is not stated, and releases, mirrors and stacked changes often depend on it.
Reading more than one ref
The post says readers should not “assume those reads describe a single point in time across the repository.” It does not say how stale one ref can be relative to another, or what tooling it uses for jobs that need two refs to agree.
Storage and tail behavior
If “packfiles are kept at every level and not discarded or garbage collected,” and levels run up to L4, a pushed object could sit in as many as five packs, one per level, before compression gains. That is our reading of the sentence, and the post gives no storage figure, retention rule or deletion policy. It also gives no tail latency and no time to first byte. Finally, the authors write that they are “currently exploring ways to share the technology that makes Pando possible with our users,” with no date, license or scope.
What to check on your own Git hosting
- Measure your advertisement.
git receive-pack --advertise-refs your-repo.git | wc -con a server-side copy shows what each pushing client downloads first. At 84 bytes per ref, 100,000 branches cost 8.4 MB per push before any object moves. - Count pushes per branch. Test 2 shows that a rule which excludes less costs most for branches pushed many times. Agent workflows that re-push one branch for every change are the ones to count.
- Check
fetch.uriprotocolsbefore assuming packfile URIs are in use. On stock Git the default is empty. - Treat OpenAI’s builds as experimental. The fork is 522 commits behind master, and some releases are labeled
codex-unstable. The patches that reached upstream are the safer place to start.
Sources
- OpenAI, “Scaling Git at OpenAI,” November 2, 2026
- OpenAI Git patch tracker (updated October 31, 2026) and the openai/git repository, including its releases
- Fork commits a3eae73013 (explicit-haves), 3f6ed83847 (
pack-objects --no-ref-delta) and a08b8d8bb7 (concurrent URI-pack indexing), and the fork’s protocol capabilities document - Upstream Git: packfile-uri technical note, protocol v2, git push, send-pack.c, fetch-pack.c and the seen branch
- Our earlier coverage: GitHub’s Git rebuild, Cloudflare’s Artifacts and OpenAI’s Habitat storage sequel








No Comment! Be the first one.