OpenAI’s Habitat Platform Turns a Deliberate Python Shortcut Into a Two-Engineer Rust Rewrite

Habitat is the storage platform behind almost everything OpenAI’s products do: every ChatGPT reply, every Codex session, every internal service that needs to read or write data. In an engineering post published September 11, OpenAI put real numbers on what running that platform at its current scale actually looks like: more than 70 million requests every second, support for over 1 billion people each week across roughly 40 geographic regions, and more than 500 petabytes of data moving through a system that began, two years ago, as a small Python library.

The scale isn’t the interesting part of the post, though. The interesting part is a bet OpenAI made early and only recently collected on. Rather than build Habitat’s backend in a fast, memory-efficient language from the start, the team deliberately kept it in Python, treated the resulting performance cost as known technical debt, and wagered that its own coding models would eventually make a language migration cheap enough to be worth deferring. In the second quarter of 2026, that bet paid off: two engineers, using OpenAI’s Codex and GPT-5.5, rewrote the entire service in Rust.

What Habitat Actually Moves

Habitat began in mid-2024 as a Python client-side library that let product engineers store and retrieve data without directly managing a database connection. Under the hood it mapped to Azure Cosmos DB. The library handled schema lookup, routing, authorization, encryption, serialization, request shaping, and connection pooling, so teams building ChatGPT features didn’t have to think about any of it themselves.

That model worked well enough that Habitat spread across OpenAI’s products with, in the post’s own words, no organized push away from self-serve Postgres and Cosmos DB. It grew because engineers chose to use it, not because they were told to.

Today Habitat is a distributed service, not a library. A client SDK talks to an Envoy proxy layer, which routes requests to a fleet of habitat-service processes, which in turn talk to a set of regionally distributed Cosmos DB accounts, a caching layer built on Valkey, blob storage, and a separate storage system OpenAI calls Nanobase. A separate change-data-capture path feeds Databricks, Rockset, and Kafka for offline analytics, keeping that traffic isolated from the online request path.

When a Client-Side Library Stopped Being Enough

By mid-2025, the library model had hit a structural limit. As OpenAI’s service count grew and Habitat’s own logic grew more complex, making a protocol change that stayed backward-compatible with every consumer became impractical.

The post gives one concrete example. To limit the blast radius of a single-region outage on its most critical data, OpenAI wanted to move that data onto regionally distributed Cosmos DB accounts. Doing it safely meant adding new routing logic behind a feature flag, waiting for the change to reach every client, shadowing the new routing against the old path to confirm the two agreed, and only then flipping the flag on. Coordinating that sequence across dozens of independently deployed services took days.

Turning Habitat into a standalone service fixed the coordination problem directly: instead of dozens of teams each needing to redeploy a shared library, OpenAI got one place to deploy, observe, and change Habitat’s behavior.

Fighting the Interpreter, Not the Database

Once Habitat had to serve traffic directly as a service, its Python implementation created a specific kind of latency problem that had little to do with database performance at all.

Scheduling Delay, Not Slow Storage

Python’s asyncio lets a service handle many requests concurrently, but the language’s global interpreter lock still means only one request executes CPU instructions at any given instant. Habitat isn’t a thin proxy; it does real CPU work on every request, including routing, compression, encryption, checksumming, downstream health checks, request shadowing, and hedging. With that much CPU-bound work competing for a single execution slot, OpenAI found that scheduling delay, not slow downstream calls, was driving its tail latency. Traces for the slowest one percent of requests showed Cosmos DB responding quickly while the request itself sat waiting for its own coroutine to be rescheduled.

The fix wasn’t algorithmic so much as observational: instrument the event loop directly by scheduling background tasks and measuring the gap between when each was supposed to run and when it actually ran. That measurement surfaced scheduling jitter reaching hundreds of milliseconds, and in some cases several seconds, even at modest concurrency per process. The mitigation was to deliberately under-load each Python process, capping the number of concurrent requests any one process handles, and to scale horizontally instead by running far more worker processes.

A Feature-Flag Library Was Quietly Stalling Every Worker

Live CPU profiling traced a specific slice of that scheduling delay to Statsig, the feature-flag and experimentation platform Habitat used for configuration. By default, Statsig polled for a refreshed config once a minute with no jitter across processes, and the config it fetched included every production rule for every service at OpenAI, not just Habitat’s own. Once a minute, every worker in every pod stopped processing in-flight requests to parse that entire file. The fix, once the profiler pointed at the cause, was ordinary: pull a smaller, targeted configuration instead of the full company-wide one, lengthen the refresh interval, and stagger refreshes with jitter so pods don’t all stall at the same instant.

The Connection Pool That Made Slow Servers Slower

The most instructive failure in the post is a connection-pooling feedback loop. OpenAI’s engineers found that Python’s aiohttp client library, as Habitat had it configured, reused whichever pooled connection had most recently been returned rather than the one that had been idle longest, a last-in-first-out pattern. That’s a reasonable default in the general case, since it lets extra connections opened during a traffic burst sit idle and eventually time out. Under load, it backfired: backend servers that were already overloaded and slow returned their connections to the pool later than healthy ones did, which made those connections the most recently returned, which made them the ones handed out first for the next request. Struggling servers kept getting sent more work precisely because they were already struggling, a self-reinforcing pattern OpenAI’s post calls a metastable failure. Switching the client’s pool to first-in-first-out reuse broke the loop and, as a side effect, reduced request-latency variance even under normal load.

Envoy does complementary work on the same problem. It upgrades Habitat’s outbound Python client connections from HTTP/1 to HTTP/2, which lets many logical requests share far fewer physical connections, and it centralizes rate limiting and circuit breaking that would be weaker if each Python process implemented its own copy.

A Narrower API Was the Point, Not a Compromise

Habitat’s client API is deliberately limited. Instead of arbitrary queries, clients get a constrained NoSQL interface built around object and edge types, with no support for unbounded scans or complex joins. OpenAI is explicit that this came from a hard lesson: earlier in the company’s history, most online data sat in Postgres, and reviewing every query and schema change by hand was manageable while the system was small. As OpenAI grew, that review process stopped scaling, and the post says it became “a frequent cause of outages where a single expensive new query on a hot path took out the database.” A general-purpose SQL interface makes it cheap to write an expensive query. Habitat’s narrower API makes expensive operations difficult to write by accident, at the cost of pushing complex querying needs onto a separate offline path: Habitat streams changes via CDC into isolated Rockset instances that each client team has to provision and scale for itself.

The Bet That Paid Off in Q2 2026

The most pointed part of the post is OpenAI’s own account of why it didn’t rewrite Habitat into a faster language from day one. The team says it knew Python’s overhead, in network latency and in CPU and memory cost relative to a local library, would not be acceptable at 100 times the scale it was designed for, and that an eventual rewrite was “almost certain.” It accepted that as technical debt anyway, because the actual priority at the time was unblocking product engineers and stabilizing the platform, not resource efficiency. Underneath that choice sat an explicit prediction: that improvement in OpenAI’s own coding models would make a future migration cheap enough to justify deferring it. In the post’s words, the team “made a calculated wager that the rapid advancement of our own coding models would simplify the technical path in the future,” betting specifically that Codex and GPT would make an eventual Python migration achievable by the time one was actually required.

By the time that bet came due, Habitat had grown into OpenAI’s second-largest internal service by core count, and its fourth-largest by Envoy footprint, with the Python service handling more than 20 million requests per second at its peak. In the second quarter of 2026, two engineers, working with Codex and GPT-5.5, rewrote the entire service in Rust. OpenAI says the Rust version now handles 95 percent of production traffic, that Python will be retired entirely within weeks of this post, and that the new service measures six times more CPU-efficient and 15 times more memory-efficient than the Python version it replaces, with lower average and tail latencies on both counts.

What’s Left for Part Two

This post covers only the request-serving layer. OpenAI says a second post will cover the storage layer itself: how Habitat’s Cosmos DB partnership handles the 500-plus petabytes and 70-plus million requests per second cited at the top of this post, along with multi-tenancy reliability and a layered strategy for read performance.

A Rare Data Point in the AI-Productivity Debate

Two engineers rewriting an entire production storage service into a different language in one quarter, with AI coding tools credited for a substantial share of the work, is exactly the kind of concrete example the debate over AI coding productivity has mostly lacked. A recent Red Hat critique covered on this site pointed to a controlled study in which developers using AI coding tools were measurably slower while believing themselves to be faster, part of a broader pattern of AI-assisted coding gains failing to show up in delivery metrics. OpenAI’s account here is a different kind of evidence: a single, large, self-reported case rather than a controlled comparison, published by the same company that makes both the coding models being credited and the blog post making the claim. There’s no independent way to check the specific six-times and 15-times efficiency figures, since they describe internal, proprietary infrastructure that only OpenAI can measure.

What is checkable is the broader pattern the post fits into. It’s at least the second major infrastructure rewrite into Rust for efficiency this year that this site has covered, following Cloudflare’s own DNS cache rewrite, which cut per-entry memory use by more than half through a series of struct-layout changes. And independent trackers had OpenAI’s underlying growth number, ChatGPT’s roughly 1 billion weekly users, converging on the same figure around the same time this summer, which is the one part of Habitat’s headline scale that outside reporting can actually confirm.

Sources

• OpenAI: Rapidly scaling online storage to serve over 1 billion ChatGPT users
• Statsig: feature flag and experimentation platform overview
• sxz.io: Cloudflare’s Big Pineapple Rewrite Turns Every DNS Cache Entry Into a Memory Tax
• sxz.io: Red Hat’s Faster-Coding Warning Turns AI Productivity Into a Measurement Problem

Image credit: A Christmas Island red crab (Gecarcoidea natalis), photographed by ChrisBrayPhotography, licensed CC BY-SA 4.0 via Wikimedia Commons; cropped and converted to WebP for sxz.io.