OpenAI’s Habitat Sequel Turns Cosmos DB Multi-Tenancy Into a Co-Managed Partnership, and Its 12x Is a Median With No Hit Rate

OpenAI has published Part II of its storage series, the engineering account of how Habitat, the storage platform behind OpenAI’s products, scaled the Azure Cosmos DB layer underneath it. Part I, which we covered in September, was about the Habitat service and its Python-to-Rust rewrite. Part II is a three-act story: one shared container where niche workloads could disrupt core login and signup flows, many dedicated containers that produced a capacity-planning burden, and a return to one shared container with fixes the post says were engineered with the Azure team. It then credits a layered read path with cutting the global read median from “roughly 120 ms to roughly 10 ms, a 12x improvement”, and closes with a change-data-capture platform and an in-house, exabyte-scale store that is still in development.

The story is worth reading, but a reader has to separate three things: what follows from Azure’s published limits, which fixes are public features and which were built for one customer, and what a median can prove. We checked the post against eight Azure Cosmos DB documentation pages and the original TAO paper from Facebook, read the figure that carries the 12x out of its own vector coordinates, and ran two small calculations. The calculations are ours, not OpenAI’s, and they are labeled as such where they appear.

The three designs and what each one broke

The post walks through three phases of tenancy on Cosmos DB. The table compresses it, with the failures in the post’s own terms.

The post is candid that the third act is unfinished: “We’re still very early in our multi-tenancy story on Habitat”. It also does not say how many months “the months since” covers. The trade-off underneath, one tenant per system versus many tenants sharing one, is not specific to Cosmos DB; our PostgreSQL row-level security tutorial shows the shared-tables end of the same choice.

Three failure modes that follow from Azure’s published limits

Each failure the post describes lines up with a number on Azure’s documentation pages, which lets us put figures on it.

Logical partition keys carry the same ceiling as physical partitions

The post says it exposed partition-key choice to clients, so products collided on common keys such as user_id or workspace_id. Azure’s partitioning page explains why that bites. Each physical partition can provide up to 10,000 request units per second, and because each logical partition maps to exactly one physical partition, that limit “implies that logical partitions also have a 10,000 RU/s limit”. A logical partition can also hold at most 20 GB. Two products that share a key therefore share one 10,000 RU/s ceiling, however large the container is.

Noisy neighbors share a physical partition

The post puts the enforcement point at the physical partition: “While documented data size and throughput limits apply to logical partition keys, the physical partition is actually where limits are enforced.” Azure confirms the mapping, saying that “one or more logical partitions map to a single physical partition”. Everything colocated on a partition draws on the same 10,000 RU/s, which is how, in the post’s account, “workloads operating on data for niche product features disrupting core login and signup flows” became possible.

Weak partitions, in numbers

The third failure mode is the least intuitive. The post puts it plainly: “The container’s provisioned throughput is distributed roughly evenly across the physical partitions for that container.” When partitions split as data grows, each partition’s share falls, and for “blob-like” use cases storage growth “alone reduces throughput capacity for active workloads even while request rates and usage remain stable”. Azure says the same thing in one sentence: “Provisioned throughput for a container divides evenly among physical partitions.”

The documented limits let us size the effect. The table is our calculation from Azure’s numbers, not a figure from the post. A physical partition stores up to 50 GB and delivers up to 10,000 RU/s. Azure’s service quotas page lists 1,000,000 RU/s as the maximum for a container with dedicated provisioned throughput (a support ticket can raise it), and gives the lowest throughput you may provision on manual throughput as the largest of 400 RU/s, one RU/s per gigabyte stored, and a hundredth of the highest throughput ever provisioned.

A container larger than about 5 TB cannot hold every partition at its ceiling inside the default container maximum, and at 100 TB and that default maximum the even split leaves 500 RU/s per partition, 5 percent of what a partition can deliver. That is the arithmetic behind the post’s complaint that storage growth alone shrinks the capacity left for active workloads.

The launch-day figure checks out, with one detail added

Figure 3 in the post compares a launch against a shared container with 100M RU/s of capacity at 80 percent utilization, which leaves 20M RU/s of headroom, and against a new dedicated container with 10K RU/s. We recomputed it. A 1K RU/s load test is 0.005 percent of the shared headroom and a 1M RU/s launch is 5 percent, while the same two loads are 10 percent and 10,000 percent of the dedicated container. The arithmetic is right. It also carries a detail the post does not mention: 1,000,000 RU/s is Azure’s documented per-container maximum before a support ticket, so a dedicated container sized for that launch would sit at the limit with no room above it.

Which fixes you can copy, and which were built for one customer

The third design rests on three changes the post lists. We looked for each one in the Azure pages.

The serverless row deserves a second look. The post is accurate about the public mode: it says that mode “would solve our weak partitions problem, but prevent us from scaling up RU-heavy use cases”. What fixed OpenAI’s problem is a variant the post says was built for it, and the post does not say whether that variant is or will be available to other customers. The portable lesson is the shape of the design, with one shared container, tiered priorities and no per-team capacity planning, and not a feature a reader can switch on.

“Co-managed” is a relationship, not a setting

The post’s own name for the result is “co-managed containers”, and it describes what that means: “shared tests, shared dashboards, direct engineering channels, and joint ownership of failure modes that only appear at our scale”. The team says it tests changes against “a production-sized but non-production Azure Cosmos DB account” and has “exposed ways to directly page each other without going through support tiers”. The post presents this as a consequence of size, since “As OpenAI became one of the largest users of Azure Cosmos DB, operating it successfully required a different model”, and claims that “To our knowledge, we are the only company which has deployed Azure Cosmos DB at this scale”. That claim is OpenAI’s own and the post offers no way to check it. The evidence it cites is one account with “10s of petabytes of data” and “more than 20M QPS”.

The two outages the post describes show what the relationship was for. In late 2024 a power outage hit the primary region of one of its largest accounts, and “we realized that our largest accounts would take several hours to fail over to a secondary region, something neither our team nor the Azure Cosmos DB team had tested at our scale”. About a month later, failing back set off a second outage when “A connection stampede to the original region overwhelmed the original region’s Azure Cosmos DB gateway”, because a quirk in the gateway-mode client sent every client back to the original region although only some partitions had moved. A failback stampede is a close relative of the herd behavior we reproduce in miniature in our cache stampede tutorial, and the mitigations that family calls for, admission limits and shedding, are the subject of our load shedding tutorial.

Reading the 12x

What Figure 5 shows when you read its coordinates

The page draws Figure 5 as a vector graphic: a millisecond axis from 0 to 160, a 37-point line, and three numbered markers for “critical data replicated into nearby geos”, “in-region caching launched” and “rust rewrite launched”. The post gives no per-step numbers, so we converted the line’s coordinates to milliseconds using the axis labels. The three marker circles come out at 108.7, 47.7 and 24.8 ms by the same conversion, which is within about 2 ms of the plateau medians below, so we trust the reading to within a few milliseconds.

Three things stand out. First, the line starts below the number in the text. The first plateau reads 108.7 ms and none of its points exceeds 115.4 ms, about 10 percent under the 120 ms in the prose. By the chart the end-to-end gain is 10.7x, from about 109 ms to about 10 ms, and the text rounds both ends in the direction of 12x. Each rounding sits inside “roughly”, but together they turn about 11x into 12x.

Second, the largest single step is the first, a drop of about 59 ms, and the post describes that change as replicating “the smallest and most latency-sensitive datasets”, with “authentication metadata that is needed on every request” as its example. A median only falls that far if a substantial share of requests got faster, which suggests that those few datasets account for a meaningful share of reads.

Third, the last step is worth about 15 ms of the 25 ms that remained. The post describes it as “removing overhead in the Habitat service itself”, so by the chart’s reading the service layer accounted for roughly 15 of the remaining 25 ms at the median (medians do not strictly add, so treat that as a rough read).

The horizontal axis is labeled only “Time”. The markers sit at about 24, 49 and 74 percent of the width and the steps end at 25.0, 50.1 and 75.0 percent, so the spacing says nothing about how long each stage took. The post does not say whether the line is raw telemetry or a drawing of it, and two pairs of consecutive points share an identical value.

What a median can and cannot say

The post is candid that the median is not the whole story: “While caching has dramatically reduced median latency, the slowest requests still require fetching uncached data from underlying storage, often across distant regions.” It gives no cache hit rate (it says only “high hit rates”), no p99, and no split of reads into hits and misses. That matters because a median has a threshold behavior: once more than half of requests are served from a nearby cache, the median is the cache’s latency and stops responding to anything else.

To show the effect we ran a toy model. It is ours, with invented latencies, and it is not OpenAI data: cache hits with a median of 8 ms, cross-region misses with a median of 120 ms (both log-normal, with spread parameters of 0.35 and 0.5), and 200,000 simulated requests per row.

In this model a 12x median gain appears between a 60 and an 80 percent hit rate, while the p99 improves by only 1.2x to 1.4x; the p99 gain passes 2x only at a 95 percent hit rate. None of this says Habitat’s hit rate is in that range. It says the 12x headline is compatible with a tail that barely changed, and the post’s own sentence about the slowest requests points at the tail as the remaining work. The numbers worth asking for are the hit rate per dataset, the p99 on the same chart, and the per-step values we had to read off the line. We applied the same line-by-line approach to Cloudflare’s D1 control plane post earlier this month.

The cache is not Azure’s integrated cache, and TAO sets the ordering rule

A name collision

Habitat’s “integrated cache” is a Valkey lookaside layer that Habitat fills and invalidates itself. Azure Cosmos DB has a feature with the same name, and it is a different thing: an in-memory cache inside the dedicated gateway, “a read-through, write-through cache with a Least Recently Used (LRU) eviction policy”. Azure adds: “Each dedicated gateway node has a separate integrated cache from the others.” A reader who looks up “integrated cache” in Azure’s documentation lands on the other one.

TAO’s rule for ordering invalidations

The post says it took “a more advanced approach to invalidation, following in the footsteps of systems like Meta’s TAO”, and uses the Cosmos DB change feed “to propagate invalidations globally within seconds”. The TAO paper (USENIX ATC 2013) is specific about ordering: “TAO embeds invalidation and refill messages in the database replication stream. These messages are delivered in a region immediately after a transaction has been replicated to a slave database. Delivering such messages earlier would create cache inconsistencies, as reading from the local database would provide stale data.” It also puts a version number in its cache maintenance messages so that one arriving after the follower was already updated can be ignored.

The reason is simple arithmetic, and the arithmetic is ours rather than the post’s. If an invalidation reaches a region g seconds before that region’s database replica has applied the write, any read of the key inside that gap refills the cache with the old value. With reads arriving at r per second as a Poisson process, the chance of at least one such read is 1-exp(-r*g): 22 percent at 1 read per second and a quarter-second gap, 71 percent at 5 reads per second, and 99 percent at 20. Hot keys, which are the ones a cache exists for, are the most exposed.

Three questions the post leaves open

Deletes

Azure’s change feed modes page says that in the default latest version mode “Deletes aren’t captured as changes”. The alternative, all versions and deletes mode, needs continuous backups: “you must have continuous backups configured for your Azure Cosmos DB account”. The post does not say which mode Habitat reads or how a deleted object leaves the cache.

Which replica drives the invalidation

The post’s CDC section says the platform uses “replica regions” and “lower-priority feed reads” to protect latency-sensitive traffic, which fits the TAO arrangement of invalidating a region from its own replica. That sentence is about capture workloads in general, though, and not about cache invalidation specifically, so the question of whether an invalidation can outrun its replica is unanswered.

What “within seconds” means

The post gives no percentile for the propagation delay. For product code that writes and then reads, the worst-case delay, not the typical one, decides whether it needs its own read-after-write handling.

Which numbers add up

These figures describe different things, a platform, a service tier and a single account, and they cannot be added or compared directly. There is no hit rate to connect the 70 million requests at the front door, over 90 percent of them reads by the post’s count, to the 20 million operations a second at the largest account. Our arithmetic on the figures that are given: if “10s of petabytes” means 10 to 50, and the partitions are full, then at 50 GB per physical partition that account holds 200,000 to 1,000,000 physical partitions. Spread over that much data, 20 million operations a second is between 0.4 and 2 operations per second per gigabyte, or on average 20 to 100 operations per second for each 50 GB partition. The average hides the shape of the load. The same total spread evenly or concentrated on a few hot keys hits the partition limits very differently, which is what the post’s throttling and tiering are for.

What comes next: an in-house store with a stated trade-off

The post ends with a plan for a “custom in-house exabyte-scale online storage system, optimized for bulky Habitat data types” that “combines the RocksDB Cloud core from Rockset with Raft-based replication”. OpenAI announced its acquisition of Rockset on June 21, 2024. The stated trade-off is the part to remember: the system “implements the same document model as Azure Cosmos DB, but uses disaggregated storage and physical replication to reduce the amount of flash capacity needed, in exchange for a lower peak qps”.

Part II explains why capacity is the constraint: broad replication “is also ultimately capacity constrained and therefore limited to a few hand-selected datasets”. Read together, the post ties the next system to that capacity limit: it spends peak throughput to save flash. The post gives no number for “lower”.

What to take from it

• If you run Cosmos DB or any partitioned store, work out the per-partition share after the splits you expect, using the documented rule that provisioned throughput divides evenly, before you size throughput for the whole container.
• If background jobs share a container with user traffic, tier them with priority-based execution, and plan for it being best-effort because the documentation says so.
• When a vendor post reports an “Nx faster” median, ask for the hit rate and the p99 before using the figure for planning.
• If you invalidate a cache from a change feed, decide up front how deletes arrive, which replica drives each region’s invalidations, and what bounds the delay.