Cloudflare Says Its GLM-5.3 Model Returned Other Customers’ Data in 155 Responses, and a Public SGLang Bug Report Had Described the Same Path Since July

Cloudflare said on Monday, October 26 that its GLM-5.3 model on Workers AI returned data from other customers’ requests in 155 responses between August 27 and September 9, all to prompts longer than 681,280 tokens. The cache pool on the model’s prefill GPUs held 681,280 tokens while its context window was 1,048,576, a mismatch Cloudflare says it knew about before launch, and a rejection path in SGLang, the open-source inference engine it runs, let over-length prompts through to a decode node that then read cache pages the request had never written. Cloudflare says it found “no evidence of deliberate exploitation,” took the model offline within 40 minutes of declaring an incident on September 9, and published 47 days later together with SGLang’s maintainers and the researcher who reported it, Gadi Evron of Knostic.

The record around the incident is where the story gets sharper. An SGLang issue filed on July 6 describes the same rejection path and includes a patch; a bot closed it for inactivity on September 5, less than four days before the report reached Cloudflare. SGLang’s own disclosure goes further than Cloudflare’s: it says a related race, in which an aborted request’s late writes land in another request’s cache, affected default prefill/decode deployments from early 2025 until v0.5.21, that the fix was not backported to earlier release branches, and that a deployer “has no reliable way to tell” whether it was exposed. Cloudflare’s post does not say how many customers’ data appeared in the 155 responses or whether anyone was told.

We read Cloudflare’s post by Kevin Flansburg and Michelle Chen, SGLang’s disclosure, Knostic’s report (dated October 27), Cloudflare’s status page entry and its GLM-5.3 model page, SGLang’s issue #30233, pull requests #36988, #38935, #38704 and #46435, its release list, and an NVD keyword search.

What Cloudflare says happened

GLM is a model from Z.ai built for agentic coding. Cloudflare serves it with SGLang in a disaggregated setup: a pool of prefill GPUs processes the prompt and builds its key-value (KV) cache, the per-request attention state a model keeps while it reads, then hands that cache over RDMA, through Mooncake, to a pool of decode GPUs that generate the answer. The KV cache is a shared pool of fixed-size pages, and Cloudflare says the pages “are not currently zeroed between requests; the next request that gets a page simply overwrites it.” That leaves the engine responsible for tracking which request owns which page.

From those times, by our arithmetic, the model was exposed for 13 days, went private 14 hours 49 minutes after the report arrived (38 minutes after the incident was declared) and stayed offline for 5 hours 40 minutes.

How a smaller cache became a leak

Cloudflare describes two overlapping problems. The first was a configuration mismatch: “We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens.” The pool was 65 percent of the window (our arithmetic), so a prompt in the top 35 percent of the window, from 681,281 to 1,048,576 tokens, fit the advertised limit but not the memory. Cloudflare says it saw the mismatch coming: “We were aware of this mismatch before launch. We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue.” The gap, it says, was in what the tests checked: “we only validated HTTP response codes and engine performance at long contexts, without validating model output and coherence.”

The second was an SGLang bug in how disaggregated mode handles a prompt that does not fit. As Cloudflare tells it, the scheduler marks such a prompt as rejected and swaps in a one-token placeholder, but the admission queues on both the prefill and decode sides checked whether a request was finished, not whether it was rejected, so the rejected request went through the handoff like any other:

1. The router told the decode node the prompt length, and decode reserved pages for a full-length prompt.
2. Prefill admitted the rejected request, ran a forward pass on the one-token placeholder and sent one page of KV to decode.
3. Decode received a “transfer complete” message and started generating while attending over every page it had reserved, of which only the first had been written by this request.
4. The remaining pages still held what earlier requests had left in them, “other users’ prompts and partial generations.”
5. The client got “HTTP 200 instead of HTTP 400,” and the answer came back faster than a prefill of that size should take.

On a single-node deployment the same rejection worked and the client received an error, which is what Cloudflare says it expected. SGLang’s account agrees that the single-node path “returned an error instead.”

Cloudflare’s first fix was configuration: the prefill pool now holds about 2.04 million tokens, roughly twice the context length (about 1.95 times, our arithmetic), so “a prompt that fits the context window now always fits the pool” and the rejection path is never reached for a valid request. It also patched the engine so that a request carrying a pending rejection is retired at the admission queue “with its original error, before any bootstrap, allocation, or KV transfer happens.” Cloudflare says GLM 5.3 was the only model affected, because every other disaggregated deployment on Workers AI has KV pools larger than its context length.

What the researcher saw

Knostic’s report says its CEO, Gadi Evron, ran into the problem on an unrelated task through OpenRouter, where the output included “paths for a machine other than ours, Chinese language tutorials, and some other suspicious output.” Knostic traced the provider to Cloudflare on September 7, switched to its own Workers AI account and says it directly observed leaks in several categories: personal names, client and company names, invoice numbers, euro amounts, payment dates, source-code information, file-system and working-directory paths, and agent task records. It describes the attacker as needing only ordinary access: “The attacker model required an ordinary paid Workers AI account, the attacker’s own API token, network access, and no victim credentials, privileged position, user interaction, or unusual configuration.”

That is an abridged copy of Knostic’s table: each prompt carried a unique marker at both ends and the model was asked to repeat it. Latency rose with prompt size up to the healthy 662,162-token request and then collapsed, and Cloudflare’s own account notes the same tell, that the answer “arrived faster than an 8-page prefill should take.” Knostic’s healthy request at 662,162 tokens and its lowest affected one at 700,166 bracket Cloudflare’s 681,280-token pool, which is our observation rather than Knostic’s claim. To rule out a hallucination, Knostic says the returned material included an absolute Linux path although every request in the investigation was issued from macOS, and that one affected response, a shell tool call, named a working directory and field names that matched a public repository whose own files described its agent stack as GLM-5.3 through Cloudflare and which received a signature-verified commit less than four minutes after the request. Knostic’s post also includes a reproduction script for what was then the vulnerable endpoint; we did not run it.

The SGLang bug report that was already public

Cloudflare writes that the bug “had been reported to SGLang by another user as issue #30233 on July 6, 2026, which Cloudflare noticed while investigating this incident,” and that it “had been flagging these types of issues with SGLang since v0.5.10.” We read the issue. Titled “[Bug] PD disaggregation: aborted prefill requests (input too long) cause decode to generate garbage from incomplete KV cache,” it lists the chain Cloudflare later described: input validation fails, the abort replaces the prompt with a one-token placeholder, the aborted request still enters the prefill queue, one token of KV is sent, and the decode side, which allocated space for the full prompt, starts from “uninitialized KV cache slots.” The reporter’s numbers show the same mismatch at a smaller scale, a 284,928-token pool and a 284,937-token prompt, and the issue includes a proposed patch that has the prefill side notify decode of the abort.

The thread has one comment, from a GitHub Actions bot, which closed the issue on September 5 with the message “This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.” The issue has no labels. Its text describes garbage output and never mentions another user’s data, leakage, tenants or security (we searched it), so the security impact was not stated there; Cloudflare’s account is what connects the same mechanism to a cross-customer leak, given that pages are not zeroed between requests.

The fix that closes the path arrived by another route. PR #36988, merged on September 6 at 02:15 UTC, was written for a VLM encoder or embedding-validation failure that aborts a disaggregated prefill request after memory has been allocated, and its description says it is “scoped to the failure path exposed by malformed VLM encoder results.” Cloudflare’s table says the same change “stops the exposure in this incident.” GitHub’s compare endpoint shows that v0.5.20, published September 18, contains the merge commit and that v0.5.19, published September 5, does not. PR #38935, which rejects such requests at admission, calls itself “a waste-and-latency fix, not a correctness fix,” and Cloudflare’s own draft, #38704, which Cloudflare says #38935 follows, was closed without merging on October 10.

In calendar terms, by our arithmetic, the issue was public 52 days before GLM 5.3 launched on Workers AI, and the first tagged release containing the fix came 22 days after launch and nine days after Cloudflare’s own fix.

SGLang says the wider race is older and broader

SGLang’s disclosure separates the incident from a family of cache-lifetime bugs in prefill/decode serving. In the family it describes, an aborted request can still have RDMA writes in flight, because RDMA work already posted cannot be cancelled; if decode frees the destination slots and the allocator reassigns them before the writes land, the late writes land inside another request’s cache. SGLang says: “Within PD deployments, default configurations were affected from the v0.4.x line (early 2025), when PD disaggregation shipped with RDMA backends, until v0.5.21, or v0.5.22 for MORI.” It adds that in typical workloads the corruption was rare, though long contexts make it far more likely, and that each occurrence was silent: “A deployer has no reliable way to tell, because the corruption leaves no error or log line.”

SGLang rates the realistic impact as “one-way cross-request leakage”: fragments of the aborted request’s context can surface in another client’s output, and the client cannot choose the victim. It says it has “not backported this fix line to earlier release branches.”

The two posts name slightly different upgrade targets. Cloudflare writes: “We highly recommend all operators update SGLang dependencies to v0.5.22 or later.” SGLang says v0.5.21 “protects Mooncake and NIXL deployments by default, for every failure kind,” that MORI deployments need a build at or past v0.5.22, and that a further fix for failed NIXL and MORI sends (#45973) ships in v0.5.23. GitHub’s release list showed v0.5.22, published October 19, as the newest tag when we checked.

Two longer-term changes are visible. SGLang has an open draft pull request, #46435, opened October 25, for an opt-in flag, off by default, that zeroes KV pages when they are freed, so that a bug reading a page it does not own would read zeros. Cloudflare says it is working on page-level integrity tags, a prototype that “cost under 1% in throughput and tail latency” in its tests. It says it tested the prototype on GLM 5.2 in mid-August but “disabled it shortly afterwards due to reliability issues.” The sources do not say whether the prototype was running when GLM 5.3 launched on August 27.

Where the accounts differ or go quiet

What operators can check

Both vendors’ posts include guidance for people who run SGLang themselves. We tested none of it, because this machine has no multi-GPU prefill/decode setup. What they say:

1. Compare the pool with the context window. Cloudflare says to measure the KV pool and compare it with the context length on every prefill and decode node, from the engine’s startup log, after every change to weights, quantization, parallelism or memory fraction, and to lower the context length if the pool is smaller. It adds: “Do not rely on a proxy to enforce the limit, and do not assume the engine will reject what does not fit.”
2. Upgrade. Cloudflare says v0.5.22 or later; SGLang’s per-backend guidance is above, and SGLang says it has not backported the fix line to earlier branches.
3. Use the new protections and watch them. SGLang recommends a router that cancels both halves of a prefill/decode pair, deferred KV release, and watching sglang:decode_deferred_kv_release_total{outcome} after rollout. Cloudflare points to the --disaggregation-enable-kv-checksum and --kv-canary options, and says they do not yet cover every model architecture; it names GLM 5.3, GLM 5.3 Flash and DeepSeek V4 as ones it is working to add.
4. Test the top of the context window and read the answers. Cloudflare’s gap was checking status codes and speed but not output. Knostic’s method, on its own account, put a unique marker at both ends of a long filler prompt and asked the model to repeat it; an answer without the marker, or one that returns far faster than a shorter prompt did, was Knostic’s sign that the submitted prompt was not becoming the state used for generation. Run anything like this only against deployments you operate.

What is still open

• How many customers’ data appeared in the 155 responses, and whether they were told. SGLang’s own post leaves this to Cloudflare’s account: “Cloudflare’s report covers its production incident and customer impact.” Cloudflare’s report gives the 155 figure and no victim count.
• Whether other hosts running SGLang in disaggregated mode, or self-managed clusters, had a pool smaller than their context window. Cloudflare’s post covers Workers AI only, and SGLang says a deployer cannot tell from logs whether its own deployment was exposed to the broader race.
• What the October 9 request to the researcher covered, given that SGLang dates its notification to September 9.

This is another Cloudflare tenant-isolation disclosure we have covered, after the Containers bug that let customers recover other tenants’ leftover disk data; in both, one customer could end up reading data that a previous customer’s work had left behind. For the architecture behind the prefill and decode pools, see our explainer on prefill/decode disaggregation, and for how quickly KV cache memory grows with context length, see our look at Reflection’s Beam serving guides.

We fetched the posts, GitHub records, status page, model page and NVD results between about 10:07 and 10:20 UTC on October 27. We did not run the researcher’s request or any request against Workers AI, did not run SGLang, and contacted none of the organizations. We left out Knostic’s reproduction script and filler-prompt recipe. Numbers marked as our arithmetic are computed from the dates and token counts the sources give; everything else is attributed to the source named.