Cross-request KV Cache Exposure in GLM-5.3 Workers AI Model and SGLang Remediation

Overview and Key Takeaways

On September 8, 2026, security researcher Gadi Evron from Knostic alerted Cloudflare to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting August 27, 2026. Cloudflare remediated the issue and restored the model on September 9, confirming no other Workers AI models were affected. Between August 27 and September 9, 155 responses were affected, all involving prompts longer than 681,280 tokens. Cloudflare and SGLang maintainers published coordinated disclosures detailing the bug, response timeline, and subsequent production hardening.

Incident Details and Root Cause

The exposure resulted from two overlapping issues: a configuration mismatch and an inference engine bug in SGLang. Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, but the prefill GPU KV cache pool only had room for 681,280 tokens. When a prompt exceeded this size, SGLang's scheduler internally rejected it and replaced it with a one-token placeholder instead of returning a client error. In disaggregated prefill/decode mode, the decode node received a transfer complete signal for the placeholder but left previous requests' KV state intact, causing decode to generate responses using stale cache pages containing other users' data.

Immediate Remediation Actions

Upon reproducing the issue via the HackerOne report within 25 minutes, Cloudflare set the GLM 5.3 model to private and took it offline. To correct the configuration, Cloudflare adjusted the GLM 5.3 setup to match GLM 5.2, expanding the prefill KV pool to hold approximately 2.04 million tokens—twice the context length—ensuring valid prompts always fit the pool. For the engine fix, Cloudflare patched SGLang so that requests carrying pending rejections are retired at the disadmission queue before any allocation or KV transfer occurs.

Upstream Coordination and SGLang Updates

Cloudflare met with SGLang maintainers at RadixArk on September 10, 2026, agreeing on upstream fixes and integrity-checking strategies. SGLang addressed the prompt-handling bug in v0.5.21, with further patches released up to v0.5.22. Upstream changes include retiring aborted prefill requests, early admission rejection (#38935), sgl-router cancellation updates, deferred KV release defaults, and bounded Mooncake transfer batches to prevent unconstrained in-flight writes.

Production Hardening and Verification

Cloudflare extended deferred release to cover prefill source pages across all failure types, updated load balancers so client disconnects do not abort transfers mid-way, and added strict pre-deployment checks to measure KV pool size against context length. All changes have been deployed to production disaggregated SGLang deployments, and operators running disaggregated SGLang are advised to update to v0.5.22 or later.