FlashStates: Unlocking Flexible Hybrid State Management for Agentic LLM Inference
FlashStates decouples recurrent checkpoint positions from KV-cache block boundaries, achieving fine-grained state reuse without dense checkpoint storage.
FlashStates vs. vLLM
In a hybrid model, attention keeps KV for every token, but each recurrent layer keeps one state that it overwrites as it reads. Resuming at a position needs the KV and a checkpoint taken exactly there; we call the pair a hybrid state.
Each agent turn re-sends its whole context. Ideally, the engine computes only the tokens added since the previous turn ended. Any earlier token it computes again is redundant prefill, caused by one of three misses.
Below 40 sessions, placement and dependency misses cause most of the redundant prefill in vLLM and SGLang. Capacity misses grow only from about 40 sessions.
Concurrent sessions
SGLang checkpoints every 256 tokens, so it has few placement misses (b). At 32 sessions, however, it keeps 4.2× as many checkpoints per request as vLLM. They compete for a fixed-size state pool, which causes its dependency misses (c).
FlashStates saves recurrent checkpoints exactly where turns resume, and keeps or evicts each checkpoint together with the KV it depends on.
The next turn resumes where the previous request ended, usually inside a KV block. ASC saves the recurrent checkpoint exactly there: at request end it captures the hybrid state before cleanup, and if the position lies inside a prefill chunk, the kernel exports the checkpoint as a side output while the chunk continues. The KV block size and the chunk size do not change, and outputs stay bit-identical.
A recurrent checkpoint is useless without the KV before it, and that KV cannot resume a request without the checkpoint. FlashStates therefore keeps or evicts each hybrid state as one unit: its checkpoint and the KV only it needs. KV pages shared with other hybrid states are freed only when the last state that needs them leaves.
Each hybrid state is priced by the prefill time it saves, measured once per model, per block of memory its eviction would free. Nothing is tuned per workload. Admission uses the same price: when memory is short, a new checkpoint is cached only if it is worth more than the cheapest one already cached. States that no request has resumed from for a while fall behind, so the newest checkpoint of each session, where its next turn resumes, is typically kept longest.
We serve GLM-5.3-Flash, Qwen3.8-Flash-Next and Nemotron 3 Super using two B200 GPUs per engine. Evaluation on the AgentX coding-agent workload shows that FlashStates achieves up to 3.02× vLLM's decode throughput and 3.14× its prefill throughput, while cutting p90 time to first token (TTFT) by up to 88%.
GLM-5.3-Flash NVFP4, 1M context
Concurrent sessions
With 512 GiB of host memory for cache and MTP enabled on GLM-5.3-Flash, FlashStates expands the Pareto frontier of the throughput–latency tradeoff, improves peak input throughput by 26%, and cuts p90 TTFT by up to 73% at the same concurrency.
@article{flashstates2026,
title = {FlashStates: Unlocking Flexible Hybrid State Management
for Agentic LLM Inference},
author = {Li, Chenxi and Xu, Weili and Zhang, Tianyi and Wang, Jue and
Zhou, Zhongzhu and Xu, Chenfeng},
journal = {alphaXiv preprint},
year = {2026},
url = {https://www.alphaxiv.org/abs/2611.flashstate}
}