Carryover Effects in KV Caches Reused in Agentic Workflows and RL Training
What does a reused KV cache do to the model that reads it?
Abstract
A carryover effect occurs when a model reads a KV cache that another computation wrote: the text or weights differ from the writer's, and the reader's next-token distribution shifts as a result. Prior work shows that reusing a cache at new positions or across weight updates can change a model's outputs. We ask what predicts the size of this effect, and we measure it in two pre-registered records: agent handoffs and policy updates.
The oracle removal fraction is the smallest share of matched tokens, ranked by deviation from a fresh cache, as in CacheBlend's analysis, whose removal brings the remaining keys' mean deviation within a cross-model map's held-out error. At agent handoffs, a cache written by the reader's own model has a removal fraction of zero on the median handoff of every cohort. The reader's per-token KL divergence compares its predictions under the reused cache and a fresh one; PipelineRL and CacheReforge report related KL divergences for stale caches. On 35 long coding-agent handoffs, each with a same-model removal fraction of zero, the reader's mean per-token KL(fresh ‖ reused) has a median of 0.14 nats, with no threshold registered.
Across GRPO policy updates in three model families, with the cache written before the first update, the same KL, as a median over sequences, stays under a registered threshold of 1e-3 nats on two held-out texts at every lag read, but on GSM8K rollouts sampled by the initial policy it is about 2,000 to 150,000 times as large as on those texts at the same lag and above the threshold after one update. With one Qwen2.5-1.5B run's policies, the KL also stays under the threshold at every lag read on the opening 1,024 tokens of fifty MLPerf Agentic Inference trajectories.
The greedy answer flip rate is the share of test problems whose final answer changes under the reused cache, a wider count than the flips reported for compressed models. In that run, the stale cache changes the greedy final answer on 71% of GSM8K test prompts after one update and on 84% to 90% at 2 to 32 updates. A reused cache should therefore be checked on the reader's predictions on the kind of text it will serve, not only on its keys or on held-out text, since in both records those checks passed while the predictions moved.