ByteDance Seed Identifies Periodic Performance Fluctuation in DeepSeek-V4 Caused by Chunked KV Cache Compression

By Wenle, from Ao Fei Si QbitAI | Official Account: QbitAI

ByteDance's Seed team has caught DeepSeek in its "divine-demonic duality."

The same problem, nothing changed, just a few irrelevant characters prepended to the input, and suddenly the model can't solve it???

And it's not just occasional glitches.

Seed researchers discovered that DeepSeek-V4's performance actually fluctuates periodically every 4 tokens depending on the position of information within the input.

What does this mean? Whether the model remembers something depends on where that thing is positioned in the input??

Add two more tokens at the beginning, and the answer might flip from wrong to correct; add two more, and it flips back again.

Alright, alright, so models are now picky about their "positioning" when answering questions too.

In 128K long-context retrieval tests, researchers found that simply changing the position of the same piece of information could cause the retrieval accuracy of DeepSeek-V4 series models to vary by up to 40.2 percentage points.

The team further discovered that this issue is related to a long-context optimization technique adopted by DeepSeek-V4—

Chunked KV Cache Compression.

This technology was originally intended to make the model process long texts more memory-efficiently and effectively.

But after compression, it seems to have made the model act unpredictably (doge).

Two More Tokens, and DeepSeek Suddenly Gets It Right

ByteDance Seed researchers initially conducted an experiment using DeepSeek's own code.

The test subject was DeepSeek-V4-Flash-Base.

They extracted an FP8 quantization function from DeepSeek-V4's official inference code and asked the model to complete the last token.

The correct answer should be 8, as the code requires completing an FP8-related type conversion.

However, the model sometimes insisted it should be 32.

To understand what was going on, researchers added a purely decorative docstring before the code, containing some repeated equals signs.

Then, they started adjusting the number of equals signs.

The code itself didn't change, the completion position didn't change, and the correct answer certainly didn't change... The only variation was the addition of these meaningless tokens at the beginning.

As a result, DeepSeek's answers began to oscillate wildly.

When the padding length fell into certain positions, the model tended to output the incorrect answer, 32.

Shift one or two tokens forward, and it started leaning towards the correct answer, 8.

Keep shifting, and the wrong answer came back—

The entire process repeats with a period of 4 tokens.

More specifically, among the 16 padding lengths tested in the paper, when the length modulo 4 resulted in a remainder of 0 or 1, the model favored the incorrect answer 32; when the remainder was 2 or 3, it favored the correct answer 8.

Researchers also tracked the probabilities assigned by the model to the two candidate answers.

At one set of positions, the average probability for the incorrect answer 32 reached 71.3%, while the correct answer 8 had only 26.4%.

Switch to another set of positions, and the situation reversed completely:

The average probability for the correct answer 8 rose to 91.5%, leaving the incorrect answer 32 with just 7.2%.

In other words, changes in the length of a few irrelevant characters at the beginning were enough to cause the model to produce drastically different judgments on the same problem.

That's hard to evaluate.

When programmers debug code, they usually check logic, variables, and dependencies first.

Now, it seems they might also need to casually check if there are two extra equals signs at the beginning.

However, a single code completion case isn't enough to demonstrate how widespread the problem is.

So the Seed team expanded their testing.

They turned their attention to a classic task in large model long-context capabilities: Needle in a Haystack.

Researchers constructed a context up to 128K tokens long, containing approximately 16,000 key-value pairs.

For example, K1 corresponds to V1, K2 to V2...

Then they asked the model to find the Value corresponding to a specified Key.

During testing, the key-value relationships remained unchanged, the question stayed the same, and the total context length was kept consistent.

Researchers focused on adjusting the position of the target information relative to the boundaries of the compression window.

They found that the accuracy curve for the DeepSeek-V4 series exhibited very obvious periodic fluctuations.

Among them, the maximum accuracy gap between different positions for DeepSeek-V4-Flash-Base reached 40.2 percentage points.

DeepSeek-V4-Pro-Base also showed a gap of 34.8 percentage points.

After post-training, the situation improved somewhat.

The gap for DeepSeek-V4-Flash-0731 narrowed to 19.1 percentage points, and for DeepSeek-V4-Pro-0813, it reduced to 14.8 percentage points.

The newer DeepSeek-V4.1-Flash-0910 saw the gap further drop to 6.1 percentage points.

But the periodic differences still persisted.

Where does this variability come from?

Looking closer, the fluctuation period for DeepSeek-V4 is 4 tokens, while for DeepSeek-V4.1, it became 2 tokens.

Researchers discovered that this corresponds exactly to the KV cache compression strides used by each generation of the model.

Well, even the periodicity of the model's performance fluctuations aligns with the underlying compression configuration.

Is the Problem in KV Cache Compression?

Let's talk about KV Cache compression.

When large models process long contexts, they need to store a vast amount of Key and Value information corresponding to historical tokens for subsequent attention calculations.

The longer the context, the greater the memory and computational overhead occupied by this cache.

Especially for long tasks involving hundreds of thousands or millions of tokens, KV Cache easily becomes a bottleneck for inference efficiency.

Therefore, DeepSeek-V4 adopts Chunked KV Cache Compression.

The idea is to divide consecutive tokens into windows and then compress the information within each window into fewer cache entries.

This way, the model doesn't need to maintain caches of the same scale for every historical token.

It saves memory and reduces the cost of attention calculations for long contexts.

But the Seed team found that the reason behind DeepSeek's "divine-demonic duality" might lie hidden within this chunking process.

Assume that every 4 tokens constitute a compression stride.

In this case, if the same piece of information appears at position 1, 2, 3, or 4 within the window, the conditions during compression may differ.

The paper refers to this position relative to the compression window boundary as Phase.

Researchers discovered that the model exhibits systematic differences in retrieval capability for information at different phases.

They named this phenomenon Phase Sensitivity.

To illustrate, consider submitting the same document to the model:

In the first layout, key numbers happen to fall into positions that the model easily retains;

In the second layout, merely adding a few characters at the beginning shifts the position of the key numbers relative to the compression window.

The content remains identical, but whether the model can subsequently retrieve those numbers may vary significantly.

Moreover, this issue cannot be simply attributed to "information being split across two windows."

Researchers found that even when both Keys and Values fall within the same compression window, retrieval accuracy can vary greatly depending on their specific positions.

This indicates that the problem involves how the model writes information into the compressed cache and how it reads from that cache afterward.

To further verify this, the Seed team trained a batch of models from scratch.

Based on the Qwen3-0.6B architecture, they constructed various KV Cache compression schemes and used full-attention models without chunked compression as controls.

The focus was solely on changing the compression mechanism to see if periodic fluctuations would emerge accordingly.

The results showed that all tested chunked compression models exhibited periodic variations corresponding to the compression stride.

Conversely, the full-attention baseline models did not show comparable periodicity.

Researchers also adjusted the window size and compression stride separately, finding that the period primarily tracked with the compression stride.

With a stride of 4, performance fluctuated with a period of approximately 4 tokens;

With a stride of 6, the period shifted to approximately 6 tokens;

With a stride of 8, the same pattern held;

...

Even without using RoPE positional encoding, or by replacing learnable compression weights with simple averaging, this phenomenon persisted.

In other words, the issue is not caused by a specific positional encoding or a particular module alone.

The design of chunked compression itself may introduce periodic retrieval weaknesses.

The Seed team further intervened in attention heads, observing how retrieval capabilities across different phases changed after removing various components.

They found that different attention heads contributed differently to various phases.

Some heads were better suited for processing information at certain positions, while others played a larger role at other positions.

Researchers termed this Phase Specialization.

This suggests that an internal division of labor has formed within the model: different attention components develop preferences for different positions within the compression window.

While this specialization aids information retrieval, it may also render certain positions relatively weak points.

The paper also analyzed the training process through simplified theoretical models, discovering that gradient flow may drive compression modules to form stable positional preferences.

This explains why such periodicity is not merely random noise. It may be a natural outcome of the model learning how to compress information.

However, issues like these might not be directly visible in standard benchmarks, as conventional evaluations typically aggregate numerous test results into a single average score.

If the model performs well at some positions but lags significantly at others, the final calculated average score may still appear decent.

Therefore, the Seed team proposed that when evaluating models using chunked KV Cache compression, one should not only look at overall retrieval accuracy but also place the same information at different compression phases and test them individually.

Nevertheless, given the inherent memory and computational costs of long-context inference, compression retains significant practical value.

But after saving cache space, the model may no longer treat information at different positions equally.

Of course, based on these experimental results, post-training and architectural iterations can indeed significantly narrow this gap.

Paper link: https://arxiv.org/pdf/2609.36322