Run an LLM for long enough and the KV cache becomes the problem. A long chat keeps adding tokens. So does an agent loop, or a service that watches some stream all day. If the cache grows with the stream, so does memory, and so does the cost of attention.
The obvious fix is a sliding window. Keep the last W tokens in the cache and throw away anything older. With trained decoder-only models this breaks almost right away, as soon as the window slides past the first few positions. Perplexity can jump by 100x within a few hundred tokens. And that happens even when the tokens you dropped were just the BOS token, some boilerplate, or a bit of punctuation.
The cause is the way softmax hands out attention. Because of it, those early tokens can matter a lot even when what they say doesn’t matter at all. StreamingLLM’s answer is to keep a few early tokens in the cache and let the rest of the window slide as usual. In the paper, reserving four tokens was enough to run sliding-window inference out to four million tokens with no collapse.
Table of contents
Open Table of contents
Where the term comes from
The name comes from Xiao et al. (2023), Efficient Streaming Language Models with Attention Sinks, which was published at ICLR 2024.
The authors wanted sliding-window attention to work on streams with no end. It kept failing. Perplexity blew up right at the point where tokens started getting evicted, and it didn’t seem to care what the surrounding text was.
So they looked inside trained decoder-only transformers: Llama, Mistral, Falcon and Pythia. Every family showed the same thing. Later tokens sent a big share of their attention to the first one to four positions in the sequence, and they did it even when those early tokens meant basically nothing. Since it turned up in every model family they checked, you couldn’t write it off as something odd about Llama. The real driver was the constraint built into softmax attention.
What an attention sink looks like
Pick a layer somewhere in the middle of a trained transformer and plot its attention map. You will often see bright columns over the first few key positions. One axis is the query tokens and the other is the key tokens, and those first few columns keep pulling weight from queries much further along.
The brightness tells you how much attention later queries give to the early keys. It shows up across heads and across layers, and you don’t need a special prompt to see it. Often the first one to four tokens soak up 30 to 80 percent of all the attention a query hands out.
Most of the time those early tokens carry very little. The BOS token at the start has no meaning of its own. The first few words of a system prompt are usually boilerplate. A stray period at position 2 isn’t holding the meaning of anything. Later tokens attend to them anyway.
No part of the architecture asks for this, and neither does the training objective. The model picks it up during training. It picks it up reliably, too, so the first few positions end up special in nearly every trained decoder you look at.
That is why eviction hurts. The model has learned to send attention through those early keys. If you drop them from the KV cache you change the computation, even though their actual text was never important.
Why softmax produces sinks
Softmax makes every attention head spend exactly one unit of attention. For a given query, the head scores every key and then passes the scores through softmax to get a probability distribution. The weights can’t be negative and they have to add up to exactly one:
Since the weights have to sum to one, a head has no way to say “there’s nothing worth looking up for this query, so I’ll just attend to nothing.” Any attention it doesn’t give to one key has to land on another.
Think about a head that has nothing useful to fetch for some query. If it spreads its attention evenly, it ends up pulling in a little bit of every value vector, more or less an average of the whole cache. That average rarely helps. The head would do better to dump the leftover attention somewhere that barely touches the residual stream.
Trained models seem to use the first few positions as that dumping ground. Every later token can see them, and they always sit in the same place. Over the course of training, the model can learn that piling extra attention there does less damage than smearing it over the whole sequence.
So given that constraint, a model can learn to park unwanted attention on a few fixed sink positions. Take the constraint away and the sink never forms.
Evan Miller’s softmax+1 proposal, from his post Attention Is Off By One, tweaks the denominator by adding an extra +1. That extra term works like a fake null position that can soak up attention nobody wants. Now attention can add up to anything from zero to one, and the spare mass goes into the null slot rather than being pushed onto real tokens.
Vision transformers have something similar. Darcet et al. (2023) found that ViTs produce outlier features in patch tokens that carry little information. Their fix was to add explicit “register” tokens whose job is to absorb that mass on purpose.
Keep the sinks permanently
StreamingLLM keeps the sink positions around instead of throwing them out. You pin the first K tokens of the sequence in the KV cache for good, and the eviction window slides over everything after them. When the cache is full, you drop the next-oldest token, not the very oldest one. The early tokens never move.
The paper usually sets K to 4. That costs four extra KV cache entries and a tiny bit more attention compute, and you still get the memory cost of a sliding window.
The point is to keep the computation close to what the model is used to from normal inference. If the model sends attention through positions 0 to 3, those positions are still there while the rest of the cache moves along. With no sinks, the window sooner or later evicts them, and the attention pattern the model learned has nowhere stable left to go.
You don’t have to use the original tokens as sinks. The paper shows you can add a few learnable “sink tokens” to the front of the sequence during pretraining or fine-tuning, and the model will send its attention there instead. That makes the sinks explicit. It also needs training, so in practice the easier thing to deploy is to just reserve the first K real tokens.
Does it work?
According to the paper, perplexity stays stable all the way to four million tokens. Using only a sliding window on Llama-2-7B, with a window of about 1024, perplexity stays near normal until a little past the original context length. A few thousand tokens later it has shot up into the thousands. At that point the model is basically broken at the eviction boundary, and Llama, Mistral, Falcon and Pythia all show the same behaviour.
Reserve 4 sink tokens and perplexity stays in the normal range for the whole sweep.
Dense attention, the baseline, also stays low until you go past the context window the model was trained on. After that it climbs, because the model is now extrapolating its positional encodings. Dense attention gives you the best quality, but for a stream with no end it costs too much. Sliding window plus sinks gets you most of the way to dense quality while paying sliding-window prices for memory and compute.
The paper tests this on several model families, so it really isn’t something only Llama does. Trained decoders grow sinks, and any sliding-window scheme that throws those positions away will fall over.
Where this shows up
If you run a sliding-window KV cache, either keep the sink positions or train explicit sink tokens. If you don’t, your cache policy can break a computation the model spent its whole training learning.
Registers in vision transformers are a similar idea in a different setting. They give the model low-information slots on purpose, so it doesn’t have to borrow real image patches for the job.
Evan Miller’s softmax+1 goes after the architecture itself. If attention already has a null slot, there’s no reason for a sink to form in the first place. But pretrained checkpoints you already have won’t get the benefit unless you retrain them.
Some interpretability work describes sinks as a no-op, or ignore, attention pattern. On that reading, some heads look at positions 0-3 whenever they have nothing useful to pull in for the current token.
Further reading
- Xiao, Tian, Chen, Han. Efficient Streaming Language Models with Attention Sinks. ICLR 2024. arXiv:2309.17453
- Darcet, Oquab, Mairal, Bojanowski. Vision Transformers Need Registers. ICLR 2024. arXiv:2309.16588
- Miller, E. Attention Is Off By One. evanmiller.org/attention-is-off-by-one.html
- StreamingLLM reference implementation: github.com/mit-han-lab/streaming-llm