You pay for every token you send a model, and the bill scales fast. A single legal review, a full customer history, a codebase dump: feed a few hundred thousand tokens into a modern LLM and the cost per query starts looking less like an API call and more like a small consulting engagement.
The pitch from vendors is that the huge context window solves your retrieval problem. Stop chunking, stop indexing, throw the whole corpus in and let the model handle it. It sounds clean. In production, it rarely holds up.
The bottleneck moved. Models used to choke on the document; now they swallow it whole, and the harder question is whether they can find the sentence inside it that matters. Understanding how attention mechanisms in AI models evolved helps explain why the naive answer keeps disappointing teams that try it.
The Window Grew Faster Than Anyone Planned For
A few years ago, engineering teams designing around a language model treated context length as the binding constraint. Every prompt was a compression exercise. Summaries of summaries, retrieval pipelines built specifically to keep the working set under a few thousand tokens, elaborate chunking heuristics: all of it existed because the model couldn't read very much at once.
That constraint has effectively lifted. Recent benchmarking shows the longest context windows have grown by roughly 30x per year since mid-2023, and the input length at which top models still reach 80% accuracy on long-context tasks has jumped by more than 250x in just nine months.
A window that once held a couple of pages now holds a small library. The engineering effort that went into squeezing prompts looks, in hindsight, like a workaround for a problem the model itself was about to outgrow.
So the natural conclusion, and the one plenty of teams jumped to, was that retrieval pipelines were obsolete. Skip the vector database, skip the reranker, load the whole thing and let the model sort it out.
The Obvious Fix Fails in the Middle
The obvious fix fails for a specific, well-documented reason. When you hand a model an enormous prompt, it does not read it evenly. Information at the very beginning and the very end gets weighted heavily. Information in the middle gets skimmed.
This is the finding from the "Lost in the Middle" work, and it has held up remarkably well across model generations. Performance is highest when the relevant passage sits near the top or bottom of the context. Bury it in the middle of a long input and accuracy falls off a cliff, even when the model is technically capable of processing the full length.
The practical consequences are ugly. A support agent built on a million-token window can miss the one clause in a contract that answers the customer's question. A code assistant fed an entire repository can confidently cite the wrong function.
The model isn't hallucinating in the classic sense. It's ignoring, or under-attending to, parts of the input it was supposed to consider. Because the output still sounds fluent, the failure is hard to catch without ground-truth testing.
The cost side compounds the problem. Every token in that window gets processed. If your users are paying per query, or your infrastructure is running the inference, dumping raw context at the model is often the most expensive way to get the least reliable answer.
What Actually Changed Inside the Model
The real progress wasn't the number on the spec sheet. It was the machinery underneath: attention getting better at deciding what to look at.
The shift has been from dense self-attention, where every token compares itself to every other token, toward sparser, hardware-aware, and hierarchically structured variants. The short version: attention learned to focus. The longer version is worth understanding, because it explains why some long-context deployments work and others don't.
None of this makes the naive "stuff everything in" pattern smart. It makes the better patterns cheaper.
Design for Focus, Not for Length
The practical takeaway for teams building on top of these models is unglamorous: design for focus, not for length. Retrieval pipelines are not obsolete. They matter more, because they decide what the model gets to attend to and where in the prompt it lands.
The Window Is a Feature, Not a Strategy
The million-token window is a real capability, and it opens doors that were closed two years ago: long-form document analysis, multi-hour transcripts, whole-repo reasoning. It doesn't replace the discipline of giving a model exactly what it needs, in the order it reads best.
Attention learned to focus. The systems built on top of it still have to do the same.
