Level 8Lesson 66⏱️ 40 min

The Attention Budget

Why a bigger context window isn't a free lunch - the model's focus is finite, and more text can make it dumber, not smarter

The Surprising Core Truth

Here's the fact that makes context engineering necessary: giving a model more text can make its answers worse. It feels backwards - more information should mean more to work with. But the model's ability to focus is limited, and every extra token spends a little of that focus.

Anthropic calls it an "attention budget." Like a person with limited working memory, the model has a finite pool of focus. Every token you add draws it down. So the goal of context engineering is not "include everything" - it's "find the smallest set of high-signal tokens that gets the job done."

Context Rot: More Tokens, Worse Recall

Researchers tested models by hiding a single fact (a "needle") inside larger and larger piles of text (the "haystack") and asking the model to find it. The result is consistent across every model: as the pile grows, the model gets worse at finding the needle. This is called context rot.

Recall accuracy as context fills up (the shape, not exact numbers):

  small context   ████████████████████  very reliable
  medium context  ███████████████░░░░░  slipping
  huge context    █████████░░░░░░░░░░░░  unreliable

It's a gradual slope, not a cliff - the model stays capable,
but it gets less precise the more you cram in.
Why this happens (one sentence): the model's architecture compares every token to every other token, so doubling the text roughly quadruples the relationships it has to juggle. Focus gets spread thin.

Lost in the Middle

There's a second, sneakier effect. Models don't pay equal attention to all parts of the context. They reliably notice what's at the beginning and the end, and they tend to miss things buried in the middle. This famous finding is literally called "lost in the middle."

Where the model actually pays attention:

  START   ███████████  strong   ← put your key facts here
  MIDDLE  ███░░░░░░░░░  weak     ← stuff gets lost here
  END     ███████████  strong   ← ...or here

So the ORDER you assemble context in matters as much as
what's in it. Bury the crucial fact in the middle of a
30,000-token wall and the model may never "see" it.
Practical rule: put the highest-signal material at the top or the bottom of the context - never in the soft middle of a giant block of retrieved text.

The Real Cost of "Just Dump Everything"

The instinct when an agent fails is to give it more context, just to be safe. It feels responsible. It usually backfires three ways:

Worse answers

Context rot and lost-in-the-middle kick in. Teams routinely find an agent does worse with a 100,000-token codebase dump than with a 5,000-token targeted slice.

Higher cost

You pay per token. A bloated context is a bloated bill, on every single turn of a long-running loop.

Slower responses

More tokens means more compute per step, so latency climbs. A chatty agent that re-reads a huge context every turn crawls.

"But Context Windows Keep Getting Bigger!"

True - models now hold hundreds of thousands or even millions of tokens. Doesn't this problem just go away? No. A bigger desk doesn't fix bad desk management:

Why bigger windows don't save you:
  • Context rot and lost-in-the-middle still apply - even at 2 million tokens, you still want the model to see just what's useful.
  • Cost and latency grow with size, so a giant context is expensive and slow even when it works.
  • The discipline is permanent: as long as windows are finite (they always are), someone must decide what enters. That someone is the context engineer.

The Mindset This Sets Up

Everything in the next five lessons flows from this one idea. Each pillar - instructions, retrieval, memory, tools - is really an answer to the question: how do I spend my limited attention budget wisely?

Carry this phrase through the whole level: the smallest set of high-signal tokens. Not the most context. The best context. Every technique you'll learn is a way to add signal or remove noise.

Hands-On: Feel the Rot

Hands-on (15 min): Take a long document (a few thousand words). Paste a specific, unusual fact into the middle of it - e.g. "the project codename is Blue Otter." Give the whole thing to an AI and ask a few unrelated questions, then near the end ask "what is the project codename?" Try it again with the fact at the very top instead. Notice whether placement changes how reliably it answers. You've just demonstrated lost-in-the-middle with your own hands - and felt why order is part of the job.
Lesson 66 Quick Reference
Attention budget

The model's focus is finite; every token spends some of it, so more context can mean worse answers

Context rot

As the context window fills, the model's ability to recall any specific fact gradually drops

Lost in the middle

Models attend to the start and end of context far better than the middle - so order matters

Smallest high-signal set

The goal: the fewest tokens that maximize the chance of the right outcome

Dumping backfires

Too much context lowers quality, raises cost, and adds latency - all at once

Big windows don't save you

Rot, lost-in-the-middle, cost, and latency persist even at millions of tokens

← From Wording to Wiring
Unlocks in ~10 min of reading