Level 8Lesson 68⏱️ 45 min

Pillar 2 - Retrieval

How outside facts get onto the desk: the difference between handing the model everything up front and letting it grab what it needs, just in time

The Question This Pillar Answers

A model only knows two things: what it learned during training, and what you put in its context window. For anything specific to your world - your files, your data, today's ticket - someone has to fetch it and put it on the desk. That fetching is retrieval, and doing it well is the single biggest source of correct answers (and doing it badly is the single biggest source of made-up ones).

Back to the analogy: retrieval is reaching into the backpack and putting the right item on the desk. The whole skill is grabbing the right thing - not dumping the entire bag out (that's the attention-budget disaster from Lesson 66).

Two Styles: Up Front vs. Just in Time

Up-front retrieval (classic RAG)

Before the model starts, you search a library of documents, grab the most relevant chunks, and load them into context. (RAG = "retrieval-augmented generation.") Fast, because the data's already there - but you have to guess what's relevant before the model has even thought about the problem.

Just-in-time retrieval (the agentic way)

Instead of pre-loading everything, you give the model lightweight pointers - file paths, search commands, links - and let it pull the actual content when it decides it needs it. Slower per step, but the model fetches based on what it's actually discovering.

UP-FRONT (RAG)
  search docs → grab top chunks → stuff into context → start
  good when: you know what's relevant ahead of time

JUST-IN-TIME
  give the model: file paths, a search tool, a database query tool
  model decides: "I need to see auth/middleware.ts" → reads it now
  good when: the relevant info depends on what the model finds

Why Just-in-Time Mirrors How You Work

You don't memorize your whole filing cabinet before starting a task. You keep an index - folders, file names, bookmarks - and pull the specific thing when you need it. Just-in-time retrieval gives the model the same superpower.

A neat side effect - "progressive disclosure": the pointers themselves carry clues. A file named test_utils.py in a /tests folder tells the model something before it even opens the file. Folder names, file sizes, and timestamps all help the model decide what's worth grabbing. It discovers the right context layer by layer, keeping its desk clear.

The Hybrid That Usually Wins

In practice the best agents mix both. Anthropic's own Claude Code is the textbook example:

Claude Code's hybrid retrieval:

  UP FRONT      drop CLAUDE.md into context at the start
                (the standing project rules - always relevant)

  JUST IN TIME  give it grep + glob + file-read tools
                (let it hunt down the exact files it needs,
                 fresh, instead of relying on a stale pre-built index)

The always-relevant stuff (project conventions) goes in up front. The depends-on-the-task stuff (which files?) is fetched just in time. Best of both: no stale index, no bloated desk.

Bad Retrieval Is the #1 Source of Wrong Answers

When an agent confidently states something false, retrieval is usually the culprit. Three classic ways it goes wrong:

Retrieval failure modes:

Too much: a search returns 4,000 hits, you dump them all in, and the real answer drowns (context overload).
Stale: your pre-built index is months old, so the model reads a deprecated function and confidently calls it. Freshness matters.
Off-target: the search grabbed plausible-looking but irrelevant chunks, and the model reasons over the wrong facts.

The cure is almost never "less retrieval" - it's better-targetedretrieval, which we'll make concrete in Lesson 71 with re-ranking.

For Coding Agents: Structure Beats Guessing

A quick real-world note (you don't need to build this, just understand it). For code, plain text search is weak - searching for a function name returns every file that mentionsit. "Code-intelligence" tools instead understand structure: they can return the one place a function is defined plus its exact call sites. In published benchmarks, giving an agent structure-aware retrieval turned tasks that timed out after two hoursinto ones that finished in under two minutes. That gap is entirely context engineering - same model, better retrieval.

Hands-On: Up Front vs. Just in Time

Hands-on (15 min): Pick a question that needs a specific document you have (a policy, a README, a spec). Try it two ways with an AI assistant. First, paste the entire document and ask. Second, paste only a table of contents or file listand say "tell me which section you'd need, then I'll give you just that." Compare the answers and the effort. You'll feel the trade-off between pre-loading everything and letting the model ask for exactly what it needs.

Lesson 68 Quick Reference

Retrieval

Fetching outside facts (files, data, search results) into the context window so the model can use them

RAG (up-front retrieval)

Search a document library and load the top relevant chunks before the model starts

Just-in-time retrieval

Give the model pointers (paths, queries) and let it pull the actual content only when it needs it

Progressive disclosure

File names, folders, sizes, timestamps act as clues so the model discovers the right context layer by layer

Hybrid

Load always-relevant info up front (CLAUDE.md), fetch task-specific info just in time (grep/file reads)

Retrieval failures

Too much, stale, or off-target retrieval is the #1 source of wrong/hallucinated answers

← Pillar 1 - Instructions

Unlocks in ~12 min of reading

Pillar 3 - Memory →