When Loops Go Wrong
The failure modes nobody warns you about: loops that lie about being done, understanding that quietly rots, and the security holes of leaving an agent running
The Dangerous Failures Are the Quiet Ones
A loop that crashes loudly is easy - you notice, you fix it. The failures that actually hurt are silent: the loop keeps running, keeps reporting success, and quietly produces garbage (or a security hole) while draining your budget. This lesson is the warning label. Read it before you ever schedule a loop unattended.
The "Ralph Wiggum" Loop
Engineer Geoffrey Huntley named this failure after the cartoon character famously oblivious to his own mistakes. A Ralph Wiggum loop is one where the agent declares "done!" before the job is actually finished - it emits a completion signal too early, the loop exits on a half-done task, and everything looks fine.
This connects straight back to Anthropic's own finding on SWE-bench: because the model couldn't see the hidden grading tests, it often "thought" it had succeeded when it had actually failed. An agent's confidence is not proof.
- You have no real verifier - you just ask a second agent to "review" in conversation instead of running an actual test suite. (That's two optimists agreeing.)
- "Done" is the agent's opinion, not an objective build/test/compile result.
- There's no hard cap, so it runs until a rate limit or a scary bill cuts it off.
Two Cousins: Goal Drift and Agentic Laziness
In a long run, the agent keeps summarizing its own history to save space, and each summary loses a little detail. Your explicit rules - "never touch the payments folder" - can quietly evaporate by turn 47. Fix: force the loop to re-read a base spec file on every iteration so the rules never fade.
The loop decides a task is "good enough" when it's only partly done. Fix: a hard /goal condition graded by a separate checker model, so "good enough" isn't the agent's call to make.
Comprehension Debt: The Bill That Really Hurts
This failure isn't technical - it's human, and it gets worse as your loops get better. Comprehension debt is the growing gap between what's in your codebase and what your team actually understands. The faster a loop ships code nobody read, the wider that gap grows.
How to keep the debt from compounding:
• Read the diffs. Every automated pull request gets human eyes on the actual changes - not just a glance at the title. • Audit your gates. Occasionally break the code ON PURPOSE to confirm your tests actually catch it. Gates rot silently. • Restrict scope. Keep loops on small, isolated, checkable changes. Never let a loop redesign your architecture.
Cognitive Surrender: The Comfortable Trap
The subtler human risk: when the loop runs itself smoothly, it's tempting to stop forming your own opinion and just accept whatever it produces. Addy Osmani calls this cognitive surrender. The exact same loop can make one person faster at work they deeply understand, and let another person avoid understanding the work at all. The loop can't tell the difference. You can.
The Security Tax
An unattended loop with access to your code and tools is a live, always-on attack surface. The more autonomous and powerful the loop, the more you must defend it. Four vectors to guard:
1. UNREVIEWED CODE - a loop ships faster than humans review, so vulnerabilities slip in. Gate every merge with automatic security scanning, dependency checks, and secret detection. 2. POISONED SKILLS - a Skill pulled from the internet can hide malicious instructions. Audit any external skill before use. 3. LEAKED SECRETS - chatty debug logs on a long run can print passwords, tokens, keys. Turn off verbose logging in production and scrub what gets recorded. 4. PERMISSION CREEP - a loop granted "read-only" today often gets "write" tomorrow for convenience. Audit its access every 30 days and claw back anything it doesn't strictly need.
The Whole Level in One Checklist
- Did it pass the 4-condition test? (Lesson 57)
- Is there an objective gate - a real test/build - not just an LLM "review"? (Lessons 57, 61)
- Are the maker and checker separate agents? (Lesson 61)
- Does it write progress to a state file? (Lesson 62)
- Is there a hard, unbypassable cap on tokens/time/iterations? (Lessons 57, 64)
- Is it blocked from architecture, auth, and payments code? (Lessons 57, 64)
- Are you actually reading every diff before merge? (Lesson 64)
Hands-On: Red-Team Your Own Loop
The agent declares "done" before the job is finished; quiet failure with no real verifier or hard cap
Rules fade across long runs as the agent summarizes; fix by re-reading a base spec every iteration
The loop calls partial work "good enough"; fix with a hard /goal checked by a separate model
The growing gap between code shipped and what the team understands - the real bill, paid at outage time
Accepting loop output without forming your own opinion; the loop can't tell good use from lazy use
Guard unreviewed code, poisoned skills, leaked secrets, and permission creep on any unattended loop