I Found 85 Problems in My Agent. Fable 5 Was Too Good for Me to Notice.
What happens when capability masks debt, and why the human read is the check nothing automates away.
I’ve been building Wiz, my AI agent, for about ten months. Fast movement guarantees slippage. Every time a tool got replaced, a workflow retired, or a skill was spun down, something stayed behind in the documentation. A note about the old API. A rule that no longer applied. A reference to a directory that didn’t exist anymore.
I never felt the problem. My daily driver was Sonnet, Opus handled the heavy lifting, and everything worked. Then Fable 5 arrived. I switched the model and things got faster, smarter, and more reliable. For weeks I felt like I’d just upgraded the entire system. That was the trap. A strong model reads a stale instruction, quietly routes around it, and delivers the right result anyway. Capability masks debt. You don’t see the rot while the engine is strong enough to fly with wrong maps.
So one evening I decided to sit down with the agent and read our constitutional files out loud, line by line. The full identity spec. Both operating contracts. The memory index. The reference docs. I gave real feedback on each section; the agent fact-checked every claim against the live system. Does this path actually exist? Does this command run? Is this tool still installed?
What We Found
The concrete findings were specific enough to sting.
A browser automation integration we retired months ago was still presented as current in two places. A task-tracking system we switched off weeks ago still appeared as “live” in a dozen different docs. An infrastructure guide told the agent to use password-based authentication on a critical system, when the real answer was the exact opposite: key-only, full stop. That’s not just stale. That’s actively wrong.
A background automation that rolled over session memory hadn’t actually run for three months because its trigger file’s format had silently drifted away from what the parsing script expected. The same operational rule was copy-pasted into four different files, which meant burning the same tokens in context every single session while burying the real load-bearing stuff that changed rarely.
One doc was still titled after an integration we’d shut down in April. Two other docs held stock notes from two months ago, loading into context every wake. Nothing was breaking, but everything was noisier and slower than it should have been.
The Numbers That Made It Real
I tracked the line counts through the cleanup.
The always-loaded memory index went from 137 lines down to 95, much denser. Working session memory went from 103 lines to 38. And because a generated planning file was embedding the working memory wholesale, fixing that source file shrank the planning document from roughly 450 lines down to 63. One cleaned file improved every downstream consumer.
Those deletions hit different when you’re building systems that load their own instruction files into context every session. Every line you remove is throughput you reclaim.
Building the Systemic Fix
Cleanup isn’t enough. Rot comes back. So I built permanent guards.
The Per-File Review Protocol
The core ritual: human reads, agent fact-checks. I opened each constitutional file (18 core docs total). For each one, I sat with it and gave raw feedback. This feels outdated. Why is this here twice? Does anyone actually use this tool anymore? The feedback was feelings, hunches, spots where something just seemed wrong.
Then the agent took over. For every claim in that file touching the live system, it fact-checked against reality. Does that path actually exist? It ran ls to verify. Does that command run? It tried it with --help and read the output. Is that tool still installed? It checked the config that gates it. Every file went through the same gauntlet: feelings from me, receipts from the agent. No claim was fixed until there was proof.
Here’s what emerged as we went file by file.
The dangerous ones were the inverted claims. A doc telling the agent to use the wrong auth method is worse than a missing doc. It’s an invisible instruction to do the opposite of reality. That’s why fact-checking matters more than cleanup.
The Fan-Out Audit
After the 18 core files were clean, I couldn’t stop. Too many satellite docs lived outside the constitution. So I built a fan-out. Five parallel audit agents, each sweeping a different cone: skill registries, command definitions, automation scripts, prompt fragments injected at wake time, generated context files. Each agent had the same list: every system we’d ever retired, every tool we’d deprecated, every API we’d shut down.
Eighty-five findings came back, each with a file:line receipt. A doc still describing a tool killed six months ago. A help-text snippet copy-pasted from an old integration. A command definition referencing a deleted script. Comments in automation code that mentioned defunct services. None of these would break the system. All of them were noise in the context window. All would compound silently if left alone.
Then came the fix stage. I spun up three fixer agents. Each one took a slice of the 85 findings and applied them (rewrite the doc, delete the line, update the comment, point the reference elsewhere). The orchestrating session did the judgment calls only. Was this fix safe? Does it need human review? Otherwise: let the agents fix it, gather the output, merge it.
The Verification Workflow
Once all edits were applied, I built a verification workflow. It ran a set of real-scenario checks against the freshly cleaned system. Not synthetic tests. Real things the agent actually does: Does the memory parser parse the new memory format? Do all internal links resolve? Does the staleness scanner run without errors? Do config mirrors match across the system? Each check was set to run three to five times, watching for drift or intermittent failures.
The workflow found a failure on the first run. One of my own updates broke a parser path. The failure is the point. A verification that cannot fail is decoration. The workflow caught the break before it went live, I fixed it, the workflow ran again, passed. That’s the discipline.
The Permanent Guards
Zero findings after this cycle. But rot comes back. So I installed two guards that stay.
First: a weekly staleness scanner. It builds our own list of retired systems (the same list the audit agents used) and greps every live doc looking for mentions. If a reference to a dead tool creeps back in, the scanner flags it. This catches accidental reintroduction: someone copies an old code snippet, embeds old documentation, reuses a retired pattern. The scanner catches it before it spreads.
Second: a fix-it-in-the-same-session rule written into the operational contracts themselves. If an audit finds something stale, don’t file it for later. Fix it now, verify it now, commit it now. Debt that doesn’t get paid in the same cycle becomes someone else’s problem.
Why This Matters
Here’s the thing about agents built on probabilistic models: determinism is not something you get from a good prompt. Determinism is something you engineer, with code-level guards and architectural checks. With harness safeguards that fail loud. With constraints that prevent silent drift.
Self-improvement loops are standard now, but they mostly check outputs and errors, not the instruction files themselves. Nobody was reading the CLAUDE.md itself. The human read is the check nothing automates away. From time to time, you have to sit down, open your agent’s constitution, and read it. Not skim it. Read it like you’re fact-checking a colleague’s work. If you feel it’s fine, that feeling might be the model’s intelligence doing the heavy lifting, not the architecture actually being sound.
A strong model can compensate for a lot. That’s a trap dressed as a feature. This is exactly the kind of rot I missed for months. Capability masking architectural debt is one of the basic ways you break your AI agent without realizing it.
The difference between a system that feels smart and one that actually is comes down to whether you audit it. I built a self-improving agent that could improve without reading its own constitution. That was a design flaw, and this audit caught it. I trust my car more than my AI agent because the car has hard constraints. Agents get hard constraints the same way: you audit, you build guards, you verify repeatedly. Hoping is the one strategy that never ships.
The Playbook
If you’re building an agent that will exist for more than a few weeks, this process is worth systematizing.
Step 1: File-by-File Review with Fact-Checking
Sit with the agent. Go through every core instruction file: identity, contracts, memory system, reference docs. For each claim about the live system, the agent fact-checks it. Does the file exist? Does the command run? Can the agent actually use this tool? Is this integration still active? Mark anything stale, contradictory, or outright wrong.
Step 2: One Home Per Rule
Don’t repeat instructions across files. Put the rule in one place. Everything else is a pointer. This cuts token cost and makes updates atomic. When an operational change happens, you update one file, not six.
Step 3: Move Dated History Out
Decisions from two months ago and session logs from last week have no business in live documentation. Archive them. If they matter for understanding why a decision was made, link to the archive. Don’t embed them in the constitution.
Step 4: Build a Staleness Scanner
List every system you’ve ever retired, every tool you’ve removed, every API you’ve deprecated. Turn that list into a weekly grep across your live docs. Staleness compounds. Catch it early.
Step 5: Verify With Real Scenarios
After any architectural change, don’t trust static checks. Run the agent through actual work. Multiple times. Does it actually behave the way the docs say it should? This is harder than writing the rule, and it’s non-negotiable.
This isn’t about perfection. It’s about the gap between what you think your agent’s constitution says and what it actually does. Capability masks that gap. A strong enough model will deliver anyway, and you’ll never know the rot is there.
Until you sit down and read it out loud.
The Guided Version
I packaged this exact protocol into a downloadable kit anyone can point their own agent at. It works with any harness: Claude Code, Cursor, Codex CLI, or a custom loop. The kit includes the file-by-file review template, the audit agent prompts, the fixer patterns, the verification workflow, and the staleness scanner configuration. If you want the guided version of everything above instead of building it from scratch, the Agent Architecture Audit Kit is $29 and included free for yearly Digital Thoughts subscribers.
If you want more experiments like this, where the failures and maintenance work are left in, subscribe to Digital Thoughts. I write about building with AI, not just using it.