Bugs moved from the end of the build to before it starts
Bugs found after the buildBugs found before coding
5.0→0.33
build-breaking bugs left for final review
1.86→0.70
fix rounds per high-risk task
279→995
merged PRs: 10 months before the harness vs 6 weeks after
Chapter 1 of 23 · Nov 2025 – Apr 2026
Review by copy-paste
The problem
AI wrote code faster than I could review it, and every review was me copying and pasting between models.
3–28→243
commits a month → commits in May
What I did
I handed the writing and reviewing to agents.
Chapter 2 of 23 · Jul 12 – 17
Three roles, one rulebook
The problem
Nothing stopped the agent that wrote the code from also being the one that checked it.
3→91
agent commits in June → in July
What I did
One rulebook gives each agent one job: build, coordinate or review.
Chapter 3 of 23 · Jul 31
Screenshots are evidence
The problem
A layout bug passed review more than once, because reviewers only read the code.
Code→The app
what reviewers read → what reviewers see
What I did
Reviews now check screenshots of the running app on iOS, web and Android.
Chapter 4 of 23 · Sep 18 – 21
The consensus judge
The problem
After a failed review, one agent proposed the fix, and the same kind of bug came back.
1→3 + 1
agent's fix after a failed review → three independent fixes, one judge
What I did
Three agents propose fixes without seeing each other's, and a judge picks one. It was the harness's first verification layer.
Chapter 5 of 23 · Sep 18 – 21
Attack the plan first
The problem
On day one, a review found 3 ways broken code could still pass my new checks.
After→Before
the code was written → any code exists
What I did
Before any code is written, a reviewer tries to break the plan, and the plan gets fixed first.
Chapter 6 of 23 · Sep 21 – 28
The judge gets expensive
The problem
Each judge round took 2–3 hours, and two of its three new bugs came from the judge's own fix.
246→2,156
lines of feature → lines of checking around it
What I did
Check in proportion to the risk, not as much as possible.
Chapter 7 of 23 · Sep 22 – 25
Only blockers reopen a build
The problem
Every review round found a new small test gap, so working builds kept reopening: about 5 hours on one task.
2.4→Blockers only
fix rounds per task → reopen a build
What I did
Only a bug that breaks the build reopens it. Smaller gaps are logged for later.
Chapter 8 of 23 · Sep 28 – 29
Test the testers
The problem
Reviews used 72% of my AI tokens, and I picked reviewers by feel.
21%→43%
of bugs found before coding → found before coding
What I did
I replayed known bugs to measure each reviewer, and kept the one that found 5 of 5.
Chapter 9 of 23 · Sep 28 – Oct 6
Write down how the top model thinks
The problem
Timing bugs were 41 of 110 serious bugs, and mostly only the top model found them.
0→13
written bug-hunting recipes → recipes, named in every high-risk plan
What I did
I wrote down how the top model found them as step-by-step recipes. Every plan names the ones that apply, and the plan review checks it.
Chapter 10 of 23 · Oct 1 – 7
Severity tiers
The problem
The bug list kept growing, and I couldn't tell which bugs mattered.
651→278
open findings → open, 373 parked
What I did
Clear severity levels: app-breaking bugs first, security in its own lane, small ones parked.
Chapter 11 of 23 · Oct 4 – 8
Retire the judge
The problem
Each judge round took 2–3 hours, and it ran about 13 times in one week.
~13→0
judge runs in one week → after Oct 6
What I did
A second fix from a different model replaced the judge.
Chapter 12 of 23 · Oct 9 – 11
A practice build first
The problem
Four different builders made the same two bugs from one plan. The plan was the problem.
5.0→0.33
build-breaking bugs at final review → after practice builds
What I did
A cheap model builds a throwaway draft, reviewers attack it, and the plan is fixed before the real build. The first one found 14 bugs.
Chapter 13 of 23 · Oct 11 – 14
A cheaper practice build
The problem
Catching bugs early made the start slow: one practice build took over two hours.
127→13–33
minutes per practice build → minutes
What I did
A smaller model, a small sample of checks, run once.
Chapter 14 of 23 · Oct 14
Audit the labels
The problem
Three different definitions of “blocker” were in use.
35→4
“blockers” claimed → held up
What I did
I checked every claim. The reviewers found real bugs; the labels were wrong.
Chapter 15 of 23 · Oct 14
A cheaper model, same catch
The problem
The top model was the expensive part of every review.
71%→87%
of the top model's reported bugs were real → for the cheaper model, which found as many (18 of 43)
What I did
With the recipes written down, a cheaper model took over the review before coding. The top model is kept for money, security and the hardest bugs.
Chapter 16 of 23 · Oct 15
Full proof after the merge
The problem
Full test campaigns tied up my Mac for hours and almost never found anything (0 of 258).
~13 h→≤ 10
of full checks on my Mac → sampled checks before merge
What I did
A small sample runs before merge. The full run happens after, in the cloud.
Chapter 17 of 23 · Oct 16
Rerun only what changed
The problem
Across 34 test campaigns, 1,254 reruns never changed a result.
1,254→Only
reruns that changed nothing → what the fix touched
What I did
After a fix, a script reruns only the checks the fix touched.
Chapter 18 of 23 · Oct 19
Run the tests once
The problem
Running every test three times cost about an hour per build, and the repeats never disagreed.
×3→×1
~68 Mac-minutes per build → repeat only timing and network tests
What I did
Run once. Repeat only the tests that depend on timing or the network.
Chapter 19 of 23 · Oct 21
One task per session
The problem
Long sessions re-read their whole history on every step.
93%→1
of a week's AI usage in 8-hour+ sessions → task, then a fresh session
What I did
Each task gets a fresh session that picks up from a written status note.
Chapter 20 of 23 · Oct 26
One meaning of blocker
The problem
A blind re-check of 300 findings showed most “blockers” were smaller bugs.
28%→1
of logged blockers really were → definition, measured
What I did
One definition, and every finding records its first and final label, so accuracy is measured.
Chapter 21 of 23 · Oct 26
Short, briefed agents
The problem
An agent carrying the whole conversation re-read it on every step. On Oct 26 that was half the day's cost.
$48–76→$0.70–8
per long-running agent → per short, briefed run
What I did
Long jobs go to a fresh agent with a written brief.
Chapter 22 of 23 · Oct 27
Catch it before coding
The problem
The first plan review alone caught 57% of serious bugs across six high-risk tasks.
57%→90%
of serious bugs caught at the first plan review → caught before the build
What I did
A practice build and a second review got it to 90%, so the real build can use a cheaper, faster model.
Chapter 23 of 23 · Oct 28
Measure the real cost
The problem
The tool reported 694K tokens for one run. Adding up its transcript gave about 110M.
694K→~110M
tokens the tool reported → actually spent
What I did
A script now counts every step, and long runs are split in two.