Verification Inside Loops: Why Unattended AI Needs a Second Pair of Eyes Built In
A loop running unattended is a loop making mistakes unattended. How verification works inside AI loops: maker-checker at every cycle, mechanical tests, verified done-conditions and the catch log that makes walking away safe.
There is a sentence in the founding essay of loop engineering that every business considering autonomous AI should tape above the desk: "A loop running unattended is also a loop making mistakes unattended."
That is Addy Osmani, writing about coding agents, but the sentence transfers to business loops without editing a word. The entire promise of a loop (it works while you do not watch) is also its entire risk, and the discipline that reconciles the two is verification: the checking built INTO the loop, so that walking away is a design decision rather than an act of faith.
Our [reflection article] covered checking as an agent habit: the chef tasting the dish. This article covers what changes when the kitchen runs overnight: why verification moves from good practice to load-bearing structure, the three checkpoints every loop needs, why the checker must never be the maker (and why that split now decides even the loop's "done"), the catch log that becomes your most valuable operational document, and the honest limits of machine-checked machines.
Why Loops Change the Verification Question
With a prompted agent, a human sits at the end of every task: reading the draft, glancing at the action, catching the wobble. Verification exists, informally, because you are there.
A loop removes you from the cycle by design. The [heartbeat] fires at 7am; the work happens; you see a summary at 9. Every informal check you used to perform has to have been designed into the system beforehand, or it simply does not exist, and the loop's mistakes compound at the same machine speed as its work. This is not a marginal change; it inverts the burden. Prompted AI is innocent until proven guilty by the human reading it. Looped AI must prove itself, cycle by cycle, to checks that were built before anyone walked away.
The analogy that carries the article: the night-shift factory. A day shift with the owner walking the floor can run on informal quality culture: someone would notice. The night shift with nobody senior present runs on built checks or it runs on luck: the gauge that stops the line at tolerance drift, the sampler that pulls every twentieth unit, the sign-off station before the loading dock. Nobody calls those checks distrust of the night crew; they are what makes an unwatched shift possible at all. Loop verification is night-shift engineering for knowledge work.
The Three Checkpoints of a Verified Loop
Verification inside a loop is not one gate but three, at different altitudes.
Checkpoint 1: Per action, mechanically. The lowest, cheapest, strongest layer: deterministic checks on everything checkable, at the moment it happens. The extracted total matches the line items. The record actually exists after the write. The email address parses. The refund is within the policy table. This is [the harness's] rule-bound code doing what rule-bound code does perfectly: comparing reality to specification, millions of times, for pennies. Design rule from the reflection article, doubled in force here: verify mechanically wherever a fact is testable; spend model-judgment only where it is not.
Checkpoint 2: Per work item, by a separate checker. The [maker-checker split], now structural: every item the loop produces (the drafted chase email, the classified document, the proposed resolution) is reviewed by a second agent with different instructions, ideally a different model family, against an explicit checklist before the loop proceeds. The reasons are the night-shift reasons: the maker cannot see its own blind spots, and there is no human at the elbow to catch what it missed. In loop deployments the checker is not an upgrade; it is the component that replaces you.
Checkpoint 3: Per run and per goal, on the stop itself. The loop's claim to be finished is verified by something other than the worker: mechanical done-conditions plus an independent check, exactly as the stopping conditions article details and exactly as the coding world's run-until-done tools implement it (a fresh model grades completion, not the one that did the work). The subtle point worth repeating: the maker-checker split applies to "done" as much as to any output, because "I have finished" is the single most convenient claim a worker-model ever makes.
three-checkpoints diagram
The Catch Log: The Document That Makes Walking Away Rational
Every verification event (mechanical fail, checker rejection, done-check dispute) writes to a log, and that log quietly becomes the most valuable operational document the loop produces. Three reasons.
It is your evidence that verification is alive. An active loop with an empty catch log does not have a perfect maker; it has dead checks ([the seeded-test rule] applies: plant a flawed item occasionally and confirm it is caught). A healthy log shows a steady trickle of catches, each one a mistake that cost nothing because it never left the building.
It is your tuning data. Catches cluster: one supplier's format causes half the extraction failures; one policy ambiguity drives the checker rejections. Reading the clusters monthly is how [thresholds rise], checklists sharpen and knowledge files get corrected: the [improvement loop] running on facts instead of anecdotes.
It is your answer to the only question that matters. When anyone (owner, auditor, customer, your own 3am doubt) asks "how do you know the unattended system is behaving?", the catch log, the escape-rate trend and the sampling records ARE the answer. Loops without them are answered by vibes, and vibes do not survive the first incident review.
A Worked Shape: Verification in the Receivables Loop
Take the [pillar's receivables loop] and x-ray just its verification skeleton.
Per action, mechanical: every invoice amount re-fetched from the ledger at draft time (never trusted from earlier context); every recipient address validated against the account record; the send-tool refuses any draft not carrying a checker-pass token. Per item, checker: a second agent, on a different model family, reviews each drafted chase against a seven-line checklist (right amount, right tone tier for this customer's history, no invented payment plans, one clear action, correct escalation if third reminder...) and rejects with reasons. Per run: done means queue empty AND every item's state written AND checker rejection rate under 10%; a rate above that trips the [circuit breaker] and pauses the loop, on the logic that a spike in rejections means something upstream changed and the humans should look before more drafts are made.
In practice, the checker catches two or three drafts a week (usually tone-tier mistakes on customers with mixed histories: exactly the judgment calls you would expect to be hardest). The escape rate, measured by sampling 10% of sent chases monthly, has held under 1%. The finance manager spends eleven minutes a day with the exception queue and the log. That is what "unattended" actually means in a well-built loop: not unwatched, but watched by design, with the human attention compressed into the minutes where it matters.
What Are the Limits of Machine-Checked Machines?
Honesty section, at full strength, because this is where over-confidence lives.
Shared blind spots are real and structural. Two models can miss the same thing for the same reason, which is why checkers on different model families help, mechanical checks outrank model checks, and sampled human review never fully retires. Osmani's formulation is the permanent one: even with the split, "done is a claim and not a proof."
Checklists drift from reality. The checker verifies against criteria someone wrote; when your policies change and the checklist does not, the loop verifies fidelity to an outdated world, immaculately. Checklist review belongs on the same quarterly calendar as every other [governance boundary].
Verification can become theatre at scale too. A checker prompted vaguely approves fluently ([the reflection article's] warning, compounded by schedule). The defences transfer: specific decidable criteria, rejection-rate monitoring (a checker that never rejects is furniture), seeded tests.
The human sample is the final backstop, forever. Some fraction of the loop's output, reviewed by a person, on a cadence, always. Not because the machines are useless, but because every layer above rests on this one epistemically: the sample is how you know the escape rate, and the escape rate is how you know everything else is working. The night-shift factory never fully retired the morning walk-through, and neither should you.
The Verification Budget: Spending Attention Where It Earns
Verification costs three currencies (compute, latency, human minutes), and mature loops allocate all three deliberately rather than evenly.
Spend compute lavishly on the mechanical layer. Deterministic checks are so cheap relative to consequences that the correct amount is "every testable fact, always." There is no economising story here worth telling.
Spend checker passes by stakes tier. Customer-facing and money-adjacent items: full checker review, every item, different model family. Internal drafts and analyst outputs: checker on a sampling basis (one in five) with full review triggered by any catch. Pure-read reporting: mechanical verification of figures plus human review of the shipped draft, no checker needed, because the [human is already the gate].
Spend human minutes where machines are weakest. The permanent sample (5 to 10% of shipped output monthly) plus the exception queue plus the catch-log read. Concentrate the sample where catches cluster and where stakes peak: sampling uniformly is fair to the work items and wasteful of the scarcest currency in the system, which is a person's attention.
And re-allocate quarterly, from the log. Categories with months of clean records earn lighter checking ([the trust dial] again); categories where catches cluster earn heavier. The end state to aim for is lopsided on purpose: near-zero human attention on the proven routine, dense attention on the genuinely uncertain, and the catch log as the referee that decides which is which. That lopsidedness is not corner-cutting. It is what "risk-based" means when it stops being a slide word.
The Buyer's Verification Questions
For any loop, vendor-built or in-house:
- "Show me the three checkpoints." Mechanical per action, checker per item, verified done per run. Missing layers are not minimalism; they are the night shift running on luck.
- "Is the checker a different model with explicit criteria?" Same-model self-review is better than nothing and worse than claimed.
- "Show me last week's catch log." Its existence, its trickle, its clusters. An empty log on an active loop is the red flag wearing a green costume.
- "What is the escape rate, and how is it measured?" The honest answer names a sampling percentage and a trend. No sampling means the number is unknown, which is an answer too.
- "What trips the quality circuit breaker?" Rejection-rate spikes should pause the loop automatically. If nothing does, the loop's bad day runs to completion.
Frequently Asked Questions
What is maker-checker in AI loops? The structural rule that the agent producing work is never the one verifying it: a second agent, differently instructed and ideally a different model, reviews every item against explicit criteria before the loop proceeds, and completion itself is judged by something other than the worker. In unattended loops this split is not best practice; it is the component that replaces the absent human.
How do you verify AI output when nobody is watching? Three built layers: mechanical checks on every testable fact at action time, an independent checker agent per work item, and verified done-conditions per run, all writing to a catch log, backed by a permanent human sampling habit that measures the escape rate. Designed before walking away, or it does not exist.
What is a catch log? The record of everything verification intercepted: mechanical failures, checker rejections, done-check disputes, with reasons. It proves the checks are alive, supplies the tuning data (catches cluster around real causes), and answers audits. An active loop with an empty catch log has dead checks, not a perfect agent.
Can two AIs checking each other really be trusted? More than one AI trusted alone, less than the marketing implies: different model families reduce shared blind spots, mechanical verification outranks both, and sampled human review remains the epistemic floor. The design goal is not certainty; it is a known, small, monitored escape rate with bounded consequences.
How much does loop verification cost? Mechanical checks are effectively free; a checker pass adds a modest model call per item (pennies); the human sample costs minutes on a cadence. Against unattended mistakes at machine speed, it is the most lopsided trade in the discipline, the [reflection economics] compounded by schedule.
What happens when the checker and maker disagree repeatedly? Revision caps end the ping-pong (two redrafts, then the human queue), and persistent disagreement is treated as signal, not friction: work the loop cannot satisfy itself about is precisely what a person should see. A spike in disagreements trips the quality breaker and pauses the run, because it usually means the world changed upstream.
Who in a small business should own loop verification? The same person who owns the loop: usually whoever held the responsibility before it was automated. Their verification duties are concrete and bounded: the daily exception queue (minutes), the weekly catch-log read (fifteen minutes), the monthly sample, and the quarterly checklist review. It is a management role over a tireless report, not a technical one, and the businesses that thrive at this treat it as a named part of a real job description rather than a hope.
The Takeaway
The loop's gift is working while you sleep; verification is what makes sleeping reasonable. Three checkpoints (mechanical, checker, verified done), one principle (the checked never supplies the check), one document (the catch log) and one permanent habit (the sample).
Build them before walking away, read the log like the operational gold it is, and hold every loop, bought or built, to the five questions. The night shift is a wonderful thing. It was never run on trust, and neither is this.
Bots and Brand Works builds the three checkpoints into every loop as standard and hands you the catch log with the keys. If you are running (or being sold) a loop, ask it our five questions, and if any answer wobbles, send us the wobble, free.
Need Help Implementing AI?
Read our FAQ section to learn more.
Resources and Further Reading
- Loop Engineering: The Complete Business Guide
- [Related: Reflection Explained]
- Related: Stopping Conditions
- Addy Osmani: Loop Engineering: Click Here
- Foundations: Multi-Agent Systems (maker-checker) · Human Approval · AI Agent Governance