Stop Wasting Human Time with AI Mistakes/Hallucinations—Use Adversarial Reviews
Why it Works So Well and How to Apply It
I do a lot of high-stakes work using AI. Whether it’s coding that affects thousands of real users, technical CAD/layout where mistakes incur tens or hundreds of thousands in cost and lost time, or investment decisions with millions of dollars at stake, I heavily rely on it.
In 80%+ of cases, there is at least one serious error that even frontier models like Claude Opus, or even Fable/GPT-5.6 Sol and other “Mythos-level” models, do not catch in their own work. A lot of these are real showstoppers.
Yet I can usually get away with a fairly quick review. Wait, huh?
I use what is called “adversarial reviews”—and, specifically, I use multiple model families (i.e., ChatGPT in addition to Claude models) to do it. This process, in my experience, catches essentially all serious hallucinations and almost all major errors. It’s not a magic bullet, but I’ll explain why it’s important, why it works, and how to use it (including a pre-made template for Claude Code).
Are you in the Bay Area? San Mateo County Libraries is hosting me for a book signing and talk at the Atherton Library for What You Need to Know About AI, moderated by Justin Kuczynski (PhD, Computational Biology and Engineering Lead at Google). It’s on Friday, August 7, 2026, at 3-4pm. Come out and say hi!
Understanding The Nature of the Problem
Obviously—or at least I hope it’s obvious—I don’t simply hit send on AI slop. My last piece was literally about people needing to take responsibility for their work, AI-assisted/generated or not. I also have a ton of agents, both running around autonomously and being key pillars of my information systems.
The ultimate gate for AI work is human review. That is why experts (as I refer to them, oracles) get so much benefit from AI. It can automate the simple parts and allow a comparatively low-effort human review to catch substantive problems. That is one way to always guarantee good output—or, at least, produce output that is no worse than human-generated.
Once you get to a certain scale, though, even low-time and low-effort review can add up quickly. You can have a lot of AI agents doing things at once. And even worse, unlike AI, humans get tired. Inevitably, a human stuck with reviewing a massive pile of AI-generated work will get sloppy. You can call it laziness, but this is something managers know well. You exhaust your decision-making and “executive” function quickly because everything is high-attention/stakes (you are the reviewer for approval) and low-skill (there’s no mechanical skill involved, whether it’s physical labor or even something like coding).
There’s a concept in psychology of “flow state”—which many describe as being “in the zone”—where there’s just enough skill and challenge involved to keep something intellectual interesting but also rewarding. Most games try to target some aspect of this where players can both experience challenge and exercise their skills but not have it either be boringly easy or throw-away-the-controller hard.
What a lot of executives—and now, everyone can be their own executive, managing an army of AI agents—deal with is not that. It’s, in fact, the opposite. You need a lot of attention and focus. But it’s not really stretching any skill where you can “get in the groove.” You just need to constantly pay attention because you are the final check on something going out, but it’s otherwise a boring task that you need to force yourself through. It’s definitely not a flow state and is mentally exhausting.
That is decision fatigue.
Now, the concept is often overvalorized in the media. This is why Steve Jobs and Mark Zuckerberg and Barack Obama always wore the same thing!
It is a real thing, though. I managed teams of people pretty early on in my career. Yes, I was often better at “the job,” whether in financial analysis/modeling or programming, than my direct reports. That’s what more experience and selection effects from promotion get you. But it’s totally untenable to literally redo your juniors’ work in order to check it. Even detailed checking becomes overwhelming as the number of direct reports—and work—scales up. Of course, this is exactly what every first-time manager tries to do. Including me.
I got the lesson beaten into me quickly by managers and by experience. You can’t do that. You need to focus on the high-level output and concepts—and find “critical points” that you can spot-check to get a strong read on whether the overall work is correct and of high quality.
It’s an important skill. It definitely helps. My teams never exceeded ten people, however.
I’ve sometimes had over a hundred AI agents outstanding, waiting for review and decisions, though. It’s a different order of magnitude where, no matter how refined you are in that review skill… it still breaks down.
Usually, this is why you have org charts and a limit to the number of direct reports. And while I do have “overseer” agents (I’ll write about this in a future article), there’s a far more direct technique that can at least stomp out most obvious AI errors and hallucinations. When combined with paying attention to certain verifiable output and looking at the high level, you can actually manage this kind of overload.
What is Adversarial Review?
We can use AI agents to check AI agents. It isn’t necessarily the final check, but it helps a lot. But why does this work? After all, it was AI that made the mistake to begin with!
Well, first off, let’s start with data on how often mistakes occur and how often this AI double-check catches them:
This is actual data from five weeks of work on a variety of subjects, including programming, CADing, writing research, and financial analysis. I would have guessed 80% anyway, but this nicely confirms my estimate on how often serious errors occur.
This checking is called “adversarial review” because you get another agent to specifically poke holes and find errors in the original agent’s work.
If you want to do the simplest version of this kind of review yourself, you don’t need to do anything fancy.
Just copy/paste work from one chat window to another. Even better, if you got work output from, say, Claude, you can take that output and paste it into ChatGPT’s chat instead with the prompt to check the work. Then take ChatGPT’s feedback and paste it back into the original Claude window. In a pinch (or avoiding multiple subscriptions), you can also just do this in multiple Claude/ChatGPT windows.
If you’re in Claude Cowork (which I highly, highly recommend for any “real” work versus chat), just ask it to dispatch a subagent to do an adversarial review. The “how” is easy: literally just tell it “Dispatch an adversarial subagent to review the work we just did.”
If you stop reading here, I promise just that step will dramatically improve the quality of AI work you need to review as a human. I have data to prove it in my own workflow (and studies) below.
But, if you stop reading, you won’t learn the underlying concept of why this works—and thus the intuition to apply this concept more broadly—or the easier way to make this part of your regular workflow.
The first reason that this works is the primary one.
Reason 1: Fresh Eyes
Most people don’t have a good sense of how LLMs like ChatGPT or Claude work under the hood. That’s why they’re surprised when they hear that AI agents get progressively (and sometimes dramatically) dumber as you talk to them more. The reason why is they are ultimately fancy auto-complete machines and the more you stuff into their “context” (both what you say to it and what it says/outputs back to you), the more they can get “confused.” A technically imprecise explanation, but it’s conceptually how it works.
It’s not the same reason why adversarial reviews work, but it’s related.
Models often fail to catch errors when the work is within their own context. There’s a nice 2026 preprint from MATS and Anthropic researchers that had models review their own generated code for errors and risky actions. They failed to catch problems quite often. Interestingly, however, the exact same model when provided with a fresh context (“new chat” if you’d like) pretty easily catches the errors. In this study, the fresh context model catches errors more or less just as often as literally using a different model altogether.
The explanation isn’t that the model with more context is “dumber” (though it is).
Agentic Degradation: More Turns Create Opportunity to Compound Errors
Any real-world task of any complexity tends to take multiple “turns.” The agent needs to go back and forth and can’t just “one-shot” an answer or work output.
For anything that isn’t trivial, the AI agent will have read a lot of information, written out a lot of self-reasoning, potentially asked clarifying questions, written a plan, revised its plan, hit errors, fixed errors, argued with itself… You get the idea. Whether this was in the service of writing code or helping you find and place a specific item into a shopping cart, there’s a lot of context that is generated through this process.
Let’s stay on the shopping cart since it’s more intuitive for non-programmers. You can get a completely wrong item placed into a shopping cart because the AI agent had searched, not quite found the right parameters, and “talked itself” into going for this item—even though it’s obviously wrong given your original request.
Let’s say you need a dairy-free and gluten-free cheese. After fruitlessly looking around, it finally finds a crunchy-hippy brand where most of its products are gluten-free, which the brand proudly proclaims on its website, but it has a vegan cheese product that… isn’t gluten-free. While it doesn’t say “not gluten-free,” a cursory inspection of its ingredients would show it’s not. But this model has gone through an entire chain of working on this problem for an hour, and it seems like this is it.
Agentic loops tend to be vulnerable to this kind of problem, whatever the application. Each step needs to be right. And the more steps you layer on, you get a higher and higher multiplicative probability that you’ll end up with some step in the chain being wrong—and often, that step will cascade to a fairly serious error in the final output.
A fresh agent who has gone through none of that process likely needs no more than 30 seconds to determine that the vegan cheese is not gluten-free. Similarly, a fresh agent that didn’t program the giant new feature you asked for can quickly determine that it has a few enormous bugs or security holes, since it didn’t “go on the journey” to create it. This is compounded with the all-too-familiar tendency of certain models to defend their own conclusions or work—which you just layer on top of natural compounding errors.
*As a fun side note, one of the reasons I originally started using adversarial reviews was because with hard enough problems—like developing AI/ML algorithms—the AI would often lie to me. It would hardcode accuracy metrics so it would “pass” my requirements. I went on wild goose chases. Obviously, a fresh agent instantly caught the shenanigans. This was around 2024. Frontier models don’t do that (much) anymore, but it’s still very useful for less intentional mischief.
AI Models Can’t Reliably Fix Their Own Work
This is fairly well-studied. Microsoft and Salesforce researchers found top models drop 39% on average when a task is underspecified and takes an uncertain number of turns. They have a hard time self-correcting: “when LLMs take a wrong turn in a conversation, they get lost and do not recover.” Separately, a 2026 ICLR paper showed that if you inject a model’s context with its own earlier errors, it tends to make more of them. The model conditions on its own mistakes and thus makes more.
A 2024 ICLR paper measured self-correction more directly. The researchers had models critique and revise their answers with no external feedback. Did that mean it caught their own errors? Well, kind of, but as a whole, accuracy went down with every model tested. Did they catch some errors? Sure. But more often, models actually flipped correct answers to incorrect ones when they self-critiqued—which makes asking a model to check its own work a very dubious idea.
Again, this does not apply to a fresh context, even if it’s the same model. However, in my own workflow, I always include not only a fresh context adversarial reviewer but also a second model, which, for simplicity, tends to be ChatGPT through Codex (ChatGPT’s version of Claude Code).
Reason 2: Different Eyes
I think it should be pretty intuitive that a fresh context of the same model is still treading a more similar path than a totally different one. While they’re all trained on pretty similar data and have similar post-training, it’s still somewhat different.
I’ve shown in some of my own tests that the same model family (e.g., Claude models like Opus and Sonnet) tends to prefer their own model family’s answers. It’s why for my “panel judging” I tend to try to bring in other models that are not being judged.
This also bears out in my own testing of adversarial review, where the fresh-eyes reviewer is usually a Claude subagent (which you can dispatch by simply asking for one in either Cowork or Claude Code).
Just from a purely qualitative perspective, ChatGPT’s models tend to be more analytical (look through and check), and Anthropic’s tend to be more empirical (build something to test directly). Nevertheless, they are somewhat different, and that diversity is valuable.
Note that I still use both model families, even though Claude is my “daily driver,” and thus the adversarial review is typically the same model that did the work. Both usually catch the same major errors—because of the fresh eyes effect—but each has a material number of instances where one model found something the other did not.
Yes, Codex did, in the past five weeks, find more errors than Claude did. But my goal is not “more.” It’s all errors, if possible, and if not, as many as possible. If I did only Codex, there were some that would also slip through.
And yes, I do still find errors myself. But it’s far, far fewer than I otherwise would have.
41 of 51 review sessions had errors, but these are usually complex, take a lot of back and forth, and have errors that pop up during many iterations. Just in preparing this piece, the adversarial reviews on my research document produced sixteen confirmed fixes. When I then ran an adversarial review on the fixes themselves, four of the sixteen had introduced new errors. That’s twenty on just one document.
With an average of around six to seven errors per review session, we’re looking at somewhere between 200 and 300 errors as a whole.
It’s literally hundreds of errors that I didn’t need to personally fix in just over a month.
An Obvious Question: Is More Better?
As I said, I only generally use Claude + Codex (ChatGPT). My Claude Code skill (pre-made template) allows you to use a lot of different models, though. If you want, it’s set up so you can plug in API keys to use Qwen, Kimi, DeepSeek, GLM, and Gemini as well.
While some diversity is good, I have often harped on the fact that most AI models are pretty similar. They use similar models, similar data, and similar compute (Nvidia GPUs, Google TPUs, etc.). A little bit gets you a lot. A lot… mostly gets you the same thing.
To quantify it, a Cornell study of over 350 models found that when two models both get a question wrong, they give the same wrong answer about 60% of the time on one of their tested benchmarks (random chance would be about a third). An Apple researcher a few months ago measured this with judge panels (like I often use): nine frontier judges from seven different companies, aggregated by majority vote, behaved like roughly 2.2 independent votes.
Nine humans didn’t equal nine votes, but they were still more diverse with 4-6 independent votes. The AI judges are very similar to each other. They’re not identical, but they’re far less different than humans.
I set it up in the pre-made skill, but just remember that what you’re looking for the reviewers to do is raise problems. You’re not asking them to vote for what’s right or wrong. The best results tend to be when the reviewers say, “these things are wrong,” and the AI agent doing the work evaluates whether or not the problems are real and then fixes the ones that are. My suggestion would be to avoid having the AI agent blindly accept everything, especially when the reviewers might also be missing some context on why a certain thing is the way it is. Data-wise from my own workflow, about one review session in four includes at least one reviewer suggestion that turns out to be wrong once the main agent tries to verify it.
Again, fresh, independent eyes to find problems—not to vote on what’s right.
Still, do I use very large panels sometimes? Yes. I’ve tried it out on various hard problems and even just played around with it for this piece (since Kimi K3, the new open-weight “Mythos-level” Chinese model, just came out) with Claude Fable, ChatGPT 5.6 Sol, Kimi K3, DeepSeek v4 Pro, GLM 5.2, and Qwen (my local 3.6 27B). While it’s interesting to see—and again, have it raise questions rather than take a “vote”—it’s not usually worth the incremental cost, complexity, and effort.
The Claude Code Skill
I’ve linked it a bunch of times already (didn’t want to hold out on you), but here it is: the Claude Code pre-made skill. The instructions are in GitHub, but really, just open a Claude Code instance, point it at the GitHub, and tell it to install it.
When done, all you need to do is type /adversarial-review when you want to do this. It’s much faster than explaining what you want every time (which is actually what I did for a long time). Just make sure you have the prerequisites, including Claude Code, and give Claude Code the GitHub address to install. Then, you’re off to the races.
Prerequisites:
Claude Code. This is a Claude Code skill. Unfortunately, Cowork and Claude Code on the web are sandboxed. It’s possible, but far more involved to make this available everywhere. Let me know in the comments if you want this cloud-accessible (basically, in the app and on Claude.ai)! Obviously, you need the subscription or an API key.
Codex. We use Codex as the second reviewer, so it needs to be installed and you need to be logged in. I personally use the $20 per month plan… and never run out.
Other Prerequisites (that Claude Code can probably install for you):
Python. Macs get it with the Xcode Command Line Tools (which you’ll want anyway—see Git below), and it’s not hard to get on other OSes either.
Git. This lets you “clone” (grab) the repo to install it. For Mac, install it through Xcode Command Line Tools. For Windows, use the Git for Windows installer. And for Linux, you can use whatever package manager. It’s honestly easier for you to just ask your favorite LLM for instructions.
And, if you want, there’s an env file that can take API keys from a very large set of models (just ask Claude Code directly—it’s “self-unpacking” or self-assembling in a way).
What If I Use Chat or Cowork?
You can’t use regular chat for this without the manual process of copying/pasting to a new window. The weakness of “normal chat” and why I tell people to never use it for “real work” is that it cannot dispatch subagents. That being said, copy/paste does work.
For Cowork, as I said, just ask it to dispatch an adversarial subagent to review the work. You won’t get Codex (unless you manually copy/paste into ChatGPT’s chat), but fresh eyes are often enough to dramatically improve the output.
The Point is the Concept
The concept is more important than the skill. All the Claude skill does is give you nicer ergonomics. If you use ChatGPT, then paste stuff into Claude’s chat. If you can’t use either, you can use various different open-weight models.
This adversarial review concept with both fresh eyes and different eyes works for all of it.
Why do we care? Tokens are cheap. Your attention isn’t.
I’ve written about how AI’s prices keep collapsing (dramatically, and even faster than compute costs with Moore’s Law)—the token side of this trade gets cheaper every quarter, especially relative to capability.
Human attention and cost stay the same—if anything, given more AI output and fewer humans to oversee it, one can argue it gets even more valuable… and hence expensive.
This is really the reason why I’ve consistently said that AI is going to make humans even more valuable. Basic operations research states that the bottleneck is what determines the throughput of a system. And inevitably, with more and more agents able to execute mechanical work, human attention is the ultimate bottleneck that gets relatively scarcer and scarcer and cannot be easily increased. If anything, given the world’s falling fertility rates, our rate of increasing this resource is getting slower.
If we have the ability to substitute AI time for human time, it’s going to become more and more worth it as we go. And, again, in the balance here, humans become more expensive, not less.
Putting aside macro musings, on an individual level, if you start using AI agents more extensively or AI for various work products more seriously, you’re going to encounter this very quickly. As per the former CEO of Reddit:
I think a new generation will suddenly understand the meaning of Paul Graham’s classic essay on the difference between Maker’s Schedule, Manager’s Schedule—except we’re all managers now.
As such, techniques like this won’t replace human attention—your attention. You are still the ultimate gate for what you send out the door, and I hope you aren’t an AI slop cannon just throwing stuff over the wall at other people. That being said, things like adversarial review should make your newfound role managing a bunch of AI agents less grindingly painful and let you get more done, in better ways.
Thanks for reading!
I hope you enjoyed this article. If you’d like to learn more about AI’s past, present, and future in an easy-to-understand way, I’ve published a book titled What You Need to Know About AI.
You can order the book on Amazon, Barnes & Noble, Bookshop, or pick up a copy in-person at a local bookstore.
