The "different eyes" is what interests me. Your own numbers say the US frontier models cluster hard, roughly 2.2 independent votes across nine judges. The real diversity injection is a Chinese model. Kimi, DeepSeek, and Qwen are trained on different data and tuned differently, so they tend to fail in different places than Claude or GPT, which is exactly what you want in a reviewer. A Claude-plus-Kimi panel might catch what a Claude-plus-Codex one misses. 100% worth testing now that K3 is out.
I have actually! Kimi K3 actually skews closer to Claude on many problems. I don’t have a large enough sample set to say definitively, but similar to my Codex/Claude comparison, that’s how it qualitatively seems.
You are right that a very different lineage helps though. Even at a far, far smaller size, Qwen 3.6 27B often finds things that GPT Sol and Fable don’t.
Super interesting find. It means simply using “Chinese model” as a diversity proxy is too limiting: K3 converging toward Claude, Qwen diverging despite its size. What actually matters is how independent the training lineage actually is, more than the flag or the parameter count. Makes me want to choose reviewers by lineage rather than by vendor. Are you seeing that in the failure modes too, or mainly in what each one catches?
Mainly in what one catches or not—I still use Claude more day-to-day, so most of the trials with other models are in large panel adversarial reviews.
I have deliberately tested Qwen 3.6 27B more extensively because it can be locally hosted. It does need to be babysat more, especially with a smaller context window but is shocking capable (it’s the subject of some of my other articles). It does tend to take pretty different paths vs GPT and Claude.
An excellent article James. Let me add my two cents of experience in this regard. First, I think it's a mistake to conflate all three models of Claude as if they are Claude. They all work differently with different capabilities. They are not one model that simply works longer. They work differently and were trained differently. I have found that if you use a different Claude model, it does a better job of finding bugs and other errors than the one doing the design and implementation. It even helps to use a different one for design and for implementation. Often the one that does the implementation will catch the design errors. That would be the same as using a different LLM I presume. It also helps to always use a more powerful model. If I am using Sonnet for something simple, I will use Opus to check it. If I am using Opus for coding, I will use Fable to check it. I have tried using Gemini 3.5 Pro to find errors and haven't had as much success as using different models of Claude. I haven't tried the other coding AIs.
I have also found that it is much easier to catch errors by asking the AI to plan with you and explain its rationale before doing any work. This allows two things. First, you can see if it missed something or something was miscommunicated or it is looking at it wrong. It's thinking is rarely complete on the first pass regardless of the model. If there is something that confuses me or I'm not sure of, I ask about it. Maybe it just ends up giving the AI clearer requirements in a larger context, but it helps.
Second, the AI that does the review can read the rationale as well as the result. Sometimes it disagrees with the rationale and comes up with a better design rather than just looking at the code or result. That surprised me but it works. I like to have discussion first before the AI starts a task, even when it's testing. It always seems like it takes multiple iterations of discussion with the AI asking different levels of questions and from different perspectives to fully vet what it is doing. And I get much better results than just saying go do this even when the task seems simple. But the same is true for myself. So maybe I am a poor communicator with AI. I find I understand my own task better once I've explained it to someone else and had a discussion around it. AI seems to work great for having that discussion for both me and the AI, at least Claude does, especially opus and above. Sonnet is still too agreeable like Chatgpt is. And still too sure that it is right.
Regarding the complexity of giving it too much to do at one time, I only deal with one feature at a time. I have found I get better results if I go through the interactive design and planning process for all parts, tasks, or changes in the feature before any coding, even if it is a big, complex feature. If there are two features with significant interaction, I do them together. I number the changes with the AI as we go. When I'm all done, I say go implement those changes and it does a better job than doing the changes one at a time or without the discussion. I believe it's because each change was planned and executed in context of the entire feature or requirements.
I love your articles with actual experience and practical advice. This is some of my experience with those issues for what it's worth.
Thanks for sharing your experience, Brent! For sure, a different, fresh context of any of the models (Sonnet or Opus or Fable—I’d leave Haiku out of this, since it’s a fairly different thing) brings the “Fresh Eyes” phenomena into play. So it definitely works and your experience completely makes sense.
That being said, it IS still true that Opus often grades Sonnet answers more positively than ChatGPT, DeepSeek, Qwen, etc… and similarly for lower levels of ChatGPT to higher levels, local 3.6 27B Qwen vs. Qwen Max. Also, as per the charts, I’ve definitely seen things missed when using the same model (generally, I only use the highest version—so recently Fable and previously Opus, and the reviewing agents are Fable or Opus).
As said in the article, it’s not saying one of these is better than another—but they take different probabilistic paths down LLM lane, so they can raise things that same-family/model don’t as often.
If you don’t mix in another model family, I do think you’ll miss certain errors that even a weak model from another one will catch (even though the biggest effect is “Fresh Eyes” not “Different Eyes”, so even with just Claude X <> Claude Y, you get more of the benefit) I’d definitely still suggest trying it out!
Though, regarding the thing you mentioned about using Gemini to check… I personally wouldn’t use Gemini or Copilot for this. For real world performance, I’ve found those two lag everything else by a lot.
Create an agent (can be Claude) that is triggered by a GitHub Action of a pull request or something else. You could even do auto adversarial reviews with it. It’s essentially the same workflow as CI and I have one on a repo that runs an E2E test and a review at the same time.
The "different eyes" is what interests me. Your own numbers say the US frontier models cluster hard, roughly 2.2 independent votes across nine judges. The real diversity injection is a Chinese model. Kimi, DeepSeek, and Qwen are trained on different data and tuned differently, so they tend to fail in different places than Claude or GPT, which is exactly what you want in a reviewer. A Claude-plus-Kimi panel might catch what a Claude-plus-Codex one misses. 100% worth testing now that K3 is out.
I have actually! Kimi K3 actually skews closer to Claude on many problems. I don’t have a large enough sample set to say definitively, but similar to my Codex/Claude comparison, that’s how it qualitatively seems.
You are right that a very different lineage helps though. Even at a far, far smaller size, Qwen 3.6 27B often finds things that GPT Sol and Fable don’t.
Super interesting find. It means simply using “Chinese model” as a diversity proxy is too limiting: K3 converging toward Claude, Qwen diverging despite its size. What actually matters is how independent the training lineage actually is, more than the flag or the parameter count. Makes me want to choose reviewers by lineage rather than by vendor. Are you seeing that in the failure modes too, or mainly in what each one catches?
Mainly in what one catches or not—I still use Claude more day-to-day, so most of the trials with other models are in large panel adversarial reviews.
I have deliberately tested Qwen 3.6 27B more extensively because it can be locally hosted. It does need to be babysat more, especially with a smaller context window but is shocking capable (it’s the subject of some of my other articles). It does tend to take pretty different paths vs GPT and Claude.
An excellent article James. Let me add my two cents of experience in this regard. First, I think it's a mistake to conflate all three models of Claude as if they are Claude. They all work differently with different capabilities. They are not one model that simply works longer. They work differently and were trained differently. I have found that if you use a different Claude model, it does a better job of finding bugs and other errors than the one doing the design and implementation. It even helps to use a different one for design and for implementation. Often the one that does the implementation will catch the design errors. That would be the same as using a different LLM I presume. It also helps to always use a more powerful model. If I am using Sonnet for something simple, I will use Opus to check it. If I am using Opus for coding, I will use Fable to check it. I have tried using Gemini 3.5 Pro to find errors and haven't had as much success as using different models of Claude. I haven't tried the other coding AIs.
I have also found that it is much easier to catch errors by asking the AI to plan with you and explain its rationale before doing any work. This allows two things. First, you can see if it missed something or something was miscommunicated or it is looking at it wrong. It's thinking is rarely complete on the first pass regardless of the model. If there is something that confuses me or I'm not sure of, I ask about it. Maybe it just ends up giving the AI clearer requirements in a larger context, but it helps.
Second, the AI that does the review can read the rationale as well as the result. Sometimes it disagrees with the rationale and comes up with a better design rather than just looking at the code or result. That surprised me but it works. I like to have discussion first before the AI starts a task, even when it's testing. It always seems like it takes multiple iterations of discussion with the AI asking different levels of questions and from different perspectives to fully vet what it is doing. And I get much better results than just saying go do this even when the task seems simple. But the same is true for myself. So maybe I am a poor communicator with AI. I find I understand my own task better once I've explained it to someone else and had a discussion around it. AI seems to work great for having that discussion for both me and the AI, at least Claude does, especially opus and above. Sonnet is still too agreeable like Chatgpt is. And still too sure that it is right.
Regarding the complexity of giving it too much to do at one time, I only deal with one feature at a time. I have found I get better results if I go through the interactive design and planning process for all parts, tasks, or changes in the feature before any coding, even if it is a big, complex feature. If there are two features with significant interaction, I do them together. I number the changes with the AI as we go. When I'm all done, I say go implement those changes and it does a better job than doing the changes one at a time or without the discussion. I believe it's because each change was planned and executed in context of the entire feature or requirements.
I love your articles with actual experience and practical advice. This is some of my experience with those issues for what it's worth.
Thanks for sharing your experience, Brent! For sure, a different, fresh context of any of the models (Sonnet or Opus or Fable—I’d leave Haiku out of this, since it’s a fairly different thing) brings the “Fresh Eyes” phenomena into play. So it definitely works and your experience completely makes sense.
That being said, it IS still true that Opus often grades Sonnet answers more positively than ChatGPT, DeepSeek, Qwen, etc… and similarly for lower levels of ChatGPT to higher levels, local 3.6 27B Qwen vs. Qwen Max. Also, as per the charts, I’ve definitely seen things missed when using the same model (generally, I only use the highest version—so recently Fable and previously Opus, and the reviewing agents are Fable or Opus).
As said in the article, it’s not saying one of these is better than another—but they take different probabilistic paths down LLM lane, so they can raise things that same-family/model don’t as often.
If you don’t mix in another model family, I do think you’ll miss certain errors that even a weak model from another one will catch (even though the biggest effect is “Fresh Eyes” not “Different Eyes”, so even with just Claude X <> Claude Y, you get more of the benefit) I’d definitely still suggest trying it out!
Though, regarding the thing you mentioned about using Gemini to check… I personally wouldn’t use Gemini or Copilot for this. For real world performance, I’ve found those two lag everything else by a lot.
Thanks, James. I'm definitely going to try Qwen. And Gemini has not been very helpful as you say. Any advice to share on git-side AI review?
Create an agent (can be Claude) that is triggered by a GitHub Action of a pull request or something else. You could even do auto adversarial reviews with it. It’s essentially the same workflow as CI and I have one on a repo that runs an E2E test and a review at the same time.