Ask a Different Model
Matt Pocock recommends putting a review agent on your codebase, then walks into the obvious follow-up and cannot answer it. "But then how do you know if the automated review agent is doing a good job, you know."
Pocock is an independent educator who builds and hands out agent skills, so the question is not academic for him. His argument is that AI ate the tactical half of programming, which moves the work to preparing the place the agent operates. "your code is the environment the agent operates in, and you should always be thinking about improving that environment" The regress stays open.
Francis Brero ran the experiment. HG Insights' VP of AI starts from the arithmetic that forced it. "So even though it's making maybe a 100 times fewer mistakes than I was, it's also shipping a 1000 times more code than I ever was. So that means there's 10 times more mistakes that are being introduced." Fewer errors per line, far more lines, more errors. So he stopped reading the output. "I was never going to be able myself to review the output and that I needed to build more adversarial systems to review the code I was generating."
What he built pairs models from different labs, one writing and one critiquing, looping until they agree. Then he ran the control. "The funny thing is I tried doing the same with another version of Opus, where I'd say, now Opus, go review the work of this other Opus thing, and it wouldn't come up with the same level of insights." Same training data, same blind spots. His conclusion is a spending instruction. "So what I found is that there is a ton of value in paying a little bit more to have another class of models review, what you're doing"
Aron Ahmadia has the half that outlasts all of it. Relativity's vice president of applied science argues the measurement is the durable asset: the model turns over in months, the system around it too, the task does not. "To build evals as a product is to say evals are a durable asset." So keep it secret, because a leaked benchmark is a compromised one: "Oh, you got to keep your test private." And resist improving it, since only a frozen test compares this year to last. He also punctures what people think they are testing. "the LLMs are almost never operating by themselves anymore. They're operating as part of a larger system."
Jordan Tigani lands in the same place from data work, where the check is cheap because an answer exists. MotherDuck's chief executive would rather assert a known number than enforce a schema. "Like, there is a right answer. And you can detect whether the, the LLM or your agents can, you know, are generating, generating that right number."
Two weeks ago the question was where the gate goes. Now it is what the gate is made of. Not the thing you are checking.
Sources: Interviews from The Pragmatic Engineer (Sep 17, Matt Pocock), Metrics that Measure Up (Sep 16, Francis Brero of HG Insights), Technically Legal (Sep 17, Aron Ahmadia of Relativity), and Dev Interrupted (Sep 15, Jordan Tigani of MotherDuck).
Engineering leaders report that reviewing agent output with a second instance of the same model fails, and that cross-model review plus private frozen evals is what holds.