Verification Is the Moat: QA in the Agentic Era
Abstract
“Boris Cherny, the creator of Claude Code at Anthropic, was asked what separates the people who get great results out of AI coding agents from everyone else. His answer had nothing to do with clever prompts. He said verification is probably the single most important thing that people do not get right. Give the model a way to check its own work, he said, and the quality of what comes back goes up two to three times. Around the same time, the researcher Jason Wei described a simple rule of thumb: the easier a task is to check, the faster AI gets good at it. Put those two ideas side by side and you get an uncomfortable picture. Anything easy to check will eventually be done by machines. What stays with us is the work that is genuinely hard to check. Generation became cheap. Knowing whether the output is actually right did not.
That is the job this talk is about, and in practice it comes down to two things. The first is evals. When a feature is powered by AI there is often no single correct answer to assert against, so the old test case with a fixed expected value stops working. An eval is simply how you check something that has no one right answer, and writing evals is quietly becoming part of the QA role rather than a data science speciality. The second is what happens to the testing lifecycle itself once agents do the work. I will walk through an Agent-driven Test Development Life Cycle, or ATDLC: the same stages every QA team already knows, from understanding the requirement through planning, case design, automation and reporting, except an agent drafts each stage and a human has to approve it before the next one can start. We will look at the real artifacts each stage produces, so it stays concrete rather than theoretical. We will also be honest about where all of this breaks, including judge models that quietly prefer their own answers, and a controlled study in which experienced developers using AI were 19% slower while believing they had been 20% faster. You will leave knowing what to try on Monday, whether you write the tests or lead the team that does.”
Key Takeaways
About Speaker
Amit Rawat is a Principal Engineer at London Stock Exchange Group (LSEG), where he leads a global team of engineers and pioneers the application of agentic engineering across the software development lifecycle, from how teams build and test to how they triage, report, and ship. His focus is on putting AI agents to work on real, repetitive engineering problems at scale while keeping humans in the loop where it matters. He is also the creator of The Agentic Engineer, where he writes and builds in public about engineering with AI coding agents and how the craft of software quality is changing in the agentic era.