Skip to main content

Your AI Coding Benchmark Score Is Not Your Software Quality

· 16 min read
Codalio Team
Codalio Team
AI app builder team

Over the past year, I have spent more time than I expected looking at AI coding benchmarks. Every few weeks, it seems, a new model arrives with another impressive result. SWE-bench scores go up, leaderboards change, and for a brief period one model appears to have established a meaningful lead—until the next release changes the picture again. As someone involved in building software products, I understand why we pay attention. If I am going to put an AI model somewhere inside a software development process, I want evidence that it can actually perform software engineering tasks.

But lately I have found myself asking a different question: what decision am I actually making with these benchmarks?

For a CTO or VP of Engineering, the real decision is rarely whether Model A can outperform Model B by several percentage points on a public benchmark. The decision is whether introducing that model into an engineering organization will help that organization produce better software, faster, without creating a quality problem somewhere else in the development lifecycle. Those questions are related to benchmark performance, but they are not the same question.

Benchmarks tell us something important, but not everything we need to know​

SWE-bench was an important improvement over earlier coding evaluations because it moved the problem closer to actual software engineering. Instead of asking a model to solve an isolated programming exercise, SWE-bench gives an agent a real repository and a real GitHub issue and evaluates whether the generated patch resolves that issue. SWE-bench Verified further improved the benchmark through human validation of the task set. That is substantially more meaningful than measuring whether a model can produce a self-contained function from a prompt.

The problem starts when we translate that result into a conclusion it was never designed to support. A benchmark can tell me that an AI system successfully resolved a particular issue according to the benchmark’s evaluation criteria. It cannot automatically tell me that I would want the resulting code merged into a production codebase.

If one of my engineers opened a pull request tomorrow and told me only that all of the tests passed, I would obviously be pleased, but I would not consider the review finished. I would want to know whether the implementation actually reflects the requirement, whether it fits the existing architecture, whether it duplicates something that already exists, whether it introduces unnecessary dependencies, whether it creates a security problem, whether the tests themselves are meaningful, and whether another engineer will be able to understand and modify the implementation six months from now.

This distinction between passing the test and producing software we would actually ship is increasingly measurable. In March 2026, METR published an experiment in which active maintainers reviewed AI-generated pull requests associated with SWE-bench Verified. Importantly, the researchers looked at patches that had already passed the benchmark’s automated grader. They found a substantial gap between automated benchmark success and whether maintainers considered those patches mergeable. Their analysis found the automated score was, on average, about 24 percentage points higher than the maintainer-derived merge decision score. Problems included code quality, failures outside the scope captured by the tests, and changes that maintainers simply would not accept into their projects.

Benchmarks tell us something important, but not everything we need to know

That finding captures a problem I think many of us working seriously with AI-generated software have begun to recognize. Functional correctness is a necessary condition for good software, but it is not a complete definition of good software.

The dangerous part is that the code is often almost right​

My concern with AI-generated code is increasingly not that it produces obviously bad code. Obviously bad code is relatively easy to deal with. The more difficult problem is that the output can be remarkably convincing.

You give the model a requirement and it produces an implementation very quickly. At first glance, the result looks good. Then you start reviewing it. It misunderstood a subtle part of the requirement, so you explain the requirement again. It corrects that problem but introduces unnecessary logic somewhere else. Eventually the tests pass, but someone who knows the codebase looks at the implementation and points out that the AI has created a new abstraction for something the system already knows how to do. Nothing about this experience feels like failure in the traditional sense. In fact, much of it feels incredibly productive. You are watching working code appear at a speed that would have seemed ridiculous a few years ago.

And yet, you are still not sure whether you should merge it.

That “almost right” problem is not limited to my experience. Stack Overflow’s 2025 Developer Survey found that 84% of respondents were using or planning to use AI tools in development, while more respondents distrusted the accuracy of AI output than trusted it. The most commonly reported frustration was AI-generated solutions that were “almost right, but not quite,” reported by 66% of respondents; 45% also pointed to the additional time required to debug AI-generated code.

I think this is one reason the conversation about AI productivity sometimes feels disconnected from what engineering teams experience in practice. Generating the implementation has become dramatically faster, but implementation was never the entire job. Understanding the requirement, fitting a change into an existing architecture, anticipating side effects, testing the right behavior, reviewing the implementation, operating it and maintaining it are all part of software engineering. Making one part of that system dramatically faster does not automatically make the entire system dramatically faster.

We may be spending too much time optimizing the model​

This is also why I am becoming less interested in model selection as the primary optimization problem for engineering organizations. I am not suggesting that models are interchangeable. They aren’t. Cost, latency, reasoning ability, context handling, tool use and coding capability all matter, and particular models can be materially better for particular tasks.

But there is a point of diminishing returns.

When several frontier models are capable enough to produce a reasonable implementation of the same requirement, another few percentage points on a public benchmark may matter much less than what happens around the model. How well did we give it context? How clearly was the requirement represented? What tools could it use? What happened when its implementation failed a test? How was the code reviewed? What security checks ran before merge? What happened when the implementation increased complexity or duplicated existing logic?

These questions become even more important as development moves toward agentic systems. There may not be a single model responsible for producing a feature. One model might participate in planning, another in implementation and another in review. Specialized agents might perform testing, security analysis or architecture checks. The underlying models may change several times over the lifetime of the development system.

At that point, obsessing over which model “writes the best code” begins to feel like the wrong level of abstraction. The model is a component of the development system. The software produced by that system is what ultimately matters.

Productivity without quality is an incomplete measurement​

The attraction of AI-assisted development is obvious: we want to build more with the same engineering capacity. Depending on whom you listen to, AI is going to make developers twice as productive, five times as productive or perhaps ten times as productive. I would love those numbers to be true. But the more I work with these systems, the more uncomfortable I become with productivity claims that measure output without measuring what happens to quality.

Code is now extraordinarily cheap to generate. Pull requests are cheap to generate. Entire features can appear remarkably quickly. That makes traditional output measures increasingly dangerous because producing more code is not necessarily the same thing as producing more value.

METR’s 2025 randomized study of experienced open-source developers illustrates how deceptive perceived acceleration can be. The developers in the study expected AI tools to make them 24% faster and, after using the tools, still believed AI had made them approximately 20% faster. The measured result in that particular experiment went in the opposite direction: developers using AI took 19% longer to complete the assigned tasks. METR is careful about the limits of that finding—it involved experienced developers working in repositories they knew well and the AI tools available in early 2025—so it should not be interpreted as evidence that AI generally makes developers slower. What interests me is the gap between perceived and measured productivity.

Productivity without quality is an incomplete measurement

It is easy to understand how that gap emerges. AI can make the beginning of a task feel extraordinarily fast. The requirement becomes a plan, the plan becomes code, and suddenly a substantial implementation exists. The cost can migrate downstream into clarification, review, debugging, architectural cleanup, security remediation and rework. If we measure only how quickly code appeared, we can conclude that productivity increased dramatically while missing the work required to make that code production-ready.

DORA’s 2025 research provides another useful way of thinking about this. Based on responses from nearly 5,000 technology professionals, DORA characterizes AI primarily as an amplifier of the engineering system around it. Organizations with strong technical and organizational capabilities can use AI to amplify those strengths; organizations with weak systems can amplify their existing problems.

That framing resonates with me. If our requirements are ambiguous, AI can implement ambiguous requirements faster. If our architecture has weak boundaries, AI can produce code across those boundaries faster. If our review process does not catch duplication, AI can duplicate code faster. If our tests validate the wrong behavior, AI can pass those tests faster. Speed is valuable, but speed amplifies whatever system we put behind it.

Productivity without quality is an incomplete measurement

What if we benchmarked the output instead?​

This has changed the way I think about AI benchmarking. Instead of beginning with the question, How capable is this model?, I increasingly want to begin with another one: What evidence would make me confident in the software this system produced?

That question immediately leads to a very different set of measurements.

Correctness and reliability obviously matter. Does the software build? Do the tests pass? What defects appear after the change is merged? Security matters. Did the implementation introduce vulnerabilities, unsafe patterns or risky dependencies? Maintainability matters. Did the AI reuse the existing system or create another version of logic that was already there? Complexity matters. Did a relatively straightforward requirement result in a straightforward implementation, or did the solution create unnecessary structural complexity?

Testing also has to be evaluated more carefully. Coverage is useful, but coverage alone is a weak proxy for whether a test suite can actually detect meaningful faults. Software-engineering research predating generative AI has already demonstrated the limitations of treating coverage as equivalent to test effectiveness. The same applies to architecture. Coupling, cohesion, dependency structure and changes in technical debt may tell us things that a functional benchmark never sees.

These aren’t theoretical categories I arrived at because AI needs a new vocabulary. In the code-quality framework I have been developing, I have deliberately tried to reuse metrics and terminology that engineering organizations already understand: test pass rate, build success, static security findings, dependency vulnerabilities, duplication, technical debt, code smells, cyclomatic and cognitive complexity, coverage, mutation testing, coupling and post-merge defect density.

That is important because I don’t think AI requires us to invent a completely new definition of good software. Much of what makes software maintainable, secure and reliable has not changed. What has changed is the volume and speed at which software can now be generated, and therefore the importance of making those quality signals systematic, reproducible and increasingly automated.

AI-generated code deserves the same engineering standards​

There is another temptation I think we should resist: creating a different quality standard for AI because of how quickly it can produce software.

If a human developer introduces a critical vulnerability, I don’t care that the feature was delivered ahead of schedule. If an AI system does it in thirty seconds, I shouldn’t care either. If a human developer duplicates an existing subsystem instead of understanding the architecture, we would catch it in review. AI-generated code should face the same expectation.

The framework I have been working on therefore starts with a simple rule: AI-generated code gets the full quality gate, not a lighter one. The thresholds should apply regardless of whether a human or AI authored the change.

This is particularly important because some quality problems are easy to hide behind superficially positive metrics. Tests can pass while maintainability deteriorates. Coverage can increase while the tests themselves remain weak. Throughput can rise while post-merge defects or rework rise with it. GitClear’s 2026 analysis of hundreds of millions of changed lines found increasing duplication, copy/paste activity, short-term churn and error-masking constructs during the period of rapidly increasing AI adoption, alongside declining refactoring signals. That observational evidence does not prove that AI caused each of these trends, but it is enough to make the quality question worth taking seriously.

The more interesting benchmark, then, is not a single score. It is a quality profile that sits next to productivity. If an AI-assisted team doubles its throughput while defects, duplication and rework also increase, I need to see both sides of that equation before calling the experiment successful.

The benchmark I actually want is the one that evaluates my pipeline​

This is ultimately where my thinking has changed the most. A year ago, I would have been more interested in asking which AI model our engineering team should use. Today, I am much more interested in asking how I know whether our AI-assisted development pipeline is improving.

That pipeline may use Claude today, GPT tomorrow, Gemini somewhere else and an open-source model for a specialized task. Those models will keep changing. Six months from now, whatever leaderboard we are looking at today may be largely irrelevant.

The evaluation system should survive those changes.

If replacing a model reduces defects and rework without sacrificing throughput, I want to be able to measure that. If a supposedly superior model produces more duplication in our codebase, I want to know that too. If improving the context we provide to the model has a larger effect on requirement adherence than switching models, that is valuable information. If adding an automated review step produces a bigger improvement than upgrading to the newest frontier model, that tells me where to invest.

This changes benchmarking from a model-selection problem into a system-optimization problem.

And that, to me, is the more interesting opportunity.

From benchmark to feedback loop​

There is also a consequence of this shift that I think will become increasingly important. Historically, we have treated benchmarks as scoreboards. We run the system, calculate a result and compare it with something else.

In an AI-assisted development environment, evaluation can become part of the development process itself. The system generates an implementation, measures it, identifies weaknesses, feeds those findings back into the appropriate agent, repairs the implementation and evaluates it again. Security findings become feedback. Complexity becomes feedback. Failing tests become feedback. Duplication becomes feedback. Eventually, perhaps even architectural and requirements-alignment signals become part of that loop.

From benchmark to feedback loop

At that point, benchmarking is no longer something we do after AI writes the software. Evaluation becomes part of how the AI writes the software.

I suspect this is where the next meaningful productivity gains will come from. Not simply from finding a model that generates code a little faster or scores a few points higher, but from becoming much better at measuring what our entire development system produces and using those measurements to improve the system continuously.

That leads to a much bigger conversation about whether our existing software-development processes are actually designed for an AI-native world. I don’t think most of them are. But that deserves an article of its own.

For now, I will keep watching SWE-bench. I will keep looking at model evaluations, experimenting with new models and paying attention when a new release makes a meaningful leap. These benchmarks remain useful because they tell us something about the underlying capabilities available to us.

I just don’t think they answer the question that ultimately matters to an engineering leader.

Can this AI-assisted development system consistently produce software that I am comfortable shipping?

That is the benchmark I want.

Sources​