How to Measure Developer Productivity When AI Writes the Code
AI can make a proposed code change cheaper to produce without making that change cheaper to trust. That distinction matters because engineering organizations do not ship suggestions, prompts, or accepted completions. They ship changes that have survived review, automated checks, deployment, and the consequences that appear afterward.
Here, verification economy names a specific constraint: AI increases the rate of proposed changes faster than an engineering system can validate them with confidence. The phrase is an editorial framing, not an established software-engineering standard; the same phrase has appeared in recent AI-economy writing, so it is not unique to DevStats.
The thesis is conditional. A randomized METR study found experienced open-source developers took 19% longer on the studied tasks with early-2025 AI tools; a later update said selection effects prevented a reliable estimate for newer tools. DORA likewise describes AI as an amplifier, not a guaranteed performance improvement.
The operational question is: can the system verify and deliver changes with confidence at the same rate it can propose them? This article turns that question into a five-part verification scorecard.
What the verification economy means for engineering
The constraint shifts where engineering capacity is spent. Code generation may become faster or cheaper in some workflows, but delivery still depends on people and systems establishing that a change is understandable, correct enough for its risk, compatible with its environment, and safe to release.
Sonar's developer survey provides one signal of this tension. In an online survey of 1,149 technology professionals who had used AI at work, respondents reported that 42% of the code they committed or contributed was generated or significantly assisted by AI; that question had 979 respondents.
The same report says only 4% completely agreed that AI-generated code is functionally correct. Another 48% completely agreed that they always check AI-assisted or generated code before committing it, and 38% said reviewing AI-generated code requires more effort than reviewing developer-written code.
Those are self-reported survey results, not repository telemetry, and the source is a vendor with a commercial interest in verification tooling. They should therefore be read as evidence of perceived verification pressure, not as proof that every organization has the same bottleneck.
DORA places that perceived pressure in a broader system. It treats AI as an amplifier: strong feedback loops can convert local acceleration into better outcomes, while weak systems can amplify instability. Its delivery guidance separates throughput from instability instead of treating speed as sufficient.
The verification economy therefore covers the whole system that converts a proposed change into one the organization is prepared to own.
Why AI breaks output-only engineering metrics
AI makes several familiar activity measures easier to increase. A team can produce more suggestions, accept more completions, open more pull requests, or change more lines without necessarily improving delivery.
That creates a measurement trap: activity can move upstream while cost moves downstream.
Accepted suggestions confirm tool use while leaving review wait, corrective work, failure rate, and deployment outcomes unknown. A rising PR count adds throughput context without measuring whether reviewers have enough capacity to evaluate those PRs well. Lines of code are an even weaker productivity signal because they reward volume independently of the value, risk, or maintainability of the change.
The SPACE framework treats developer productivity as multidimensional and warns against reducing it to a single activity measure. DORA applies the same principle at the delivery-system level: throughput and instability must be interpreted together.
METR provides a counterexample to automatic productivity claims: in the studied setting, experienced developers took 19% longer with AI. The result does not generalize to engineering as a whole, and METR later cautioned against extrapolating it to newer tools because of selection effects. It shows why the effect must be measured rather than assumed.
The scorecard needs at least three kinds of evidence:
| Signal | What it can tell you | What it cannot tell you alone |
|---|---|---|
| AI usage | Whether people are using AI tools | Whether engineering outcomes improved |
| Flow | How changes move through coding, pickup, review, merge, and deploy | Whether faster flow is producing stable outcomes |
| Downstream quality/rework | Whether delivered changes create corrective work or failure | Which upstream cause produced that outcome without reliable linkage |
The five metrics in a verification scorecard
No single metric captures verification. The scorecard needs complementary signals for flow, review demand, corrective work, and concentration of human attention.
These five metrics are proposed as a measurement framework, not as industry standards. Their definitions need to be adapted to the events an organization can actually observe.
| Metric | Core question | Unit | Minimum instrumentation | Main failure mode |
|---|---|---|---|---|
| Verified Throughput | Are we delivering more changes without buying more downstream correction? | changes/period + rate | deploy + quality gates + outcome linkage | weak attribution from failure back to change |
| Verification Time | How long does a change wait and work through required verification? | elapsed time | ready-for-verification + review + required checks | double-counting overlapping review/check time |
| Review Demand per Active Reviewer | Is review demand growing faster than the reviewer pool? | reviews/reviewer/period | review requests + active reviewer definition | treating individual participation as productivity |
| AI Rework Rate | Do AI-assisted changes generate more corrective work than a comparable cohort? | percent | PR-level AI provenance + deploy + rework linkage | using user-level AI adoption as PR-level attribution |
| Reviewer Concentration | Does verification depend on a small number of people? | share/index | reviewer distribution by team/repo/window | calling concentration "saturation" without capacity data |
1. Verified Throughput
Verified Throughput asks how many delivered changes remain acceptable after the verification and post-deploy window you have defined.
A defensible implementation reports both a count and a rate:
verified changes in period
and
verified changes / deployed changes
The definition of verified determines the metric. State which gates count, which downstream outcomes disqualify a change, and how long the observation window lasts. A change might need to pass required CI and human review, then remain free of a linked rollback, hotfix, or qualifying incident for a declared window W. There is no universal value for W; it must fit the system being measured.
DORA's separation of throughput from instability follows the same principle: faster delivery should not be interpreted without a counter-signal for negative outcomes. Verified Throughput extends that logic for a verification-first scorecard; it is not an existing DORA metric.
Weak causal linkage is the limiting factor. If an incident cannot be reliably linked to the change that caused it, describe the metric as incomplete rather than repair it with guesswork.
2. Verification Time
Verification Time measures elapsed time from "ready for verification" until every required pre-merge verification step is complete.
That can include human pickup, review, CI, tests, security checks, or other mandatory gates. Because some happen in parallel, use elapsed wall-clock time between two defined events as the top line:
verification complete timestamp - ready for verification timestamp
Use queue time, active review time, and automated-check time as diagnostic components rather than blindly summing them.
Use this metric to ask: where does the time saved during generation reappear?
If coding time falls while Verification Time rises, the system may have shifted rather than removed work. If both fall while downstream outcomes remain stable, the organization has stronger evidence of system-level improvement.
PR cycle-time systems commonly observe pickup and review stages, but an organization should not call that full Verification Time unless all required verification events are represented.
3. Review Demand per Active Reviewer
Review Demand per Active Reviewer measures the pressure entering human review without pretending that review capacity is known when it is not.
A simple version is:
PRs entering review in period / active reviewers in period
or, when review requests are directly observable:
review requests in period / active reviewers in period
Read the result with PR size or another change-complexity signal. Twenty tiny changes and twenty large changes do not represent the same review burden.
Review Demand per Active Reviewer replaces the original name Review Capacity Ratio because capacity has units. If an organization knows the planned number of review hours per week, it can estimate demand hours divided by planned capacity hours. Without that denominator, "capacity ratio" would overstate what the data knows.
Keep this metric at team, repository, or service level. A low personal review count is not evidence of low productivity, and a high one is not evidence of high performance. SPACE's multidimensional model is one reason to avoid turning participation data into individual rankings.
4. AI Rework Rate
AI Rework Rate compares corrective work after AI-assisted changes with corrective work after an appropriate comparison cohort.
A conceptual formula is:
AI-assisted deployed changes with defined rework inside W / total AI-assisted deployed changes
The same calculation can be run for a manual or less-AI-assisted cohort.
AI Rework Rate fails without change-level AI provenance. Knowing that a developer uses an assistant does not prove a specific PR was AI-assisted. Tool adoption, accepted suggestions, AI-generated lines, and AI-assisted PRs are different populations.
Define rework as a traceable corrective event, such as a linked rollback or hotfix. Comparison cohorts also need context because AI-assisted and manual changes may differ in task type, codebase, risk, or developer experience. A rate difference warrants investigation; it does not automatically establish a causal effect of AI.
Without PR-level provenance and reliable rework linkage, AI Rework Rate should remain unavailable rather than approximated from user-level adoption.
5. Reviewer Concentration
Reviewer Concentration asks whether a disproportionate share of verification depends on a small group of people.
It can be expressed as a simple top-share measure, such as the percentage of reviews handled by the top 20% of reviewers, or as a concentration index such as HHI. Whichever method is chosen should be documented and kept stable over the comparison window.
Concentration and saturation are different measures.
Concentration is observable from the distribution of reviews. Saturation means demand is approaching or exceeding capacity. You cannot infer saturation unless you also know something meaningful about available capacity.
Rising concentration can signal specialist bottlenecks, ownership gaps, or a review system that relies on too few people. It should prompt a systems question: Why does verification depend on this narrow set of reviewers? The answer should guide process changes, not a leaderboard.
Instrumentation comes before attribution
Any comparison between AI-assisted and manual changes starts with provenance.
Usage telemetry can tell you that people are using AI. It can count suggestions, acceptances, chat sessions, insertions, or similar events when the tool exposes them. But user-level adoption is not enough to compare AI-assisted and manual changes.
For stronger attribution, prefer change-level evidence in roughly this order:
- explicit PR or change metadata that marks AI assistance;
- tool audit events reliably linked to a PR or commit;
- commit/change metadata with a documented attribution method;
- user-level AI adoption only as contextual information, not as PR-level provenance.
Downstream attribution has the same requirement. A rollback or incident is not useful for Verified Throughput or AI Rework Rate unless the measurement system can relate that outcome to the relevant change with an acceptable level of confidence.
A before-and-after comparison can confound AI adoption with changes in architecture, staffing, review policy, release cadence, or workload. Segment by context, preserve a baseline, and make causal claims only when the study design supports them. When instrumentation cannot support the comparison, report not measurable with current instrumentation.
How to read the scorecard as a system
The scorecard becomes useful when metrics are read in tension with each other.
| Pattern | Possible interpretation | Next question |
|---|---|---|
| Throughput rises and Verification Time rises | More changes are reaching a constrained review/validation system | Which verification component is accumulating time? |
| Throughput rises and rework/failure rises | Acceleration may be unstable | Which change types or cohorts create the downstream correction? |
| Review demand rises and Reviewer Concentration rises | Verification may depend on a narrowing group | Is ownership or specialist knowledge becoming a bottleneck? |
| AI usage rises while delivery outcomes stay flat | Adoption is increasing without demonstrated system gain | Where is time or effort being reallocated? |
| Verification Time falls while failures rise | Gates may be weakening or being bypassed | Which checks changed, and did coverage or review depth fall? |
These are diagnostic interpretations, not causal conclusions.
Rising throughput plus rising Verification Time does not prove AI created the bottleneck. It tells you verification is taking longer while output rises; then segment by AI provenance, repository, work type, PR size, or another relevant dimension.
A single "AI productivity score" is therefore a poor target. DORA's throughput/instability split and SPACE's multidimensional view support reading a portfolio of signals instead of collapsing the system into one number.
Measurement guardrails for the AI era
AI measurement should help teams improve the delivery system, not make individuals optimize for whatever number management happened to expose.
Five guardrails keep the verification scorecard useful:
- Measure the system before the person. Prefer team, repository, service, or value-stream units for review demand, rework, and verification.
- Never equate activity with productivity. Suggestions accepted, commits, lines changed, PRs opened, or reviews completed can be context. None is a standalone productivity measure.
- Publish denominators and windows. A rate without its population and observation period is difficult to interpret and easy to game.
- Pair speed with consequence. Faster coding, review, or deployment only becomes a stronger outcome when instability and corrective work do not rise with it.
- Use local baselines before universal thresholds. The verification scorecard is a framework. It does not come with defensible universal "good" values for these five metrics.
These guardrails direct attention to where work waits, how trust is established, where corrective work appears, and which system constraint needs more capacity or better automation.
What DevStats can observe and what still has to be derived
DevStats already exposes several signals that fit the verification scorecard, but the framework should not be confused with five prebuilt product metrics.
For review and flow, DevStats surfaces code review activity and PR size, and breaks PR cycle time into stages such as Coding, Pickup, Review, Merge, and, when deployment data is available, Deploy. It also surfaces throughput and activity signals such as merged PRs, opened PRs, reviews, commits, and code changes.
For AI-assisted work, DevStats can compare identified AI-assisted and manual PRs across delivery-flow signals such as cycle time and throughput. It also surfaces quality-related signals for AI-assisted work, including review patterns, bug-rate signals, and security scan results.
For downstream correction and stability, DevStats surfaces rework as part of its engineering analytics. When deployment and incident tracking are configured, it can also calculate delivery-instability signals such as Change Failure Rate and Mean Time to Recovery.
| Scorecard metric | What is observable | What still needs derivation or configuration |
|---|---|---|
| Verified Throughput | throughput, deploy context, incident/rework signals | definition of verified, observation window, and reliable outcome-to-change linkage |
| Verification Time | pickup/review/cycle-time stages | complete required-check timeline if CI/security/test completion is not represented |
| Review Demand per Active Reviewer | review activity and PR size | active-reviewer denominator and chosen demand definition |
| AI Rework Rate | AI-assisted/manual cohorts and rework signals | reliable join between AI provenance, deployed change, and rework window |
| Reviewer Concentration | reviewer participation distribution | chosen concentration method and stable analysis window |
The practical value comes from connecting AI adoption, delivery flow, review pressure, rework, and stability without reducing engineering performance to activity alone.
If your organization is trying to understand whether AI is improving the whole delivery system or just moving work downstream, explore DevStats.
A verification-first way to evaluate AI engineering
Ask one question:
Can the engineering system validate and deliver trustworthy changes at the same rate it can propose them?
Use AI usage as context. Read throughput against instability, relate review demand to reviewer concentration, and wait for reliable PR-level provenance before comparing AI-assisted and manual work.
When verification keeps pace and downstream correction stays stable or improves, a higher proposal rate supports a system-level gain. When queues, reviewer concentration, rework, or failures rise, the scorecard shows where the expected gain was absorbed.
That is what the verification economy adds to AI productivity measurement: visibility into the cost of establishing trust.