Is our AI spend paying off?

We pay for AI coding assistants and leadership is asking whether it is worth it. I want an honest answer that goes from usage to delivery outcomes to money. ## Context - Assistants we pay for: [Cursor, Claude Code, GitHub Copilot, Devin] - Seats and monthly price per tool: [for example 40 Cursor seats at $40, 25 Copilot seats at $19]. DevStats does not know our prices, so use exactly what I give you. If I gave none, do not stop: run usage and outcomes, leave the money columns empty and tell me what to fill in. - Rollout date: [rollout date]. If I do not know it, use the first month with AI activity in DevStats and say so. - Scope: [all squads | squad names] - If I left anything in [brackets], keep what reads as a default (a period, a number of days, the first of several options) and ask me for the rest in one go, before pulling any data. - Write everything, the deliverable included, in the language of this prompt. ## Pull this data from DevStats 1. The AI Activity tool for each assistant, last 90 days: active users, acceptance rate, AI lines accepted and committed, share of AI code in commits, sessions, PRs authored by agents and how many merged, PRs reviewed by AI and suggestions applied 2. The same AI Activity per player (use List Players), to split people into heavy users and light or non-users 3. Delivery outcomes for two equal windows, the 90 days before the rollout and the last 90 days: Throughput, PR Cycle Time with stage breakdown, Code Review (PR size, comments per review), Work Breakdown (rework share), DORA Metrics (change failure rate) 4. Player Metrics for both windows, to compare the two groups of people ## How to analyze - ROI is a chain: usage, then outcome, then money. Most AI reports stop at usage. Do not. - Start with seat utilization: active users over seats paid, per tool. Unused seats are the only certain waste and the cheapest win. - Acceptance rate and AI lines prove the tool is being used, not that it helps. Never present them as a benefit. - Before vs after on its own is weak, because other things changed too. The stronger read is heavy users vs light users in the same period, each compared with their own numbers before the rollout. Say clearly that this is correlation: heavy users may already have been the fastest people. - Look for the cost of speed. PR size, review time, rework share or failure rate going up after the rollout means AI is producing more code that is harder to review. Throughput that ends up parked in the review queue is not a gain. - For agent-authored PRs, compare merge rate and time to merge with human PRs. A low merge rate is review burden without output. - Do not convert lines of code into hours saved. Use merged PRs per person per month or cycle time, against cost per active seat, and give a range instead of a single number. If the effect is not visible yet, say "not measurable yet". That is a valid answer. - Autocomplete, agents and AI review do different jobs. Do not compare tools on a single metric. ## Deliver A two-page memo for leadership and finance: 1. **Verdict** — one sentence: paying off, too early to tell, or not visible 2. **What we pay for and what is used** — table: tool, seats, active users, utilization, monthly cost, cost per active user 3. **What changed since the rollout** — five numbers, heavy users vs everyone else, with the caveats 4. **Side effects** — what got worse, if anything 5. **Recommendation per tool** — keep, expand, reclaim seats, or change how we use it, with the monthly impact in money 6. **What to measure next quarter** — to turn correlation into something closer to proof