Hey All, real john for a minute. No card deck today (I know shocking right?)… hold on. Ok, 8 of Diamonds, spiderman deck from Theory11. Anyway, we are measuring so much and using AI to generate even more, but there was a book a while ago called, “Measure what matters”. The title says a lot, and assumes even more. Do we even know what matters? This week, I talk about making numbers look great, but also making numbers look real. Let’s get into it.
A few weeks ago, I pulled up a dashboard I’d built to feel good about myself.
It was tracking AI assisted output across a few projects I’m close to. Pull request open commits merged into feature branches, lines of code touch per week. Every number was up and to the right. I remember sitting there thinking, This is it. This is the velocity everyone’s been promising. I almost took a screenshot to brag about it.
Then I asked a different question. Not “how much code did we generate this week”, but “how much of it is actually running in front of a customer right now.” And the second number didn’t move nearly as much as the first. Some weeks it didn’t move at all.
The gap is the whole story, and almost nobody’s dashboard is built to see it.
The number nobody’s tracking.
Circle CI ran the math across 28 million workflows this year, and the pattern is almost embarrassing once you see it stated plainly. Feature branch throughput, the AI, the stuff AI is generally great at the drafts, the scaffolding the first pass is up over 15 %. Main branch throughput, the code that’s actually validated and shipped to real users is down nearly seven percent industry wide. Only about 70 .8 % of main branches attempt even succeed. Well, under the 90 % you’d want to see. And when somebody breaks it in production, mid-size teams are sitting there for close to three hours on average before it’s recovered.
We didn’t get faster. We got a faster front half and a slower back half, and most of us are only measuring the front half.
I lived a version of this before AI ever entered the picture. Early in my career, I learned the hard way that we shipped a lot of code this sprint, and we shipped a lot of value this sprint are two completely different sentences. And leadership loves to conflate them because the first one is so much easier to put in the slot. AI didn’t invent that temptation, it just handed us a much bigger number to be tempted by.
What gets me that isn’t a tooling failure, quad code, cursor, whatever you’re running. None of that’s a problem. The problem is that we built our review processes, our CI pipelines, our sense of what done means for a world where code was the bottleneck. It isn’t anymore. Judgment is. And judgment doesn’t scale just because generation did.
I think about this every time I look at what I built for cash critters. 50 bucks a month, mostly solo. And the thing that keeps it alive isn’t that I write code fast. I do now faster than I ever have. It’s that it I still personally decide what’s allowed to touch me. That decision doesn’t get automated. It can’t. The day I let it, I stop having a product and start having a pile of commits.
What to actually do about it.
If you’re leading the team right now, here’s where I’d start putting attention, in order:
Stop reporting velocity by itself. Pair every PR’s open or commits merge metric with a ship to production number in the same breath. If you can’t say both in one sentence, you don’t actually know how you’re doing.
Watch main branch success rates like it’s revenue. Below 85, 90 %, and you’re not moving fast, you’re accumulating a bill you haven’t opened yet.
Time your recovery, not just your deploys. How long from something broke to its fix tells you more about your engineering maturity than anything else on the board.
Give someone explicit ownership of the merge decision, not a committee, a person. AI can draft the code, a human still has to decide it’s ready to matter.
Ask your team out loud, “what shipped this week?” Not what was written, what shipped. Watch how long the pause is before someone answers.
None of this is complicated. It’s just unpopular because we’re generating more feels a lot better in a board meeting than we’re being more careful about what goes out the door, but careful is what actually compounds.
The tools got faster. That part’s real. And I’m not here to tell you to slow down on purpose. Constraints are a feature, but manufactured ones aren’t the same as real as the real ones. What has to get faster now is our judgment about what’s worth shipping, because that’s the part that never got automated and never will.
Measure what all actually reach the customer. Everything else is just a number that makes you feel good in the meeting.
Go build something amazing, and then make sure it actually ships.
Next Week: an autonomous AI agent broke into a prodcution system in five days flat, and nobody could agree on whose job it was to stop it. That one’s coming.
John Mann is the founder of Startups and Code LLC, a software engineering executive, and the guy who built Casht Critters for $50/month because contraints are a feature, not a bug. Subscribe for weekly takes on AI, startups, and building things that matter.