Code Coverage Is a Vanity Metric Unless You Read What It's Actually Measuring

I've sat in a release review where someone pointed at 92% coverage as proof a module was safe to ship. The one bug that shipped that release was in a branch technically inside that 92%, a line that had absolutely been executed during a test run, just never actually checked for the right outcome. Nobody had lied about the number. The number just wasn't measuring the thing everyone in the room assumed it was.

Execution isn't verification

A test that calls a function and asserts nothing about what comes back still counts as covering that function's lines. Coverage tools measure execution, not correctness, and they have no way to distinguish a real assertion from a function call nobody bothered to have an opinion about. The number goes up either way.

Where it actively misleads

Error handling is the classic gap. A try/catch block gets marked covered because the happy path ran straight through the try, while the actual catch, the part that only matters when something goes wrong, never executed once. High overall coverage sitting next to an untested error path is a specific and common combination, and it's exactly the kind of thing a single aggregate percentage hides on purpose, because it's an average, and averages are built to hide exactly this sort of unevenness.

What I'd rather look at

Mutation testing gets closer to the actual question: does a deliberately introduced bug actually break a test. If you flip a comparison operator or change a return value and every test still passes, coverage was lying to you about that line the whole time. It's more expensive to run than standard coverage, expensive enough that I wouldn't put it on every PR, but running it periodically against your most critical modules tells you something coverage alone never will.

None of this means the coverage number is useless. It's a decent early-warning signal for code nobody's testing at all. It's a bad late-stage signal for whether code that looks tested actually is, and that's the exact point where teams tend to over-trust it.

This is the same theme running through QA metrics that actually matter: the easiest number to report is rarely the one that tells you the most, and coverage is the clearest example of that in most teams' dashboards. If your test automation spend is being justified with a coverage number and nothing else, that's worth a second look before the next ROI conversation happens, and it's part of what we dig into in a test automation audit.