QA Metrics That Actually Matter (and the Ones That Don't)

"We have 2,000 tests and a 98% pass rate" sounds like a healthy QA process.

It tells you almost nothing about whether the product ships fewer bugs than it did last quarter. Both numbers are easy to report and easy to game, which is exactly why they're the two metrics QA dashboards default to.

Why test count and pass rate mislead

Test count rewards writing more tests, not better ones. A suite can double in size by adding shallow duplicates that check the same path from a slightly different angle.

Pass rate rewards tests that never fail, which includes tests that are too weak to catch anything, and quarantined flaky tests that got excluded from the count entirely.

Neither number tells you what actually matters: are bugs reaching users, and if they are, why did testing miss them.

Metrics that actually correlate with quality

• Escaped defect rate. Bugs found in production versus bugs found before release. This is the closest thing to a direct measure of whether testing is working, because it measures the thing testing exists to prevent.
• Defect density by area. Bugs per feature or module, not just a site-wide total. A flat total hides that one module is consistently the source of most production issues, which is exactly the module that needs more test investment, not an even split.
• Mean time to detect. How long between a bug being introduced and being caught, whether by a test, a review, or a user. Shrinking this number matters more than growing raw coverage, because a bug caught in an hour costs a fraction of one caught in three weeks.
• Flaky test rate. The percentage of test runs that fail for reasons unrelated to a real bug. A rising flaky rate is a leading indicator that the suite is about to lose the team's trust, well before anyone reports it as a problem. See flaky tests: root causes and how to actually fix them for what drives this number up.

How to actually calculate these

None of the four above need special tooling. They need a bug tracker with consistent tagging and someone who reads the numbers monthly, not quarterly.

• Escaped defect rate = bugs reported in production ÷ (bugs reported in production + bugs caught pre-release), for a given period. A team that catches 45 bugs before release and ships 5 to production has an escaped rate of 10%. Track that number over time, not as a one-off snapshot.
• Defect density by area = bugs tagged to a module ÷ some size measure for that module (story points shipped, or lines changed, if you want a rougher but faster proxy). The absolute number matters less than which module consistently comes out worst, quarter over quarter.
• Mean time to detect = the average gap between a bug's introduction commit (via git bisect or release tagging) and the ticket that reported it. This is the most annoying one to instrument and the most valuable. It's the number that tells you whether your test layers are actually positioned early enough to matter, which is the whole premise behind shift-left testing.

Code coverage deserves the same skepticism as pass rate

Code coverage percentage gets treated as a quality target more often than any other metric on this list, and it has the same failure mode as pass rate: it's trivial to inflate without adding real protection. A test that calls a function and asserts nothing still counts toward coverage.

100% coverage with weak assertions catches fewer real bugs than 60% coverage with assertions that actually check behavior.

Use coverage as a gap finder. "This error-handling branch has zero tests" is a genuinely useful signal, but it's not a target number teams should get evaluated against. The moment coverage becomes the target, teams optimize for the metric instead of the thing it was a proxy for, which is exactly what happened to pass rate and test count.

Where this connects to delivery metrics, not just QA ones

Escaped defect rate and mean time to detect aren't QA-only numbers. They feed directly into a team's change failure rate, one of the four DORA metrics engineering orgs already track for delivery performance.

A QA process that's actually working shows up as a falling change failure rate at the delivery level, not just a better-looking QA dashboard. If your QA metrics and your delivery metrics tell different stories, one of them is measuring the wrong thing.

A reporting cadence that actually gets read

A metric nobody looks at is worse than useless, because it creates a false sense of visibility. Weekly is too frequent for trend metrics like escaped defect rate and mean time to detect: a single release doesn't move them enough to mean anything, and reporting noise as if it were signal trains people to stop reading.

Monthly, with the trend line shown alongside the current number, is usually the right cadence for those two. Flaky test rate is the exception. Check it every release, since it can spike from a single bad merge and quietly compound if left for a month.

What to report instead of a dashboard of vanity numbers

Report escaped defects and where they came from, every release. Report defect density by area, so investment decisions are based on where bugs actually cluster. Track mean time to detect as a trend, not a single number.

Skip pass rate as a headline metric entirely. It belongs in a build log, not a quality report, because a green suite that isn't catching real bugs is worse than a suite that fails honestly.

Getting this right usually means rethinking what a QA process measures itself against, not just adding another dashboard. That's the kind of structural review we run alongside test automation and manual testing engagements.