Regression Testing Strategy: What to Run When You Can't Run Everything

Nobody decides on purpose to stop running the full suite on every PR. It happens gradually: the suite grows, the pipeline gets slower, someone adds a skip for "just this one time," and eighteen months later the team is running a subset by accident instead of by design. I'd rather see a team decide this deliberately than drift into it.

Risk-based selection, not random sampling

Tie test selection to what actually changed. If a PR only touches the billing module, running the full mobile onboarding suite against it isn't extra safety, it's extra time with no extra signal. Test impact analysis tools can map changed files to the tests that exercise them automatically; where that's not available, a manually maintained map of module to test ownership gets you most of the same benefit with more upkeep.

The full suite still has a home

It just isn't the PR gate anymore. A nightly run, or a pre-release run, is where the complete regression suite belongs, catching whatever the faster PR-gating subset was never meant to catch. The PR check is a first filter, fast enough that nobody's tempted to skip it, not the only line of defense.

Let failure history do some of the prioritizing

A test that's caught a real regression in the last year deserves to run earlier and more often than one that's been green for two years straight without ever failing for a real reason. That second test is either protecting something genuinely stable, or testing something that doesn't matter enough to break, and it's worth finding out which before deciding where it belongs in your pipeline.

I've done this on a suite where a meaningful chunk of tests hadn't failed legitimately in over a year. Moving them to the nightly lane instead of the PR gate cut gating time noticeably, and the number of regressions that made it further downstream didn't go up. The tests weren't wrong to have, they were just in the wrong lane.

Where full coverage on every run still matters

Not everything should get the risk-based treatment. Payment flows, auth, anything where a missed regression is expensive enough that the extra runtime is cheap by comparison, deserves to run on every PR regardless of what changed, because the cost of being wrong here is not symmetric with the cost of being wrong on a settings page nobody's touched in months.

If your pipeline has grown past the point where the full suite fits comfortably in a PR check, that's usually a sign it's time to make this decision on purpose. We cover the sharding side of the same problem in CI/CD pipeline optimization, and a lot of the flakiness that makes teams distrust partial suites traces back to the same root causes in flaky tests. It's the kind of audit we run in a test automation engagement.