Testing Feature Flags and Progressive Rollouts Without Losing Track of What's Live

Every feature flag doubles the number of states your application can be in. Ten flags in a codebase means, in theory, over a thousand possible combinations of on and off. Nobody tests all of them, and nobody should. The actual goal is testing the combinations that can occur together in practice, which is a much smaller and much more specific list than the combinatorial math suggests.

Flag hygiene is a testing prerequisite

A flag left at 100% or 0% for months after its rollout finished isn't harmless, it's an untested code path. The team stopped thinking about it as a variable, but the branch is still in the code, still reachable if the flag configuration ever changes accidentally, and nobody's regression suite is covering the "flag flips back" case anymore because nobody remembers it's still a possibility.

Every flag needs an owner and a removal date at creation time. A flag with no plan to be deleted becomes permanent conditional logic that nobody budgets test time for, which is a different problem than the one flags were introduced to solve.

What to test at each rollout stage

Flag off: this is your baseline. It has to keep passing the full regression suite, unchanged, for the entire rollout period. A regression here means the flag's code isn't as isolated as the team assumed.

Flag on, internal cohort: a full functional pass against the new behavior, the same depth of testing any new feature gets before it ships to anyone.

Percentage rollout: this stage isn't for finding new functional bugs, it's for confirming the rollout mechanism itself, monitoring for error rate and latency shifts, and having a verified rollback path. Treating a percentage rollout as more functional testing usually means the actual testing that matters here, watching real production signal, doesn't get the attention it needs.

Full rollout and flag removal: the flag and the old code path should be deleted together, not staggered. A flag left in the code after reaching 100% is exactly the kind of forgotten branch that causes the hygiene problem above.

The rollback test almost everyone skips

Turning a flag off should be as safe as turning it on, but it rarely gets tested as a scenario on its own. If the new feature wrote data in a new format, a shape the old code path doesn't know how to read, rolling back the flag doesn't roll back the data. That's a broken production incident waiting for the first time someone actually needs the rollback, usually during an outage, which is the worst possible time to discover it doesn't work.

Write the rollback as its own test case: enable the flag, let it write real data, disable the flag, and confirm the old path still functions against whatever state the new path left behind. This single test case catches a class of incident that otherwise only shows up in a postmortem.

Getting this right is part of the same discipline that makes shift-left testing work: catching the rollback failure in a test environment costs a few minutes, catching it during an actual incident costs a lot more than that.