Chaos Engineering for QA: Testing What Happens When Things Actually Break

The incidents I get called into after the fact almost never trace back to the main feature breaking. They trace back to a downstream dependency degrading in a way nobody had actually tried on purpose before it happened for real. The main feature was fine the whole time. Something it quietly depended on wasn't, and nothing in the test suite had ever asked what would happen if it weren't.

Failure isn't just error codes

A well-behaved test double returns a clean 500 or a timeout exactly on cue, because that's easy to script. Real infrastructure fails messier than that: a connection that hangs indefinitely instead of erroring, a response that's half-written and then drops, a dependency that's technically up and returning garbage instead of an honest failure. Testing only the clean failure case, the one that's convenient to simulate, misses most of what an actual production incident looks like.

Start with fault injection, not full chaos

You don't need a program randomly killing production nodes to get real value here. Injecting a deliberate delay or a dropped connection into one dependency, in a controlled test environment, and checking whether your system degrades gracefully or falls over completely, gets you most of the benefit with a fraction of the risk and none of the production blast radius. This is a test you can run in CI, not an operational program you need a dedicated team to run.

What "graceful" actually has to mean

A fallback response, a clear error surfaced to the user, a retry with backoff, or a circuit breaker that stops hammering an already-struggling dependency instead of making it worse. Each of those is a testable requirement, not just a design intention someone wrote in a doc once. If nobody's written a test for the failure path, the failure path is unverified code, no matter how thoroughly the happy path is covered, and that's the same blind spot we've written about in code coverage: a line existing in the codebase and a line actually being checked for correct behavior are not the same claim.

Where this pairs with what you already have

If you're already using service virtualization for microservices, you have most of the infrastructure needed to inject these failure modes already in place. A virtualized dependency that can return a clean response can just as easily be configured to hang, drop, or return garbage on command, which is exactly the scenario a real staging environment almost never produces for you on its own. It's part of what we cover in load and performance testing once the happy-path suite is solid and the real question becomes what happens when it isn't.