Mobile Testing
Simulators Catch Most Bugs. Real Devices Catch the Ones That Matter
Simulators are fast, cheap to scale in CI, and fine for most of a mobile test suite.
The mistake is assuming they're sufficient for all of it.
What simulators get right
Most functional bugs don't care about hardware. Navigation flows, form validation, API integration, and state management all get caught by a simulator just as well as a real device, at a fraction of the cost and time.
- Fast enough to run on every pull request.
- Easy to parallelize across OS versions and screen sizes.
- No device farm scheduling or physical unit maintenance.
What only shows up on physical hardware
- Camera and sensor behavior. Simulators fake these inputs. Real lighting conditions and sensor noise expose bugs that never appear in a simulated feed, because the simulated feed was never noisy to begin with.
- Battery and thermal throttling. A feature that performs fine for two minutes can degrade once the device heats up under sustained real load, something a simulator that never actually gets warm can't reproduce.
- Network switching. Moving between Wi-Fi and cellular, or dropping to a weak signal, behaves differently on real radios than on a simulated connection that just swaps a flag.
- Push notifications and background execution. OS-level battery optimization and background app limits are inconsistent across real manufacturers and are hard to reproduce in a simulator that doesn't run the manufacturer's actual power management stack.
Running the same mobile suite on simulators and real devices, for everything?
That doubles run time without catching anything new, since most of that suite is already tested correctly on the simulator. We split coverage by failure type instead of by test count.
Why the split isn't just "run everything twice"
Running the same suite on both simulators and real devices, for everything, defeats the point. It doubles run time without changing what gets caught, since most of that suite is testing exactly the functional behavior simulators already handle correctly.
The split that actually works separates by failure type, not by test count. Simulators own CI. Real devices own the specific categories above, plus whatever devices and OS versions your actual user base is on, which usually isn't the newest flagship phone the team happens to have.
A practical split
Run the full functional suite on simulators in CI on every commit. Run a smaller, targeted suite on real devices before a release ships, covering the sensor-dependent flows, the older OS versions real users are still on, and any device models with a known history of problems.
That split keeps CI fast without giving up the coverage that actually prevents the bugs that show up in the field instead of in a pull request. This is the model we build mobile testing suites around, sized to how often a given category actually breaks rather than testing everything everywhere out of caution.