Testing on real devices vs simulators in mobile development

Special thanks to Alex S. and Adina V. for making this article possible with their technical know-how. 😃

When we set out to build products that people love to use, it was part of the job that we’d do the design, the code, and the quality assurance before launching any app to a store.

Today, we’re a team of 60+ people, and while the process has gotten heavier on the strategy side (after all, who wants to build a product nobody wants to use?), some things have stayed the same.

We’ve always integrated continuous testing into our mobile development practices. Embedding quality into every phase of the software delivery lifecycle is integral to making sure that the product people get to use does what it’s supposed to. Our testing team’s overall goal is to reduce risks for the founding startup and improve the quality of the application.

To this day, technical issues are the number one reason why users uninstall an app in the first 30 days of usage. One of our goals is to make sure that doesn’t happen.

At Tapptitude, testing is part of the project at any complexity

Whether we’re building a proof of concept, your MVP, or iterating on your already established product, testing is always part of the development process. In fact, as the product gets more complex, we work with you to develop testing processes that cover all requirements to establish quality checks. We believe that your product deserves the same attention to detail at every development stage. Every part of the app, every screen and every feature should be solid, so in the end, they all come together to build a product that people use without a hitch, getting value out of it and, dare we say, even some delight. The delivery of that value shouldn’t be hindered by the device the end-user is holding in their hand.

Many of the clients we’ve worked with have asked us to talk more about our testing processes and how we test on real devices and on simulators or emulators. We’re always happy to explain to them how each applies to their own case, and how each helps us in:

•  functional testing – where we test that the features behave as they are supposed to, and
•  non-functional testing – where we test that the software behaves as it’s supposed to on different devices and usage parameters.

Testing on real devices

The advantages

1. Testers can use real devices to test mobile applications in the real environment. This is the most significant advantage, as developers can observe the exact behaviour of the application, which the end-users would also see.
2. Just as we like to do, most testing teams try to have the latest releases and the most commonly used devices on the market, both for iOS and Android. We also test on unreleased beta versions for each OS.
3. Real devices allow the tester to see how sensors such as GPS, proximity sensors, life sensors, or force touch gestures behave during app usage.
4. Using the tested app on a real device also helps the tester to evaluate battery performance issues like battery drainage or overheating.
5. You can test incoming interruptions like push notifications, messages, or calls, or see how the app behaves while the app runs on standby or in the background while other apps are running. All of these are normal behaviours a real user would also experience.
6. Testing on real devices is faster compared to emulator/simulator-based testing.
7. Real devices are useful for testing your application under less than optimal network conditions. How does your app respond if the network is dropping in and out, for instance? How much of the CPU are you going to use for your app? How much memory? How responsive is it in these instances?
8. The tester interacts directly with the device and can control the user interface in the same way the real consumer does (with gestures such as scrolling, double-tapping, pinching, and other gestures).

The disadvantages

It’s unlikely that any mobile development agency has all available devices in the market also available for testing. We usually test the most used versions of the OS. We don’t keep all devices that we own up-to-date so we can cover all of them.

Compared to iOS, where the number of devices is limited, Android testing is dependent on the high number of manufacturers that create Android devices and customised operating systems based on the open-source system.

• We test the compatibility on different devices, OS versions, screen sizes, and also manufacturers. We cannot cover them all, but we prioritise them based on real-world usage.
• Ensuring the availability of any specific Android device for testing is difficult. There are thousands of mobile devices out there, and maintaining a proper testing pool with a wide range of devices demands enormous investment.

Testing on simulators or emulators

The simulators run on Xcode (iOS) and the emulators run on Android Studio (Android).
Both of them serve to compile the code of the application.

The difference between the two is that Android emulates (replicates or reproduces) real mobile device software, hardware, and the OS in order to test and debug applications within another software/hardware platform, while iOS simulates (imitates or mimics) the internal behaviour of a device, but does not emulate hardware or work on the OS.

The advantages

1. It’s easy to set up for each platform.
2. It’s easy to choose the OS version (including the next unreleased beta version for each platform) and the screen size so you can test compatibility.
3. It’s a good fit to test user flows.
4. There’s no cost involved, as you don’t have to buy any real devices.
5. Emulators and simulators are better than real devices at debugging (finding functionality hiccups that would annoy the end-user). This makes them most suited for testing mobile apps in the initial stages of development.

The disadvantages

Both tester community opinions and our experience show that you might get some false negatives as you work with simulators and emulators.

1. As mobile emulators simulate both hardware and software, they are slow. As a result, the testing process becomes more time consuming.
2. Mobile simulators and emulators do not take into consideration essential factors like battery drainage, overheating, and internal conflicts with other applications. This may result in incomplete testing of applications, which are then unfit for release.
3. Simulators are not capable of identifying the cause of abnormal behaviours in an app. The tester can’t determine whether the unexpected behaviour is due to the simulator or a bug in the application itself.
4. Emulators and simulators are less suitable for testing mobile apps in the later stages of development when products become more complex.
5. Emulators are extremely slow and fail to match the fast speed of simulators.
6. There are some functionalities that simulators and emulators cannot mimic: features such as push-up notifications, incoming calls, or a device’s battery, camera, or sensors.

How QA testing is done in real life

A case study on Wellory

Wellory is a tech company on a mission to make personalized nutrition accessible for all. They were connecting beta users with nutrition coaches, and the main call to action was to share photos of their meals.

The users got coaching via messaging to build healthier habits based on what they were already eating. The initial scope of work was to build MVPs for both users and nutritional coaches, for both iOS and Android platforms. That’s four mobile applications. Wellory decided to launch on iOS and Android because they tested their core product for nine months before building custom apps.

We also built a web admin, which is where Wellory manages administrative tasks between coaches and clients and has dashboards set up. We’ve since built an additional web coaching app, which gives the coaches the flexibility to work on desktop or mobile, depending on their preference.

How we did the testing on real devices

While testing Wellory for the initial iOS release, we used a diverse list of iPhone models and software versions available in-house. We wanted to make sure the end-users would not face any issues using the app.

As more users joined the app, Wellory entered a new phase where we started development for Android users. That was the point where more complexity was introduced since Android devices come with particular specifics.

Only by installing the app in development on a device can we truly see if it operates as intended, if there are any compatibility issues, and if the app interactions meet the guidelines.

Since Wellory is a messaging-based app, real devices facilitate testing on different network circumstances. That allows us to make sure that the messages are delivered when the connection is re-established. To test In-App Purchases, testing subscriptions required real devices with sandbox test accounts in order to investigate unpredictable results.

Another of Wellory’s functionalities is sending pictures to a personal nutrition coach and receiving notifications. We test device permissions manually, a feature that cannot be tested on virtual devices.

Another standard that cannot be tested on virtual devices is how users (and all of us, really) keep devices in Standby mode. We have to make sure that the app responds as expected while the device is idle or while other apps are running.

We use real devices for final regression check-ups before release to make sure that the app behaves as expected for the end-user. Real devices allow us to better assess apps, especially when it comes to performance and UX.

When we tested on simulators or emulators

As a general rule of thumb, the tendency is to use real-device testing at the advanced stages of the development and virtual devices while testing early in the cycle. For any product, end-users own a variety of devices and there’s no avoiding this reality. We use emulators and simulators for certain types of testing, which helps our teams maximise coverage. That’s one of the biggest reasons that we use them in conjunction with real devices.

For our project Wellory, we used virtual devices in UI testing on different screen resolutions, to find basic bugs that affect the layout of the app and certain internal hardware functions, since they are more suitable for debugging. Testing an app that made it into the top mobile app designs of winter 2021 also showcases our dedication to make sure that the app displays and works as it was designed for everyone, no matter their device or screen size. The project team compiled a series of different OS resources and screen sizes not available at hand, to align with targeted product audiences while testing faster and releasing high-quality apps with fewer defects.

Ready to build your first mobile app?

Let's talk about your idea

Conclusion

Emulators and simulators can both prove to be an optimal solution for mobile app testing, but only in the initial stages of development. The closer you get to launch, and the more complex your product becomes, the more you’ll want your testing team to rely on real devices as they build their testing plans. In the long term, this approach has proven to be more reliable and efficient.

Keep in mind that a good QA team will work with all the tools at their disposal as they develop their testing plans. They will include simulators, emulators, and real devices in their plans. But they will also look at the big picture: the audiences the product addresses, in order to compile the most used device types when it comes to operating systems and screen sizes.

There is no perfect recipe that will make sure your product will launch bug-free. Our commitment as your product partner is to look at:

• the needs of your product
• the stage you’re at in development
• the market you’re addressing and
• the resources you have

and offer you the best plan and the best recommendations there are to make sure the product you release will work at its best, on most devices, and most of the time.