• info@helpingtesters.com
  • helpingtesters

SUBSCRIBE TO OUR NEWSLATER





Types of Mobile Devices for Testing: Emulators, Simulators, and Real Devices

February 15, 2018
mobile devices for testing, different types of mobile devices for testing, emulators vs simulators vs real devices, emulators vs simulators, difference between emulators and simulators, difference between emulator and simulator, Mobile devices, cloud based mobile testing, cloud-based mobile testing

Once the mobile application development is complete, carrying out pre-launch testing is a crucial step before releasing it to the market. Testing the application on different types of mobile devices benefits the developers in many ways, few are:

  • Application is released defect free, hence there is less chance of recalling it from the market
  • Can confidently communicate to customers that it is the trust-worthy application for buying
  • Usability can be experienced much before the target users
  • Performance, Security, Reliability can be enhanced as the next course of action, after functionality and usability

The application has to be tested on all the highly-scoped devices in the market – Android phones, iOS, Blackberry phones, tablets, and iPads. But the question here is that, can we accommodate all such devices for testing? Yes, but EXPENSIVE. So, what is the alternative for such real devices? Now, the answer is Mobile Emulators and Mobile Simulators.

To be more precise:

  • Application could be easy to use on one device and might be bit difficult in another
  • Navigation could be easy with touch screen device rather than QWERTY keyboard device
  • User experience may vary with different screen sizes, resolutions, and touch sensitivities
  • Application behavior may change during different conditions of the device – battery and network bandwidth

Testing all these on all the different mobile devices is tedious is near to impossible. Say, you want to test the application on 100 different devices and the possibility is that only one device can be used for testing at a time. So now, how much time do you alone need to complete the testing on all the 100 devices? Do you really have such an adequate time to accommodate entire testing? Or even if you share the testing with 100 different testers (max), do all of them provide the same standard of results as you do? In such cases, it is preferred to use Mobile Emulators and Mobile Simulators for testing the application.

Mobile Emulators and Mobile Simulators are primarily the software programs that are designed to provide the replica of important features of any of the mobile devices, except battery and camera. Other features that are very difficult to mimic/replicate in Emulators and Simulators are the Incoming Calls and messaging interruptions and memory usage. These terms can be used interchangeably as they are very similar in nature.

Difference between Mobile Devices: Emulators and Simulators

Mobile Emulator based testing

Mobile Simulator based testing

Aims to mimic the external state of the mobile devices:

  • Device Software
  • Device hardware
Aims to mimic the internal behavior of the mobile devices:

  • Running application
  • Operating System / Software
Usually provided by the device manufacturer Usually provided by either the device manufacturer or some other companies
Written in machine-level assembly languages Written in high-level languages
More suitable for debugging Difficult to debug
Performance-wise it is actually slow, as emulating the hardware makes the software to sun slow Performance-wise it is very fast
Complete re-implementation of the original software Partial re-implementation of the original software
Example: Google’s Android SDK Example: Apple’s iOS Simulator

 

Advantages of Mobile Emulators / Simulators based testing and Real Devices based testing

Mobile Emulators:

  • Simulates both hardware and software aspects of the real device
  • Apps run unmodified – Mimics the device to the state that it gives the feeling of running the application on real device
  • Unexpected behaviors are easy to figure out
  • Free and Open Source
  • Inexpensive
  • Can be connected to IDE for early testing during development phase itself

Mobile Simulators:

  • Simulates only software and hence faster
  • Easy to set-up and use
  • Behavior of the application can be analyzed thoroughly
  • Unexpected behaviors are easy to figure out, but sometimes it portrays correct behavior as incorrect, i.e., false positives
  • Free and Open Source

Real Devices:

  • Provides real insights of the application under real environment and real conditions (battery, network)
  • Screen resolutions and brightness can be easily tested under different lighting conditions
  • Performance is very fast when compared to virtual ones as above
  • Interoperability testing is easier
  • Replicating issues are comparatively easier on the devices
  • All the correct behavior are considered as correct i.e., no false positives
  • Push notifications, built-in features like geolocation, camera, Calls, and SMS, etc, and built-in sensors like orientation, Compass, Wi-Fi connectivity, and Bluetooth detections etc, are privileged to test with the application.
  • Application’s issues with battery draining can be tested in real-time
  • Application correlation with other applications can be tested
  • Interruptions can be easily tested
  • Memory usage patterns with respect to application can be analyzed effectively
  • Application’s reliability can be highly assured

Disadvantages of Mobile Emulators / Simulators based testing and Real Devices based testing

Mobile Emulators:

  • Slow, as they mimic both hardware and software of the device
  • Battery draining / overheating or conflicts with other applications cannot be considered into account for testing
  • Setting up good and effective emulators takes time and expensive
  • They may be incompatible with application or application’s elements, sometimes, which needs to apply patches to application or purchase software patches to emulator until it is feasible to use on emulators
  • Only few Operation Systems are supported
  • Interruptions, push notifications cannot be tested
  • Suitable only for functional testing
  • Not suitable to validate performance of the application for longer period of time

Mobile Simulators:

  • Hardware is not taken into consideration
  • Way the applications run might be little different
  • Uses incomplete data, results generated are difficult to analyze
  • They may be incompatible with application or application’s elements, sometimes, which needs to apply patches to application or purchase software patches to simulator until it is feasible to use on simulators
  • Interruptions, push notifications cannot be tested
  • Suitable only for functional testing
  • Not suitable to validate performance of the application for longer period of time

Real Devices:

  • As there are thousands of mobile devices, creating the testing pool with all possible devices is difficult and becomes very expensive – Risks the profitability and viability of the overall project
  • Maintenance of the device is difficult
  • Time-consuming
  • Difficult to access the devices that are allowed only in foreign countries
  • Difficult to connect to IDE when early testing has to be started during development phase itself
  • In case of Mobile Web, every time user has to either manually type the URL to access the web page or bookmark once and use it every time or share it through possible other applications and use it every time through the application from which it was shared. All these make use of lot of memory space and impacts the overall performance

Mobile Cloud Testing: Solution to Mobile Emulators / Simulators based testing and Real Devices based testing

Even though there is the availability of Mobile Emulators / Simulators and Real Devices alone, we still need the best solution to use all the advantages and overcome disadvantages of Mobile Emulators / Simulators and Real Devices. So the best possible solution is Mobile Cloud Testing, which means testing mobile applications in the cloud. Mobile Cloud Testing allows the tester to access thousands of Mobile Devices with wide-range of Android and iOS emulators, simulators, real devices, multiple OS, and versions of OS directly from any browser. It eliminates the need to buy emulator, simulator, and real devices, but still enables to test the application through on them through the browser. It also helps to carry out:

  • Manual Functional tests
  • Automated Tests
  • Load testing
  • Performance testing
  • Monitoring mobile applications

Mobile Cloud Testing also accelerates Time to Market and Return-on-Investment. Business can be kept up-to-date with respect to customer needs and updated technology. Cross-functional teams (different mobile testing teams) located in multiple locations can work on the same project and same environment, resulting in transparency to view which tests were run altogether and which devices were used for testing. This helps in achieving high cross-platform compatibility for the application under test.

Conclusion

It is very important to decide/strategize which type among these mobile devices to use at different stages of testing. Emulator or Simulator or Real Device alone cannot be used for the entire testing phase. Combination of these has to be strategized strongly to maintain quality assurance and standards. In simple terms, it should be the Optimum combination of both. The focus should be on planning at which phase the Emulators / Simulators should be used and at which phase Real Device should be used. Again using the number of real devices depends on covering market needs and managing those.

Best Practices to follow:

  • Use Emulators / Simulators (and few real devices if needed) during the development phase, so that debugging the issues would become easier and eliminates maintenance costs at the later stages. Use Real devices once the application is ready to be tested and/or when there is a need for Sanity, Smoke, Regression, Performance, Interoperability, and network feasibility testing.
  • Developers to use Emulators / Simulators for faster execution during the development phase. Testing team to use real devices to ensure overall quality assurance goals and targets
  • Mobile Cloud Testing

Hello, this is Nandini.

About the author

Nandini KS editor

Hello, this is Nandini.

Leave a Reply

Your email address will not be published.