Mobile app testing pre-planning

It was already 4 years since Julia had started her career as a QA tester and she felt quite confident with any task.

At one of the regular planning meetings, the company Product Owner announced that their application was going to obtain its mobile version as the market needed it. Julia was assigned to be in charge of all testing activities for this new area of the application testing.
The first thing that she was expected to do was to create a list of possible challenges and solutions for the next meeting to discuss and create a strategy for testing. This was a new experience for Julia but she understood that the time demands these changes and this was a great opportunity for her to acquire new skills and knowledge. The research started.

During her research, Julia created the following report:

Challenge #1. The Biggest Mobile Testing Challenge is Devices

There are approximately thousands of different mobile devices through which our customer may want to access our mobile app. So, testing of our mobile application should take into account all these devices. Ignoring some of the devices equals to loosing numerous potential customers.

Possible solutions:

1. We can test exclusively using real devices;
2. We can test exclusively with emulated devices;
3. We can use a combination of each.

Real devices

Of course, nothing can substitute real devices in full extent to be sure of all potential situations and bugs. However testing with real devices is expensive, as we will have to buy all of them.

Emulated devices

The benefit of emulated devices is that we can switch among device models, simply loading a new device profile that displays our application the same way that the real device would.

The emulators were designed with testing in mind. They have full instrumentation to provide detailed diagnostics about the protocols that go back and forth between client and server at the various levels of the stack. Single platform with frequent updates of device profiles can be used to test any device on the market without buying it and this makes emulated devices very cost effective.

Real devices + Emulated devices

Unfortunately, emulated devices still lack some individual characteristics that only the real device have (providing of the pixel-perfect accurate rendering, performance and speed specific to different devices, sensitiveness to the ambient conditions that can impact the behavior of the device).

A good way out is a mixture of both emulation and real device testing. Early in the development cycle we can achieve most goals at a relatively low cost due to the high speed and device diversity that an emulator can provide and while we don’t need the pixel-perfect rendering of an actual device. The benefits are gained by increasing the number of test cases and device types covered in the test suite. Then we’ll test on real devices later in the development cycle to validate that the applications are functioning as expected and certify that all development requirements and objectives have been met.

Challenge #2. Regional Network Challenge

Every country has its specific network operators. Totally there are about 816 mobile network operators worldwide. Each mobile operator may support multiple network technologies like LTE, CDMA, GSM. Most network operators have inserted mobile Web proxies (or gateway) to regulate how, when, and if their customers are able to connect to a particular site.

Besides the technologies, there are also location challenges that are no less important. To be able to test the full network stack on a particular operator’s network infrastructure, we must be connected to the target network. We should also take into account that the radio signals are not strong on cellular networks. Thus, if we want to test against Telecom Italia, we must be in Italy, and if we want to test against China Unicom, we must be in China. But traveling around the world for the testing needs is not obligatory at all with the following options.

Possible solutions:

1. We can bypass the lower layers of the network and simply test over the Internet or LAN
2. We can use the real network by using either phones or modems.

Network Bypass

To use this technique we will need a device emulator, as most real devices are not capable of doing this.
We can use TCP/IP to connect directly to the server and ignore the GPRS tunneling systems used by network operators. We may need to check first if our device emulator can perform network bypass by using the Internet. It can be nice to look for device emulators that are able to access the operator’s proxy to perform a more realistic test. With the network bypassing we will not use and thus pay for airtime.

Real Networks

It is possible to test against real networks. One way is to use real devices at the target location, but again traveling is costly. There is another option – real device in the cloud. This type of solution still can be expensive because to the cost of the device we need to add the cost of the remote control hardware, remote control software, and local software. However, it is possible to rent testing time on a resource supported by a vendor that lets us access remote real devices at any time we need it. All we have to do is to open an account and buy testing time with a given model of device.

Challenge #3. Manual or automated scripting

We will need to choose the method that will used to actually execute the test scripts. Script execution can either be manual or automated. As there are many different mobile devices which have different structures, the automated scripting should be created so to be suitable for any device. A script with strict keystrokes on an iPhone will not be effectively performed on a Samsung mobile phone as their user interfaces are absolutely different. The good news is that most automated testing software provide high-level scripting functions such as “go to URL” or “send SMS” that are not dependent on the particular menu structure of the target device. Most device emulators can be used for automating test execution with a higher-level, abstracted scripting language that is not device dependent.

Cost of automated scripting is typically higher than the cost of a single manual execution of a test script, however, if run on a periodic basis it can really save time and money.

Besides, many automated scripting tools have a special capability that can test an entire site with a single command. This means a set up of quick test that will look at every page of our mobile app for errors and inconsistencies.

***
The next meeting Julia was armed with the above mentioned list of possible challenges and options of solutions. The discussion went productively and testers together with developers marked the skeleton of future testing plan. They made many compromises in selecting combinations of techniques. It was decided to search for device emulators for testing that have the instrumentation and the network options that can give a lot of flexibility, make sure that these emulated device solution contain a high-level scripting solution. In addition, it was agreed to search for a vendor to get access to remote real devices. Julia knew that new challenges would come, but none seemed impossible to overwhelm.