Mae West once said, “ You only live once, but if you do it right, once is enough.”
Why are we quoting an American actress from the 1930s in an article about Automated mobile app testing? Hold on. We are getting there.
In this Age of Apps, with an estimated 2 million apps on the App store and 2.9 million apps on the Google Play store, you don’t get a second shot at getting your customer’s attention.
So Mae West’s quote can be rephrased as “Your App only gets one chance, and you have to do it right because once is all you get .”If your app freezes or crashes, the number of customers who will give it a second chance drops to 79%, and, if you or your app is having a bad day and it crashes again, the number of repeat users drops down to a whopping 16%, according to this Compuware study.
Talk about the pressure of getting it right the first time. And this pressure is not just for the developers, but a group of people who always work in the shadows and ensure that the app is running smoothly before it goes into production — the testing team. The testers are also known as the ninjas in the world of software development, (one of our testers added this last bit)!
Although testing is seen as overhead by the customer and an annoying part of the development process by the developers, like taxes or traffic fines, testing is a necessity and very much a part of our lives. So deal with it.
Software testing has been around for a few years, but mobile app testing is a relatively new kid on the block and has picked up steam only in the past five years.
In this article, we will look at:
The Waterfall model for software development had a lot of takers early on in the industry, and, by the time smartphones infiltrated into our daily lives and changed the way we interact with each other, the Waterfall model was deeply entrenched in the development psyche.
For complete newbies into this space, a Waterfall model represents a form of Software Development Life Cycle where development flows smoothly, much like a waterfall.
So we have the entire development phase split into well-defined stages, and the project managers have a large amount of control over who is doing what task at which point of time.
For instance, there is the Requirement Gathering stage, the Design stage, the Coding stage, the Integration stage, the Testing stage, the Installation Stage, and finally, the Maintenance stage.
While this method of development gave a lot of structure to the process, in the case of mobile app development, it came with its own set of challenges.
For starters, the needs of the customer keep changing when it comes to a mobile app, and this is a complete no-no in case of the waterfall model. When you need your product to have additional functionality, like a “, Submit a review” button that you thought of at the last moment, the Waterfall model fails, and you can’t have the developer go back and change the code in an earlier stage.
Another issue with the waterfall model is that you can only see the completed mobile app in the end, and if the customer is unhappy with the output, the PM has no choice but to begin from ground zero.
As mobile apps grow more and more complex, the waterfall model is slowly beginning to fall apart, to be quickly replaced by Agile software development.
In Agile Software development, the entire development stage of the app is split into chunks called sprints, and the customer takes part in all the sprint meetings of the team, held every one or two weeks.
With this method, the development and testing happen in tandem, which means a lot of time saved for the customer. You write a piece of code, test it, show it to the client, gain overall feedback, and bam, you are done. You can now move to the next sprint.
Agile also meant that testing was now a collective responsibility, with not just the QA but the developers also sharing responsibility if the code is broken. In the agile model, finding bugs is easier, and since there is better communication between the tester and the developer, complex issues are nipped at the bud.
Since teams now work in sprints, incorporating changes to the product is easier in the Agile method. Customers are in the loop on how the development progresses, and there are no surprises when the final product is shipped.
With Agile, the build times are fast, and that means rapid deployment, which calls for faster testing methods.
For instance, imagine you are building a game such as Candy Crush Saga.
You need to ship new levels out every week, and if you don’t, your gamers are going to move out to the next copy cat game that promises an additional 15 levels.
Your development team pushes out the levels in time, and you need to try now and find the bugs in the new level before the end of the week.
This is where Automated Test cases come to the rescue.
With automated regressive testing, your team can test the new level for all possible bugs, while making sure the previous levels of the game are still stable.
Here’s a quick checklist on when you should use automated testing and when to go manual
For Repetitive Features — Go with automated testing
For Multiple Devices — Automated testing beats manual testing
For Simple Apps — Manual testing because you need fewer resources
While we have seen how Automated tests make the whole development process more manageable, adding fuel to the fire are the obvious limitations of manual testing a mobile app.
According to Steve McConnell’s book, Code Complete, there is an average of “15–50 errors per 1000 lines of derived code.” For a sophisticated mobile app with thousands of lines of code, this translates to a software tester’s worst nightmare.
Here are a few more reasons manual testing breaks in case of a mobile app:
We now know that manual testing falls flat on its face when tasked with testing a complex mobile app, thanks to all the reasons mentioned above.
The way out? Automated tests. Automating the testing of your mobile application helps you overcome challenges, such as:
Testing whether your application performs the way it was intended when a set number of users all use it at the same time is one of the most crucial tests you can perform on your mobile app. Automated testing helps speed up this aspect of the testing life cycle.
Consider this for an instant, a 5-minute downtime in 2013 at Google cost the company an estimated $545,000, according to this report by Coralogix.
So we can’t stress how important it is to have your app up and running all the time, and the way to ensure that your app meets the speed, stability, and scalability requirements are by subjecting it to rigorous performance tests.
Automating the performance tests means you can have your test cases run simultaneously from a large number of devices, something which is practically impossible if you are running a manual test.
Automating the performance test also means you can run Endurance tests ( ability to handle the expected load over a long period of time), Spike tests (handling unexpected spikes in the number of simultaneous users), Data volume, and Load tests relatively in a simple manner.
Functional testing helps to check if the app functions according to the design (duh!). In a functional test, all that you need to know is if the app is working the way you intend it to work.
In the real world, customers are not just running your app, but 15 different apps at the same time, expecting all of them to function without a glitch.
So you have a customer who is not just playing Candy Crush Saga but is checking his/her Instagram feed in between, texting a friend over WhatsApp, and attending important calls, all at the same time. Your app needs to stand the test of all this, and more.
With functional testing, you are not going into the deep layers of the app, such as the source code, but are just testing how user-friendly the app is.
Developers need a benchmark to test how every function, a “developer-free test,” if you may. Agile development means running tests on the features every day, and this, in turn, means automation.
So functional tests include:
We have been using big words and things have been getting a bit too technical. Let’s take a deep breath, step back for a moment. Now, try and understand why you need to automate your mobile app testing with an analogy.
We looked around the office to find inspiration to write this one, and we found one that every office goer or remote worker has in his/her workplace, chairs.
Imagine you are a manufacturer of chairs. The chairs can be anything ranging from black cushion ones with excellent lumbar support that makes you feel like you are sitting on a cloud or a simple plastic chair that is a sure-fire way to mess up your spine. It doesn’t matter.
All that matters is that you own a factory that makes chairs. Lots and lots of them. This is similar to developing an app that has a lot of lines of codes and different functions.
Your little chair factory makes a thousand chairs a day. Now you know that for every 100 chairs you make, two chairs are defective. A broken leg, back support that is messed up, etc.
This defect represents a bug in the app, a bug that your testing team is supposed to catch.
Now inspecting all the thousand chairs thoroughly to find the few defective chairs might be cumbersome, but can you automate this process?
Can you, for instance, buy a shiny new machine with laser technology that scans each chair in a matter of seconds and tells you if it is defect-free or not?
If your answer is yes, then it probably makes sense to automate it, depending on how established your factory is and if you can invest in a fancy laser machine that inspects chairs.
Again, this depends on the number of chairs that you are producing. If you are making eighty chairs a day, it doesn’t make financial sense to invest in a machine, since inspecting eighty chairs can be done by a couple of factory workers.
This is the same case with Automated testing. Investing in a complex automated testing framework that requires specialized knowledge to implement might not make a lot of sense if you are building a simple app with only a few features that you can easily test manually.
If you are a bit more niche player in the field, say building the Rolls Royce of chairs, you don’t need an automated chair testing machine.
This is because if you are an experienced carpenter, there is a high probability that the chairs you produce will be defect-free. Like a Rolls Royce.
But the probability that your developer is an expert carpenter is very rare (not literally, we are just connecting the analogy), and more often than not, you need to test your app for bugs.
If you are a player in the middle segment of chairs, which means you produce premium quality chairs that are mostly defect-free, then automation becomes a necessity. This is similar to an app that is used by more than 1000 users and needs to function flawlessly.
With a mid-range chair, there is an element of exclusivity, and when people pay for your chair (or app), they expect it to be defect-free. So you cannot afford to risk your reputation as a top chair builder (or an app developer) by hiring a few manual testers when the testing needs to be automated.
So we know that automated tests are the way to go if you have a big project with deep pockets and the resources to invest in the long run. But this does not mean that automated testing will completely replace manual testing of mobile apps.
(Yes, they do exist but no company will reveal them)
Automated app testing also comes with a small set of challenges that can only be overcome by manual testing. These include:
Despite these minor drawbacks, automated testing is still probably the best solution if you are planning to build a scalable, responsive app using the Agile model, and we would highly recommend automating your mobile app’s testing.