Review of Best Mobile Automated Testing Tools

Every application development team recognizes the value of thoroughly testing an application for quality assurance prior to deployment. Overlooked bugs can cost companies thousands of dollars. Moreover, releasing broken builds and features can frustrate and alienate the users and clients. Manual testing can be useful in a lot of cases, but it is a slow and human resource-intensive process.

Automation

Mobile Automated Testing on the other hand, can speed up the whole testing process by expanding your test coverage and, as a result, providing better clarity and high quality code. Automated testing can save the project owners a great deal of time anmoney, as it only requires a fraction of the time needed for manual testing. That’s why every QA engineer is sooner or later faced with the question of how to automate their workload.  The next question is how to choose the right automation tool for the project at hand.

Native applications are the ones that are written using the iOS or Android SDKs. Mobile web applications are web applications that are being accessed using a mobile browser. Hybrid applications are defined by a wrapper around a “webview” — a native control that enables interaction with web content. So, these criteria need to be taken into account during the evaluation process.

Our goal was to create automation test coverage for both iOS and Android native applications. Our team was previously involved in automated testing of a web version of this application using RSpec, Selenium and Capybara. Therefore, we acquired experience working with these tools. The main factors during the tool selection process were: ability to use this tool for automating both iOS and Android tests, reliability, ability to work with simulators and emulators as well a work with real devices. The tool had to be well documented, easy to set up and maintain, ad we really wanted to use the latest technology.

Commercial Automation Options

Most available frameworks have some sort of limitations in object/image/text recognition or in object instrumentalization. Usually the open-source frameworks do not have a unique solution for platform independent automation, which is why we chose to look into commercial products. The most popular commercial mobile cloud solution are Devices Anywhere and Perfecto Mobile.

These tools demonstrated the best performance of automation on real devices. Both of these tools use their proprietary scripting language which is more of a pseudo code than a programming language. However,  they also support many other programming languages. So if the goal is to have a unique platform and application-independent solution without any limitations – which mobile cloud solutions can offer – the user can definitely use these two tools

Open Source Solutions

There are a few most popular open-source mobile automation tools that specialize in iOS or Android. These include: Frank (iOS), UIAutomation (iOS),  KeepItFunctional (iOS),  Robotium (Android),  UiAutomator (Android),  Selendroid (Android) and others. For our project we were looking for a multiplatform tool. And there were three of them that met our criteria: Calabash, Appium and MonkeyTalk.

Appmium

At first Appium looked pretty promising – it works with both iOS and Android and helps test native, web and hybrid applications. However,  Appium is only good as long as the QA tests using simulator or emulator. If the user would like to test on real devices, then Appium  has very basic support for real device testing. It requires to use a source code which is not always available. Appium’s concept is really nice but it is being undermined by poor documentation, outdated examples and some hybrid app issues (selendroid dependency – there were reported issues with coordination between Appium and Selendroid API). It is simply painful to set up locally (and we cannot only depend on Sauce Labs as one needs to debug the tests locally).

Monkey Talk

MonkeyTalk seemed to be more user friendly and better documented.  It provides a very flexible scripting, great recorder, iOS/Android/Hybrid applications testing in one tool, the ability to run on simulators/emulators/devices even without USB connection over Wi-Fi, ability to use Jenkins or any other Continuous Integration solution. Many people who used to work with this tool provided great reviews for it. However, we experienced too many issues while using it (crashes, instrumentation failures, and some of the better features being unavailable for iOS (on-the-fly instrumentation).

Robotium vs Calaba.sh

A lot of QA experts recommend to use a combination of two or three tools for mobile testing. For example, they strongly suggest using Robotium for Android native applications, but Calabash-Android has several advantages over pure Robotium:

• You don’t have to write your tests in Java – you can use a more flexible language (Ruby) which usually fits the tests better.
•  You can control your tests from a computer instead of a device – making interaction with the environment easier (eg. if you want to modify something on the server, etc)
• Calabash has “higher level” commands, so starting with the testing is usually easier
•  You can write your feature files in a natural language

Eventually the only one Mobile automated testing tool for mobile applications that met all of our criteria was Calabash. It is well-documented, easy to setup and maintain, gets updated regularly, supports iOS and Android, runs on devices untethered, and allows you to write tests that are readable by everyone.

To start development of the automated scripts using combination of Calabash, Cucumber and Ruby, the user should first have a better understanding of what Cucumber and Calabash are.

Calabash, Cucumber & Ruby

Calabash is cross-platform tool, that enables the user to write and execute automated-functional acceptance tests for mobile applications, supporting Android and iOS Native apps. It is open source and free, and has a company, Xamarin, backing and developing it. Calabash could be compared to Selenium WebDriver. However, it is important to realize that interacting with a web app from a desktop computer is vastly different than interacting with a native app using a touch screen. Calabash provides APIs that are geared toward native apps running on touch screen devices.

Cucumber

Here is an easy to use Android and iOS testing harness that works using Cucumber feature scripts. It is simple to  and has lots of power because it is a pure Ruby testing DSL. Ruby makes testing much easier than writing in Java. Java is a great language for high performance applications, but a poor language for writing test scripts, where compactness and simplicity trump execution speed. Calabash succeeds because it is simple and offers the full expression that comes from ruby and all the available ruby libraries.

Cucumber is a method of writing tests, described in “behavior driven development with elegance and joy.” A cucumber test is often two files, the “features,” (a human readable set of instructions, text files written using Cucumber jargon and saved with a .feature extension) and the “steps,” a set of code which implements the tests. For example, you might have a test like this:

Feature: Customer registration

Scenario: As a new customer I can register given I start the registration process

Then, you have step definitions, the code which performs these actions as a test. Your step definitions for this test might look like this:

Touch “button” is Calabash code to touch a button in your mobile app. These steps match your features and you can do more advanced things like have a Regexp which matches and extracts parameters you then use it in your steps. There is also a large library of standard steps which Calabash provides for you to do the most common things involved with interacting with a mobile application.

So, basically Calabash is a set of scripts and an API which sets up your mobile app on a real device (or emulator) and then runs though the tests you’ve defined. Then, Calabash takes your features and converts them to  the special commands and runs your tests.

Working with Calabash

When creating automated scripts for our application, we started out with installation of Ruby 1.9.3 with RVM, then we created a new gemset for automated testing and installed the calabash-cucumber gem:

cd /path_to_xcode_project_directory
rvm --rvmrc --create use 1.9.3@calabash
gem install calabash-cucumber

Then we generated features folders for both Android and iOS

calabash-android gen
calabash-ios gen

Then we integrated Calabash with our mobile application (we followed these native instructions https://github.com/calabash/calabashios/blob/master/README.md and

https://github.com/calabash/calabash-android/blob/master/README.md)

And after these easy steps, we focused on actual script development for our mobile Android and iOS native applications.

Here is the example of our scripts:

@common_test

Feature: Forgot Password feature

Scenario: As a valid user I can reset my password

• I agree with EULA
• I create a valid customer “C0”
• I open Forgot Password page
• I populate Forgot Password page as “C0” customer
• I send password on Forgot Password page
• I confirm sending password
• I read message that password is reset and close it

Scenario: I can not reset my password with invalid data

• I agree with EULA
• I open Forgot Password page
• I set “ssn” of “Customer” “C0” to “123450000”
• I populate Forgot Password page as “C0” customer
• I send password on Forgot Password page
• I confirm sending password
• I read warning message that password is not reset and close it

Given(/^I open Forgot Password page$/) do

macro 'I wait for the "Forgot Password?" button to appear'

macro 'I press "Forgot Password?"'

macro 'I wait for 4 seconds'

macro 'I wait for "title" on "ForgotPasswordPage"'

end

Given(/^I send password on Forgot Password page$/) do

macro 'I wait for "send_password_button" on "ForgotPasswordPage"'

macro 'I wait for 4 seconds'

macro 'I press "send_password_button" on "ForgotPasswordPage"'

macro 'I wait for 2 seconds'

endv

Given(/^I populate Forgot Password page as "(.*?)" customer$/) do |customer_object_name|

macro 'I enter "email" into "email_address" field on "ForgotPasswordPage" as "' + customer_object_name + '" customer'

macro 'I wait for 2 seconds'

macro 'I enter "first_name" into "first_name" field on "ForgotPasswordPage" as "' + customer_object_name + '" customer'

macro 'I enter "last_name" into "last_name" field on "ForgotPasswordPage" as "' + customer_object_name + '" customer'

macro 'I enter "date_of_birth" value into "date_of_birth" "date" field on "ForgotPasswordPage" as "' + customer_object_name + '" customer'

macro 'I enter "ssn4" into "ssn_last_four" field on "ForgotPasswordPage" as "' + customer_object_name + '" customer'

end

Given(/^I confirm sending password$/) do

macro 'I wait for "confirm_button" on "ForgotPasswordPage"'

macro 'I press "confirm_button" on "ForgotPasswordPage"'

macro 'I wait for 2 seconds'

end

Given(/^I read message that password is reset and close it$/) do

macro 'I wait for "password_email_sent" on "ForgotPasswordPage

macro 'I press "ok_button" on "ForgotPasswordPage"'

end

Given(/^I read warning message that password is not reset and close it$/) do

macro 'I check "password_not_reset" message on "ForgotPasswordPage" and press "ok_button"'

end

Given(/^I go back to Log In page$/) do

macro 'I press "go_back_to_login_page_button" on "ForgotPasswordPage"'

end