Monday, February 27, 2017

Setting Up Calabash: Page Modelling For Mobile Tests

Welcome back, this is the second post in the series dealing with setting up Calabash. The first post dealt with setting up your environment in preparation for writing the tests. This post will deal with setting up your test solution for page modeling, and some lessons I learned from my first-page modeling driven test.


1. Page modeling comes from Selenium but is a pattern used to model the behaviour of pages to code objects.
2. A specific Calabash directory structure has to exist for an XPlat (cross platform) page modeling approach to a test suite.
3. Cucumber.yaml controls the dependencies and platform specific flow of execution
4. Feature files contain plain language actions which can map to acceptance criteria of a story
5. Shared step files contain code that acts as an interface of how the plain language steps should be executed
6. Page models contain platform specific (iOS, Android) implementation of properties and methods used in shared steps. 
7. It's easier to write iOS tests first, but we need to stick to IDs as automation hooks and not break the page modeling pattern.
8. If we follow the page modeling approach, we can implement tests for the second platform in about 1/3 of the time of the first!

Page Modelling: What Is It And Why Should I Use It?

The page object modeling pattern originates from Selenium and has been widely adopted in GUI (Graphical User Interface) automation. A link to the original Selenium documentation can be found in the links section below. Page Object Modelling (POM) uses an object driven design to describe the behaviors of the page or screen in question. This type of design separates the test from the implementation and allows easier maintenance and higher reusability of code. This last point becomes very evident in the mobile space. Once written, two-thirds of a test which runs against one platform (ex. iOS) can be reused for another platform (ex. Android).

How Do I Set Up My Solution For Page Modelling and XPlat Execution?

The default calabash feature generation command ("Calabash-iOS gen") should set up your directory structure, without anticipating page modeling or cross platform (XPlat) execution. This means that you should end up with something that looks like below.
Default generated Calabash-iOS directory
But to prep our solution for XPlat (Cross platform) execution, we need to add a few things. Remember, the goal of page modeling is to reduce maintenance and increase code re-use.  We will need some shared elements and some that are platform specific.

Step 1: Generate An Appropriate Directory Structure

The XPlat page modeling approach in Calabash assumes that both iOS and Android tests execute a very similar test flow, and so share features and step definitions. You will see that any ".feature" or "_steps.rb" files will not be defined in platform specific folders. 

Your directory structure should follow the following breakdown
  • Features
    • android
      • pages
        • AndroidPageModel.rb
      • support
        • app_life_cycle_hooks.rb
    • ios
      • pages
        • iOSPageModel.rb
      • support
        • 01_launch.rb
    • step_definitions
      • shared_Steps.rb
    • support
    • features.feature
My test solution layout

Test Execution Flow

Upon executing a test command (for example "Bundle exec cucumber -p ios"), cucumber decides which pieces of the solution are necessary for inclusion, and then which test platform to execute against. We will now take a look at which pieces in the solution (directory) structure control this flow.

Important Test Execution Flow Concept: Cucumber.yml

Scope: Shared between iOS and Android
Purpose: Define which files to include in test flow execution
Details: This file uses the "-r" parameter to include files and directories in the compilation. It is important to include only the platform specific files for test flow execution. Ex. Do not include any "ios" specific pages in the android profile and vice versa. 

Important Test Execution Flow Concept: Feature File (X.feature)

Scope: Shared between iOS and Android
Purpose: Define feature flow in domain specific language. 
Details: This file reads in plain English, but needs to match up to step definitions. Each one of the scenario defined steps need to be matched to a step definition.

Important Test Execution Flow Concept: Shared Steps Definition

Scope: Shared between iOS and Android
Purpose: Define what each step means. Acts as an interface, which will later have to be implemented by the page models (individually on iOS and Android).
Details: The shared steps file acts as an interface which defines how the behaviors necessary to execute the actions from the feature file will be executed. For example, carrying out the action "Given the app has launched to the tutorial" means waiting up to 30s for the TutorialPageModel to load. 

NOTE: In order to maintain separation of concern with respect to platform specific implementation, all steps in the shared steps have to be custom written steps. A step definition file which is used for an XPlat approach cannot contain platform specific canned calabash steps.  

Important Test Execution Flow Concept: Page Model Files

Scope: Specific to iOS and Android
Purpose: Implement the platform specific actions that are defined in the shared steps.
Details: Each page object file carries out the actions that the shared steps defined through an implementation of methods. The page object files take advantage of the Calabash API (one for iOS and one for Android), to carry out the actions defined in the shared steps. The page object files need to inherit from their respective base classes (ABase, IBase) in order to be recognized as page objects.

Step 2: Write Shared Features

Shared features provide a plain English definition of how the application feature should behave. These steps have to be written in the "Given When Then" format, in order to define what the expected result of an action should be. Shared features should be able to be written very early in the lifecycle of the tests, and even the application under test. My team has seen success in educating the entire team, with respect to how to write proper acceptance criteria, for features in a "Given When Then" format, since those criteria translate very well into feature files in Calabash. The Agile Alliance has a great write up on using the "Given When Then" format for acceptance criteria to which a link is included in the links section. Additionally, since these criteria are written in plain English, any member of the team who is familiar with the purpose of the feature, can write them.

Step 3: Write Shared Step Definitions

As a reminder, shared step definitions provide an interface of what it means to carry out the actions necessary for the behaviors defined in the feature definition. This means that some code is written into this file, and it is not 100% readable, but at the same time, the code implemented here is still not platform specific. A shared step definition used for XPlat Page Object Modelled test suites requires the use of all custom steps, to ensure that no platform-specific steps that cannot be shared between both implementations exist. 

Step 4: Implement Step Definitions Through Page Models on iOS

During compilation of my test suite, I chose to implement iOS tests simply because a few of my friends from the team had already compiled tests for Android.

Choosing this approach, turned out to be a good one, as it saved me a bit of upfront work. I knew going into this project that I wanted to stick to the approach of using IDs for both platforms (iOS and Android) as automation hooks. For the iOS platform, Calabash uses the "AccessibilityID" property as the default automation hook. Now I don't know about your app, but mine did not contain many accessibility ids.

So after wrestling with XCode a bit, I was able to add them to my application under test and implement a pattern in my page objects of using the id field as the default automation hook. Because of this approach, in general, I was able to use the API call which looks for "marks" on properties retrieved by Calabash. This API call (documented here and referenced in the links section under the API Query Syntax) filters objects by ids, contentDescription or text. Simply, it looks for a wide variety of hooks and given I ensured to enter IDs on iOS it worked for that platform. The beautiful thing was that on Android because most IDs are essential to the application flow, I was able to maintain very similar page models, and just change the IDs!

IMPORTANT NOTE: Execution on iOS Requires Linking of Calabash To A Debug Version Of The Application Under Test!

I will mention this in a subsequent post about debugging and actual execution of the tests, but it is worth noting that before you can actually run Calabash-iOS tests, you will need to link the Calabash framework to your application. This is necessary because, at its essence, Calabash is essentially a person in the middle which is allowed to make calls to the application under test through explicit permissions. A tutorial on how to link iOS Calabash to your application can be found here and in the links section below. Without doing this, the tests will not run. 

Step 5: Implement Step Definitions Through Page Models on Android

In Closing...

Implementing the iOS page models (and steps, and features), debugging and testing them locally and on Xamarin Test Cloud, took about a week. 

"Re-implementing" the entire flow for Android...1 day. Mind == Blown. This process really sold me on the value of using proper automation hooks (ids) and page modeling. I know, I know, it doesn't seem possible. There's only one way to experience this joy...try it! 


Setting Up Calabash: Environmental Pre-Reqs (OSX)

Hey there Internet friends! It's me again! For the first quarter of this year, I've committed to helping a few of my team members with some mobile automation, and so the next few posts will be focused around that! So far, coming from a desktop automation background, I've learned a crap ton and solved a lot of quirky problems. I've learned a lot of lessons that I'd like to share with you, and selfishly document for my own future use :) Today's post will be focused on how to setup your environment, and which tools worked best for me. The goal was to get from zero to login tests, for iOS and Android, using Calabash...first step: Figure out how to setup the environment.


1. Get a Mac , it's best BRO..but really it helps a lot for debugging iOS and Android. Also if your team lets you, you can add accessibility IDs (automation hooks) via XCode!
2. Get a physical Android device. Dealing with emulators and Google services is a pain.
3. Get VS Code for editing. Light weight, extensible due to a great plugin model.
4. Sign up for a cloud testing environment and use it to ensure your tests work on devices that are not your own. 
5. Use RBEnv to install Ruby 2.3.1 (for Xamarin Test Cloud)
6. Use Bundler to install Calabash
7. Setup Calabash dir structure for iOS

Environmental Pre-reqs

Before we begin coding, I'd like to share with you my environmental setup. This is important information, as before we begin , we want to ensure we are setting ourselves up for success and minimal configuration. 

Get a Mac

Unfortunately, it's easiest to work with iOS and Android on a Mac. I'm not saying that OSX is bad, or I'm a Windows fanboy, it was just unfortunate that I needed to get another device, given the majority of non-mobile automation I wrote, was in Visual Studio for the Windows environment.

Having said that, while working with Android and iOS tests is theoretically possible on a Windows device, having a Mac gives you the flexibility to work with the source code of the application to do things like link the Calabash-iOS framework or add accessibility id's for automation hooks.

Worked for me!

Get an Android Device

Devices or emulators are necessary for script debugging. While Calabash-iOS works really well with the iOS simulator from XCode (another reason to have a Mac), emulating an Android environment is not exactly easy if you need to call into Google services. If your application does not make any calls to Google services (ex. maps) then an emulator could work for you. But to be safe, I advise you to get a physical Android device and use that for script debugging. It will save you a lot of configuration effort and environmental debugging.

Thanks for lending it to me Mobile Team!

Choose an Editor

Calabash scripts are usually executed through the console, but need to be written in an editor of some sort. Some of my colleagues like to use Android Studio, or Sublime, but I chose Visual Studio Code. Up until this mobile testing thing, I spent the majority of my time in Visual Studio proper, and found that the keyboard shortcuts translated nicely. Additionally, VS Code (Visual Studio Code) provided a rich plugin environment, through which I installed Ruby specific plugins that aided the dev cycle through things like IntelliSense. VS Code also installed very quickly and launches very quickly, so overall I really like it as an editor. Links can be found below.

Simple, fast, editor

Get a Cloud Testing Service...ASAP

As far as I'm concerned, an automation script does is not done until it runs on a foreign environment. Only then are you sure whether or not you've incorrectly incorporated environment specific variables, which allow the script to only run on your machine/device/etc. In order to prevent this habit from forming, it is essential to have access to a lab of devices. Some teams have access to physical device labs in house. My team has started using a cloud environment to perform the same role. 

A quick plug for Xamarin Test Cloud. It's relatively cheap (so far) (about $100/month with a 30 day trial) and performs really well. It is quick, and provides all the logging you'd want (device logs, script logs, screenshots). See the links section for a reference to their site. 

I cannot stress enough that before you consider a script complete, you need to ensure it runs on an environment that is not your machine! I run all mine on XTC (Xamarin Test Cloud). That step is part of my personal definition of done.

"worked on my machine" is NOT GOOD ENOUGH

Script Pre-Reqs

So now that we've discussed the environmental setup, we should get into some script specific setup pre-requisites. 


RBEnv is a Ruby version manager. It is important to realize that on Mac OSX, Ruby is installed by default, but the version of Ruby by default impacts your entire system, so it's not recommended that you mess with it and its' Gems. Enter RBEnv. Through the use of this tool, you can control which version of Ruby you are running per each environment you are working with. By installing a setting a specific version of Ruby for your environment when developing scripts, you are ensuring that the version specified is what is used for your Calabash environment. This is important because it keeps your version of Ruby for Calabash and its' Gems separate from your main Ruby install. This separation ensures that if you mess up and uninstall or accidentally upgrade Gems in your Calabash environment, you will not impact the rest of your machine. 

Additionally, some script specific dependencies (such as Xamarin Test Cloud) run a specific version of Ruby (2.3.1), so you will want to control which version is installed for the environment to stay consistent. I've provided a link on RBEnv setup from the Xamarin website below. Please note, in the link that describes the install of RBEnv and specific Ruby versions, there exist two ways of installing Ruby for Calabash.. a "non Experienced dev" and "Experienced Dev" path. Unfortunately after trying to figure out why the non experienced dev way was not working for me (Calabash Sandbox), and a bit of googling, I realized that that way is no longer working and will not be supported going forward. So you have take the hard way. Once you've completed the RBEnv Ruby setup, you should be ready for Bundler.

Sorry...easy way not supported anymore :(


"Bundler provides a consistent environment for Ruby projects by tracking and installing the exact gems and versions that are needed. Bundler is an exit from dependency hell, and ensures that the gems you need are present in development, staging, and production. Starting work on a project is as simple as bundle install." (

Bundler is Ruby dependency manager, and an easy way of ensuring that all of the environments that your project is running in (your machine, cloud environment, Arthi's machine, Cynthia's machine) are using the same Ruby Gems and the same versions of those Gems. Given you have Ruby installed on your environment, Bundler is really easy to install. The instructions are on the front page and consist of running one command in the terminal (Mac). 

The dependcies for your project are controlled by a "Gemfile". The Gemfile has to reside in your project directory, and contains the Ruby Gems necessary to be installed for your project to be executed successfully. I've provided a link to the Stackoverflow article I used to figure out how to create a Gemfile. I've also provided the Xamarin article I used to create the contents of the Gemfile for Calabash. 

Gemfile example

Setup Calabash

Now that we installed the proper Ruby version using RBEnv, installed Bundler, and created a Gemfile we are ready to setup our Calabash project. The first thing we need to do, is install the necessary Gems. This is very simple to setup. Assuming you are in the directory and a Gemfile exists, all you have to do is run the "bundle install" command in the console. The output of the command should show you which Gems were installed. 

At this time, the last thing to do is to create the Calabash directory structure for the tests. Run the Cucumber command "Calabash-ios gen" to create a skeleton for tests. This command should create a directory structure which includes a feature folder, and some others. If the command successfully executed, then you should see the feature folder in your test directory.  

Calabash Test Directory Structure

Now that we've learned what we need to get up and running with Calabash on iOS and Android, we can start getting familiar with the solution structure. That is when things start to get interesting :)