Thursday, March 30, 2017

Setting Up Calabash: Debugging Page Models


Hey There! Thanks for visiting! This post is the 4th in the series focused on mobile automation, specifically Calabash. In the first two posts, we found out how to setup the Calabash framework for testing iOS and Android apps and why and how we should use the page object modeling pattern in our mobile tests. The third post dealt with setting an automation delivery channel via Jenkins and Xamarin Test Cloud.

This post is going to be focused on my process for debugging page models during test creation. We'll look at the Calabash console, and how to ensure your page model behaviors are performing the expected actions. We'll focus on an iOS page model, but the same principles can be applied to Android page models. If you need help setting up your page modeling project, or understand more about the page object modeling pattern for mobile testing, click on the links below to navigate to my previous posts!


  1. When starting to write a test suite, write the feature, then the step definitions then the page models
  2. Do not use and pre-canned steps or Calabash-Android or Calabash-iOS API steps in the step definitions
  3. Write iOS models first, enter accessibility IDs if possible. Copy structure of iOS page models to Android
  4. Debug page models by checking properties in Console (Android and iOS)
  5. Debug entire test by running locally, then on a different machine (ie. Xamarin Test Cloud)


1. This post assumes you have setup your project for page object modeling
2. This post assumes you have been able to create at least one-page model
3. This post assumes you have kept the tools outlined in the Environmental Pre-Reqs post below and can run Calabash tests.

Process For Writing Tests

First thing is first. Before we get into debugging, it is important to identify a good process for writing tests. This process is important to outline because it will identify pieces of the process that only have to be written once for both platforms.

1. Scenario (Cross Platform... ie. Write Once)

In Calabash page modeling, a scenario can be shared between iOS and Android. It specifies the business logic flow you are trying to test. A scenario is stored in the ".feature" file and controls the flow of the test for both platforms.

2. Step Definitions (Cross Platform ie. Write Once)

Step definitions match up with the scenario flow. This step identifies what specific actions are going to be taken. Notice, the step definitions do not use prebuilt in ruby steps, nor do they use API calls to the iOS or Android Calabash APIs. This is a very important detail, as it allows for true cross-platform re-use. 

3. Page Models (Platform Specific)

A page model implements the step definitions. This is where the fun starts. A platform specific implementation of an object uses the Calabash iOS or Calabash Android API to make calls specific to the platform. Because we've gone through steps 1 and 2, we can now focus on how to ensure the page model works appropriately for each platform, with very little re-work.

Page Model Composition 

A page model should be made of up two main sections. One for properties of the model, one for behaviors. The properties section defines how the elements of the page model are found, and the behaviors define what the page model can give a user access to. I find it easiest to initially identify the properties and behaviors which I am interested in for the immediate test, instead of trying to identify all properties and behaviors of a page. I think this works better because once I have the page model created and working for one property and behavior, it is easy to replicate for others as new test flows require. 

Page Model Property Identification

Each property on the page model has to be somehow found by the Calabash testing framework. I found it best to use the API filter "marked". As described in a Xamarin Calabash Query Syntax Example, this filter uses the accessibility ID property in iOS and the ID property in Android to filter elements. I believe that using accessibility IDs in iOS and IDs in Android is the best way to hook on to elements, but as described in the link I sent above, there are many other ways to filter (retrieve) elements. The key is to ensure you have the elements available for use. 

Debugging Page Model Properties and Behaviours

So now we know that we have to figure out what is available for us to use and then test that we are using it correctly. But how exactly do we do that? 

Enter, the console. The Calabash console was to me the most helpful debugging tool for iOS and Android. It can be started from a terminal window by running the command "Bundle Exec Calabash-IOS Console" or "Bundle Exec Calabash-Android Console". To launch your application, you will need to then run the command "start_test_server_in_background". Xamarin does a pretty great job of explaining how to use the functionality in their Calabash-Android Console Wiki Article and their Calabash-IOS Console Wiki Article.

The basic premise is this: The console will tell you in real time whether your property retrieval commands or behaviors work. 

Example 1: Check if you have access to a property
Let's say we wanted to see if the above property called "UILogin" was available for our use. IE. if called by the page model, it would be able to be acted on. In order to ensure we had the correct property, we would want to run a query on the application and ensure it retrieved the correct property. We would need to follow the below steps: 

Step 1: Launch Terminal
Step 2: Navigate to your project directory (ie. CD directory)
Step 3: Launch the Calabash console (let's assume iOS) using the command "bundle exec Calabash-iOS Console"
Step 4: Launch the test server by executing the command "launch_test_server_in_background", which for iOS would launch your default iOS simulator and then your application under test. 
Step 5: Ensure the screen that contains the property is on screen

Step 6: Run the command query "view UIButton marked:'UILogin'"

If our command returns the property, with its' available values, we have correctly identified the property! Otherwise, we should get a return value that specifies "false".  

We should repeat this step for all properties on a page model before we move on to behaviors.

Example 2: See if A Behaviour Works  
Similarly to how we need to check that a property can be retrieved, we need to check that the Calabash API functions we are calling, work within the behavior methods specified in our Page Model. The basic debugging flow is the same as for checking a property, but instead of focusing on ensuring our properties are available for action, we will now ensure that the actions we want to perform are correctly setup for the screen we are testing. 

Step 1: Launch Terminal
Step 2: Navigate to your project directory (ie. CD directory)
Step 3: Launch the Calabash console (let's assume iOS) using the command "bundle exec Calabash-iOS Console"
Step 4: Launch the test server by executing the command "launch_test_server_in_background", which for iOS would launch your default iOS simulator and then your application under test. 
Step 5: Ensure the screen that contains the behaviour is on screen

Step 6: Use the Calabash-iOS api to run the contents of a behaviour. For example if we wanted to see if a swipe with specific start and finish coordinates, based on the location of an element works: query "swipe :left, :query => "* marked:'UITutorial'", :offset => {:x => 100, :"swipe-delta" =>{:horizontal => {:dx=>500, :dy=>500}} }"

If our query command works, we should see the command's action (for example a long swipe) executed on the iOS simulator (or hooked up to the Android device). 

Debugging The Entire Flow (Local)

Once we've been able to debug the page model properties and behaviors, we should perform some full test flows. This can be accomplished by simple initiating the tests from the console by running the regular "bundle exec Calabash-ios" or "bundle exec Calabash-android" commands. I found that the majority of my debugging time was focused around figuring out how to find the correct properties and behaviors to use at the page model level. However, I did note that in some timing issues with respect to screen loads messed up my test flow when I ran it from start to finish. I found it very helpful to use the platform specific wait helpers (Calabash-iOS Wait Helpers, Calabash-Android Wait Helpers) to allow my test to keep up with the application flow. 

Final Test: Submit To Xamarin Test Cloud

Finally, after I got my tests to run fine on my machine, I submitted them to Xamarin Test Cloud, for a final check. After all, an automated test should run with the same results on multiple machines, right? Submission to Xamarin Test Cloud required a bit of customization from the base Xamarin guidance for Calabash test submissions to Xamarin Test Cloud. Specifically, I needed to specify which profile I was going to use (ie. Android or iOS)

An example of a command I used from the terminal:

 test-cloud submit /Path/To/AppUnderTest.ipa  mYaPpGuiDfRoMxTc --devices dEvIcEiDfRoMxTc --series "TestSeries" --locale "en_US" --user --profile ios --config /location/of/cucumberyml/on/my/machine/cucumber.yml

I would then see the results listed in the terminal, of the run executing and call it a day if it passed!

So that's basically it that's how I debugged my tests. Easy right? ;) For most of the time during the debugging flow, I felt like the dude at the top of this post :)


Wednesday, March 29, 2017

Setting Up Calabash: A Jenkins Build For Calabash Test Execution On Xamarin Test Cloud

Greetings and Welcome Back! 

In this third post for mobile tests, I will outline how to make a Jenkins build that submits pre-existing Calabash tests to Xamarin Test Cloud. The purpose of this post is to walk you through the process to provide an automated way of running your iOS and Android focused tests on Xamarin Test Cloud, on a nightly basis.


1. You need a working Jenkins server instance to run a Jenkins XTC Calabash Test Run

2. You will need an RBENV plugin configured on your Jenkins instance

3. You will need to specify the Ruby version as 2.3.1 (for now) in your Jenkins build, and setup a     Jenkins specific folder for RBENV on your Jenkins machine.

4. You will need to supply a custom bash script to specify what to supply to XTC (example below)


You will need access to a functional Jenkins instance, and it helps to have administrator privileges, for the purpose of installing plugins. You will need to install the RBENV plugin before creating the build.  You will also need to create a new "Freestyle" Jenkins build.


I decided to write this blog post due to the fact that I could not find any Xamarin Test Cloud guides to submitting Calabash tests to Xamarin Test Cloud using Jenkins. However, I used Jeffry's post as a starting point for setting up my build.

Setting Up A Jenkins Powered Calabash Test Run On Xamarin Test Cloud

General Section

Almost no changes from default. Give your build a good name that reflects the test app and the platform you are testing against.

Jenkins General Section

Source Code Management

You will need to specify your test code repository. In my case, it is a Git repository. If you do not see your source code repo types available, it is possible your Jenkins instance does not have the necessary plugin. You will need to add it to Jenkins if your screen does not look like the screenshot below.

Jenkins Source Code Management

Build Triggers

Nothing crazy once again. In this section, you can specify when your build kicks off. I have mine set for a nightly kick off at 3 am ish.

Jenkins Build Triggers

Build Environment

This section is very important. First, the "rbenv build wrapper" checkbox has to be checked. Second, in the advanced settings for the rbenv build wrapper, it is important to specify the current XTC Ruby version, which is outlined in this blog post. Third, you want to ensure that in the preinstall gem list, you specify bundler and rake.

Finally, you have to ensure that you have created a folder in your Jenkins' machine $HOME location, to be used for the Jenkins RBENV bits. This is a dedicated folder for RBENV bits for Jenkins, so you do not want to leave it as the default RBENV location on your Jenkins machine. If this folder is not present your build will fail.

Jenkins Build Environment


This section is where the build script actually runs. As per Xamarin guidance, I've moved the build script to a separate bash script, and checked it into my test repository. So the screenshot below only shows the execution of the build script. Notice that I use an environment variable to CD into the script workspace on the Jenkins machine to execute the script.

Jenkins Build Section

Build Script

An example of my build script can be seen in the screenshot below, but there are a few sections worth writing about. First, it's imperative you specify where to pick up your .ipa or .apk from. I've specified a hard coded directory in my script ("APP_FILE") but it is possible to setup a Jenkins build variable to hook up your application's build output directory location.

Second, it is necessary to specify your test run details. The variables "TEST_SERIES", "LOCALE", "DEVICE_SET" control which tests you will be running, which locale you are running against and which devices you will be executing on. You also have to specify what your Xamarin Test Cloud user account is ("XTC_USER") and what your API key is ("API_KEY"). All of your Xamarin Test Cloud details can be obtained through initiating a manual test run on Xamarin Test Cloud. Details as to how to do that can be found toward the end of this article.

Finally, it is imperative to specify which platform specific pieces you need Calabash execution to load. This detail is specified through the Cucumber.yml and the profile flag. I set these two details through the definition of the "TEST_RUN_CONFIG" and "EXECUTION_PROFILE" variables.

Xamarin Test Cloud custom bash script

If you've done everything correctly, you will see that the build runs, and in the console output, pulls your code from the git repo, downloads and installs Ruby, then the gems, then starts XTC execution. You will see test cloud execution start in the Jenkins console output. It is indicated by appearance of the XTC process, including dependency verification, digest calculation, upload, validation and finally a run.

It is worth noting that after the build is executed, a link to the results is located in the build console output, which gives a really nice way to get to a nice XTC dashboard, if you are already examining Jenkins builds on a daily basis. Alternatively, Xamarin Test Cloud sends email notifications of executed test runs, so you can be notified that way too.

Xamarin Test Cloud notification

And that's it! If you are interested in a specific build log example, one is posted here, but aside from that, thanks for reading and happy testing!


Setting Up A Xamarin Build On Jenkins
Submitting Tests To Xamarin Test Cloud
Example Build Log