Thursday, March 30, 2017

Setting Up Calabash: Debugging Page Models


Hey There! Thanks for visiting! This post is the 4th in the series focused on mobile automation, specifically Calabash. In the first two posts, we found out how to setup the Calabash framework for testing iOS and Android apps and why and how we should use the page object modeling pattern in our mobile tests. The third post dealt with setting an automation delivery channel via Jenkins and Xamarin Test Cloud.

This post is going to be focused on my process for debugging page models during test creation. We'll look at the Calabash console, and how to ensure your page model behaviors are performing the expected actions. We'll focus on an iOS page model, but the same principles can be applied to Android page models. If you need help setting up your page modeling project, or understand more about the page object modeling pattern for mobile testing, click on the links below to navigate to my previous posts!


  1. When starting to write a test suite, write the feature, then the step definitions then the page models
  2. Do not use and pre-canned steps or Calabash-Android or Calabash-iOS API steps in the step definitions
  3. Write iOS models first, enter accessibility IDs if possible. Copy structure of iOS page models to Android
  4. Debug page models by checking properties in Console (Android and iOS)
  5. Debug entire test by running locally, then on a different machine (ie. Xamarin Test Cloud)


1. This post assumes you have setup your project for page object modeling
2. This post assumes you have been able to create at least one-page model
3. This post assumes you have kept the tools outlined in the Environmental Pre-Reqs post below and can run Calabash tests.

Process For Writing Tests

First thing is first. Before we get into debugging, it is important to identify a good process for writing tests. This process is important to outline because it will identify pieces of the process that only have to be written once for both platforms.

1. Scenario (Cross Platform... ie. Write Once)

In Calabash page modeling, a scenario can be shared between iOS and Android. It specifies the business logic flow you are trying to test. A scenario is stored in the ".feature" file and controls the flow of the test for both platforms.

2. Step Definitions (Cross Platform ie. Write Once)

Step definitions match up with the scenario flow. This step identifies what specific actions are going to be taken. Notice, the step definitions do not use prebuilt in ruby steps, nor do they use API calls to the iOS or Android Calabash APIs. This is a very important detail, as it allows for true cross-platform re-use. 

3. Page Models (Platform Specific)

A page model implements the step definitions. This is where the fun starts. A platform specific implementation of an object uses the Calabash iOS or Calabash Android API to make calls specific to the platform. Because we've gone through steps 1 and 2, we can now focus on how to ensure the page model works appropriately for each platform, with very little re-work.

Page Model Composition 

A page model should be made of up two main sections. One for properties of the model, one for behaviors. The properties section defines how the elements of the page model are found, and the behaviors define what the page model can give a user access to. I find it easiest to initially identify the properties and behaviors which I am interested in for the immediate test, instead of trying to identify all properties and behaviors of a page. I think this works better because once I have the page model created and working for one property and behavior, it is easy to replicate for others as new test flows require. 

Page Model Property Identification

Each property on the page model has to be somehow found by the Calabash testing framework. I found it best to use the API filter "marked". As described in a Xamarin Calabash Query Syntax Example, this filter uses the accessibility ID property in iOS and the ID property in Android to filter elements. I believe that using accessibility IDs in iOS and IDs in Android is the best way to hook on to elements, but as described in the link I sent above, there are many other ways to filter (retrieve) elements. The key is to ensure you have the elements available for use. 

Debugging Page Model Properties and Behaviours

So now we know that we have to figure out what is available for us to use and then test that we are using it correctly. But how exactly do we do that? 

Enter, the console. The Calabash console was to me the most helpful debugging tool for iOS and Android. It can be started from a terminal window by running the command "Bundle Exec Calabash-IOS Console" or "Bundle Exec Calabash-Android Console". To launch your application, you will need to then run the command "start_test_server_in_background". Xamarin does a pretty great job of explaining how to use the functionality in their Calabash-Android Console Wiki Article and their Calabash-IOS Console Wiki Article.

The basic premise is this: The console will tell you in real time whether your property retrieval commands or behaviors work. 

Example 1: Check if you have access to a property
Let's say we wanted to see if the above property called "UILogin" was available for our use. IE. if called by the page model, it would be able to be acted on. In order to ensure we had the correct property, we would want to run a query on the application and ensure it retrieved the correct property. We would need to follow the below steps: 

Step 1: Launch Terminal
Step 2: Navigate to your project directory (ie. CD directory)
Step 3: Launch the Calabash console (let's assume iOS) using the command "bundle exec Calabash-iOS Console"
Step 4: Launch the test server by executing the command "launch_test_server_in_background", which for iOS would launch your default iOS simulator and then your application under test. 
Step 5: Ensure the screen that contains the property is on screen

Step 6: Run the command query "view UIButton marked:'UILogin'"

If our command returns the property, with its' available values, we have correctly identified the property! Otherwise, we should get a return value that specifies "false".  

We should repeat this step for all properties on a page model before we move on to behaviors.

Example 2: See if A Behaviour Works  
Similarly to how we need to check that a property can be retrieved, we need to check that the Calabash API functions we are calling, work within the behavior methods specified in our Page Model. The basic debugging flow is the same as for checking a property, but instead of focusing on ensuring our properties are available for action, we will now ensure that the actions we want to perform are correctly setup for the screen we are testing. 

Step 1: Launch Terminal
Step 2: Navigate to your project directory (ie. CD directory)
Step 3: Launch the Calabash console (let's assume iOS) using the command "bundle exec Calabash-iOS Console"
Step 4: Launch the test server by executing the command "launch_test_server_in_background", which for iOS would launch your default iOS simulator and then your application under test. 
Step 5: Ensure the screen that contains the behaviour is on screen

Step 6: Use the Calabash-iOS api to run the contents of a behaviour. For example if we wanted to see if a swipe with specific start and finish coordinates, based on the location of an element works: query "swipe :left, :query => "* marked:'UITutorial'", :offset => {:x => 100, :"swipe-delta" =>{:horizontal => {:dx=>500, :dy=>500}} }"

If our query command works, we should see the command's action (for example a long swipe) executed on the iOS simulator (or hooked up to the Android device). 

Debugging The Entire Flow (Local)

Once we've been able to debug the page model properties and behaviors, we should perform some full test flows. This can be accomplished by simple initiating the tests from the console by running the regular "bundle exec Calabash-ios" or "bundle exec Calabash-android" commands. I found that the majority of my debugging time was focused around figuring out how to find the correct properties and behaviors to use at the page model level. However, I did note that in some timing issues with respect to screen loads messed up my test flow when I ran it from start to finish. I found it very helpful to use the platform specific wait helpers (Calabash-iOS Wait Helpers, Calabash-Android Wait Helpers) to allow my test to keep up with the application flow. 

Final Test: Submit To Xamarin Test Cloud

Finally, after I got my tests to run fine on my machine, I submitted them to Xamarin Test Cloud, for a final check. After all, an automated test should run with the same results on multiple machines, right? Submission to Xamarin Test Cloud required a bit of customization from the base Xamarin guidance for Calabash test submissions to Xamarin Test Cloud. Specifically, I needed to specify which profile I was going to use (ie. Android or iOS)

An example of a command I used from the terminal:

 test-cloud submit /Path/To/AppUnderTest.ipa  mYaPpGuiDfRoMxTc --devices dEvIcEiDfRoMxTc --series "TestSeries" --locale "en_US" --user --profile ios --config /location/of/cucumberyml/on/my/machine/cucumber.yml

I would then see the results listed in the terminal, of the run executing and call it a day if it passed!

So that's basically it that's how I debugged my tests. Easy right? ;) For most of the time during the debugging flow, I felt like the dude at the top of this post :)


No comments:

Post a Comment