Setting Up Calabash: A Jenkins Build For Calabash Test Execution On Xamarin Test Cloud

Greetings and Welcome Back! 

In this third post for mobile tests, I will outline how to make a Jenkins build that submits pre-existing Calabash tests to Xamarin Test Cloud. The purpose of this post is to walk you through the process to provide an automated way of running your iOS and Android focused tests on Xamarin Test Cloud, on a nightly basis.

TLDR;

1. You need a working Jenkins server instance to run a Jenkins XTC Calabash Test Run

2. You will need an RBENV plugin configured on your Jenkins instance

3. You will need to specify the Ruby version as 2.3.1 (for now) in your Jenkins build, and setup a     Jenkins specific folder for RBENV on your Jenkins machine.

4. You will need to supply a custom bash script to specify what to supply to XTC (example below)

Pre-Reqs

You will need access to a functional Jenkins instance, and it helps to have administrator privileges, for the purpose of installing plugins. You will need to install the RBENV plugin before creating the build.  You will also need to create a new "Freestyle" Jenkins build.

Background

I decided to write this blog post due to the fact that I could not find any Xamarin Test Cloud guides to submitting Calabash tests to Xamarin Test Cloud using Jenkins. However, I used Jeffry's post as a starting point for setting up my build.

Setting Up A Jenkins Powered Calabash Test Run On Xamarin Test Cloud

General Section

Almost no changes from default. Give your build a good name that reflects the test app and the platform you are testing against.

Source Code Management

You will need to specify your test code repository. In my case, it is a Git repository. If you do not see your source code repo types available, it is possible your Jenkins instance does not have the necessary plugin. You will need to add it to Jenkins if your screen does not look like the screenshot below.

Build Triggers

Nothing crazy once again. In this section, you can specify when your build kicks off. I have mine set for a nightly kick off at 3 am ish.

Build Environment

This section is very important. First, the "rbenv build wrapper" checkbox has to be checked. Second, in the advanced settings for the rbenv build wrapper, it is important to specify the current XTC Ruby version, which is outlined in this blog post. Third, you want to ensure that in the preinstall gem list, you specify bundler and rake.

Finally, you have to ensure that you have created a folder in your Jenkins' machine $HOME location, to be used for the Jenkins RBENV bits. This is a dedicated folder for RBENV bits for Jenkins, so you do not want to leave it as the default RBENV location on your Jenkins machine. If this folder is not present your build will fail.

Build

This section is where the build script actually runs. As per Xamarin guidance, I've moved the build script to a separate bash script, and checked it into my test repository. So the screenshot below only shows the execution of the build script. Notice that I use an environment variable to CD into the script workspace on the Jenkins machine to execute the script.

Build Script

An example of my build script can be seen in the screenshot below, but there are a few sections worth writing about. First, it's imperative you specify where to pick up your .ipa or .apk from. I've specified a hard coded directory in my script ("APP_FILE") but it is possible to setup a Jenkins build variable to hook up your application's build output directory location.

Second, it is necessary to specify your test run details. The variables "TEST_SERIES", "LOCALE", "DEVICE_SET" control which tests you will be running, which locale you are running against and which devices you will be executing on. You also have to specify what your Xamarin Test Cloud user account is ("XTC_USER") and what your API key is ("API_KEY"). All of your Xamarin Test Cloud details can be obtained through initiating a manual test run on Xamarin Test Cloud. Details as to how to do that can be found toward the end of this article.

Finally, it is imperative to specify which platform specific pieces you need Calabash execution to load. This detail is specified through the Cucumber.yml and the profile flag. I set these two details through the definition of the "TEST_RUN_CONFIG" and "EXECUTION_PROFILE" variables.

If you've done everything correctly, you will see that the build runs, and in the console output, pulls your code from the git repo, downloads and installs Ruby, then the gems, then starts XTC execution. You will see test cloud execution start in the Jenkins console output. It is indicated by appearance of the XTC process, including dependency verification, digest calculation, upload, validation and finally a run.

It is worth noting that after the build is executed, a link to the results is located in the build console output, which gives a really nice way to get to a nice XTC dashboard, if you are already examining Jenkins builds on a daily basis. Alternatively, Xamarin Test Cloud sends email notifications of executed test runs, so you can be notified that way too.

And that's it! If you are interested in a specific build log example, one is posted here, but aside from that, thanks for reading and happy testing!

Links

Setting Up A Xamarin Build On Jenkins
Submitting Tests To Xamarin Test Cloud
Example Build Log