Background: Testing PyramidIn the quality world, we like to often refer to the testing pyramid. As Marty Fowler states in a blog post from 2012 (http://martinfowler.com/bliki/TestPyramid.html), the testing pyramid is a visual representation of a proposed test strategy, which follows the belief that the majority of automated tests should be performed at the unit level, followed by service level tests (or integration tests) and finally by GUI tests. Although Mr. Fowler doesn't include them in his blog post, I also like to include manual tests at the very top of the pyramid, since as much as I love to have automation provide me with a pulse of how my application is behaving in predefined situations, I don't believe that we can ever get away from investigating odd, complex scenarios, through the human touch.
|Testing pyramid by Scott Allister (https://watirmelon.com/tag/testing-pyramid/)|
|Testing pyramid by Martin Fowler (http://martinfowler.com/bliki/TestPyramid.html)|
For the past few years, I've motivated my team to work really hard on UI tests, as we came to a belief that they were the biggest bang for our buck. But wait, you may say, Maciek, isn't that the opposite of what the testing pyramid states? Well, yes, yes it is.
Having said that, we made a conscious decision due to our circumstances. Our team members (quality champions) needed to first learn how to automate through code (see my previous posts about Top Gun), while providing value. We came from a world where the majority of the team did not know how to write code driven automation and we wanted to not only provide immediate value to our business partners (through quality), but also to provide a career long skill to our team members. That is what code driven GUI automation has given us.
After two years of training on the job, we have created a team that runs and maintains approximately 1500 automated GUI tests, which are run on a nightly basis, in a lab environment that is not their own machine :) The tests are relatively stable and execute at a regular pass rate of about 85-90%. All of the tests are hand rolled, and follow a programming pattern. Our mission for covering the highest level of our testing pyramid (automated GUI tests) and teaching our mostly non technical QA team how to write programs for testing is nearly complete. Our team is now focusing on perfecting the craft of writing automated tests, and seeing that they have a reliable suite of tests to run at any time by anyone. It is really AWESOME.
Our team is re-focusing and starting to learn how to write integration tests within our environment and I'm confident that we will see that because of the skills picked up in GUI automation with respect to coding (in the same language that our application is written in (C#)), they will be able to knock out that level of testing much quicker than the GUI tests.
So with that left to the team, a while ago I decided that I needed to focus on figuring out our biggest mission yet.
Unit Testing: The Final Frontier
Pardon the Star Trek pun, but for our team, unit testing our main application has always been the elephant in the room. Our main business application is relatively old, and not super easy to test. About 6 months ago I started investigating why, and realized that our implementation of the framework we are using for business logic, doesn't allow for easy unit testing. We basically make use of a lot of private constructors, and methods, without interfacing. We are getting better at this, but need immediate solutions to provide unit test coverage.
I've heard all the arguments: Why don't you just re-write your application, or just interface everything, or even just plain out "don't do it. Tell your business that you will not do it that way". We know we have a problem, and are tackling it from different directions. I will not get into all of them right now, but would like to focus on what I think is the hardest one to fix: unit testing an application which cannot not expose public methods for all of essential pieces and does not have interfaces to be used for implementation for tests.
Enter Microsoft Fakes(https://msdn.microsoft.com/en-us/library/hh549175.aspx).
MS Fakes is a framework from Microsoft which basically allows you to isolate code for unit testing, at run time. Although fairly heavy, fakes allowed us to provide unit tests for our application which was not very test friendly. Our developers and some quality champions, have been successfully writing unit tests using this framework for about a year.
The above described framework gave us a way to unit test. It gave us the ability to start covering the unit test level of the testing pyramid. Some argue it is not the best unit testing framework, and that it is a very heavy handed approach. While I agree in theory, for us in practise it is the best that we can do for our current situation. And so, dear reader, I will summarize our current situation to you:
- Our quality champions are now writing fully automated, regularly running (scheduled) GUI tests
- Our quality champions are starting to write fully automated, regularly running (scheduled) integration tests
- Our quality champions (mostly our developers) are starting to write unit tests. And we've proved out that even though we are using a pretty heavy weight unit testing framework, it can be ran in an automated way, and measured to see how much progress we are making.
I think we've breached the last frontier of the testing pyramid, and are starting down the path of full stack test coverage. But to find out how we've figured out running MS Fakes based unit tests, you'll have to read the next post.
Stay tuned :)