Part 4: Results Of The Efforts Top Gun Kicked Off
1. As a result of focusing on changing a mindset, which was kicked off with the Top Gun training, we have created automation focused ninjas that created over twenty thousand tests that are executed on a nightly basis.
2. As a result of focusing on changing a mindset through the Top Gun training, we have created a support army (of Devs, business analysts, etc.) for our automation focused ninjas that help traditional QAs with writing automated tests.
3. We are not done with creating armies of testing ninjas. We understand that the quality struggle is real and we have to evolve as the world around us does. To be effective Quality Champions, we need to be involved in all aspects of testing, from UI automated tests to unit tests, to performance and security tests.
4. If you want to create armies of ninjas, you need to start with training one ninja. Change is hard, excuses are easy. Dedicate yourself to taking the first step, whatever that may mean for your team.
5. Chances are that you've made it through all the blog posts in this series, and as a reward, I'd like to share with you all the material with respect to running the Top Gun training. From lesson plan and quiz examples to powerpoints and evaluation examples. Please peruse it on my Github page, and feel free to use what you want to design a test automation training of your own :)
Was It Worth It?
As you may have noticed in this series of blog posts, our team has gone through a lot of trouble to focus on ensuring test automation is our primary method of testing. But what have we achieved? What outcomes have we seen? What does the road ahead look like?
|Courtesy of Martin Fowler's website: The basic premise, the more your automated tests are focused on the UI, the more expensive and slow they will be.|
It is a representation of the percentage of time and investment a team spends on different types of tests and the return each layer of testing provides. I like it because it paints a good picture of roughly how much effort is going into automating tests and into which levels of test automation. We will not focus on the details but on the holistic picture. 4 years ago, when we started talking about automation, we literally ran one script, which had to be babysat and "spacebarred" through (ie. the space bar key had to be hit at critical points in the test to keep it going). I know, because I watched in amazement as this script was described to me as our "automation". As of today, our team has over 20,000 tests...that run...EVERY NIGHT.
Reducing Daily Risk
We run 514 automated tests that test the user interface, 2,867 automated tests that test the logic layer through user flows (integration tests) and 16,153 unit tests, which test very granular pieces of logic, such as individual methods, or custom properties on classes.
|Our current test automation test pyramid make up|
We communicate our results on a daily basis through an automated email, have set up dashboards for individual teams and even are working on bots to do the work for us.
|Dashboard for test results for one of our teams|
When we see consistent failures in our automated test results, our quality champions act immediately, as they understand their teams and leadership care about the results.
Because of the above activities, we find issues early on in our cycle and usually, not in production.
Test Automation For Everyone
By maintaining a central test code repository, we can tell who is checking in scripts and know that our focus on pushing for cross-functionality is working. In the last year, we've had 36 individual team members checking into our automated test suites, of which 22 were not traditional quality champions. Our team members do care about quality, and are practicing their skills in contributing to our efforts!
|Contributors to our test automation source control area in the last year|
The Road Ahead
Our journey in test automation began in 2014, and the last four years have brought us many lessons learned, struggles, and rewards...but we are not done yet. As software development changes around us, we have to as well.
While in 2014 the concept of automated tests was fresh and hot, now it is simply the ante to play. We are seeing DevOps push the envelope with respect to the speed of delivery, and testing has to keep up. We are seeing SecDevOps push the envelope forward with respect to security testing, and we have to involve our selves in those efforts and don't forget about performance, and load tests :)
As you can see, our mission is not simply to learn something new, the real mission, is to continuously learn something new, and be comfortable with being uncomfortable.
While we still have a long way to the moving target of all appropriate things automated, we have made what I consider monumental strides in the right direction, and I think it is because of the process and investment which I described to you in the last few posts.
Changing mindsets is no easy feat, and it took us about 4 years, but you now have the opportunity to take our lessons, and learn from them!
To spread the quality butter "Simply" follow the following steps
1. Convince leadership the juice is worth the squeeze
2. Prepare and deliver a focused, purposeful, measurable training
3. Empower trainees by focusing them on performing tasks learned in training, and reward them for sharing the knowledge learned to empower other team members to learn as well.
4. Struggle, encourage, fail, change, repeat until goals are met...then set higher goals ;)
What Is Your First Step?
The above probably seems like a lot of to do....because it is. But the first step for you or your team is to make a conscientious decision to do it. Change is hard. Technology is hard. Excuses are easy.
Take the first step, acknowledge the fact that you need to focus on quality, by focusing on the hard problem that is instilling a culture of automation first testing. What to do depends on how far along your team is, but the attitude change, which arguably the most necessary piece of your transformation, starts with you.
|As Jocko Willink of "Extreme Ownership" advises, do not let the moment pass. Start your team's automation journey, today.|