Tuesday, December 29, 2015

Upgraded To Visual Studio 2015: What It Meant For Me.


In a previous post I talked about the benefits of VS 2015, and that I was ready to take the plunge. Welp, my hand was forced about three weeks ago. Our build server was upgraded to TFS 2015, which meant that it was the perfect time to upgrade my IDE as well. I did about two weeks ago.

As part of the upgrade testing, I wanted to ensure that I could effectively write a test in VS 2015, compile it, check it in, associate it with a test case, and be able to run the test case in the test lab environment.

The Problems

GUMP: I needed to update a crap ton of dependencies. At the end of the day, automated test cases rely on multiple assemblies to run. For example, test attributes, assertion classes all stem from the UnitTest and UITesting assemblies. So without them, our tests don't run. More specifically, without the correct versions of those assemblies our tests don't even get added to the test runner!

1. Dependencies Built Into CodedUIExtensionsAndHelpers

My initial problem after upgrading to VS 2015, and attempting to run a test case manifested itself through not being able to run any tests on my local machine. More specifically, not even being able  to hit a debug point. The test suite was building fine, but as soon as I'd go to run it, I could not hit any breakpoint, and the test suite would seem to not do anything.

How I Diagnosed This
At first I thought something was wrong with my test runner, and so I started trying to run test cases via the ReSharper test runner. This did not help. I not saw that ReSharper treated my tests as inconclusive, but still did not run them. This problem was fairly hard to diagnose as I was not getting any explicit exceptions, or obvious errors. After some consultation with one of my senior dev buddies, I was able to figure out what the problem was using the call stack window. I was able to see where the coded ui test script started crashing. Turns out that the package that we use for providing helper methods and framing for Coded UI elements (CodedUIExtensionsAndHelpers) had a dependency on test DLLs, which were compiled using VS 2013. This put me in a bit of a pickle, as I did not really own the source code to that package.

How I Fixed This
Because the developer who owns these packages is a former colleague of ours, I emailed him, texted him and called him up to see if he could push a version of the extensions package which was compiled via VS2015. I waited a few hours (lol), then a day or so, (perhaps a weekend?), and after no response got impatient. I figured that I could pull down the source code (since it's on GitHub @ https://github.com/spelltwister/CodedUIFluentExtensions), recompile it, and push the DLL along with our code. After a lot of promising to push the proper way after proving that recompiling would work to my dev buddy who was helping me, I was able to show that after recompiling with VS 2015 and updating all the references in the CodedUIExtensionsAndHelpers package, I could build locally and debug. I then promptly forked the source of CodedUIExtensionsAndHelpers to my own Github (https://github.com/mkonkolowicz/CodedUIFluentExtensions), recompiled it, repackaged and updated all the references so I could update the DLL from Github. So the end result was that the package no longer had dependencies on VS2013 asssemblies, and I could debug locally.

2. Dependency Versions In The Checked In Projects

After figuring out that I could build locally, I assumed that I will be able to run the nightly runs with no problems. After all, everything else should just upgrade itself right? Welp, I was wrong, and spent a long time figuring that one out. I was able to build and run locally, but all my test runs on the test lab environment kept failing 100% of the time with error messages specifying that a specific version of an assembly was still missing (see the screenshot below for the specific message).

How I Diagnosed This
The build process for QAAutomation, takes all the code necessary for our test cases (which includes dependency dlls such as UITesting.dll) and drops them into a shared folder from where they are used by the test agent during the test run. I knew that if the error above was true, then there should be something interesting to see in the drop folder. I navigated there and inspected the files. I noticed they were all there, but I also noticed that the file versions were not correct, and they were all still pointing to the VS 2013 versions (V12.0.x)

How I Fixed This
I figured that if my projects all contained references to the correct versions of the referenced assemblies, I'd be all good. For 90% of the troubleshooting time I focused on the one big project (ATLAS) and spent a lot of time messing with references, as even after removing, cleaning, re-adding checking in, building and running our test suites, but I still could not get the proper versions. I then realized, that more than one project was being built by the build process that was dropping dependencies in to our drop location. After checking that project out, updating the references I was finally able to get the tests added to the test run. 

3. Lack Of Support For VS2015 On Lab Environment

So then I really got excited...I was able to get all the proper versions of the files I needed in the drop location, which theoretically meant I could push them all and run my tests right? Nope. Still having problems. This time, it was one I was completely unfamiliar with...

"Error adding test case [100372] to test run: Unable to load the test container '\\mi\DFS\TSI-ArchivesTemp\TFSDrop\Area51\QAMain\QAMain_20151228.3\atlasguitests.dll' or one of its dependencies. Error details: System.TypeLoadException: GenericArguments[0], 'Microsoft.VisualStudio.TestTools.UITesting.WpfControls.WpfControl', on 'CodedUIExtensionsAndHelpers.PageModeling.PageModelBase`1[T]' violates the constraint of type parameter 'T'.
System.TypeLoadException: GenericArguments[0], 'Microsoft.VisualStudio.Test"

I was completely flabbergasted by this error and had no idea what to do about it. I referred to my dev buddy, who basically told me that to google it. I mean at this point I googled the heck out of this whole process, so this advice did not seem like it would help...

How I Diagnosed This

But after googling, I did find something. Well, specifically, he did. We found this post: http://blogs.msdn.com/b/gopinath/archive/2015/02/27/test-agents-support-for-visual-studio-2015.aspx?CommentPosted=true#commentmessage. And that post, got us back to a running state. But I was not happy about it. Basically what the post says, is that VS2015 does not support agents, which means our test lab environment just won't work with V14.0 assemblies. The gump of the fix is that we have to trick the assemblies being shipped to the test lab environment to look like V12.0 (VS2013) assemblies, and only then will the tests run.

How I Fixed This
I followed the instructions in the post above and pasted in a bunch of assembly binding redirection into each of the projects' I'm working app.configs. I still have a few to fix (here's looking at you Nexsys), but all in all, the hack to get our tests running works as last night the biggest chunk of our tests (ATLAS) ran again. 

What I've Learned From the Experience

1. Microsoft makes mistakes, even the new Microsoft

It seems that MS has forgot to update agents for VS 2015. I'm not sure why, perhaps it's because not much noise is made from developers about GUI testing? I don't know too many that love that domain...but still Coded UI and test lab is a supported MS technology, so it seems to me that it just didn't get the love that the rest of their dev ops system received in this latest update. It's clear to me that this is the case, and MS needs to fix this. As we say, every client, every time no exceptions, no excuses. Even if it is the fat ugly kid in the MS family of test offering.

2. Perhaps it's good to explore other options

Being tied to one testing platform restricted our test suite to be non functional for about a week worth of time. I understand in the long term, that is not such a huge deal, but it seems that our automation goes with whatever MS does, and I'm not sure I like that. Now to be fair, I don't think Visual Studio will ever disappear, but Coded UI may not get as much love down the road. For the short term, I have a few things on the radar, but for the long term, having the ability to add in other things to our arsenal is a must.

That's all for now quality fanatics.

Over and Out.

Interesting Links Mentioned Above
1. http://blogs.msdn.com/b/gopinath/archive/2015/02/27/test-agents-support-for-visual-studio-2015.aspx?CommentPosted=true#commentmessage (The hack)


  1. First let me say I am sorry for all the trouble you are having. I need to know more about your environment. I run our CodedUI during releases on test machines all the time. When you say Lab Environment are you talking about TFS Lab Management or just an environment you created for test on your hardware?

    When I am building a machine to use as part of my continuous testing pipeline I start with a virgin machine. I then install Visual Studio 2015 Enterprise and our build agent. With just those two pieces in place you should be able to run all of our test types including coded UI. As you already discovered any third party dlls you use must be available as well. But all the core libraries would have been installed with Visual Studio.

    When I test a Web application I normally don’t install the app on my test machine. So I never feel bad installing Visual Studio there. My test machine is just the executor of my test that are exercising a different system under test.

    I am guessing but it looks like you are testing a web application. If so Selenium is a fantastic tool set for web testing and supported by Visual Studio and our build system.

    1. Thanks for the comment Donovan. Yea, we're using TFS Lab Management. We migrated our build agents from 2013 to 2015. We have both 2013 and 2015 (enterprise) on the build agent. The test machines which we are running our tests on, get a fresh install of our app. The problem I am facing is that when the tests are attempted to be run on the test machine (not the build agent), the 2015 DLL versions are not being used, even though they are in the drop folders.

      I am going away for the next week, but after, may start investigating solutions other than CodedUI.

      Thanks for your reply!