Upgraded To Visual Studio 2015: What It Meant For Me.
BackgroundIn a previous post I talked about the benefits of VS 2015, and that I was ready to take the plunge. Welp, my hand was forced about three weeks ago. Our build server was upgraded to TFS 2015, which meant that it was the perfect time to upgrade my IDE as well. I did about two weeks ago.
As part of the upgrade testing, I wanted to ensure that I could effectively write a test in VS 2015, compile it, check it in, associate it with a test case, and be able to run the test case in the test lab environment.
The ProblemsGUMP: I needed to update a crap ton of dependencies. At the end of the day, automated test cases rely on multiple assemblies to run. For example, test attributes, assertion classes all stem from the UnitTest and UITesting assemblies. So without them, our tests don't run. More specifically, without the correct versions of those assemblies our tests don't even get added to the test runner!
1. Dependencies Built Into CodedUIExtensionsAndHelpersMy initial problem after upgrading to VS 2015, and attempting to run a test case manifested itself through not being able to run any tests on my local machine. More specifically, not even being able to hit a debug point. The test suite was building fine, but as soon as I'd go to run it, I could not hit any breakpoint, and the test suite would seem to not do anything.
How I Diagnosed This
At first I thought something was wrong with my test runner, and so I started trying to run test cases via the ReSharper test runner. This did not help. I not saw that ReSharper treated my tests as inconclusive, but still did not run them. This problem was fairly hard to diagnose as I was not getting any explicit exceptions, or obvious errors. After some consultation with one of my senior dev buddies, I was able to figure out what the problem was using the call stack window. I was able to see where the coded ui test script started crashing. Turns out that the package that we use for providing helper methods and framing for Coded UI elements (CodedUIExtensionsAndHelpers) had a dependency on test DLLs, which were compiled using VS 2013. This put me in a bit of a pickle, as I did not really own the source code to that package.
How I Fixed This
Because the developer who owns these packages is a former colleague of ours, I emailed him, texted him and called him up to see if he could push a version of the extensions package which was compiled via VS2015. I waited a few hours (lol), then a day or so, (perhaps a weekend?), and after no response got impatient. I figured that I could pull down the source code (since it's on GitHub @ https://github.com/spelltwister/CodedUIFluentExtensions), recompile it, and push the DLL along with our code. After a lot of promising to push the proper way after proving that recompiling would work to my dev buddy who was helping me, I was able to show that after recompiling with VS 2015 and updating all the references in the CodedUIExtensionsAndHelpers package, I could build locally and debug. I then promptly forked the source of CodedUIExtensionsAndHelpers to my own Github (https://github.com/mkonkolowicz/CodedUIFluentExtensions), recompiled it, repackaged and updated all the references so I could update the DLL from Github. So the end result was that the package no longer had dependencies on VS2013 asssemblies, and I could debug locally.
2. Dependency Versions In The Checked In ProjectsAfter figuring out that I could build locally, I assumed that I will be able to run the nightly runs with no problems. After all, everything else should just upgrade itself right? Welp, I was wrong, and spent a long time figuring that one out. I was able to build and run locally, but all my test runs on the test lab environment kept failing 100% of the time with error messages specifying that a specific version of an assembly was still missing (see the screenshot below for the specific message).
How I Diagnosed This
The build process for QAAutomation, takes all the code necessary for our test cases (which includes dependency dlls such as UITesting.dll) and drops them into a shared folder from where they are used by the test agent during the test run. I knew that if the error above was true, then there should be something interesting to see in the drop folder. I navigated there and inspected the files. I noticed they were all there, but I also noticed that the file versions were not correct, and they were all still pointing to the VS 2013 versions (V12.0.x)
How I Fixed This
I figured that if my projects all contained references to the correct versions of the referenced assemblies, I'd be all good. For 90% of the troubleshooting time I focused on the one big project (ATLAS) and spent a lot of time messing with references, as even after removing, cleaning, re-adding checking in, building and running our test suites, but I still could not get the proper versions. I then realized, that more than one project was being built by the build process that was dropping dependencies in to our drop location. After checking that project out, updating the references I was finally able to get the tests added to the test run.
3. Lack Of Support For VS2015 On Lab Environment
So then I really got excited...I was able to get all the proper versions of the files I needed in the drop location, which theoretically meant I could push them all and run my tests right? Nope. Still having problems. This time, it was one I was completely unfamiliar with...
"Error adding test case  to test run: Unable to load the test container '\\mi\DFS\TSI-ArchivesTemp\TFSDrop\Area51\QAMain\QAMain_20151228.3\atlasguitests.dll' or one of its dependencies. Error details: System.TypeLoadException: GenericArguments, 'Microsoft.VisualStudio.TestTools.UITesting.WpfControls.WpfControl', on 'CodedUIExtensionsAndHelpers.PageModeling.PageModelBase`1[T]' violates the constraint of type parameter 'T'.
System.TypeLoadException: GenericArguments, 'Microsoft.VisualStudio.Test"
I was completely flabbergasted by this error and had no idea what to do about it. I referred to my dev buddy, who basically told me that to google it. I mean at this point I googled the heck out of this whole process, so this advice did not seem like it would help...
How I Diagnosed This
But after googling, I did find something. Well, specifically, he did. We found this post: http://blogs.msdn.com/b/gopinath/archive/2015/02/27/test-agents-support-for-visual-studio-2015.aspx?CommentPosted=true#commentmessage. And that post, got us back to a running state. But I was not happy about it. Basically what the post says, is that VS2015 does not support agents, which means our test lab environment just won't work with V14.0 assemblies. The gump of the fix is that we have to trick the assemblies being shipped to the test lab environment to look like V12.0 (VS2013) assemblies, and only then will the tests run.
How I Fixed This
I followed the instructions in the post above and pasted in a bunch of assembly binding redirection into each of the projects' I'm working app.configs. I still have a few to fix (here's looking at you Nexsys), but all in all, the hack to get our tests running works as last night the biggest chunk of our tests (ATLAS) ran again.
What I've Learned From the Experience
1. Microsoft makes mistakes, even the new MicrosoftIt seems that MS has forgot to update agents for VS 2015. I'm not sure why, perhaps it's because not much noise is made from developers about GUI testing? I don't know too many that love that domain...but still Coded UI and test lab is a supported MS technology, so it seems to me that it just didn't get the love that the rest of their dev ops system received in this latest update. It's clear to me that this is the case, and MS needs to fix this. As we say, every client, every time no exceptions, no excuses. Even if it is the fat ugly kid in the MS family of test offering.
2. Perhaps it's good to explore other optionsBeing tied to one testing platform restricted our test suite to be non functional for about a week worth of time. I understand in the long term, that is not such a huge deal, but it seems that our automation goes with whatever MS does, and I'm not sure I like that. Now to be fair, I don't think Visual Studio will ever disappear, but Coded UI may not get as much love down the road. For the short term, I have a few things on the radar, but for the long term, having the ability to add in other things to our arsenal is a must.
That's all for now quality fanatics.
Over and Out.
Interesting Links Mentioned Above
1. http://blogs.msdn.com/b/gopinath/archive/2015/02/27/test-agents-support-for-visual-studio-2015.aspx?CommentPosted=true#commentmessage (The hack)