Integrating SonarQube Metrics into a TFS build – Intro

I have been working on getting SonarQube integrated into our nightly build which is run out of TFS. If you don’t know what SonarQube is check it out. Its a good way to centralise information about your code which is good for capturing and tracking metrics for those people who care about KPIs (if we must have them at least they can be about something positive towards code quality) whilst giving those people that don’t a central place to see issues with code quality etc that they can improve upon.

Previously we had a separate Jenkins build reading source from TFS but we wanted to get rid of this additional build and just do it once in TFS – seemed a bit wasteful and time consuming to sort out Jenkins as well as TFS. It turns out I have had pretty good timing as (at the time of writing) it is only recently that Sonar can handle the upload of Microsoft’s test results and coverage files (.trx and .coverage respectively) – http://docs.codehaus.org/display/SONAR/C%23+Plugin.

Continue reading

Integrating SonarQube Metrics into a TFS build – Part 3

Integrate the “Sonar Runner” into your build.

  1. Introduction
  2. Make TFS output the required test and coverage results files.
  3. Setup your project to use Sonar.
  4. Integrate the “Sonar Runner” into your build.

The bit that actually does something

So far we have only been doing the setup required to get us ready to run Sonar. Stage 3 is all about actually getting SonarQube scans to run in your build. Assuming you have previously followed stages 1 and 2 this is now just a case of doing a little work to your build template.

There is still a little prep work that we need to do before we can get SonarQube scans running.  The coverage files (which are binary) are not understood by SonarQube, we need to convert them into Xml files that it can understand. This can be achieved using an executable – CodeCoverage.exe.

Continue reading

Integrating SonarQube Metrics into a TFS build – Part 2

Setup your project to use Sonar.

  1. Introduction
  2. Make TFS output the required test and coverage results files.
  3. Setup your project to use Sonar.
  4. Integrate the “Sonar Runner” into your build.

Stage 2 of integrating SonarQube into a TFS build involves setting up the necessaries on your build server, updating/configuring SonarQube itself, creating (or in some cases sourcing) the required scripts and assemblies and adding the required scripts to your project. As mentioned in the introduction, I wanted to stay away from using a ‘custom assembly’ solution meaning the steps to achieve SonarQube integration involve calling a batch file to do the scan and upload.

You may need to upgrade a few of your SonarQube plugins if you haven’t already.

If you want to be able to upload your .trx and .coverage files to SonarQube (see previous post for how to generate and get at these in your build) you will need to be using at least version 3.1 of the C# plugin – http://docs.codehaus.org/display/SONAR/C%23+Plugin.

This upgrade brings with it a few issues:

  • When upgrading the plugin – make sure you consult the link at the bottom of the page under the title “Upgrading from version 2.1?” as there is a specific approach for upgrading the plugin (I won’t cover that here since they have documented that themselves but you can get to it HERE)
  • Stylecop is no longer embedded into the Sonar runner so you will need to install that on your build machine if you wish to use it.
  • Likewise if you want to use FXCop make sure that is installed on the build machine also.
  • Various other breaking changes were made such as sonar property file changes (some new mandatory fields) and a few older test runners etc no longer supported. Essentially check with other people before upgrading as it is almost certain that by upgrading you will break their scans if they don’t do any work to keep up.

Continue reading

Integrating SonarQube Metrics into a TFS build – Part 1

Make TFS output the required test and coverage results files.

  1. Introduction
  2. Make TFS output the required test and coverage results files.
  3. Setup your project to use Sonar.
  4. Integrate the “Sonar Runner” into your build.

Sonar Supports .trx and .coverage files but TFS does not generate them …

SonarQube now supports the test and code coverage results files that are generated in the Microsoft stack. This is a recent feature addition and one that removes the need to run unit tests in the Gallio test runner as was required previously.

As good as this sounds the TFS build uses the Visual Studio Test Runner which doesn’t generate any .trx or .coverage files to upload to Sonar. Additionally using the outdated MSTest runner cannot generate .coverage files (although it can generate .trx files) so this does not solve the problem either.

VSTest can generate results files – but at the expense of publishing them

VSTest is a more modern test runner from Microsoft that has the ability to run tests against any test framework. See here for a quick run down of it. Looking at VSTest I could see that there was the ability to achieve the generation of both the .trx file and the .coverage file so there was the option to call this executable directly rather than use the in built TFS test runners. Again this was problematic as VSTest can publish results to TFS OR create the files, not both at the same time.

MSTest.exe can publish test results from a .trx file without re-running them.

Luckily we can still publish the test results to TFS without the need to re-run them using MSTest.exe. It would be nice (if someone from Microsoft reads this please take note) if we could have all of this functionality in one place rather than some of it being in the new executable and some in the old but we are where we are so using both seems to be the answer for this requirement.

Always some caveat or compromise

One ‘issue’ I did encounter was the lack of test ‘passes’ and ‘fails’ being written to screen and build log file. This was a compromise I was willing to make as I still get the test results anyway so didn’t really care that the log doesn’t display such info- if you feel differently to me maybe this isn’t the approach for you.

Continue reading