Difference in codecoverage report of VSTS build and SonarQube

Question

I have a set of .Net projects with a couple of test projects. I ran a build and from VSTS build i see that my Lines covered % is 45.24(8213/18153) but when i see the same from SonarQube, it shows me 13.3%(361/2717). I'm wondering how does sonar display different metrics compared to VSTS. Also the number of binaries analyzed in code coverage differes from VST to SonarQube. Is this expected?

• SonarQube version: 6.7.1, Community Edition
• SonarC# version: 6.8.2(build 4717)
• SonarQube VSTS task version : 3.2.0(4.1 version doesn't work at all)
• SonarScanner for MSBuild: 4.1
• Solution Version: VS 2015
• VSTest Console.exe version: 14.0.25420.1

Please let me know if any further details would be required to investigate further.

Answer 1

The difference in the numbers is most likely caused by SonarC#'s calculation of Executable Lines. It is explained in detail in this blog post.

In brief: Executable Lines counts are fed into the coverage denominator for files not represented in your coverage report. Native tools simply ignore their existence when calculating that percentage. Your coverage number is lower in SonarQube because it's taking all your files into account.

Addendum: My answer is (correct and) language-agnostic. duncanp's answer is MSDN-specific and thus likely more relevant in this context.

Answer 2

In addition to the suggestions above, there are a couple of other reasons why the overall coverage numbers reported in VSTS will be different (assuming you're using the standard VS code coverage tool and haven't provided any special configuration).

The default behaviour of the VS code coverage tool is to include the test code when calculating the coverage numbers i.e. the number reported in VSTS is the coverage of [test code + product code], whereas the figure reported in SonarQube only takes into account the product code.

So if VS reports an overall coverage number of, say 80%, it means that 80% of your lines of [test + product] code are being executed. However, it's normally the case that almost all of your test code is being executed, so the code coverage on your test code is likely to be higher than 80%, which means the code coverage on your product code will be less than the 80% reported (the last time I analysed a VS code coverage report for a solution in detail, the coverage on product code turned out to be 10% lower than the summary figure reported by the build).

In addition, the VS coverage numbers will also include assemblies that are not part of your solution if the pdb files are available (I'm looking at an example of a coverage report that includes the third-party fluentassertions.core.dll and fluentassertions.dll assemblies, both of which are showing as >95% not covered).

VSTS makes it very easy to set up collecting code coverage - you just click a checkbox in the build definition, and the number appears on the build summary page. Intuitively, you might expect this number to be the coverage of your product code (which is what SonarQube shows). However, it almost certainly isn't the case out of the box. The inclusion of test code will tend to skew the number upwards, whereas any third-party assemblies with pdbs that have been included will tend to skew the number downwards.

If you want to see exactly which projects and assemblies are included in the coverage report the simplest thing to do is download it by clicking on the build summary page and open it up in VS.

It is possible to change the default VS behaviour to get the coverage on product code, but it's not particularly straightforward. See Customize code coverage analysis on MSDN for more information.