Unity Code Coverage Package - Discussion

I just tried it out and just gotta say, wow, this is an awesome tool to have at your disposal! I try to work with TDD where possible so I thought I had 100% test coverage in some of my submodules and with the help of this I immediately spotted som holes that could have meant bugs could creep in.

I really like that I can see the history too

It's nice that I can place the report in a folder of my choice as I want to test submodules separately and store the report in each submodule. Would it be possible to add some concept of "presets" so that I can define one test report for each submodule and then easily switch between them? As it is right now I would need to do this manually and must remember wehere each submodule stores its report.

Also, it warns for poor performance when being turned on, is this only when you generate the report or does it have some kind of constant overhead?

I'll try it out more and get back to you with more questions probably

 

Thanks for the clarification! I'll turn it off when not actively TDD:ing then

Regarding presets, I'm sure whatever you guys come up with will be awesome. I don't think it is a super necessary feature, just nice to have.

 

Is possible to use it in unity 2017?

Answer:

Unfortunately not as the Code Coverage API was introduced in Unity 2019.2

 

Hello, I used code coverage tool in Unity 2020.1 today and I want to use this tool for a long time. However, I have encountered several problems. I don't know if it's my own operation problem or a tool bug.

I wrote a demo to learn to use code coverage tool, which is a class including function add, sub, mul and div and a Print function:

Then I wrote a test class in playmode, including the test function of function add, sub, mul and div(not including print function).

Then I ran the playmode test and generated the code coverage,which is out of my expectation:

The functions which are tested by the test function are not covered and the print function which is not tested by the test function is covered!

Then I found out that the reason why the Print function is covered is that I invoked that function in Start function. I can't understand why this is handled like this: Shouldn’t coverage be the part of the code being tested? The invoke of Print function is not in the test code.

 

There is a second question. I added some log infomation in the add, sub, mul and div functions:

Then I ran the test, the result surprised me again:

The coverage is 100% now! I did nothing but add log statement in the class! I can't understand the result like this. Can you troubleshoot the answer? Thank you very much!! Looking forward to your reply!!

(The website doesn't allow me to upload source code so I have to use picture instead, forgive me.)

 

It works!! The coverage is 100% whether I have added logs or not.
However, I still have a question. The meaning of coverage here is the coverage of the test code or the full coverage of all the code runs? As I didn't write the test code of Print function but I invoked that function in Start function, the coverage increased still. I am a bit confused.
Once upon a time I used googletest to get the coverage of my C++ unit test, its coverage is only about the code I covered in my test code.
Looking forward to your reply!!! Thank you very much!!

 

@Unity-Nikos ok, so here's another slightly dreamy feature request Would it be possible to somehow use the test coverage report with SonarQube? That would really make this fit in well at work where we use SonarQube for static code analysis and over time test coverage measuring on all builds. Not sure how many would be interested in this but I suspect many in the industrial segment where Unity is part of a larger ecosystem would be...

 

Hi, Nikos. I get a new question. I want to use batchmode to run unit test and get coverage report. How can I change Unity to debug mode using command line?

 

I changed to use Unity2019.3.0b12, which doesn't have the little bug at its right bottom conner and the problem is solved!

 

Hi Nikos, I have a new problem here. Can Code Coverage only be used to determine the coverage of which part of the code to test? There is a part of code in my project that is automatically generated and does not require test coverage, but if I simply specify Assembly-CSharp to include they will be included in my coverage as well. Is there a way to solve it? Looking forward to your reply!

 

May I ask if the using of the command line to filter out directory is available now?

 

Ok, thank you very much! You helped a lot!!!

 

As I understand it, the related tech this relies upon was added in 2019.2, not 2019.3. Does that mean we can manually install in .2 ?

(Guess who is stuck with being standardized on 2019.2 for the next 6+ months ...)

 

This tool is a little gem. It gives you a "birds-eye view" of your code.
I used the code coverage option on a game I work on for a long time.
I have found many non-obvious mistakes I did (as well as many obvious ones), very fast, just by running the game as usual and checking the code coverage results.

You can change the name to "Bad code revealer"
Much appreciated!

 

Greetings!

It seems no matter what I try, I\m unable to generate the coverage report in batch mode.

I'm running it like this:

Code (csharp):

1.  
2. /Applications/Unity/Hub/Editor/2019.3.0f6/Unity.app/Contents/MacOS/Unity \
3.                     -batchmode \
4.                     -projectPath . \
5.                     -runTests -testPlatform editmode \
6.                     -testResults ./build/editmode-results.xml \
7.                     -enableCodeCoverage -coverageResultsPath ./coverage \
8.                     -coverageOptions 'enableCyclomaticComplexity;generateHtmlReport;generateBadgeReport' \
9.                     -logFile /dev/stdout
10.  

It does create the /coverage/<project-name>-opencov/EditMode folder, but it's always empty.

On the other hand, it works just fine when being run from within the Unity editor.

Is there something obvious that I'm missing, or is this a known issue in the current version?

Thank you in advance for any help!

 

Thank you so much for your reply, chrisj!

I'm actually setting up my CI workflow and, while the editmode test results report is generated, I don't have any tests yet.

This is probably the reason why it's not working for me.
I assumed it would mimic the coverage report I get in the editor, showing 0% coverage.

I'll definitely try again with some tests and come back here with my results.

Again, thank you for your help.

 

chrisj, I can confirm that was the issue indeed - the coverage report is generated if at least one test is present.

Thanks!

 

Greetings!

I have some problems when try running UT in Unity and get code coverage. My command's that I'm running it like this:

Code (CSharp):

1. /Applications/Unity/Hub/Editor/2019.3.5f1/Unity.app/Contents/MacOS/Unity
2.         -batchmode \
3.         -projectPath . \
4.         -runTests -testPlatform editmode \
5.         -enableCodeCoverage
6.         -coverageResultsPath {path_to_coverage_folder} \
7.         -coverageOptions 'enableCyclomaticComplexity;generateHtmlReport;generateBadgeReport' \
8.         -logFile {path_to_coverage_folder_logs}

But, I don't know why in the coverage folder was empty. I checked log and found the error, that like

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. (Filename: Library/PackageCache/com.unity.testtools.codecoverage@0.2.3-preview/Editor/CoverageFormats/OpenCover/OpenCoverReporter.cs Line: 685)

Some more information:
Unity version: 2019.3.5f1
Code coverage version: codecoverage@0.2.3-preview.
And the TestResults_xxx.xml file is still generated.

Is there something that I'm missing?

 

@Unity-Nikos My report does not appear to include coverage of [UnityTest] tests, see the screenshots:

LocalPlayerTest ran and was successful, yet it is marked as not covered

And yet, when I look at the coverage report, it is all red. Note that all my [Test] code is green, but all my [UnityTest] is red.

 

@Unity-Nikos - Does this package work if the tests are run through the TestRunnerAPI?

 

@Unity-Nikos Here is how to reproduce the issue:

1) Clone this repo: https://github.com/MirrorNG/MirrorNG
2) Open it in Unity 2019.3.7
3) Enable coverage in editor settings if you haven't
4) Open Code Coverage window and enable Generate HTML Report and AutoGenerate Report
5) Open the test runner
6) Click on PlayMode
7) Click Run All
8) Tests run and should all pass,
9) Expand MirrorNG -> Mirror.Tests.Runtime.dll -> Mirror -> Tests -> NetworkClientTest
10) observe LocalPlayerTest passes
11) Report is generated, open index.htm (attached here)
12) Expand Mirror.Tests.Runtime
13) Click on Mirror.Tests.NetworkClientTest
14) Click on LocalPlayerTest
15) Observe the method body is all red

 

Thanks that worked.

I suggest displaying an error if a [UnityTest] returns anything other than IEnumerator,

 

@Unity-Nikos It seems like neither `System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage` or `UnityEngine.TestTools.ExcludeFromCoverage` attribute is being respected by the package. Whenever I open up the report in SonarQube, anything I've marked with that attribute is still being included in the coverage report.

Answer:

We have added this to our backlog to fix. Thanks for reporting it.

Edit: This is fixed in the next version of the package 0.4.0-preview

 

@Unity-Nikos

I am using Unity 2020.1b8 and I have found a different behavior than the 2019.3.x version.

It turns out that in 2019.3.x I get 100% coverage in a script but in 2020 it tells me that there are several lines that are not covered when I know that they are.

Researching it turns out that it is because in Unity 2020 you can switch between Debug/Release (Bottom right) and if it is in Release it seems that the behavior is different.

I understand that it's a bug? Or is it that to analyze the code coverage you must be in Debug mode? It's very confusing, actually.

Answer:

Hi @bdovaz - thank you for the feedback. You are correct that Debug mode is required in order to obtain accurate code coverage information. This is not a bug as in Release mode the code is optimized and therefore not directly represented by the original code.

There is a warning in the console and the Code Coverage window that explains that and we have added a popup message in the upcoming version of the package that is displayed when running tests or coverage recording on Release mode, which lets you switch to Debug mode too. For more information see Using Code Coverage with Code Optimization

 

Hi, I'm testing my code and it seem to me that some line are counted as coverable whe nthey shoud not.



Basicly all the code of the class is covered but I endup with a 60ish % coverage due to the blanl/comment/signature lines.

Edit : NVM, I cleared all coverage data and regenerate the report, it's fine now.

 

Hi @Unity-Nikos

I'm using the coverage package for my project and I have question regarding the CRAP computation (I 'm new to that CRAP ).

My understanding is that code coverage should be taken into account in the calculation. basically if the code is complex but it's well covered by test it's less CRAPpy than if it was not.

The coverage report for my project returns some CRAP warning but it does not seem to take into consideration the code coverage.

CRAP results


Coverage of methods



If I take the first one and apply the formula to get the 56 score I just have to do 7^2 + 7, so cov(m) = 0

For the second i have also to consider cov(m) = 0 to get the 42 score 6^2 + 6. where in fac I have 100% coverage so score should be 6^2 * (1 - (100/100))^3 + 6 = 6 so not CRAP.

Answer:

We have added this to our backlog to fix. Thank you for reporting it.

Update:

Fixed in 0.3.1-preview

 

can't like my own post so thanks for the answer

 

Is there a reason that all assemblies are not included by default in the coverage options? As a corollary, is there an API for adding assemblies for coverage via script?

Answer:

Hi Jamie_Robertson, the idea behind not selecting all assemblies by default is that a high percentage of users would want to measure the code coverage of their code base and not of unity's & packages' code base. Instead only the assemblies that live under the Assets folder are selected by default.

Adding an API to specify the assemblies makes sense. Thank you for the suggestion, I have added it in our backlog.

Other ways to include all assemblies are:

1. Press 'Select All' in the 'Included Assemblies' dropdown
2. If running unity from the command line in batchmode, you can pass

   -coverageOptions assemblyFilters:+*

 

That answer makes sense. I was aware of the `assemblyFilters` options. I was specifically curious about the editor behaviour.

Now that you mention the `assemblyFilters` option though, is the default value for that argument all assemblies? If that's the case, then would it not make sense then the default value in editor also be all assemblies?

Update:

Fixed in 0.3.1-preview

• If `assemblyFilters` is not specified in `-coverageOptions` in batchmode, include only the assemblies found under the Assets folder

 

I have some questions:

1. Is it possible to generate the Code Coverage report if I run my Play Mode tests in a platform besides the Editor? In other words, if I have Android specific code and run the test using the "Run all in player (Android)" option, will I get a report that includes these lines?
2. Is it possible to generate a test build that executes only specific tests? It is clearly possible in the Editor using the "Run Selected" option, but I can't find a way to do the same thing in another player.
3. Is it possible to label lines as untestable? For example, if I surround my code with #if UNITY_ANDROID and #endif and I test using the Editor, those lines should be excluded from the test.

Answers:

Thanks Arthur

1. Unfortunately not at the moment. We plan to add support for playmode tests in the player next year.
2. It is not possible from the UI to execute specific tests in player testing at the moment, but it is something that we will be adding to the test framework soon.
3. If your code is surrounded by an #if block that isn't defined, then the compiler won't be generating any IL for that code, so it should already be excluded both from the tests and the coverage. If you run the tests in the editor and you are testing in Android platform, UNITY_ANDROID would be defined and therefore the code would be included in the coverage calculation & report.

Note that in the next version of the package 0.4.0-preview you will be able to use the UnityEngine.TestTools.ExcludeFromCoverage attribute to exclude Assemblies, Classes, Constructors, Methods and Structs from the coverage calculation.

 

@Unity-Nikos - Glad to see the fixes going in for `ExcludeFromCoverage`! Thanks for the quick follow up.

Curious, do you have a timeline for a non-preview release?

Answer:

Hi Jamie,

we are currently in the process of getting the package out of preview. I will make sure to update on the progress in this thread.

Edit:

The Code Coverage package has now been released. You can find the new introduction thread here: https://forum.unity.com/threads/1073831

 

@Unity-Nikos There seems to be a regression with 0.4.1-preview.
I am unable to build for android with it, while there seems to be no issue with 0.4.0-preview.

The error:

Code (csharp):

1. ArgumentException: The Assembly WindowsBase is referenced by ReportGeneratorMerged ('Packages/com.unity.testtools.codecoverage/lib/ReportGenerator/ReportGeneratorMerged.dll'). But the dll is not allowed to be included or could not be found.
2. UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List`1[T] alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[] foldersToSearch, System.Collections.Generic.Dictionary`2[TKey,TValue] cache, UnityEditor.BuildTarget target) (at /home/bokken/buildslave/unity/build/Editor/Mono/AssemblyHelper.cs:158)
3. UnityEditor.AssemblyHelper.FindAssembliesReferencedBy (System.String[] paths, System.String[] foldersToSearch, UnityEditor.BuildTarget target) (at /home/bokken/buildslave/unity/build/Editor/Mono/AssemblyHelper.cs:198)
4. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&) (at /home/bokken/buildslave/unity/build/Modules/IMGUI/GUIUtility.cs:189)
5.  

P.S. I have create a bug report (Case 1306557)

Update:

Thank you for filing a bug report. You can fix this for now by selecting only the Editor platform in the import settings of the ReportGeneratorMerged.dll in Packages > Code Coverage > lib > ReportGenerator. I will make sure this is fixed in the next version of the package.

Edit:

This is fixed in 0.4.2-preview. Apologies, I didn't realise that you can't do this if the package lives in Library/PackageCache.

 

For others struggling with this error and noticing that you can't actually edit the .dll file settings as shown in the screenshot: you can edit them by first copying them over to a folder under Assets. After editing them, use File Explorer to drag them back to their original location.

Or just wait for the fix, which is what I'm going to do.

Update:

Fixed in 0.4.2-preview

• Updated Include Platforms to Editor only in the ReportGeneratorMerged.dll settings. Fixes an Android build error introduced in 0.4.1-preview (case 1306557)