Search Unity

Unity Code Coverage Package Preview

Discussion in 'Testing & Automation' started by Unity-Nikos, Nov 14, 2019.

  1. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    Use the Code Coverage preview package with the Test Runner to gather and present test coverage information. When you run your tests with code coverage enabled, you can see exactly which lines of your code are executed, in addition to whether the tests have passed or failed. For more details see Using Code Coverage with Test Runner.


    Code Coverage window

    Once a test run has completed, the Code Coverage package will generate an HTML coverage report showing which lines of your code are covered by tests. Code Coverage currently supports EditMode and PlayMode tests. It also allows you to track the code coverage changes through time.


    HTML report

    If you don't have any tests, don't worry! The Code Coverage package offers a Coverage Recording feature which allows capturing coverage data on demand.

    coverage-recording.gif

    The Code Coverage package is available as a preview package via the Package Manager for Unity 2019.3 and above. See Installing Code Coverage. Please make sure to enable preview packages in the Package Manager, otherwise it won't be listed.

    Take note that preview packages are not considered ready for production use and might change before being verified for release.

    upload_2019-11-14_13-52-3.png
    Installing Code Coverage

    For more information see the Code Coverage package documentation.

    We look forward to your questions and feedback!
     
    Last edited: Dec 6, 2019
    konsnos, WAYN_Group, Zipe92 and 12 others like this.
  2. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    I just tried it out and just gotta say, wow, this is an awesome tool to have at your disposal! I try to work with TDD where possible so I thought I had 100% test coverage in some of my submodules and with the help of this I immediately spotted som holes that could have meant bugs could creep in.

    I really like that I can see the history too :)

    It's nice that I can place the report in a folder of my choice as I want to test submodules separately and store the report in each submodule. Would it be possible to add some concept of "presets" so that I can define one test report for each submodule and then easily switch between them? As it is right now I would need to do this manually and must remember wehere each submodule stores its report.

    Also, it warns for poor performance when being turned on, is this only when you generate the report or does it have some kind of constant overhead?

    I'll try it out more and get back to you with more questions probably :)
     
  3. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    Hi Mikael,

    thanks for the feedback and suggestions!

    Presets:
    This is a good idea and agree that having separate presets for separate reports would give more flexibility. Currently the results are saved under a <ProjectName>-opencov folder, so with presets this could be updated to <ProjectName>-<PresetName>-opencov. What we would not be able to do is to reflect the presets in the report (i.e. a single report with different presets)..for now at least. I will discuss it with the team and add it to the backlog, and post any updates here.

    Code Coverage performance overhead:
    When you check Enable Code Coverage in Preferences > General you enable access to the interface for the code coverage data that Mono exposes - see https://docs.unity3d.com/ScriptReference/TestTools.Coverage.html and https://docs.unity3d.com/Packages/c...0.2/manual/TechnicalDetails.html#how-it-works. This adds some overhead to the editor and lowers the performance, so it is not recommended to leave it on if you are not doing coverage testing. We are exploring changing this in the future. We will post any progress here.

    Thanks
    Nikos
     
    Last edited: Nov 18, 2019
    GarthSmith and Mikael-H like this.
  4. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Thanks for the clarification! I'll turn it off when not actively TDD:ing then :)

    Regarding presets, I'm sure whatever you guys come up with will be awesome. I don't think it is a super necessary feature, just nice to have.
     
    Unity-Nikos likes this.
  5. Aaron_Chan

    Aaron_Chan

    Joined:
    May 5, 2017
    Posts:
    2
    Is possible to use it in unity 2017?
     
  6. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    Hello, I used code coverage tool in Unity 2020.1 today and I want to use this tool for a long time. However, I have encountered several problems. I don't know if it's my own operation problem or a tool bug.

    I wrote a demo to learn to use code coverage tool, which is a class including function add, sub, mul and div and a Print function:
    upload_2019-11-21_19-14-44.png
    Then I wrote a test class in playmode, including the test function of function add, sub, mul and div(not including print function).
    upload_2019-11-21_19-15-37.png
    Then I ran the playmode test and generated the code coverage,which is out of my expectation:
    upload_2019-11-21_19-16-15.png
    The functions which are tested by the test function are not covered and the print function which is not tested by the test function is covered!

    Then I found out that the reason why the Print function is covered is that I invoked that function in Start function. I can't understand why this is handled like this: Shouldn’t coverage be the part of the code being tested? The invoke of Print function is not in the test code.
     
  7. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    There is a second question. I added some log infomation in the add, sub, mul and div functions:
    upload_2019-11-21_19-17-43.png
    Then I ran the test, the result surprised me again:
    upload_2019-11-21_19-18-9.png
    The coverage is 100% now! I did nothing but add log statement in the class! I can't understand the result like this. Can you troubleshoot the answer? Thank you very much!! Looking forward to your reply!!

    (The website doesn't allow me to upload source code so I have to use picture instead, forgive me.)
     
  8. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    Hi DavidWei1. I suspect it is due to Code optimization been set to Release mode. Code Optimization was introduced in 2020.1. Code Optimization mode defines whether Unity Editor compiles scripts in Debug or Release mode. Debug mode enables C# debugging and it is required in order to obtain accurate code coverage. To ensure Code optimization is set to Debug mode you can do one of the following:
    • Switch to Debug mode in the Editor (bottom right corner, select the Bug icon > Switch to debug mode)
    • Using the CompilationPipeline api, set CompilationPipeline.codeOptimization = CodeOptimization.Debug
    • Pass -debugCodeOptimization to the command line
    Please see https://docs.unity3d.com/Packages/c...ml#using-code-coverage-with-code-optimization

    Let me know if this helps. We are planning to do this automatically in the future.

    Thanks
    Nikos
     
  9. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
  10. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    It works!! The coverage is 100% whether I have added logs or not.
    However, I still have a question. The meaning of coverage here is the coverage of the test code or the full coverage of all the code runs? As I didn't write the test code of Print function but I invoked that function in Start function, the coverage increased still. I am a bit confused.
    Once upon a time I used googletest to get the coverage of my C++ unit test, its coverage is only about the code I covered in my test code.
    Looking forward to your reply!!! Thank you very much!!
     
  11. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    @davidwe01 since Class1 is a MonoBehaviour, depending on how you initialise it in your playmode test (I cannot see that bit in the screenshots), I suspect that Start is triggered as the script gets added to the Scene, or perhaps it is already added in the Scene before you run your test
     
  12. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    @Unity-Nikos ok, so here's another slightly dreamy feature request :) Would it be possible to somehow use the test coverage report with SonarQube? That would really make this fit in well at work where we use SonarQube for static code analysis and over time test coverage measuring on all builds. Not sure how many would be interested in this but I suspect many in the industrial segment where Unity is part of a larger ecosystem would be...
     
  13. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    Hi, Nikos. I get a new question. I want to use batchmode to run unit test and get coverage report. How can I change Unity to debug mode using command line?
     
  14. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    I changed to use Unity2019.3.0b12, which doesn't have the little bug at its right bottom conner and the problem is solved!
     
  15. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
  16. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    @Mikael-H try pointing SonarQube to the <project-name>-opencov folder located under CodeCoverage in the project's root folder or under the Destination folder.
     
    Mikael-H likes this.
  17. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    Hi Nikos, I have a new problem here. Can Code Coverage only be used to determine the coverage of which part of the code to test? There is a part of code in my project that is automatically generated and does not require test coverage, but if I simply specify Assembly-CSharp to include they will be included in my coverage as well. Is there a way to solve it? Looking forward to your reply!
     
  18. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
  19. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    May I ask if the using of the command line to filter out directory is available now?
     
  20. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    It will be in the next version of the package, 0.2.1-preview. We will post an update to this thread when this is live.
     
    Last edited: Nov 27, 2019
  21. DavidWei1

    DavidWei1

    Joined:
    Nov 21, 2019
    Posts:
    8
    Ok, thank you very much! You helped a lot!!!
     
    Unity-Nikos likes this.
  22. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    [Code Coverage package - version 0.2.1-preview is out]
    • Added pathFilters batchmode coverageOption
    • Improved globbing for pathFilters and assemblyFilters - see https://docs.unity3d.com/Packages/c...overage.html#using-code-coverage-in-batchmode
    • Added new sections and examples in documentation
    • Added warning and button to switch to debug mode, when using Code Optimization in release mode in 2020.1 and above
    • Added confirmation dialogs when selecting Clear Data and Clear History buttons
    Documentation: https://docs.unity3d.com/Packages/com.unity.testtools.codecoverage@0.2
     
    GarthSmith and Warnecke like this.
  23. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    431
    As I understand it, the related tech this relies upon was added in 2019.2, not 2019.3. Does that mean we can manually install in .2 ?

    (Guess who is stuck with being standardized on 2019.2 for the next 6+ months :)...)
     
  24. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    Technically yes, but there might be untested scenarios as we officially support 2019.3 and above. But feel free to try it and let me know if you experience any issues and I will try to help.
     
    a436t4ataf likes this.
  25. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    [Unsupported - Package installation in 2019.2]

    To get the package in 2019.2:
    • Install it in a new project in 2019.3
    • Go to the project's folder into Library/PackageCache and copy the com.unity.testtools.codecoverage@0.2.x-preview folder
    • Paste it into your 2019.2 project's Packages folder
    Please note that the Code Coverage package is officially supported in 2019.3 and above.

    It is highly recommended having the latest version of 2019.2 installed.
     
    Last edited: Feb 8, 2020
    a436t4ataf likes this.
  26. Ippokratis

    Ippokratis

    Joined:
    Oct 13, 2008
    Posts:
    1,519
    This tool is a little gem. It gives you a "birds-eye view" of your code.
    I used the code coverage option on a game I work on for a long time.
    I have found many non-obvious mistakes I did (as well as many obvious ones), very fast, just by running the game as usual and checking the code coverage results.

    You can change the name to "Bad code revealer" :)
    Much appreciated!
     
    Last edited: Dec 8, 2019
  27. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    [Code Coverage package - version 0.2.2-preview is out]
    • The default Included Assemblies are now only the assemblies found under the project's Assets folder, instead of all project assemblies
    • After the report is generated, the file viewer window highlights the index.htm file, if Generate HTML Report is selected
    • Fixed unassigned CodeCoverageWindow.m_IncludeWarnings warning in 2019.3
    Documentation: https://docs.unity3d.com/Packages/com.unity.testtools.codecoverage@0.2
     
    Last edited: Dec 18, 2019
    Paulo_Mattos and GarthSmith like this.
  28. Unity-Nikos

    Unity-Nikos

    Unity Technologies

    Joined:
    Sep 30, 2015
    Posts:
    36
    Updated package codename to "Bad Code Revealer", from the "Oath to Quality" series
    (Thanks Ippokratis!)
     
    Ippokratis likes this.
  29. Marakino

    Marakino

    Joined:
    Aug 31, 2018
    Posts:
    4
    Greetings!

    It seems no matter what I try, I\m unable to generate the coverage report in batch mode.

    I'm running it like this:

    Code (csharp):
    1.  
    2. /Applications/Unity/Hub/Editor/2019.3.0f6/Unity.app/Contents/MacOS/Unity \
    3.                     -batchmode \
    4.                     -projectPath . \
    5.                     -runTests -testPlatform editmode \
    6.                     -testResults ./build/editmode-results.xml \
    7.                     -enableCodeCoverage -coverageResultsPath ./coverage \
    8.                     -coverageOptions 'enableCyclomaticComplexity;generateHtmlReport;generateBadgeReport' \
    9.                     -logFile /dev/stdout
    10.  
    It does create the /coverage/<project-name>-opencov/EditMode folder, but it's always empty.

    On the other hand, it works just fine when being run from within the Unity editor.

    Is there something obvious that I'm missing, or is this a known issue in the current version?

    Thank you in advance for any help!
     
  30. chrisj_unity

    chrisj_unity

    Unity Technologies

    Joined:
    Jun 26, 2018
    Posts:
    1
    Hi Marakino,

    I just tried your command line on a simple test project using the same version of Unity and the latest version of the coverage package (0.2.2-preview). It's working correctly for me.

    When you run the command, do you get a ./build/editmode-results.xml file? If so, could you check the file to make sure that some edit mode tests were actually run?
     
  31. Marakino

    Marakino

    Joined:
    Aug 31, 2018
    Posts:
    4
    Thank you so much for your reply, chrisj!

    I'm actually setting up my CI workflow and, while the editmode test results report is generated, I don't have any tests yet.

    This is probably the reason why it's not working for me.
    I assumed it would mimic the coverage report I get in the editor, showing 0% coverage.

    I'll definitely try again with some tests and come back here with my results.

    Again, thank you for your help.
     
    chrisj_unity likes this.
  32. Marakino

    Marakino

    Joined:
    Aug 31, 2018
    Posts:
    4
    chrisj, I can confirm that was the issue indeed - the coverage report is generated if at least one test is present.

    Thanks! :)
     
    chrisj_unity likes this.
unityunity