Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice

Test Runner EditorMode / PlayMode tests and code coverage with NUnit

Discussion in '5.6 Beta' started by bdovaz, Dec 25, 2016.

  1. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    Hi,

    As you some know (like @QA-for-life @ElvisAlistar @Alex-Lian) I'm building our continuous integration system that can build projects (Jenkins) and perform code analysis and coverage (Sonar). https://forum.unity3d.com/threads/sonarqube-and-unity-code-quality.444490/

    I was looking after PlayMode tests that didn't made it into 5.5.

    I'm now using it in 5.6 beta and works really great.

    I see that you now use NUnit 3 (3.6).

    I also want to get the code coverage results but in "command line arguments" page there is no command available for code coverage (only for unit test execution with -runEditorTests): https://docs.unity3d.com/Manual/CommandLineArguments.html

    I can't do it myself from outside editor because: "System.Security.SecurityException : ECall methods must be packaged into a system module."

    Now I'm trying to do it with:

    Can you add it please? It shouldn't be difficult.
     
    Last edited: Dec 25, 2016
    bdovaz, Dec 25, 2016
    #1
    Ash-Blue, Mikael-H and sdgormley like this.

  2. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    bdovaz, Jan 4, 2017
    #2

  3. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    Bump.

    I insist @QA-for-life @ElvisAlistar @Alex-Lian. It's a really important matter. Unit testing without code coverage it's useless.

    I've in our pipeline, nightly builds from Jenkins that triggers a build step that runs the code analysis and coverage that reports it to SonarQube but I can't run tests that use Unity code because of this.

    Can I have at least an answer? Can I chat with any of you, please? Thanks.
     
    bdovaz, Jan 9, 2017
    #3

  4. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Code coverage is a just a metric like many other and unit tests have a great value by themselfs. We don't have a code coverage tooling in the plans for the nearest future.
     
    Tomek-Paszek, Jan 10, 2017
    #4

  5. bdominguezvw

    bdominguezvw

    Joined:
    Dec 4, 2013
    Posts:
    96
    Is there no workaround for this @Tomek-Paszek? It would be really useful if we could run a code coverage analysis by ourselves.

    Now we only can a code coverage analysis on code that don't use Unity API (System.Security.SecurityException : ECall methods must be packaged into a system module.).
     
    bdominguezvw, Jan 10, 2017
    #5

  6. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Some time ago we evaluated few open source solutions but haven't found anything that would work with Unity code out of the box. We haven't tried commercial solutions.
     
    Tomek-Paszek, Jan 10, 2017
    #6

  7. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    Can't you at least enable it with the solution I proposed (Visual Studio it's used by almost anyone)? Code coverage is something advanced that it's not going to be used by people starting to learn unity, it's mostly used by professional developers working on companies and wouldn't be hard to understand that it requires more advanced solutions like Visual Studio because any company developing applications using Unity uses Visual Studio IDE.

    The solution I proposed should work, it works for me on test that don't use unity API and it would work on the other tests if the are executed inside unity itself.

    If you don't evaluate Visual Studio solution (largely used by developers) it's going to be something that it's going to be on standby always...
     
    bdovaz, Jan 10, 2017
    #7

  8. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    @Tomek-Paszek we've been able to reach a 20% code coverage on our codebase on .NET API code but we can't pass that because when I'm going to analyze code coverage involving Unity API it crashes with the f****** "System.Security.SecurityException : ECall methods must be packaged into a system module." exception.

    It there nothing that we can do on our end? A simple workaround/guide/tips/help would be nice. Or it's impossible without Unity staff involvement?
     
    bdovaz, Jan 14, 2017
    #8

  9. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    "System.Security.SecurityException : ECall methods must be packaged into a system module." happend because you are trying to invoke Unity API outside of the Unity environment where binding with the navive layer don't exist (thus "ECall methods..." exception). There is no workaround for that nor a way to simply enable it in VS, for example. Most of Unity API needs to be executed inside the editor or player.
     
    Tomek-Paszek, Jan 16, 2017
    #9

  10. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    That's why I need that you implement the minimal possible to execute it from editor via command line.

    It's really so much work? Please talk about it internally.
     
    bdovaz, Jan 16, 2017
    #10

  11. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Not sure what you expect us to make working? There is no magical button we can switch to turn code coverage on.
     
    Tomek-Paszek, Jan 16, 2017
    #11

  12. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    I know that.

    I mean that if it's possible to ingrate Visual Studio code coverage tool, some other open source tool or you need to develop one on your own.

    I only need some technical explanation to understand what's the work involved in this.
     
    bdovaz, Jan 16, 2017
    #12

  13. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    To my knowledge, a code coverage tool usually is implemented with profiling the code. That means the tool usually weaves in code marks (during compilation code is added to every method call or even to every line of code). That happens either during compilation or after, where the assembly is modified (which essentially gives you a different, slower version of the same assembly). In case of Unity you also need to remember that it runs on Mono and not every tools that is designed for .Net will work automatically with Mono.

    This is not a hard thing do to (from a technical point of view) but it's messing with a system that makes it very likely to break many other things.

    I can refer you to https://feedback.unity3d.com/ which we use to prioritise work on new features.
     
    Tomek-Paszek, Jan 17, 2017
    #13
    GarthSmith and LeonhardP like this.

  14. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    I see.

    But seeing that it's not a hard thing to do it could be a nice challenge for the test runner & qa team to achieve on the upcoming versions.

    I opened this thread because in this version I'm seeing movement in this area with Play Mode tests and NUnit library upgrade so I had the feeling that you could listen to my suggestions. If you want to chat about this feel free to do it because I have a Jenkins+Sonar combo on our codebase that is ready to try this when you work on it.

    Unity it's getting bigger and developers are doing more and more complex projects that need tools like this.
     
    bdovaz, Jan 17, 2017
    #14
    Playcorp likes this.

  15. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    bdovaz, Jan 31, 2017
    #15
    GabLeRoux likes this.

  16. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,052
    @Tomek-Paszek I have filled a bug regarding test runner batch mode.

    (Case 892798)

    Please, check it out.

    Do you have any news regarding code coverage? Any internal conversation. Microsoft it's going open source with many tools regarding code coverage as you can see.
     
    bdovaz, Mar 19, 2017
    #16