Search Unity

  1. Unity 2019.1 is now released.
    Dismiss Notice

Feedback for LogAssert class

Discussion in 'Testing & Automation' started by liortal, May 8, 2018.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,179
    Howdy,

    Today was my first attempt of using the LogAssert class. I had spend a looong time to figure out why my test isn't working as i expect it to. Here is feedback about this class after trying (and failing) to use it properly today:

    * First off, i tried looking for "logassert" on the forums and feedback site. Found 0 results.. this class doesn't really look like a popular one at the moment (which is great, since it can be easier to modify it without pissing off too many people!)
    • All tests have this "hard-wired" behaviour that will basically fail them in case an error is logged to the console. It could've been a lot nicer (and cleaner) to be able to toggle this behaviour directly from the [Test] or [UnityTest] attributes. Something like this:
    Code (CSharp):
    1. [UnityTest(IgnoreLogErrors = true)]
    2. public IEnumerator MyTest()
    3. {
    4.     // ...
    5. }
    This behaviour is related to the test itself, why should another class be responsible for turning this behaviour off? also, this way allows for more freedom (turning off for a specific test).
    • In case i'm ignoring exceptions / error logs (so they don't fail my test), any Asserts i make will not cause the test to fail as well:
    Code (CSharp):
    1. [UnityTest]
    2. public IEnumerator Test()
    3. {
    4.     yield return null;
    5.  
    6.     LogAssert.ignoreFailingMessages = true; // with this, asserts below will not fail the test
    7.  
    8.     Assert.That(0 == 2, "0 isn't equal to 2");
    9.     Assert.AreEqual(0, 2, "0 isn't equal to 2");
    10. }
    This test passes (only tested for PlayMode test, but i suppose it'll be the same for editor tests as well). Not sure this is the desired behaviour... how can i fail the test now basically ?
    • The Expect method (!!) is my biggest rant here. since it's a static method, i somehow assumed that setting it to expect a particular log type + msg will mean that *any* log messages like that will not fail the test. In reality, it only matches a single log message and then moves on.
    For example, in my test i had set something like this:
    LogAssert.Expect(LogType.Error, new Regex("duplicate key for.+"));

    During my test, any number of messages from that form might be logged, and i don't want them to fail my test. With the current API, this only matches a single log message, but any subsequent error logs fail the build.

    I realize this might be similar to other test frameworks, but it was still confusing to use and i had to spend a lot of time figuring out why it didn't work.

    This can be solved in any number of ways, for example:

    - LogAssert.ExpectOnce / LogAsset.ExpectAny
    - LogAssert.Expect(string, int) --> expect a log message for the given number of times (-1 for any number?)
    - Updating the documentation to stress out this fact. Currently this is the description (not sure if it's clear enough):
    • Another issue - the Expect method will not fail the test in case a log that matches the expectation is seen, but will fail in case it's missing. basically, there's no API to say "don't fail my test in case this error message is shown". Should be added somehow to make it easier to ignore certain error messages.
    Please consider this feedback when making changes to LogAssert or to the unit test framework in Unity :)
     
    Last edited: Jan 6, 2019
    chauteville likes this.
  2. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    214
    Thanks for the feedback. All very valid points and we should implement as much of it as possible.
     
    liortal likes this.
  3. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,179
    The test framework and other supporting libraries should probably be open sourced, so i can fix it for myself locally (as well as push those changes over to you, in case you're interested).
     
  4. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    214
    It will become a Unity package soon, so you will be able to change it locally.
     
  5. mpartel

    mpartel

    Joined:
    Mar 9, 2013
    Posts:
    2
    I was surprised to find that that multiple
    Expect
    calls will expect the log messages to appear in that order. This ought to be documented and ideally made optional.
     
    liortal likes this.
  6. mpartel

    mpartel

    Joined:
    Mar 9, 2013
    Posts:
    2
    And there appears to be something weird with calling
    NoUnexpectedReceived
    multiple times. This is the minimal test case I came up with:

    Code (CSharp):
    1.  
    2. LogAssert.Expect(LogType.Log, "Hi");
    3. Debug.Log("Hi");
    4. LogAssert.NoUnexpectedReceived();
    5.  
    6. LogAssert.Expect(LogType.Log, "Hi");
    7. Debug.Log("Hi");
    8. LogAssert.NoUnexpectedReceived();
    It complains about the second log message not being expected.
    Remove either of the calls or change either of the log messages and it stops failing.
     
  7. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,179
    Any interesting updates about the test runner or test framework in general? @ElvisAlistar
    I am actually using it more often nowadays and i have a few other things that could be useful.

    Just open source it and i will do the rest :)
     
  8. vestigial

    vestigial

    Joined:
    May 9, 2015
    Posts:
    24
    I have a related question: inside a test, I am catching an exception. I use
    Code (CSharp):
    1.            LogAssert.ignoreFailingMessages = true;
    so my test does pass. However, I still get a LogType.Exception printed in the console. Functionally, not a problem, but when I run all my tests its a little annoying to see a bunch of errors printed, even though my tests pass. Is there a way to stop Unity from writing an error to the console, when an exception is caught? (and not rethrown)
     
  9. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,179
    how are you using ignoreFailingMessages? are you setting it up in the beginning of the test?
    You're catching the exception, but it's still logged to the console?
     
  10. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,179
    any updates you're able to share with us?
     
  11. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    17
    We are still working on it actively, but I cannot give any specific timeframe.
     
  12. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    189
    It's been almost a year. What's the delay? Should we continue using UnityTestFramework (since it's apparently not getting updates) and do our own integrations of NUnit instead?

    Core features are still missing - can't mock MonoBehaviour even using the NUnit features specifcially created for doing so - but when we build this stuff itself (without any of Unity's code) it Just works. I found a couple more core bugs in Unity 2018's implementation of the Test framework - stuff that makes it almost completely useless (Unity outputs the wrong error messages and actively corrupts the code its running!). Lost a few hours discovering that. What gives?
     
  13. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,179
    i really wish they would open source the whole thing. i am really into unit testing and would love to donate any fixes or improvements to this framework for the sake of all Unity devs!
     
  14. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    17
    Hey

    Internally in Unity we are using the Unity Test Framework heavily. Currently we run 4 million test runs daily on that framework. It is a test framework that we are betting on internally and thus you can be sure that we will keep supporting it.

    The iteration time for us to fix issues and apply feedback to the framework is currently not good enough and that is our main motivation to move the framework into a package. It is however, due to our heavy usage of the test framework that it is taking a long time to make the switch to a package. We plan to ship it as a package in 2019.2.
     
    liortal likes this.
  15. jonbro5556

    jonbro5556

    Joined:
    Jan 1, 2013
    Posts:
    23
    I just want to add a request for having more access to this. I know this isn't the most active part of the forum - thought I would throw another vote here for package + better access.

    The feature that I am really missing is being able to run tests on standalone platform builds without having the editor kick off the tests. Also, it would be great to not need a network connection for capturing the output.

    The pipe line I am working in means that I can't have dedicated editors per each device running the test builds, and I can't be guaranteed network access between my devices and the network either. In addition not all platforms yet support "Run all in Player" - it would be nice to short cut this.
     
  16. gtzpower

    gtzpower

    Joined:
    Jan 23, 2011
    Posts:
    246
    In regard to the following line causing tests to pass even when asserts fail:
    Code (CSharp):
    1. LogAssert.ignoreFailingMessages = true;
    I just spent about 4 hours tracking this issue down and came here to post about it. Sad to hear it has been unresolved in more than a year. We have unit tests that cover 3rd party plugins, some of which are logging errors that don't concern us. Without that line of code, our tests fail due to these errors that we don't care about. With it, our tests never fail even if they should.
     
    liortal likes this.
  17. Stormy102

    Stormy102

    Joined:
    Jan 17, 2014
    Posts:
    478
    We do use LogAssert but the only use we have for it is Unit Testing our in-game dev console. Therefore we know exactly what LogError input we're going to pipe through it. It would be helpful to just blanket ignore X number of tests or turn it off altogether. The code in LogAssert.Expect works exactly as (pardon the pun) expected, so porting that to work with LogAssert.ignoreFailingMessages shouldn't be too hard.
     
  18. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    17
    We are working on something for this. It will allow you to split the building of the player from the actual run, thus being able to do the build on a machine without the device and then copy the resulting player to a device/a machine with the device and execute it. We then have some hooks that will allow you to process the test result yourself.
     
    5argon likes this.
  19. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,178
    This would be great to send the build to Firebase Test Labs devices. Will the build be compatible with XCTest as well? Since it seems to be a framework Firebase Test Labs supports for iOS build.

    A small suggestion, please allow any bundle name other than com.UnityTestRunner.UnityTestRunner. It affects Firebase library in particular and I have to make a new Firebase project with this bundle name and have to manually switch in a custom credential because this lib compares bundle name.