Feedback for LogAssert class

Howdy,

Today was my first attempt of using the LogAssert class. I had spend a looong time to figure out why my test isn't working as i expect it to. Here is feedback about this class after trying (and failing) to use it properly today:

* First off, i tried looking for "logassert" on the forums and feedback site. Found 0 results.. this class doesn't really look like a popular one at the moment (which is great, since it can be easier to modify it without pissing off too many people!)

• All tests have this "hard-wired" behaviour that will basically fail them in case an error is logged to the console. It could've been a lot nicer (and cleaner) to be able to toggle this behaviour directly from the [Test] or [UnityTest] attributes. Something like this:

Code (CSharp):

1. [UnityTest(IgnoreLogErrors = true)]
2. public IEnumerator MyTest()
3. {
4.     // ...
5. }

This behaviour is related to the test itself, why should another class be responsible for turning this behaviour off? also, this way allows for more freedom (turning off for a specific test).

• In case i'm ignoring exceptions / error logs (so they don't fail my test), any Asserts i make will not cause the test to fail as well:

Code (CSharp):

1. [UnityTest]
2. public IEnumerator Test()
3. {
4.     yield return null;
5.  
6.     LogAssert.ignoreFailingMessages = true; // with this, asserts below will not fail the test
7.  
8.     Assert.That(0 == 2, "0 isn't equal to 2");
9.     Assert.AreEqual(0, 2, "0 isn't equal to 2");
10. }

This test passes (only tested for PlayMode test, but i suppose it'll be the same for editor tests as well). Not sure this is the desired behaviour... how can i fail the test now basically ?

• The Expect method (!!) is my biggest rant here. since it's a static method, i somehow assumed that setting it to expect a particular log type + msg will mean that *any* log messages like that will not fail the test. In reality, it only matches a single log message and then moves on.

For example, in my test i had set something like this:

LogAssert.Expect(LogType.Error, new Regex("duplicate key for.+"));

During my test, any number of messages from that form might be logged, and i don't want them to fail my test. With the current API, this only matches a single log message, but any subsequent error logs fail the build.

I realize this might be similar to other test frameworks, but it was still confusing to use and i had to spend a lot of time figuring out why it didn't work.

This can be solved in any number of ways, for example:

- LogAssert.ExpectOnce / LogAsset.ExpectAny
- LogAssert.Expect(string, int) --> expect a log message for the given number of times (-1 for any number?)
- Updating the documentation to stress out this fact. Currently this is the description (not sure if it's clear enough):

• Another issue - the Expect method will not fail the test in case a log that matches the expectation is seen, but will fail in case it's missing. basically, there's no API to say "don't fail my test in case this error message is shown". Should be added somehow to make it easier to ignore certain error messages.

Please consider this feedback when making changes to LogAssert or to the unit test framework in Unity

 

The test framework and other supporting libraries should probably be open sourced, so i can fix it for myself locally (as well as push those changes over to you, in case you're interested).

 

I was surprised to find that that multiple

Expect

calls will expect the log messages to appear in that order. This ought to be documented and ideally made optional.

 

And there appears to be something weird with calling

NoUnexpectedReceived

multiple times. This is the minimal test case I came up with:

Code (CSharp):

1.  
2. LogAssert.Expect(LogType.Log, "Hi");
3. Debug.Log("Hi");
4. LogAssert.NoUnexpectedReceived();
5.  
6. LogAssert.Expect(LogType.Log, "Hi");
7. Debug.Log("Hi");
8. LogAssert.NoUnexpectedReceived();

It complains about the second log message not being expected.
Remove either of the calls or change either of the log messages and it stops failing.

 

Any interesting updates about the test runner or test framework in general? @ElvisAlistar
I am actually using it more often nowadays and i have a few other things that could be useful.

Just open source it and i will do the rest

 

how are you using ignoreFailingMessages? are you setting it up in the beginning of the test?
You're catching the exception, but it's still logged to the console?

 

any updates you're able to share with us?

 

It's been almost a year. What's the delay? Should we continue using UnityTestFramework (since it's apparently not getting updates) and do our own integrations of NUnit instead?

Core features are still missing - can't mock MonoBehaviour even using the NUnit features specifcially created for doing so - but when we build this stuff itself (without any of Unity's code) it Just works. I found a couple more core bugs in Unity 2018's implementation of the Test framework - stuff that makes it almost completely useless (Unity outputs the wrong error messages and actively corrupts the code its running!). Lost a few hours discovering that. What gives?

 

i really wish they would open source the whole thing. i am really into unit testing and would love to donate any fixes or improvements to this framework for the sake of all Unity devs!

 

I just want to add a request for having more access to this. I know this isn't the most active part of the forum - thought I would throw another vote here for package + better access.

The feature that I am really missing is being able to run tests on standalone platform builds without having the editor kick off the tests. Also, it would be great to not need a network connection for capturing the output.

The pipe line I am working in means that I can't have dedicated editors per each device running the test builds, and I can't be guaranteed network access between my devices and the network either. In addition not all platforms yet support "Run all in Player" - it would be nice to short cut this.

 

In regard to the following line causing tests to pass even when asserts fail:

Code (CSharp):

1. LogAssert.ignoreFailingMessages = true;

I just spent about 4 hours tracking this issue down and came here to post about it. Sad to hear it has been unresolved in more than a year. We have unit tests that cover 3rd party plugins, some of which are logging errors that don't concern us. Without that line of code, our tests fail due to these errors that we don't care about. With it, our tests never fail even if they should.

 

We do use LogAssert but the only use we have for it is Unit Testing our in-game dev console. Therefore we know exactly what LogError input we're going to pipe through it. It would be helpful to just blanket ignore X number of tests or turn it off altogether. The code in LogAssert.Expect works exactly as (pardon the pun) expected, so porting that to work with LogAssert.ignoreFailingMessages shouldn't be too hard.

 

This would be great to send the build to Firebase Test Labs devices. Will the build be compatible with XCTest as well? Since it seems to be a framework Firebase Test Labs supports for iOS build.

A small suggestion, please allow any bundle name other than com.UnityTestRunner.UnityTestRunner. It affects Firebase library in particular and I have to make a new Firebase project with this bundle name and have to manually switch in a custom credential because this lib compares bundle name.

 

For those following this thread - check out my new Unity Test Extensions project on GitHub.
It contains the LogAssertEx class that provides some improvements that were missing in the original LogAssert.

Right now it offers basic improvements, but will keep on getting more stuff soon

 

Wow, that's 3999935 more tests than I run! I think I'm almost catching up

Is there a way to use LogAssert, but have a clear console at the end of the test? I have a function that logs an error when you use it wrong, and a test that uses LogAssert.Expect to make sure the error is logged, but it's cluttering the debug console every time I run the full test suite. It seems like "pull the error I just checked for out of the log" would be a useful command, at least for my workflow. Maybe it exists and I'm just using it wrong. As liortal said:

I had the honor of posting the first LogAssert question on the Answers site (as far as my search turned up), but I haven't had much luck there, so I thought I'd bomb the suggestion thread here as well.
(Just re-reading the thread, I noticed that this question was already brought up. Bump!)

 

You quoted me, so i came

Just out of general curiousity - why do you care so much about the console at the end of your tests?

Errors that appear there were expected, otherwise they sould fail the test.

You want to filter the console from expected messages? I think that you could implement something like this yourself by implementing the ILogger interface.

 

It would be great if there were a way to only have to handle Errors and Asserts, and ignore Logs when using LogAssert. Our log is verbose, and when I'm expecting an error, I have to register a lot (too many) of Log type logs.

 

I would like to have a LogAssert.Contains (like Assert.Contains) if not the complete log message is know. Is there any way to do that with the current features that I haven't seen?

 

LogAssert allows using a regex from what i remember. This sounds like what you need...

 

Could you please give an example, e.g. here?

 

@ActiveSim Here's an example of what i meant:

Code (CSharp):

1. LogAssert.Expect(LogType.Exception, new Regex("Assertion failure.*"));

This assertion will catch all cases of an exception with the text containing "Assertion failure", but doesn't have to match it precisely.

 

More fun: you can't stuff the

LogAssert.ignoreFailingMessages = true

call into a SetUp function. By the time any log calls happen in your test, it will have been reset back to false. Manually putting that line into each test works, so if you have multiple tests where you're expecting errors you have to call this for each test.

This is not documented, and totally unexpected. What's the point of setting a global "don't mess with my tests" if it resets itself?

 

is it me or the LogAssert.Expect doesn't work with Debug.LogError ? (Unity 2021.1.27f1 - Win)
it works fine if I use Debug.Log but not with Debug.LogError...

 

It works well for me. Did your code look like:

Code (CSharp):

1. LogAssert.Expect(LogType.Error, "your log content");

and is

Code (CSharp):

1.             LogAssert.ignoreFailingMessages = false;

 

The original suggestion of this post, to add a LogAssert.Expect where you can specify "Any number of times" would be really useful. Any news since it was proposed back in 2018?

 

+1 to this

 

It seems in the Expect() method, There's no option for ANY string message. I tried using NSubstitute's Arg.Any<string>() but no dice. It would be great to have an option to not care about what the message is.

 

It also takes a regex, just set it to match any string...

Code (CSharp):

1. LogAssert.Expect(new Regex(".+"));

 

Thanks. That's helpful! Although it assumes you know how to Regex.. haha. I'm happy to copy paste that though!

 

Hi all!

Seting

LogAssert.IgnoreFailingMessages = true

in a

[SetUp]

method not working as expected for PlayMode tests.

Under the hood

LogScope

exists in many instances with

IgnoreFailingMessages = false

by default.
Any call

LogAssert.IgnoreFailingMessages = true

change the property only for "current" instance of

LogSope

.

If your game log any error at startup you will always getting failed test run with "Unhandled log message"!

LogScope

u have no chance to set

LogAssert.IgnoreFailingMessages = true

for instance of that

LogSope

.

Actual for Test Framework 2.0.1-exp.2

---
As quick fix you can intro real static property for

LogAssert.IgnoreFailingMessages

and check tham in any

LogScope