Question preprocessor #define set when running in testrunner?

Hi there,
I'm using the testrunner to run some unit-tests. (Edit-Mode)
Since there are parts of the code that are not initialized as it would during gameplay I would like to add preprocessor-blocks to prevent problems. Is there any #define set when starting the testrunner? Can't find anything about this in the documentation.
I tried adding a define in the test's asmdef, but this is also active in playmode so it won't help me here.

Any suggestion would be appreciated.
Thx

 

I'd much rather talk about the non-initialized issue you're having.

It's better to make tests work without #ifdef, especially if that #ifdef TESTING were to be in the runtime code. I would consider that a code smell.

If you rely in some runtime class participating in tests for some reason, but it isn't designed to participate in tests, there are several options:

• refactor the class to work with tests
• have the class provide safe defaults when not (fully) initialized
• test against an interface and make a mock class for testing that provides testable values

 

I would also suggest trying to rely as little as possible on whether tests are running or not. I find the cases where doing this is a good idea very limited. One example would be allowing to override dependencies (in a DI framework) while running tests, but not under normal conditions.

Having a pre-processor symbol would mean that code would need to be recompiled before running the tests, so it would be slow!

However, there is one way to check if tests are running or not, which is checking

TestContext.CurrentTestExecutionContext?.ExecutionStatus == TestExecutionStatus.Running

. However, you'll need to jump through a few hoops to actually use this. One is that you can't access this directly in player code (unless you are making a player tests build), and will have to use e.g. a

[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.SubsystemRegistration)]

editor-only method to set up a delegate to read the status. The other is that Unity doesn't clear this properly after finishing test runs, so you'll have to also reset it yourself! (I've reported a bug, and I believe it's been fixed in UTF 2.) You can use the same callback to do that (set the execution context to null).

 

I also advise NOT modifying game runtime code to check if tests are currently executing. This is a bad idea.
Instead, if your test code depends on things that must be initialized prior to running tests, consider adding setup logic that executes before your tests, using the SetUpFixture attribute:

Code (CSharp):

1. using NUnit.Framework;
2. using UnityEngine;
3.  
4. [SetUpFixture]
5. public class EditModeSetupFixture
6. {
7.     [OneTimeSetUp]
8.     public void Setup()
9.     {
10.         // Executes before all tests
11.     }
12.    
13.     [OneTimeTearDown]
14.     public void Teardown()
15.     {
16.         // Executes after all tests
17.     }
18. }

This setup fixture is executed before all tests in its current namespace (or more derived namespaces).
Since this class is not defined in a namespace, it will execute before all tests

This is a good central location for doing one-time init as well as cleanups.

 

Contrary to all the "this is a bad idea" ... how do you write clean API / library code that works correctly when being used by 3rd party calls that MIGHT be run inside a TestRunner test?

The one that just tripped me up: Undo.* API doesn't work correctly while a TestRunner EditMode test is running (TestRunner is like Application.playmode ... it requires you to disable/avoid using Undo.* APIs).

We can detect application isplaying in code, using the API's specifically for detecting that. That also covers us for TestRunner PlayMode tests (they update the API call to tell you that it's playing).

So far ... I'm struggling to find a clean, fast way to detect TestRunner.isRunningAnEditModeTest

 

I've written Undo.* test code to test my code's undo works fine with Unity's Undo system.
Note that this is for Edit mode tests. In Playmode tests you can't use editor API. What you said is incorrect: EditMode tests aren't like Playmode - this is only the case for Playmode tests.

All you really need is to check if you are in the editor AND playmode isn't active:

Code (CSharp):

1. if (Application.isEditor && Application.isPlaying == false)
2. {
3.     // your code here
4. }

 

I think you misread what I said, or misunderstood it.

1. playmode can be detected. This is true for both hitting Play, and for running playmode tests.
2. editor mode can be detected.
3. editor-mode WHILE running TestRunner EditMode Tests cannot be detected
4. Undo.* APIs work corretly in Editor mode but NOT in editor-mode with EditorTests
5. Undo.* APIs don't work in Play mode, and don't work in PlayMode tests - but this is fine, becuase Unity reports both those cases as 'running play mode'

 

What do you mean by that anyway? What exactly is not working correctly?
If the Undo fails to do its job in the edit-mode test, it's either the test setup or the undo code that's broken.
You may want to try a simple test, such as changing a serialized value in a MonoBehaviour and then call Undo in the test and then assert if the previous value has been restored. This should definitely work in edit-mode tests.

Yes, it can be. How are you trying to detect this? Please post your code.

The code being tested is not aware that it is being called from within a test. Ideally it should not have such a check!
If you feel that it ought to check that, for example to skip a time-consuming database connection process, you should change that because you don't want special test-mode cases in production code. Instead you could use a mock class for example that implements or wraps the target class but performs some shortcuts.

You could also implement the main TestRunner callback methods in a class to be informed when the test run starts and ends. I used that to write and reset EditorPrefs values to ensure tests run at full speed as described here.

 

Unity themselves couldn't get their code to run without being "aware that it is being called from within a test" -- so I don't get what's contentious here? It's a known issue with the Undo API(s). I assume that fixing it is sufficirently painful/difficult that no-one at Unity wanted to take up the challenge.

(For what little it's worth: I've found bugs in Unity's Undo API before, reported them, got them fixed. Some of them incredibly simple. It in no way surprises me that Undo fails in the presence of TestRunner tests)

My goal now is to find a clean workaround that means I and others can write code that doesn't corrupt the Scene (which is what eventually happens when you're relying on transform.parent to be correct and find that it isn't). That means: detecting when a EditMode TestRunner test is running.

For anyone else coming to this thread in future: I've found a workaround that's working but is unpleasant: check the call-stack for the presence of TestRunner code (this is obviously bad: reading the callstack is expensive! But at least you can wrap it in #if UNITY_EDITOR ... #endif so it won't affect your runtime/player builds)

Code (CSharp):

1. if( Application.isEditor
2.  && !Application.isPlaying
3.  && Environment.StackTrace.Contains( "UnityEngine.TestRunner" ) )

... seems (so far) to reliably give you "I am currently inside an EditMode test that is being run by something/someone".

 

Like I said, you could just set an EditorPrefs value within the TestRunner callbacks rather than using stacktraces.

Also, you shouldn't be corrupting a scene because if you do tests properly, you will load and run tests in a test scene and then discard it. I made myself an Attribute for test methods so that I can tell it to create an empty scene, or with which objects and components to fill it, or which existing (test-only) scene to load.

If you run your tests on production scenes, you're prone to shoot yourself in the foot really bad.

 

You're missing the point. Many of the core unity APIs - for instance this one: https://docs.unity3d.com/ScriptReference/Transform.SetParent.html - are capable of (and frequently do!) indirectly corrupting the scene if invoked in Editor outside of playmode. That is a built-in feature of Unity, unrelated to tests. It is a side-effect of how the Undo API was originally designed and implemented (instead of hooking the existing APIs, it duplicates them) and we are stuck with it. The easiest way forward is to delete all calls to (e.g.) Transform.SetParent and replace them with calls to EditorSafeTransform.SetParent, where EditorSafeTransform has #if UNITY_EDITOR ... [use the Undo API] .. [else use the playmode API] ... #endif (although it's a little more complex than that).

 

Imagine this: I am not writing any of the tests. I do not have access to the tests. I do not run any of the tests. I can't access the tests - they are in a dependent Assembly, so by definition: my code cannot ever see or interact with them, not even using Reflection.

But ... if I can detect that TestRunner itself is running .. not by me, but by someone else ... then I can make sure *my* code doesn't accidentally corrupt the scene (because I can avoid using API calls that are known to be broken while TestRunner is running).

 

Is this issue documented somewhere?

I find it hard to believe that such a commonly used method could corrupt the scene but only in tests. Maybe it's a potential issue, I don't know. But I've not seen or heard of a thing like that. Nor have I seen scripts safeguarding themselves against being run in a test, which I'd expect if it were a common issue.

I am familiar with the occasional developer on a ghost hunt trying to work around issues that don't actually exist (typically the perceived issue was actually a symptom or follow-up of something else). But then again, I do know the very rare case where that developer was onto something ...

That makes me curious. If this can be reproduced I'd like to see it in action myself just because I can't believe it. At least I'd like to see some reports of this issue.

 

Regarding the original question, I too have an issue that would be solved with a define. I have a RuntimeInitialiseOnLoad method which runs some code that is not compatible or necessary for a test runner environment, and I wanted to conditionally compile the attribute to ensure it didn't cause issues during testing.

Unfortunately, it turns out that my attempted solution, ITestPlayerBuildModifier, only runs on full player builds, not compilations for runtime tests.