Bug Unity sometimes terminates prematurely when running tests from command line

I recently created a sample UI test and am trying to run it from the command line. The command I'm using is as follows:

unity -runTests -projectPath <project path> -testResults <result file path> -testPlatform playmode -editorTestsFilter TestCloseButton

However, the Unity editor sometimes quits after showing the splash screen without running the tests. This issue isn't 100% reproducible but seems to happen more often after the computer is rebooted. However, subsequent runs tend to complete without problems.

Is this a bug?

For what it's worth, I'm running Unity 2017.2.0p2 on Windows 10 Pro. The only reason I haven't upgraded to the latest version is that it's not yet supported by our team.

 

@DannyChiaIntel Check if unity lists the tests once opening, otherwise you might encounter the same bug as we do.
Ignore the check marks for succeeded, fails and unknown tests - the tests itself must be listed.
In that case you need to add one step before and reimport whatever is necessary (careful the posted script is missing a reimport of the asmdef files, but I will add it).

If it is indeed a bug, file a bug report and attach a small project reproducing the error.

 

Thanks for the response. I suspect it's the same issue as I don't recall seeing the test runner window whenever the editor has quit prematurely.

Unfortunately, I can't really share any projects as our code is confidential. Is there anywhere I can go to check the editor logs?

 

@DannyChiaIntel uploading a large project would not help much anyway as it is unlikely that unity is looking at it in the first place. However, try to reproduce it in a new project. We are doing so as well, but so far I was not able to reproduce it from a small new project, therefore it has to do with something more complex and likely I guess refactorings, different serialization/deserialization, namespace changes and assembly changes all play into it.

My biggest bet so far is, that something of the following of Unity is bugged:

• I do not know how much state the editor has itself, but it has to be either a bug in it's state or a bug in the /Library folder. I have diffed both the .cs, .asset and their .meta files before and after reimport and they are identical.
• The error happens for us in one particular project, the project is much larger and more complex using all the features available through asmdef files - while projects using simpler configurations do not show the problem.
• I am pretty sure the order of reimport causes the bug as reimporting the scripts after the prefabs does not fix it, so the scripts and scriptableobjects have to be reimported in the correct order first and because the script is reimported incorrectly I guess the test runner does not detect them. The bug is likely the same for scripts/scriptable objects and prefabs.
• As the order matters I assume the dependencies of the asmdef configurations cause it. A wild guess would be, that it is fixed by reimporting because some assemblies are succesfully compiled, then reused the second time when the system is "hot".

I have updated the script we are using as additional build step before any real CI/CD but be careful, that takes quite a while and I assume there are a lot of unecessary recompilations triggered that way. For our project this procedure adds about 30 minutes to each build configuration.

 

@DannyChiaIntel Out of interest - can you specify if you are using a Mac or Windows machine for running playmode tests?

The reason i am asking is we have a CI server environment and i'm having issues with running playmode tests on Windows based machines.

 

I'm using Windows 10.