Save Dialog cancels Batch Mode test run

In one of our projects, we can't run our (edit mode) tests in batch mode on our CI (Jenkins) server, because for some reason Unity decides that before running the test, the current (Untitled) scene needs to be saved. But in batch mode that results in cancelling the tests altogether.
Part of the log file:

[Project] Loading completed in 335.912 seconds
Project init time: 335.785 seconds
Services packages init time: 0.000 seconds
Package Manager init time: 0.998 seconds
Asset Database init time: 7.336 seconds
Global illumination init time: 0.269 seconds
Assemblies load time: 1.975 seconds
Unity extensions init time: 0.001 seconds
Asset Database refresh time: 205.563 seconds
Scene opening time: 0.077 seconds
GraniteSDK > Be sure to place a Granite Manager in your scene. The editor automatically registered a GraniteManager but this will not happen in-game.
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:LogWarning(Object)
GraniteSDK.EditorUpdate:Initialize()
GraniteSDK.EditorUpdate:UpdateEditorCamera()
GraniteSDK.EditorFunctions:OnEditorUpdate()
UnityEditor.EditorApplication:Internal_CallUpdateFunctions() (at C:\buildslave\unity\build\Editor\Mono\EditorApplication.cs:200)

(Filename: C:/buildslave/unity/build/Editor/Mono/EditorApplication.cs Line: 200)


Hashing assets (1 files)... 0.000 seconds
file read: 0.000 seconds (0.005 MB)
wait for write: 0.000 seconds (I/O thread blocked by consumer, aka CPU bound)
wait for read: 0.000 seconds (CPUT thread waiting for I/O thread, aka disk bound)
hash: 0.000 seconds
Connect to CacheServer 10.10.3.14:8126
Disconnect from CacheServer
Refreshing native plugins compatible for Editor in 0.61 ms, found 4 plugins.
Preloading 0 native plugins for Editor in 0.00 ms.
Running tests for EditMode
Executing tests with settings: test mode = EditMode
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
UnityEditor.TestTools.TestRunner.CommandLineTest.Executer:InitializeAndExecuteRun(String[]) (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\Executer.cs:42)
UnityEditor.TestTools.TestRunner.CommandLineTest.TestStarter:UpdateWatch() (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\TestStarter.cs:48)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions() (at C:\buildslave\unity\build\Editor\Mono\EditorApplication.cs:200)

(Filename: C:/buildslave/unity/build/Extensions/TestRunner/UnityEditor.TestRunner/CommandLineTest/Executer.cs Line: 42)

Cancelling DisplayDialogComplex: Scene(s) Have Been Modified Do you want to save the changes you made in the scenes:
Untitled

Your changes will be lost if you don't save them.
This should not be called in batch mode.
UnityEditor.SceneManagement.EditorSceneManager:SaveCurrentModifiedScenesIfUserWantsTo()
UnityEditor.TestTools.TestRunner.EditModeLauncher:Run() (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\TestLaunchers\EditModeLauncher.cs:28)
UnityEditor.TestTools.TestRunner.Api.TestRunnerApi:Execute(ExecutionSettings) (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\Api\TestRunnerApi.cs:25)
UnityEditor.TestTools.TestRunner.CommandLineTest.Executer:InitializeAndExecuteRun(String[]) (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\Executer.cs:43)
UnityEditor.TestTools.TestRunner.CommandLineTest.TestStarter:UpdateWatch() (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\TestStarter.cs:48)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions() (at C:\buildslave\unity\build\Editor\Mono\EditorApplication.cs:200)

[C:\buildslave\unity\build\Editor/Platform/Windows/EditorUtility.cpp line 1255]
(Filename: C:/buildslave/unity/build/Extensions/TestRunner/UnityEditor.TestRunner/TestLaunchers/EditModeLauncher.cs Line: 28)
​

It's not entirely clear to me what causes the Untitled scene (which is empty, not loaded from disk) to have changed but that may be due to the Granite plugin we use.
The dialog appears when I run Edit Mode tests in the editor too, and cancelling it causes the tests not being run there as well. Only choosing "don't save" will actually start the tests.

How can I work around this issue? Or is this a bug that can be fixed in the test framework? In batch mode, the default choice in this case should be "Don't Save" instead of "Cancel", I think.

 

Are you sure this is the root cause for the test to fail?

In my understanding, batch mode "overrides" any dialog by simply preventing it from being displayed (since there's no user who could interact with the editor).

Can you attach the full log ?

 

Yes the override CANCELS the hidden dialog instead of choosing "Don't Save". I've updated the question with some more of the log file.

Also, the tests don't fail, they aren't run and Unity sits idle until we manually cancel the CI job.

 

Near the top of your log: "GraniteSDK > Be sure to place a Granite Manager in your scene. The editor automatically registered a GraniteManager but this will not happen in-game."

It seems this GraniteSDK might add something your scene automatically.

Auto-cancels seems like bad choice of behaviour in batch-mode regardless.

 

@frank-ijsfontein did you fix this issue of the edit mode tests not being run? We are experiencing the same issue with our TeamCity server where the build just hangs because of this exact same issue

 

@hugeandy No this issue was never fixed. We decided not to run the tests in CI anymore for this project. And then the project was finished so I never revisited it. And Granite was bought and assimilated by Unity so this specific case will not likely come back to haunt us in other projects.

In the mean time I have learned some more about doing Play Mode tests and loading a specific scene before running those (Play Mode) tests.
I've never tried it, but it may be a solution to also load a specific scene before running Edit Mode tests? One that doesn't trigger changes that want to be saved.

 

This sounds like a great idea, I'll give it a go. Although it's very annoying it's needed, if it gets us going again then I'm happy!

 

Hey man I know I'm late for the party but..
Did you fix this issue by adding new scene before automation?
I did, I even saved all loaded scenes in the script but the result is same.. any help would be grateful.

 

@WhiteDoggy I've been racking my brain to remember about this but I can't, and I can't find any implementation of the suggested fix in the project. I have a feeling that in the end it was something else causing the issue that we either fixed or removed.

Sorry to not be more helpful!

 

we are also facing the same issue and not sure how to fix it. it's random, sometimes it goes on its own by a retry/rebuild sometimes it persists longer.
is this something caused by the project or Unity being Unity?