Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Save Dialog cancels Batch Mode test run

Discussion in 'Testing & Automation' started by frank-ijsfontein, Dec 5, 2019.

  1. frank-ijsfontein

    frank-ijsfontein

    Joined:
    Sep 11, 2015
    Posts:
    16
    In one of our projects, we can't run our (edit mode) tests in batch mode on our CI (Jenkins) server, because for some reason Unity decides that before running the test, the current (Untitled) scene needs to be saved. But in batch mode that results in cancelling the tests altogether.
    Part of the log file:

    [Project] Loading completed in 335.912 seconds
    Project init time: 335.785 seconds
    Services packages init time: 0.000 seconds
    Package Manager init time: 0.998 seconds
    Asset Database init time: 7.336 seconds
    Global illumination init time: 0.269 seconds
    Assemblies load time: 1.975 seconds
    Unity extensions init time: 0.001 seconds
    Asset Database refresh time: 205.563 seconds
    Scene opening time: 0.077 seconds
    GraniteSDK > Be sure to place a Granite Manager in your scene. The editor automatically registered a GraniteManager but this will not happen in-game.
    UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
    UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    UnityEngine.Logger:Log(LogType, Object)
    UnityEngine.Debug:LogWarning(Object)
    GraniteSDK.EditorUpdate:Initialize()
    GraniteSDK.EditorUpdate:UpdateEditorCamera()
    GraniteSDK.EditorFunctions:OnEditorUpdate()
    UnityEditor.EditorApplication:Internal_CallUpdateFunctions() (at C:\buildslave\unity\build\Editor\Mono\EditorApplication.cs:200)

    (Filename: C:/buildslave/unity/build/Editor/Mono/EditorApplication.cs Line: 200)


    Hashing assets (1 files)... 0.000 seconds
    file read: 0.000 seconds (0.005 MB)
    wait for write: 0.000 seconds (I/O thread blocked by consumer, aka CPU bound)
    wait for read: 0.000 seconds (CPUT thread waiting for I/O thread, aka disk bound)
    hash: 0.000 seconds
    Connect to CacheServer 10.10.3.14:8126
    Disconnect from CacheServer
    Refreshing native plugins compatible for Editor in 0.61 ms, found 4 plugins.
    Preloading 0 native plugins for Editor in 0.00 ms.
    Running tests for EditMode
    Executing tests with settings: test mode = EditMode
    UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
    UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    UnityEngine.Logger:Log(LogType, Object)
    UnityEngine.Debug:Log(Object)
    UnityEditor.TestTools.TestRunner.CommandLineTest.Executer:InitializeAndExecuteRun(String[]) (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\Executer.cs:42)
    UnityEditor.TestTools.TestRunner.CommandLineTest.TestStarter:UpdateWatch() (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\TestStarter.cs:48)
    UnityEditor.EditorApplication:Internal_CallUpdateFunctions() (at C:\buildslave\unity\build\Editor\Mono\EditorApplication.cs:200)

    (Filename: C:/buildslave/unity/build/Extensions/TestRunner/UnityEditor.TestRunner/CommandLineTest/Executer.cs Line: 42)

    Cancelling DisplayDialogComplex: Scene(s) Have Been Modified Do you want to save the changes you made in the scenes:
    Untitled

    Your changes will be lost if you don't save them.
    This should not be called in batch mode.

    UnityEditor.SceneManagement.EditorSceneManager:SaveCurrentModifiedScenesIfUserWantsTo()
    UnityEditor.TestTools.TestRunner.EditModeLauncher:Run() (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\TestLaunchers\EditModeLauncher.cs:28)
    UnityEditor.TestTools.TestRunner.Api.TestRunnerApi:Execute(ExecutionSettings) (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\Api\TestRunnerApi.cs:25)
    UnityEditor.TestTools.TestRunner.CommandLineTest.Executer:InitializeAndExecuteRun(String[]) (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\Executer.cs:43)
    UnityEditor.TestTools.TestRunner.CommandLineTest.TestStarter:UpdateWatch() (at C:\buildslave\unity\build\Extensions\TestRunner\UnityEditor.TestRunner\CommandLineTest\TestStarter.cs:48)
    UnityEditor.EditorApplication:Internal_CallUpdateFunctions() (at C:\buildslave\unity\build\Editor\Mono\EditorApplication.cs:200)

    [C:\buildslave\unity\build\Editor/Platform/Windows/EditorUtility.cpp line 1255]
    (Filename: C:/buildslave/unity/build/Extensions/TestRunner/UnityEditor.TestRunner/TestLaunchers/EditModeLauncher.cs Line: 28)
    It's not entirely clear to me what causes the Untitled scene (which is empty, not loaded from disk) to have changed but that may be due to the Granite plugin we use.
    The dialog appears when I run Edit Mode tests in the editor too, and cancelling it causes the tests not being run there as well. Only choosing "don't save" will actually start the tests.

    How can I work around this issue? Or is this a bug that can be fixed in the test framework? In batch mode, the default choice in this case should be "Don't Save" instead of "Cancel", I think.
     
    Last edited: Dec 5, 2019
  2. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Are you sure this is the root cause for the test to fail?

    In my understanding, batch mode "overrides" any dialog by simply preventing it from being displayed (since there's no user who could interact with the editor).

    Can you attach the full log ?
     
  3. frank-ijsfontein

    frank-ijsfontein

    Joined:
    Sep 11, 2015
    Posts:
    16
    Yes the override CANCELS the hidden dialog instead of choosing "Don't Save". I've updated the question with some more of the log file.

    Also, the tests don't fail, they aren't run and Unity sits idle until we manually cancel the CI job.
     
    Last edited: Dec 5, 2019
  4. Yacuzo

    Yacuzo

    Joined:
    Sep 22, 2016
    Posts:
    27
    Near the top of your log: "GraniteSDK > Be sure to place a Granite Manager in your scene. The editor automatically registered a GraniteManager but this will not happen in-game."

    It seems this GraniteSDK might add something your scene automatically.

    Auto-cancels seems like bad choice of behaviour in batch-mode regardless.
     
  5. hugeandy

    hugeandy

    Joined:
    Nov 2, 2016
    Posts:
    135
    @frank-ijsfontein did you fix this issue of the edit mode tests not being run? We are experiencing the same issue with our TeamCity server where the build just hangs because of this exact same issue
     
  6. frank-ijsfontein

    frank-ijsfontein

    Joined:
    Sep 11, 2015
    Posts:
    16
    @hugeandy No this issue was never fixed. We decided not to run the tests in CI anymore for this project. And then the project was finished so I never revisited it. And Granite was bought and assimilated by Unity so this specific case will not likely come back to haunt us in other projects.

    In the mean time I have learned some more about doing Play Mode tests and loading a specific scene before running those (Play Mode) tests.
    I've never tried it, but it may be a solution to also load a specific scene before running Edit Mode tests? One that doesn't trigger changes that want to be saved.
     
  7. hugeandy

    hugeandy

    Joined:
    Nov 2, 2016
    Posts:
    135
    This sounds like a great idea, I'll give it a go. Although it's very annoying it's needed, if it gets us going again then I'm happy!
     
  8. WhiteDoggy

    WhiteDoggy

    Joined:
    Sep 10, 2020
    Posts:
    17
    Hey man I know I'm late for the party but..
    Did you fix this issue by adding new scene before automation?
    I did, I even saved all loaded scenes in the script but the result is same.. any help would be grateful.
     
  9. hugeandy

    hugeandy

    Joined:
    Nov 2, 2016
    Posts:
    135
    @WhiteDoggy I've been racking my brain to remember about this but I can't, and I can't find any implementation of the suggested fix in the project. I have a feeling that in the end it was something else causing the issue that we either fixed or removed.

    Sorry to not be more helpful!
     
  10. hamza_unity995

    hamza_unity995

    Joined:
    Apr 19, 2022
    Posts:
    8
    we are also facing the same issue and not sure how to fix it. it's random, sometimes it goes on its own by a retry/rebuild sometimes it persists longer.
    is this something caused by the project or Unity being Unity?