Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Unity Test Runner is much slower if Unity is minimized or the screen is locked

Discussion in 'Testing & Automation' started by Revolter, Jun 28, 2019.

  1. Revolter

    Revolter

    Joined:
    Mar 15, 2014
    Posts:
    217
    3 minutes in focus VS 30 min with locked screen on Windows 10
    Unity 2017.4.16
     
  2. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    I wonder what are your results if you run your tests from the command line.
     
  3. Revolter

    Revolter

    Joined:
    Mar 15, 2014
    Posts:
    217
    also around 3-4 minutes
     
  4. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Well, there is a workaround... dont run tests and then minimize :)
     
  5. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    226
    @Revolter This sounds very much like something we fixed in later Unity versions. Can you try with the latest Unity 2018 version and let me know if you see the same behaviour?
     
  6. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,353
    This is very visible in 2018.4.2f1. I've got a two-monitor setup, and if I tab to the other monitor, the speed of the progress bar immediately slows down massively.

    I don't quite have the time to test 2018.4.3f1 right now, but unless you think the fix hit in that exact patch, I'd say it still there in 2018.
     
  7. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    226
    Please submit a bug report if you haven't done that already. Mention the case number here, so I can follow up on it! Thanks!
     
  8. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,353
    It's a bit hard, it doesn't seem consistent.

    I've only been able to repro it in our full game, not in a repro project, and only sometimes - it seems that it happens when Windows Defender is doing a scan.

    Anyways, that means that it's probably an OS issue? If the system is low on resources, it makes sense that it prioritizes the window that has focus over windows that doesn't.

    The difference is pretty staggering - 10 seconds when Unity has focus, 60 when it doesn't. But, as I said, only reproducible when there's a virus scan going on, and only in one, large project.

    I'm not going to spend any more time trying to create a repro, as this doesn't really affect me that much. But it does for sure happen, it's just hard to tell why and when.

    Here's a gif of it in action. The cancel button has a blue outline when Unity has focus: bug.gif
     
  9. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    226
    Yeah, that could be it. Internally some of us are adding our local repo clones, project folders, and related processes as exceptions to Windows Defender so that it never kicks in when we're working with those. Sure, there's some "risk" involved with that, but I think the speed gains we get from it are significant enough that we're willing to take it.
     
  10. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,353
    That's already done. Rider has started giving an option to automatically configure the repo and related files to be excluded by Defender.

    I have tried to exclude Unity, but I can't quite figure out if you're supposed to exclude the app name (Unity) or the executable name (Unity.exe), or the path to the executable or something else. Any insight?
     
  11. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    226
    I don't quite know how Defender handles these exclusions. I've excluded both the path and the executable just to be sure.
     
    Baste likes this.
  12. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,932
    I am seeing this issue in Unity 2022.2.17f1 and Rider 2023.1 (April 4th).

    When I run all tests, I get about 5-7 times slower test runs in Rider vs TestRunner:

    TestRunner: 3-4s
    Rider: ~20s

    When I start to run all tests in Rider and immediately tab into Unit Editor, the tests finish in 4s as expected.

    I've opened an issue here: https://youtrack.jetbrains.com/issu...slower-in-Rider-vs-TestRunner-in-Unity-Editor

    I have tried several Unity and Windows settings (see issue details) but none made a difference.
     
    Last edited: Apr 30, 2023
  13. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,932
    UPDATE: I checked the reverse situation: running TestRunner in the background.

    Indeed TestRunner runs a lot slower when immediately tabbing to another app after clicking on "Run All" in the Unity Editor. The difference was 2.5s focused Unity editor vs 15s Unity edit in background in three subsequent runs. With 6x slower timings this seems just the same difference I was observing running tests in Rider.

    Thus this seems to be a Unity issue, or worse, one regarding Windows 11 behaviour slowing down background processes. I will submit a report.
     
  14. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,932
    WORKAROUND #1: Keep moving the mouse while tests run in the background. There are scripts/tools that can do this for you. HOWEVER this only works when the mouse is moved over the Unity Editor window, so not really applicable as it does not speed up running Unity TestRunner tests from the IDE (Rider).

    WORKAROUND #2: I noticed that changing Preferences => General => Busy Progress Delay value by clicking & dragging on the label will make tests run faster in Rider (5s instead of 20s). Note that you have to drag to change the value, entering a value directly won't work. Also it does not matter what the value is or how much you change it. I hope to find out what setting/method this calls behind the scenes in order to call this every time tests start, but had no luck so far.

    I also tried a few more Windows settings and opened a superuser question in hopes for a fix or workaround.

    I did some tests with Preferences => General => Interaction Mode as I noticed that TestRunner execution speed depends on it. Here are results for my project running all tests:

    | Interaction Mode | Time to "Run all" |
    | Default | 2.9s |
    | Monitor Refresh Rate (8 ms) | 3.7s |
    | No Throttling | 2.1s |

    No Throttling (or: Custom with 0 ms Frame Throttling) makes tests run about 30-40% faster! However I don't wish throttling to be enabled all the time. I'd rather hope Unity could just generally disable throttling during test runs if we can't do it ourselves.
     
    Last edited: May 2, 2023
  15. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,932
    I tried changing the corresponding EditorPrefs key "EditorBusyProgressDialogDelay" before running tests but this apparently does not trigger the code that runs when changing the value in Preferences which apparently makes the tests run faster.

    If anyone has any idea how to disassemble the editor code for the General Preferences or where to find it (string search didn't reveal anything) it may help because then I could actually call that (possibly internal) method.
     
  16. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,932
    UPDATE: I can confirm that the "background slows down tests" issue no longer occurs in Unity 2023.1.0b14 (beta) and 2023.2.0a12 (alpha). TestRunner runs just as fast in background as it does in foreground. Speeding up tests within Rider however seem to require the following script.

    I wrote this script for use in test assemblies that automatically sets Interaction Mode to "No Throttling" while tests are running. This reliably speeds up TestRunner execution by 20-40% and beginning with the 2023 tech streams it also considerably (~5x) speeds up running tests from within Rider.

    Here's the script in full, it should work if you put it in any folder that's compiled by a unit test Assembly Definition
    Code (CSharp):
    1. // This work is Public Domain (CC0) - "No Rights Reserved"
    2. // License: https://creativecommons.org/publicdomain/zero/1.0/
    3.  
    4. using System.Reflection;
    5. using UnityEditor;
    6. using UnityEditor.TestTools.TestRunner.Api;
    7. using UnityEngine;
    8.  
    9. namespace CodeSmile.Tests.Utilities
    10. {
    11.     [InitializeOnLoad]
    12.     public class FasterTests
    13.     {
    14.         static FasterTests() => ScriptableObject.CreateInstance<TestRunnerApi>().RegisterCallbacks(new Callbacks());
    15.  
    16.         private class Callbacks : ICallbacks
    17.         {
    18.             private const string ApplicationIdleTimeKey = "ApplicationIdleTime";
    19.             private const string InteractionModeKey = "InteractionMode";
    20.  
    21.             private int m_UserApplicationIdleTime;
    22.             private int m_UserInteractionMode;
    23.  
    24.             private static void UpdateInteractionModeSettings()
    25.             {
    26.                 const string UpdateInteractionModeMethodName = "UpdateInteractionModeSettings";
    27.  
    28.                 var bindingFlags = BindingFlags.Static | BindingFlags.NonPublic;
    29.                 var type = typeof(EditorApplication);
    30.                 var method = type.GetMethod(UpdateInteractionModeMethodName, bindingFlags);
    31.                 method.Invoke(null, null);
    32.             }
    33.  
    34.             public void RunStarted(ITestAdaptor testsToRun) => SpeedUpTestRunner();
    35.             public void RunFinished(ITestResultAdaptor result) => ResetInteractionMode();
    36.             public void TestStarted(ITestAdaptor test) {}
    37.             public void TestFinished(ITestResultAdaptor result) {}
    38.  
    39.             private void SpeedUpTestRunner()
    40.             {
    41.                 Debug.Log("Set Interaction Mode to 'No Throttling' during tests.");
    42.                 SetInteractionModeToNoThrottling();
    43.                 UpdateInteractionModeSettings();
    44.             }
    45.  
    46.             private void ResetInteractionMode()
    47.             {
    48.                 Debug.Log("Reset Interaction Mode to user settings.");
    49.                 SetInteractionModeToUserSetting();
    50.                 UpdateInteractionModeSettings();
    51.             }
    52.  
    53.             private void SetInteractionModeToNoThrottling()
    54.             {
    55.                 m_UserApplicationIdleTime = EditorPrefs.GetInt(ApplicationIdleTimeKey);
    56.                 m_UserInteractionMode = EditorPrefs.GetInt(InteractionModeKey);
    57.                 EditorPrefs.SetInt(ApplicationIdleTimeKey, 0);
    58.                 EditorPrefs.SetInt(InteractionModeKey, 1);
    59.             }
    60.  
    61.             private void SetInteractionModeToUserSetting()
    62.             {
    63.                 EditorPrefs.SetInt(ApplicationIdleTimeKey, m_UserApplicationIdleTime);
    64.                 EditorPrefs.SetInt(InteractionModeKey, m_UserInteractionMode);
    65.             }
    66.         }
    67.     }
    68. }
     
    Last edited: May 2, 2023