Why do my tests run so slowly on Jenkins/Cloud Build?

I have a few playmode tests that load each level, watch for errors, then unload the level. They're super simple. The RunsCleanly() test just presses Continue on the controller, and walks the player character forwards for 6 seconds. It's sequenced behind other tests so we already know the level is loaded.

On my local machine, RunsCleanly takes 8 seconds once it's done its setup and teardown. On the jenkins server and cloud build, it takes over a minute and sometimes even longer.

I'm on Unity 2019.4.5. The jenkins server is a t2.large aws instance using ebs. The command line I'm using to run the tests is

-batchmode -projectPath "." -runTests -testResults playmodetests.xml -testPlatform playmode -logFile ${WORKSPACE}\playtestlog.txt

The unity editor log is clean. I can see it starting the level and completing the test. It's just taking forever. I don't think it's loading time, because this test only starts once the level is fully loaded, and ends before we unload the level.

What could be slowing it down? How can I speed it up? Is there any more info I can get you?

 

does it ever complete? you said locally it takes 8 seconds. how long does it take to complete on EC2 ?

 

Over a minute, and sometimes even longer. It by default times out at 3 minutes.

 

And what does the Unity log say? did you try to log the times in the Editor log to see whats taking it so long?
e.g: use an example code such as this one:

Code (CSharp):

1. public class LogWithTime : ILogHandler
2. {
3.     private ILogHandler logHandler;
4.    
5.     public LogHandlerLior()
6.     {
7.         this.logHandler = Debug.unityLogger.logHandler;
8.     }
9.    
10.     public void LogFormat(LogType logType, Object context, string format, params object[] args)
11.     {
12.         logHandler.LogFormat(logType, context, $"[{DateTime.Now.ToString("hh.mm.ss.ffffff")}] {format}", args);
13.     }
14.  
15.     public void LogException(Exception exception, Object context)
16.     {
17.         logHandler.LogException(exception, context);
18.     }
19. }
20.  
21. [InitializeOnLoad]
22. public class EditorClass
23. {
24.     static EditorClass()
25.     {
26.         Debug.unityLogger.logHandler = new LogWithTime();
27.     }
28. }

 

as I mentioned in the OP the log is clean. I can see it starting and stopping the test in the log, with the timestamps that unity inserts, and the timestamp on jenkin's test output is accurate.