Unity testing broken

Unit tests take so long to run they are effectively unusable. Just opening the test runner window takes forever.

I started a unit test while writing this message. It's still not finished.

 

Won't do any good I don't think filing a bug report if I can't repro it with a small enough project. Mine is too big to send in and I wouldn't even know where to start getting a repro. The problem is obviously in work being done not directly related to Nunit. It takes about 45 seconds just to get to where it runs the test, which itself completes in the expected time.

I suspect it's doing things which cause multiple passes of the general reload phase or parts of it. The project is big enough where a code change results in 30 seconds of waiting until you can play. So connecting the dots there it seems to be connected.

 

Hmm it actually just locks up the editor and profiler. Both when opening the window and when running the test. When running it locks up for around 20 seconds, shows the diallog 'running such and such test' then immediately locks up again for another 20 or so seconds then completes.

Like it's literally trying to get a lock on something and eventually times out. The cpu at that time is just spinning at a steady 30% but no disk activity.

 

Same here. I tried using the profiler with no success.
Took over a minute to open.




It's the same thing when changing from "EditMode" to "PlayMode". Over a minute to switch.
Messages will be shown, this time

Skipping profile frame. Receiver can not keep up with the amount of data sent
UnityEditor.TestTools.TestRunner.TestRunnerWindow:ShowPlaymodeTestsRunnerWindowCodeBased()

Skipping profile frame. Receiver can not keep up with the amount of data sent

The profiler will have a "blank" frame (no data) in the hiccup spot.

This has been happening for me in all 2018 builds, but I couldn't reproduce on a small project so never got around to submitting a bug report.

 

Maybe relevant: using the filter dropdown to the right of the search field is just as slow as changing tabs or re-opening the window. Takes a long time. From 20s to over a minute.

 

Not familiar but shouldn't be a problem. Getting late here so I probably won't get back with any info until tomorrow my time.

 

@superpig it's similar to this.

https://forum.unity.com/threads/hug...ormance-discovering-and-running-tests.512535/

 

I'm posting the relevant screenshots of the meaty part (29s delay).
If you need the full dump, let me know how send I can send it privately. Those contain a lot of info.

Here's the gallery: https://imgur.com/a/uwo4t
I let the Performance Analyzer download symbols from Unity's symbol servers and use the local copies as well.


Edit: I only have a few dozen tests in this project. They are isolated to two custom assemblies (one for runtime, and one for editor tests).

 

> So... we're a bit stuck without Unity's profiler.

Yeah. I hope the profiler gets some love eventually. I used to be able to get a better view of assembly reloads and compilation, but in recent versions (with .NET 4.x) I can't get anything useful from it for assembly reloads. This was especially useful to find stuff with InitializeOnLoad causing slowdowns.


Here's two mode examples of what the profiler shows when opening the test runner. It's pretty random what it can see.






> Best thing to do for now is just to wait for Harald I think.

I think so too.

 

What would probably help and a feature request anyways, is that we be able to have some way of specifying the location of tests. Similar to how outside of Unity best practice is a separate project for tests.

One potential issue with scanning an entire project is that I don't want third party asset tests running when I run my tests. I'm assuming right now that would happen. It's also not a real issue right now because hardly any asset creators create tests, but that's another issue

 

@SebastianHjorth

>Can you please try profile editor, open up the test runner window, store the profiling session and attach the file here with the version of Unity used?

Sure, give me a few minutes.
I'm curious about that, since it gets pretty hard to debug assembly reload issues.

>Skipping profile frame. Receiver can not keep up with the amount of data sent. Does this only happen when profiling playmode?

No. It's in edit mode and load is pretty normal (<50%). I find it pretty strange too, since I only ever saw when debugging a remote player with deep profiling enabled. It also only shows once or twice. When I saw this with a built player, it would usually be spamming like crazy.

 

Here's the session:
https://www.dropbox.com/s/qrtwvzuvl6g6jzp/PROFILER SugoiDev 2018.03.28.data.7z?dl=0

Edit: I'm on Unity 2018.1.0b12 right now, on Win 10 x64


Since you guys are looking at the list view, there's a minor annoyance I noticed a while ago.
Check out this git:


Click the inner "Aftertile" is collapsing the outer one, when it should only collapse the inner one.

 

I'll create a small repro later today and submit a report. Should I ping you back here with the number?

 

@LeonhardP

Alright. Took a while to see it on fogbugz. Case #1019376

 

> Also that log you posted is profiled using Deep profiling and Editor profiling, correct?

Yes.


> Did it print that discard frames error?

Yes, once.