Bug (Case 1256947) App launch loading times have increased from 6 seconds to 100+ seconds in 2020.2

Reported at: (Case 1256947) [XR] Oculus Quest build start of app and async single scene loading is vastly slower on 2020.2.0a15

The load time at the start of our Oculus Quest app and the loading time using async scene load between scenes is almost 2 minutes+ for very simple scenes even though the priority of loading is set to high. On earlier engine versions this load time was closer to 20-30 seconds.

This behaviour is only observed when at least 3 scenes are included in the build. However, when building an empty scene on it's own, the load time is fine. If we add multiple other scenes in the build with that empty scene, the load time dramatically increases.

Often this has led to 100+ seconds in load time for the app start when on prior engine versions such as 2020.1 or 2019.x the load time was 3-6 seconds (during the Unity splash screen)

I backported the project from 2020.2.0a15 to 2020.1.0b12 and saw the load times immediately went back to their normal speeds. See this comparison video I made.

See screenshot below from our bug report: Time to first swap is 105 seconds and the screen is just black that whole time, no splash screen showing even though splash screen is set to show.

 

It is very important to note also that the Unity splash screen doesn't show if you build and run with about 5+ scenes in the build from this bug case. If you build with 1 scene only such as 'Lobby', the splash screen shows on app launch.

Splash screen XR bug acknowledged here: https://issuetracker.unity3d.com/is...y-splash-image-doesnt-show-up-with-new-xr-sdk

I cannot recreate this slowdown in editor, in editor on a PC the scenes load just as fast as they always did. On the Oculus Quest, when loading scenes there is barely any update of what is going on in android logcat, I just have to sit and wait 20-100 seconds for the app to start or for a scene to load.

 

I have made an example repro project with one version using 2020.1.0b12 beta and one version using 2020.2.0a15 alpha. There is also a video in the .zip file showing me running through the apps so you know what to do and can see the load times for yourself in a much simpler reproducible project.

All you need to do is open the project in the right unity version, build and run on the Quest and copy what I do in the video.

https://drive.google.com/file/d/1CLGXlPYJN5VuAk-Q3nZcqKDpw9VWJS6H/view?usp=sharing

I hope QA or Unity staff can look into this severe performance regression, I have done my best to troubleshoot it myself.

 

Issuetracker link for this, if it affects you too please upvote it on the issue tracker
https://issuetracker.unity3d.com/issues/xr-quest-significant-increase-in-async-scene-loading

 

Hoping to see an update on the issue from this soon! Staying on beta releases now since this issue, as one project nears completion I can no longer experiment so much with the 2020.2 alpha

 

Will the issue be added to known issues section of the 2020.2.0 Alpha releases? I can't see issue id 1256947 here in the release notes for the current alpha.
https://unity3d.com/unity/alpha/2020.2.0a21

 

Probably in the next release notes, already released builds don't get updated notes as far I know.

 

Original issue tracker here seems to be updated, glad it is fixed, hoping for a backport to 2020.1 or 2020.2 so i can avoid alpha versions for a while!
https://issuetracker.unity3d.com/is...04.864603206.1593435964-1770994880.1575461548

 

@LeonhardP Can I request a backport of this please? To 2020.2 versions at least, and 2020.1 also if it occurs there or even has a chance of occuring.

 

Probably the same issue - Quest (XR) build, upgraded the project from an older unity version, not using URP (at least not explicitly - don't have the package in the package manager).

Scene loading times went from 1s to ~10s - using 2020.2.0b7 (not sure if it was fixed in that release - tried to read the release notes but didn't found anything related).

Should I upgrade to 2020.2.0b9?

Many thanks <3

 

We're also experiencing insane load times on Quest on 2020.2, not using URP, regular built-in and absolutely trivial scenes. I'll try to reproduce and send a report.

 

Same on Android phone. At first I though it was the shaders preloading, but its the same when they're cleared. Tried deleting the library folder and recompile it as well, didn't help. I'd say a 10x time increase is about right.

Edit - This is happening in both 10.1.0 and 10.2.0 version of URP.

 

I too have a non URP, regular android AR Foundation project that has much slower loads too. Bit fed up of these easy to catch regressions!

 

Found a related one @LeonhardP : https://issuetracker.unity3d.com/is...ression-takes-way-longer-to-open-than-default

Also I took some measurements to get to the bottom of this, as we're effectively blocked on multiple current projects (x4-x10 startup time regression = unhappy clients).
I kept the scene deliberately simple but there's already clearly regressions visible.

Unity 2019.4
Texture Compression "ASTC"
Compression "Default"
Backend: IL2CPP

Mi 9 (no XR)
startup: 2.08s

Quest (no XR)
startup: 2.08

Quest (with XR)
startup: 4.54
build time: 62s

Quest (with XR and empty loader scene)
startup: 4.81
build time: 85s

Unity 2020.2
Texture Compression "ASTC"
Compression "Default"
Backend: IL2CPP

Mi 9 (no XR)
startup: 2.06s

Quest (no XR)
startup: 3.25

Quest (with XR)
startup: 4.53
build time: 65s

Quest (with XR and empty loader scene):
startup: 8.01
build time: 120s

Summary
From the above, to me it's clear that

1. XR startup time adds (or multiplies, not sure) quite a lot of time to general startup
2. It seems that 2020.2 regresses heavily with multiple scenes (an empty scene is scene 0, then that just loads another scene in OnEnable).
3. Resources.Load / Shader.Find et al are massively slower in 2020.2

---

I went a bit deeper and profiled this in a larger project.
There is a number of really weird places in terms of timing; some of them seem to contribute to the slowdowns.

Not sure what's going on here, "SetUserDefinedSettings" alone is taking between 0.5 and 1s in 2020.2



URP Volume Framework does an entire assembly walk just to initialize itself (0.5s)



But the actual regression here seems to be Resources.Load; every single Resources.Load call takes > 3 seconds!!


another example - note the blocking time is almost identical



And one more, this call internally loads ~10 shaders in resources (from the Oculus integration) and thus easily blocks for 30 seconds:

 

@LeonhardP I reported a new case (and another one with additional data):
(Case 1292862) [Android] Performance regression: scene load times in builds 2x-10x slower.
(Case 1292878) [Android] Performance regression, part 2: scene load times in builds 4x slower

Would be great if you can make sure that this is not simply flagged as duplicate of the similar-sounding ones. This is not using URP, just regular built-in; for example, this one seems to be URP-specific. The repro project outlines what I believe are three different issues contributing to these slow load times (as also seen in the profiler screenshots above)

In the repro case there's a 2-4x slowdown on 2020.2 compared to 2019.4, depending on what you measure.
It seems that the issue scales linearly, either with scene count or asset count or amount of code; as others mentioned here in actual production projects we're seeing 10x slowdowns.



EDIT: I did one more test, and added first the official Oculus Integration, and then the URP. This paints an even clearer picture. All of the following tests have identical setup and content. I think we can all agree that this is a production blocker for any Android app, especially Quest.

Unity 2019.4
Quest, no XR: Load Time 2s
+ XR: Load Time 4s
+ empty loading scene: Load Time 4.5s
+ Oculus Integration: Load Time 5s
+ URP 7.5: Load Time 5s

Unity 2020.2
Quest, no XR: Load Time 3s
+ XR: Load Time 4.5s
+ empty loading scene: Load Time 8s
+ Oculus Integration: Load Time 19s
+ URP 10.2: Load Time 28s

Reported this again so Unity has more data to test against. (Case 1292878)

 

Same problem, a real blocker for release.
Not using URP (just the default), running on 2020.2.0b7 - scenes loading went from half second to around 7-12 secs (XR too).

I wonder what was changed so much in the XR aspect.

Couldn't find your issue @fherbst, can you please share a link so I could upvote?

 

I will try to reproduce, but I'm pretty sure it's not URP specific and that's sounds like a similar issue to mine on a very close timeline/versions.

** Looks like those might be related:
1. https://forum.unity.com/threads/cas...-times-on-oculus-quest-in-2020-2-0a15.915776/

2. https://issuetracker.unity3d.com/is...55.908068696.1605586066-1104644099.1603304312

3. https://issuetracker.unity3d.com/is...04.864603206.1593435964-1770994880.1575461548

Is there a beta fix for one of them? Or as @ROBYER1 suggested on (3) - just for the alpha 2020.2.0a18?

 

@Elmak reports only become Issue Tracker Links once QA has reproduced the issue; once they did I can share a link. (Given previous experience, many of such "actually bad issues" are intransparently handled internally and never made into public issues.)

If there's one person that can make sure this gets attention and a public link to vote on it's @LeonhardP

 

Many thanks Leonhard, that's good to hear

And also thanks to the OP and @fherbst for the research, now we know that some heroes wear yellow shirts.

 

So our game is also impacted by this, it seems every scene or prefab load is about 100x slower, I actually thought the build had hardlocked, but yeah it takes a couple of minutes to launch from a few seconds previously.

Just my two cents, I believe Android and mobile in general is a big market for Unity, heck, according to your own CEO, more than half of the games in the mobile market are built using Unity.
So having a regression like this one, that's been open for months, and is still not fixed on the latest beta, is a pretty big concern for us; a bug that can be reproduced by simply building for a target platform and launching a test app, should not have been shipped in a build, not even a beta build.

I understand testing all your target platforms for every new version of the engine probably hurts your velocity, but shipping builds like these hurt your image and the confidence and trust developers put into this engine.
When we're updating to the latest engine version with regressions like this, who knows what else may have slipped through the cracks that we'll run into in a few weeks or months?

 

I too am shocked I was the first to report this regression, given I was just faffing about with an Android Oculus Quest build. There are hundreds of android mobile app developers using your engine outside of my niche VR development scenario that this bug affects.

 

I feel you all, I'm refreshing every day cause I wish to launch my app and that's the only blocker for me (downgrading now just to upgrade later and resolve some already resolved bugs is not an option for me, so I'm just waiting )

 

If you were wondering if this is fixed in beta 14, it's not. (but apparently beta 15 will have a fix)

 

Thanks for the update as I didn't have time to check recently. Issues like this which take months and months to resolve are major blockers for my projects.

Especially when developing for VR/AR where features are in flux forcing me to move to newer engine versions against my will to get the latest plugins. Hitting something like this pushing back projects for 6 months+ is really unacceptable.

 

. . . .
______

@LeonhardP Do you know if it's gonna be fixed in 2020.2b15? I can wait another week, but more than that for a release blocker is just too much...

 

The issue: "Android: Loading the Viking Village scene takes a couple of minutes (1276114)"
Is not longer at: https://unity3d.com/unity/beta/2020.2.0b14
But it is on: https://unity3d.com/unity/beta/2020.2.0b13

I'm not that familiar with Unity's issue tracking system - is it ok that it is not mentioned in 2020.2.0b14?

 

It's still there but the description is not the same, it's the first line in the list of known issues:
Android: Fixed issue when accessing streamable assets time is increased due to incorrect compression settings (1276114)
Fixed in 2020.2.0b15.

 

That's great, thanks @sl1nk3_ubi

 

@ROBYER1 @fherbst @sl1nk3_ubi and all the other heroes on this thread - thank you!
I can confirm that the problem was solved for me on 2020.2.0f1 <3

I can finally release my stupid game for Quest 2

 

I am so pleased, upgrading to that engine version now too, hope the launch goes well

 

Apple just returned our ios app for super long loading times. App has been built with 2020.2f1. And seems like it is related to some kind of sprite compression settings!!!

 

@Foriero the load time regression mentioned in this thread was specifically for Android; can you please open a bug report for yours?

 

I've done a couple few builds on IOS today with the same apps I was noticing the android load times with and saw no load time regression on IOS. Hopefully in your case it is a specific compression or asset you can isolate - if you see it happening in an empty project when you reproduce it for the bug report let me know.

As fherbst said, getting a QA bug report in for it will help a lot.

 

I just stumbled upon this, we also just got rejected for super long loading time on iOS. Currently made a build that we think is working (at least better) by unchecking the "Show Splash Screen" checkbox (don't ask me how the hell that can make a difference).

Using 2019.4.16f1 though.

(Sorry that I hijacked this thread abit)

 

Sounds like it isn't a new issue then, are Apple starting to report apps that have the Unity splash screen?
Does the app load fine on your end?

 

For what it's worth, in our case, Unity 2020.2.0f1 did not fix the long loading times we were experiencing in the beta versions on Android.
We have a unity support ticket open to get to the bottom of this, but just so you know, there may still be another regression out there.

 

The app does not load "fine" on our end, if we enable "Show Splash Screen". It adds about 15-20 seconds to the normal load time (normal load time for our app is 4-6 seconds). We tested this on multiple devices.
NOTE: I'm not talking about "Show Unity Logo", see below:

 

Quote the case number here please? @LeonhardP this sounds quite urgent!

 

Hi, we have found out what made our case loading long. We have included master and its variant SpriteAtlas into the same build. Obviously Unity has hard time to resolve it in players.

 

@LeonhardP In our case it has been more or less our mistake in that we included in the build two identical atlases.

 

I'm dealing with what I thought was a similar problem. The splash screen was blank. In the end it was just a silly mistake I made. But maybe you're making the same silly mistake?

The OVRManager object in Oculus (under the latest integration tools) contains a texture field called 'System Splash Screen'. Entering a texture into that field overrides the default Unity splash, and if the texture is missing you get no splash screen.

 

@fherbst, how did you capture the startup profiling data? I'm not able to connect profiling early enough in the process to catch everything, and often for reasons unknown, I can't even connect the profiler to my Quest. I'm having an issue that sounds basically identical to this.