Starting async load causes massive lag spike

This has had be stumped for a while now. In certain places in my game, I'm trying to asynchronous load the next scene in the background while the user is doing something else. This helps cut down the loading screen time the user sees.

I understand async loading is a 2 step process. Step 1, load data for the scene from disk into memory. This step is the one that's able to be done asynchronously. Step 2, activate the new scene (e.g. call Awake and OnEnable for all enabled objects in the new scene). This step is not asynchronous.

This is all well and good. I can start the async load and disallow scene activation so step 1 takes place when the user doesn't notice. Then, when the user decides to proceed to the new scene, I show a loading screen, then a allow scene activation.

The problem is that in certain circumstances, simply calling Application.LoadLevelAsync(sceneName) causes a massive 6 second stall in the application. So far, it seems like this only happens if you start an async load too quickly after a previous load operation. This stall shows up in the profiler under Loading.ReadObject > Loading.Lock PersistentManager. Unfortunately, I see no way to know how soon is too soon. And having a random 6 second stall is obviously unacceptable for the end user. What's going on here? How do I prevent this?

The particular scenario in my game is as follows:
1) Application opens to the Main Menu
2) User chooses to start a game
3) A "Garage" scene is loaded synchronously where the user can choose a motorcycle and rider
4) As soon as the Garage is loaded, asynchronous loading of the first racetrack scene is started
5) Application immediately stalls for 4-6 seconds.

I believe that if I delay 4) long enough, the stall will not occur. It seems as though the previous load operation is still performing some operation in the background and attempting to start a new load causes the application to wait for the previous operation to finish before starting the new one and continuing execution.

I've attached a screenshot of the profiler when this occurs. The first spike is the Garage scene activation. The second one is some initialization going on in that scene. There's a 1 second delay. Then there's a large spike from simply starting an async load (with scene activation disabled). I hacked in the delay to make the spike clearer.

EDIT: I tried delaying the async load for a full 30 seconds and the stall still occurs.

 

Are you profiling this in a standalone build or in the Editor? Especially in 4.x the async calls are actually pretty synchronous in the editor.

 

The screenshot and numbers are from the Editor. It's greatly reduced in a build, but the problem still exists. I'll get some exact numbers tomorrow.

 

Looking at that profile it appears that all of the time is going into an AmplifyMotion call. Do you still get the stall if you disable AmplifyMotion? I wonder if they have some kind of bad interaction with async loading?

 

If you look at the stack in the profiler you'll see the time is all spent in the Loading methods, not in the Amplify script. The Loading stuff shows up in seemingly random places. Before I added the delay it was showing up under GameObject.SetActive.

 

Yeah, I saw that. It's just curious that it shows up in Amplify at all. If those are internal functions, I'd expect to see them called by other Unity functions, not just directly from a custom script. Even if it moves around a lot, it'd be interesting to see what actual function calls are responsible for triggering that Loading Lock. Doing so might help find a way around the problem. While writing SECTR STREAM, I had to do stuff like that a lot, just isolate bad behavior and find an alternative implementation that avoided the internal issue.

 

After testing with a build, this issue does still occur. It doesn't stall for multiple seconds, but there are bunch of heavy frames that end up dropping the average framerate from 180 to 25 on my machine. I've attached screenshots of each of the >50 ms frames. It appears most of the hits are from parsing shaders.

This makes for a crummy user experience and I'm wondering if I should avoid using async loading altogether. I'd really hate to have to avoid an engine feature because it performs so poorly, though.

Any chance someone from Unity could comment on what's happening here and/or how to prevent it?

EDIT: I've confirmed this occurs in 4.6 builds as well.

 

I'm still trying to understand this issue if anyone has any input.

 

That shader compile has to happen sometime. You can try adding a call to PrewarmShaders at the start of the game or level load, but I don't know how comprehensive that will be for objects in other scenes. You may need to have all shaders referenced in the main scene and then call prewarm to get rid of the hitches.

 

I thought shaders were compiled at build time? Why does this happen at runtime? And why every time the game is run instead of just the first?

 

It depends on your target but the gles spec doesn't allow precomputation and it's impractical on many regular GL systems.

 

This is running on Windows, so it should be using DirectX 10, not OpenGL.

 

Seems like they should be precompiled. I'd still try the call to prewarm shaders to see if it helps. Clearly, something thinks it needs to compile shaders.

 

Any improvement on this subject? I'm facing the same issue but on Android (while profiling the device, as it happens on build too)

 

I'm getting a similar issue (v5.0.0f2 beta).

I get about 3ish huge spikes w/ a lot of time spent in Loading.ReadObject > Loading.Lock PersistentManager. Anyone know what this routine does?

I doubt this has anything to do with shader compiles, because I do a Shader.WarmupAllShaders() before the spikes happen. It happens in the midst of doing my async level loads.

 

I'm also getting the same issue on 4.6.3f1. It happens right when the Application.LoadLevelAsync call is made and it seems very unlikely that it has anything to do with Prewarming shaders since I set async.allowSceneActivation = false; and the scene does not switch over to the new one until about 5 seconds later.

 

Unity 5 has the same issue

 

I've put in a bug report for this (#677082) but haven't heard back so far.

 

We're having the same bug, but it seems to be only in a built player. We load small terrain chunks asynchronously and it works quite well, but every now and then Loading.LockPersistentManager shows up and takes a lot of time. It happens when we access the ResourceRequest.asset value. When I decompile UnityEngine.dll, it shows that ResourceRequest.asset calls Resources.Load(string path, Type systemTypeInstance);

I guess it has to do with some kind of multithreaded lock. Resource.Load must try to access some locked value while another background loading is taking place in another thread, making the main thread wait for this loading to end. It seems to happen less frequently in FixedUpdate(), but it may be an illusion.

No matter where I put the ResourceRequest.asset call, it still happens. It would be pretty nice if Unity would fix this. I might try to make a small project to submit a bug, but it's a bit tricky.

 

This is a really annoying issue... I have created a Unity Answers for it just in case some kind of Unity God exists and knows the answer.

There.

 

Have you tried querying ResourceRequest.asset after `yield return resourceRequest`? If ResourceRequest.isDone is true asset is loaded and ready.
Please post a bug number to here when you submit it.

 

My actual code is this (in a coroutine):

Code (CSharp):

1.         var request = Resources.LoadAsync<GameObject>("Tiles/" + name);
2.         yield return request;
3.         yield return new WaitForFixedUpdate();
4.         var asset = request.asset;

I added a WaitForFixedUpdate() because it seemed to happen less frequently, but it may be an illusion. It seems to only happen in a build, too.

I will try to create a small project with the issue and submit a bug later today.

 

I'm experiencing this same problem in the latest 4.6.7p1 Unity client. Occurs in the editor and in builds when I call the Application.LoadLevelAsync().

 

Is it possible to extract a repro project? Bug report is very appreciated.

 

According to artaka, there is already the issue 677082 in the database. Should I also enter a bug?

 

Unfortunately there is no repro project for this issue. If you can extract or share you project, file a bug please.

 

Problem is my project is 32 gb in size, A man's gotta do what a man's gotta do, I'll start the upload overnight

 

I'll try to put something together, but as alexandre said, our project is quite big. Is it okay to upload an 8-gig project?

 

Anything on this? I'm having the same issue.

The lock appears to come from different places when I try different things. At first it was coming from the UI sprite start up stuff, then I delayed the UI start up and now it's coming from other places. Looking at 2-3 sec in unity and up to 20-30 seconds!!! in a PS4 build.

I've done all sorts of testing, even tried to make a cut down project so I could submit something but I'm so swamped.

The one interesting thing I've found is this....

First scene is Main Menu.
Main Menu calls Application.LoadLevel(MainLoader)
MainLoader scene objects(UI, Player, not much else) load and start up
MainLoader uses LoadLevelAdditiveAsync(FirstWorld)
20-30 second pause
All the stuff in FirstWorld loads in and starts up within 1 second with slight fps drop(as intended/expected)

This is where it gets interesting to me... IF I make the MainLoader the initial scene(exclude the main menu), no other changes, the pause is MUCH less, to where it's unnoticeable outside of the profiler.

 

Hi!

Is it the stall in Loading.LockPersistentManager? Which Unity version?
Could you please share Profiler's screen?

Besides that, it could be:

• Mix of sync and async methods (like LoadAsset and LoadLevelAsync).
• Implicit internal call of Resources.UnloadUnusedAssets during LoadLevel call which causes assets being loaded as a part of main level and used in FirstWorld to be reloaded again.

Could it be reproed on other platforms?

 

I'll have to check if that still happens in the most recent version of Unity. Will check it out soon next week and come back here.

 

Hey! Thanks for the reply.

Unity version is 5.1.3p1 at the moment. The issue does occur on OSX and Windows as well. I'm confirming with the Mac people on the timing but on Windows it's 2-3 sec pause like from within the editor.

I'm checking into other loading stuff but as far as I know there is nothing else being loaded(confirming with our tech artist still). But we are not using asset bundles nor loading any resources manually.

Although, as explained above, we do sort of have a mix of sync and async. Load(MainLoader) happens in MainMenu, and once MainLoader starts up, it calls LoadLevelAdditiveAsync(FirstWorld).

Below is a screen shot I posted the other week when I was blaming this on ngui(thread here http://www.tasharen.com/forum/index.php?topic=13558.0), I'll try to get you some fresh ones today but I've been working on fixing this so things are a little different now.

Thanks!!



PS> How do I determine if my issue is related to your 2nd bullet point??

 

Looking for your sync/async thing and I just noticed....

The LoadLevel(MainLoader) is coming from(last line) a coroutine in MainMenu. Is that a "mix of sync and async" like you mention?

 

I took it out of the coroutine, didn't change anything....

I can't remember exactly what I did to get the pause out of the UI stuff and into the Camera but that is where it's at now. I think maybe I atlased the UI sprites(using unity's sprite system).

Regardless, the problem happens in the same spot but comes from the cam now in the profiler....




ALSO.... This happened just now, first I've seen it, could not reproduce....

 

Well,

I've tried everything I could think of with no luck. We are really stuck on this and it sucks. Async add works SUPER smooth everywhere else we use it(in between each level) but not on this first scene. Driving me nuts.

I removed the MainMenu, which we thought had eliminated the problem previously. However, that just moves the long pause to the very start of the startup for the game, making it less noticeable but still there.

The MainLoader only has a few things it in, mainly the UI and Player. I thought maybe it was something to do with shaders so I took every single VFX off of the player as well as completely removed the UI from the project. Did not change anything. ;-(

 

I have the same problem. I'm using LoadLevelAsync and I set Application.backgroundLoadingPriority to lowest possible (i.e. ThreadPriority.Low). Get a lag spike in the editor and in the engine. It seems to either be connected to the number of different textures OR the number of different shaders. The scenes where I get the lag are the scenes with most stuff going on.

Anyway, there seems to be some really big hiccup in the loading. The scene management in Unity seems to be pretty broken. :-/ So many bugs and irregularities.

 

The spikes I get now seem to be caused by GPU locks. In fact, it's very similar to JGriffith issue. If you would open the GPU profiler, you would see those spikes clearly, probably flagged as RenderTexture.SetActive, MeshSkinning.Update or ReflectionProbe.BlendCubemap.

Here's another thread about it. I sent a bug report about this. http://forum.unity3d.com/threads/lo...reates-spikes-on-the-gpu.382436/#post-2486290

 

Oh interesting. Can you link the bug report too so I can upvote it?

[Edit] Oh found it in the thread you linked.
Here it is: https://fogbugz.unity3d.com/default.asp?765766_qbd1keock6sshi07

 

Bump.

This is still happening in 5.3.2. The only workaround so far is to load all my scenes at startup, then unload them in order to avoid the huge spike. Unsurprisingly, this takes a lot of memory and make the initial loading really, really long.

 

Yea that actually works! This means that the problem is probably very bound to how asset loading is handled (such as textures, etc).

 

Any progress on this bug? I have the same problem which is stopping me from go live on an android phone game.

Cheers

 

Upvote this: https://issuetracker.unity3d.com/is...c-creates-huge-performance-spikes-for-the-gpu

But to be honest don't expect a fix anytime soon (we filed the issue in February). The issue has been there for ages. We simply ended up adding loadings in our game, because Unity

 

Wow glad to hear I`m not the only one, I was testing the async scene loading like this
1-.Game loads main menu scene
2-.main scene has a game object with a script that async loads world 1 on Start()
3-. the play button just allows scene activation wich is supposedly already loaded in memory

of course it is unnoticeable in the pc, but when I deploy to my Phone to do some testing, the game literally hangs the first 8 seconds on the main screen then it runs like butter.

 

Necro'ing this thread.

Some things I've found that drastically improve async scene loading, reducing lag spikes / frame stutter.

1, Don't prewarm particle systems. When prewarm is checked Unity will try to prewarm all your particles when the scene is loaded. If you have a lot of systems this is going to take a while. The scene will not load until all these are complete.

2, Don't make anything static / avoid batching. This seems counter intuitive, but if you make things static, like your backgrounds, LoadAsync will try to batch them, one of the steps in that is a sort. The sort is very expensive.

3, Remove all prefabs. Yep, I know. Having prefabs causes a step noted in the profiler as Prefabs.MergePrefabs. This was taking 40ms. Removing all prefabs, through GameObject > Break Prefab reduced this to 5ms.

(I'll try to keep adding to this list as I find stuff.)

Making these adjustments dropped LoadAsync from 550ms to 60ms, nearly an order of magnitude. It's still not low enough, and it causes a GC alloc which I don't like, but this is much better.

@alexeyzakharov Not sure if you still work on scene loading, but is stutter free scene loading possible when building games on the scale of something like INSIDE or Ori? I know they achieved it with those games, but my understanding is that both teams received extensive dev support from Unity. With INSIDE having access to the Unity source code, resulting in a heavily modified version of the base Unity 5 engine.

 

Trying to squeeze some more perf from LoadSync. Can someone from Unity explain what I can do to optimize things?

OnLoad, we're spending the majority of time in BuildPipleline.Interfaces validating types and methods. What is the source of this? What can I do to reduce this?



TIA!

 

I'll add my 2 cents because this stuff is hard. I don't have any specific comments about the spike above but I have some experience here and thought I would share.

On the subject of shaders. I can comment a little bit about why prewarming is the only real option.
Shaders do take longer than they should to upload and are fairly synchronous in that they cause a loading stall and I think this has been a known thing for the Unity devs for quite some time, but I can't comment to much about that other than I have seen this be a recurring theme.

However, on PC and Mac, although your shaders are "compiled" to a compact intermediate form the graphics card you are using on that machine is variable (unlike say, on a console where the hardware is fixed and does not change) This means that the shaders still do get re-compiled from the intermediate format to the card specific version at runtime for the card being used on the machine that is playing your game and this has a very significant cost. Ultimately shaders are therefore a blocker to most asynchronous loads, the recursive nature of shaders make this even more painful as heavily recursive shaders can (in some situations) cause a loading stall while a portion of the shader that was missing (due to an import statement) is fetched from disk.

Another thing to think about for anyone attempting to stream levels:

Audio is big, ensure you are background loading everything you can to avoid loading it when you are streaming in your levels, although in some cases that is impractical, you might try loading that data asynchronously in an asset bundle.

Avoid resource files, those are probably some of the slowest loading forms of assets in the Unity engine, if you can use asset bundles instead, do it resource files are extremely expensive.

Think about what is unique about the level and what is common to the game and, where you can, strip the common bits out into asset bundles and manage them independently.

Unloading is a big problem, or at least it has been in my experience, this stuff is changing all the time as the Unity team improves things. When I was working on some of this stuff, unloading was only accomplished through unload unused assets or Destroy. We ended up having to build a custom system to destroy assets across frames. Textures are a huge challenge here in that they are often shared and you need to understand how they are shared!

In a streaming world it is a question of when you will GC not if. Ideally your game will never allocate during gameplay because you have been carefully pooling and tracking this stuff right? You will know your memory high water mark, be monitoring your memory use and be running the GC behind every scene transition you can! Every fade, every major player idle, every cut scene, every time you detect you can run the GC proactively so you will to avoid it running when you don't want it to run, during gameplay...

Compress everything you can! This has a double benefit, when your game runs on a spinning platter you seek less, often fragment less AND it can take less time to get it off disk once the data is queued up.
Test load performance in the player regularly on the slowest loading machine you can! Ensure you have a SLOW spinning platter in that machine.
Track your scene size and ensure you scenes are not growing above your average throughput.

Finally your Awake calls and Start calls can be part of the problem. If you try to access resources inside of those calls you can trigger a load stall if the asset is not yet fully loaded. Be careful!

One trick is to use editor scripts to trade GetComponent<> and cross asset references for private serialized fields. This allows Unity to fixup the links at load time and thus load assets in depencency order rather than you having to do it during Awake or Start. This can, if done well, help a great deal.

 

Great notes @IXNet, thanks for sharing.

I'd like to hear a Unity dev weigh in here as well, but I'm starting to feel it's just not practical (for me at least, small two person team) to tackle streaming levels. The Unity tech just doesn't seem evolved enough. Fair assumption?

 

The more I think about this the more confused I am.

How can LoadAsync be an async operation if it has a blocking "integrate" phase? I'd barely call myself a dev, but even to me that just seems broken. Having a blocking phase means it's not async, yes? Wouldn't a correct implementation of integrate (and any other blocking operation) parcel out its work over multiple frames?

In a less hopeful light it feels like no matter what I do (aside from getting the blackbox that is integrate down to 10ms or so), there will be a blocking phase i.e. frame stutter, when loading async. Of course that makes no sense either, if I could get it down to 10ms or so then I wouldn't need an async operation.

Anyone out there that can unpack the thinking here for me? And, if possible suggest a way to work around what seems to be a serious flaw in the LoadAsync impl?

TIA!

 

Hi,

Thanks for raising these questions!
I would like to clear up some things first.

Prefabs.MergePrefabs is Editor-only activity which is relevant to playmode only, in a player, there is no prefabs merging. This instantiates prefabs into scene which is being loaded and applies local modifications, because scene file (*.unity) contains only references to prefab assets. Scene which is built into a player has prefabs instantiated at build time - all prefabs are flattened. Runtime performance and playmode performance has differences.
But obviously Prefabs.MergePrefabs performance affects playmode experience, and we are investigating potential optimization.

Same here, this is Editor only build pipeline caching activity which should happen only once after domain reload. If you see it on each scene load - this is a bug, which should have been fixed (could be not backported to version you use).

Regarding,

From my knowledge source code modifications are usually game specific and not applicable to a generic game.
Which is what ECS approach should solve by allowing you to control data loading and most of the integration parts, which are now blackboxed behind Object and LoadAsset.
There are are some fundamental issue with async loading in the Editor. E.g., any sync loading might be blocked by async loading, because Editor itself or plugins can do sync loading during playmode (e.g. fetch an icon) on a main thread. Profiler might help you to see where it is originating from and do a workaround in a mean time.
The same applies to runtime - don't mix sync and async loading - even small object might block main thread, because large scene is being loaded on preloading thread.

 

@alexeyzakharov Thanks for the insights, very helpful.

Which is what ECS approach should solve by allowing you to control data loading and most of the integration parts, which are now blackboxed behind Object and LoadAsset.

Should I take that to mean that achieving perfectly smooth async loading will not be possible for me until ECS is shipped. It's still in the works, yes? ETA this year?

(FWIW, I'm stuck on 2017.1 b/c more recent builds have such poor Animation Editor performance I can't use them.)

The same applies to runtime - don't mix sync and async loading - even small object might block main thread, because large scene is being loaded on preloading thread.

I don't believe I am doing any mixing, but maybe there are places this is happening that I haven't thought of. Perhaps you could list areas I should be careful to avoid / look for? Other than save game data which is only loaded once at start up, I don't get anything from disk, use any resources, write to PlayerPrefs, or use any asset bundles.