Search Unity

Occlusion culling CPU spike in profiler

Discussion in 'Editor & General Support' started by Arkensis, Mar 30, 2015.

  1. Arkensis

    Arkensis

    Joined:
    Jan 27, 2013
    Posts:
    13
    Hi, it seems that occlusion culling in Unity 5 creates unusually large spikes on the CPU in the profiler. I've noticed this problem in multiple scenes, in multiple projects, and have reproduced it in a simple scene with a few cubes in it.

    There is only one camera, one directional light and 9 cubes using the standard specular shader. The cubes are marked as occluder and occludee static, and the occlusion culling settings are default.

    Here is a screenshot of the scene :

    OcclusionCullingBug.PNG

    I saw a post with a similar problem, which was solved by removing occlusion culling on a second camera that should not have used it. However, in this case I only have one camera, and it needs occlusion culling.
     
    NGC6543 likes this.
  2. DarthDisembowel

    DarthDisembowel

    Joined:
    Jun 11, 2014
    Posts:
    53
    I'm also encountering this spike caused by WaitingForJob (though mine is under Culling/SCeneCulling/CullAllVisibleLights/WaitingForJob).

    I need the occlusion culling as I am creating a large detailed environment with many rooms, so clearing the baked occlusion is not an option. I'm going to try being more selective about what I flag as Occluder Static and see if that helps, but I have my doubts. I'll post my results here.
     
  3. DarthDisembowel

    DarthDisembowel

    Joined:
    Jun 11, 2014
    Posts:
    53
    Quick update: no, reducing the number of occluders made no difference.
     
  4. cadellinman

    cadellinman

    Joined:
    Dec 3, 2014
    Posts:
    16
    Having the same problem myself.

    Checking the profiler in an attempt to optimise my scene, and noticed that whenever the camera turns, there is an enormous spike in CPU usage coming from Camera.Render/Culling/SceneCulling/WaitingForJob. Whatever it is, it seems to take up 87% of the CPU for a brief moment, dropping the framerate from a smooth 70 to 10 or lower.
     
  5. Wizard-Games

    Wizard-Games

    Joined:
    Jun 4, 2013
    Posts:
    34
    I have the same issue, even if I remove Occluder static, and ocludee static from all objects, turn off occlusion culling on the camera and clear the oc data, I still get a spike every minute or so.
     
    Last edited: Apr 28, 2015
    Dionysos1111 likes this.
  6. cadellinman

    cadellinman

    Joined:
    Dec 3, 2014
    Posts:
    16
    Hmm. I had initially assumed it was something to do with my DetailGroup scripts (hiding and unhiding groups of objects based on distance to reduce distance calculations) but evidently not if others are having the same problem.
     
  7. Wizard-Games

    Wizard-Games

    Joined:
    Jun 4, 2013
    Posts:
    34
    I submitted a bug report. I will let you know what they say.
     
  8. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    64
    We're seeing the same issue popping up in random spikes, interested in seeing if this is a bug or just something that we can optimize somehow.
     
  9. cadellinman

    cadellinman

    Joined:
    Dec 3, 2014
    Posts:
    16
    Still no word on their part? I have been optimising my project across the board and have it running at 120fps, but these occluder spikes are still causing dropped frames in there.
     
  10. Wizard-Games

    Wizard-Games

    Joined:
    Jun 4, 2013
    Posts:
    34
    No reply as of yet.
     
  11. cadellinman

    cadellinman

    Joined:
    Dec 3, 2014
    Posts:
    16
    Taken a closer look at the profiler; it seems that the WaitingForJob process that is causing these spikes is actually running on every frame, albeit at a much lower CPU usage. It runs constantly at about 0.5ms, spiking to 20ms at seemingly random intervals. This background process takes up something like 11% of the CPU usage, in an 80fps scene, with the spikes taking up over 80%.
     
  12. cadellinman

    cadellinman

    Joined:
    Dec 3, 2014
    Posts:
    16
  13. cadellinman

    cadellinman

    Joined:
    Dec 3, 2014
    Posts:
    16
    I also submitted a bug report, and attached a basic scene with nothing but cubes that shows the glitch. Hopefully that should do it.
     
  14. phasiclabs

    phasiclabs

    Joined:
    Nov 13, 2013
    Posts:
    126
    We are seeing stalls in Camera.Render->Culling->SceneCulling->WaitingForJob - our game is stalling once every couple of seconds (the stalls are for 10-16 seconds). We re running Unity 5.0.1
     
  15. Wizard-Games

    Wizard-Games

    Joined:
    Jun 4, 2013
    Posts:
    34
    I think we need someone with a pro account to submit a ticket.
     
  16. phasiclabs

    phasiclabs

    Joined:
    Nov 13, 2013
    Posts:
    126
    We made this problem go away by a combination of setting the Bounce Intensity on all our dynamic lights to zero (I think this stops them contributing to global illumination), and not continually enabling and disabling lights (just turning there Intensity to 0 rather than fully disabling them seemed to improve performance).
     
    kk99 likes this.
  17. fafet

    fafet

    Joined:
    Sep 2, 2013
    Posts:
    3
    I found that unmark light disabled/enable during runtime as static dit the trick for me.
     
  18. ElKrullo

    ElKrullo

    Joined:
    Feb 24, 2013
    Posts:
    17
    I have the same problem in 5.0.2f1 and 5.1.1f1. Although my game is 2D without any light sources so I guess I have to fiddle around and see if there's another solution. I'll make sure to come back here if I find anything.
     
  19. crudeMe

    crudeMe

    Joined:
    Jul 8, 2015
    Posts:
    81
    Oh! I thought it was because I switch off static gameobjects that was baked in occlusion data. Is it still a bug?
     
  20. LeonH

    LeonH

    Joined:
    Oct 15, 2014
    Posts:
    92
    I am still seeing these spikes as of 5.2.1p1.

    Please note that 'WaitingForJob' is a very generic token and does not really identify a specific problem. We need to dig deeper to look at what job (from Unity's multithreaded job system) is taking longer than we think it should. To do this, when you are looking at a CPU frame in Unity's profiler, click the drop down in the upper left corner of the bottom pane and select 'Timeline.' In the timeline view, you can see what is happening on each thread, in actual chronological order from left to right. You can zoom in/out with the mouse scroll wheel to get a better look at the names and timings of specific markers. If you zoom in so far that you can no longer see the entire frame, there is a scroll bar at the bottom of the pane that lets you pan left and right.

    Anyway, in this comment I attached an image of a spike frame in a profile I just checked. Have a look!

    In this frame there are two problem areas that are causing large WaitingForJob stalls, which I've circled in red. The first one is caused by long CullSceneDynamicObjects and CullDynamicObjectsWithUmbra markers. The second one is caused by long Shadows.CullShadowCastersLocal and Shadows.CullShadowCastersWithoutUmbra markers. Note that all of the smaller marker boxes around those markers are the exact same types of jobs, they just normally take a lot less time to complete. On a non-spikey frame, they are all around the same short duration. Also note that there are different spike levels on different frames. The worst frames are as illustrated in this image, where a job in the dynamic object occlusion culling system has spiked and also a job in the shadow caster culling system has spiked. In other frames, only one of those two subsystems have a job that spikes, resulting in a spike that is not as severe as when both have spiked.

    I don't have any solutions to offer. I haven't really started experimenting with content to see if there is anything obvious that could be causing this.

    Also, I haven't seen any patch notes that mentioned this, but @karl.jones mentioned in another thread that Umbra was updated in 5.2.1p2. It didn't sound like the update involved anything related to performance though, so it probably did not affect this. Karl, any chance you could poke around into this one, too? :)

    EDIT: I should probably add, in case it matters: this is using legacy deferred rendering for all cameras.
     

    Attached Files:

    Last edited: Oct 20, 2015
  21. LeonH

    LeonH

    Joined:
    Oct 15, 2014
    Posts:
    92
    Ok, actually, the more I look at other profiles of spikey frames, the more I think maybe the problem isn't specific to any system. I see occasional weird frames with a job that take 20-30 times longer than typical in pretty much all the systems that have jobs on worker threads. For example, I also see (though much more rarely), super large spikes in PhysX jobs:
    Code (CSharp):
    1. PhysX.Sc::Scene::ccBroadPhase,
    2. PhysX.Sc::Scene::postBroadPhase,
    3. PhysX.BPWorkTask,
    4. PhysX.Sc::Scene::islandGen
    5. (bunch of other Culling related tasks)
    6. etc
    Maybe there is just a more general problem with the job system and/or an issue with memory access or management, and the culling systems exhibit problems more frequently simply because they take advantage of the job system more heavily than any other systems do?
     
  22. pd-heks

    pd-heks

    Joined:
    Jan 15, 2015
    Posts:
    2
    We experienced more or less the same spikes as you describe LeonH (WaitingForjob taking 20-30 times longer at random points), and after some investigation it turned out to be a thread preemption problem in our case.

    We had a higher priority thread running on the same cores as the worker threads and this thread could take up to 4-5ms to complete its task before relinquishing control back to the thread scheduler. If a job had begun on one of the shared cores, and had been preempted by our higher priority thread, it would intermittently stall the main thread with a long running WaitingForJob.

    I don't know if this is the same in your case, but I thought I'd share my findings :)
     
  23. LeonH

    LeonH

    Joined:
    Oct 15, 2014
    Posts:
    92
    Hey thanks! That's really good info to keep in mind. In our case, we aren't intentionally doing anything outside the main thread...I guess maybe it's possible something in our game code is causing Unity itself to kick off some work in another thread, but I don't think that is happening.

    Actually, my results are from a console platform and I am a relative n00b to anything outside of standalone...Thinking more about what you said, perhaps something in the way the platform's dev/testing environment is set up is causing this and it's not an actual problem in a release/retail environment. I'll have to investigate that.
     
  24. pd-heks

    pd-heks

    Joined:
    Jan 15, 2015
    Posts:
    2
    Our problem occurred on console too. The SDKs for most consoles usually ship with profiling tools which make it pretty straightforward to profile/debug this particular behaviour - just sample over a suitable time interval and then look at the CPU timeline to identify thread work.
     
  25. GenOli

    GenOli

    Joined:
    Apr 21, 2013
    Posts:
    135
    I got this as well, making my game unplayable.
     
  26. danbrani

    danbrani

    Joined:
    Nov 22, 2012
    Posts:
    45
    Any updates on this?
     
  27. NGC6543

    NGC6543

    Joined:
    Jun 3, 2015
    Posts:
    198
    This still happens on Unity 5.3.1p1. Any words from Unity?
     
  28. Alabatross

    Alabatross

    Joined:
    Dec 15, 2012
    Posts:
    223
    Any update on this? I dont even use occlusion culling or have any objects marked static, and im getting hits from 30-100ms on 5.3




     
    Last edited: Jan 13, 2016
    dreasgrech likes this.
  29. Morfeuskiev

    Morfeuskiev

    Joined:
    Oct 10, 2013
    Posts:
    122
    Frustum and Occlusion Culling is 100% broken in Unity 5.2 and 5.3... It's incredible when Culling takes more than a Drawing object.

    I have the same problems and i dont know how i can public game with low FPS.
     
    Stardog and dreasgrech like this.
  30. N1warhead

    N1warhead

    Joined:
    Mar 12, 2014
    Posts:
    3,525
    I noticed my issue was coming from LOD Objects not culling. Might be everyone elses issues as well. As soon as I turned off Occluder Static it went back to normal.
     
  31. NGC6543

    NGC6543

    Joined:
    Jun 3, 2015
    Posts:
    198
    You mean that Occluder Static objects should not use LOD?
     
  32. N1warhead

    N1warhead

    Joined:
    Mar 12, 2014
    Posts:
    3,525
    Idk, i just know it fixed it for me. They still occluded fine, etc. But it certainly got a normal frame rate back.
     
  33. blunzn

    blunzn

    Joined:
    Jan 14, 2015
    Posts:
    10
    I am having this problem on IOS. Even in a completely empty 2D scene, without any game code running, there are spikes in WaitingForjob regularly, taking 4-5ms on an iPad mini2. By turning off occlusion culling, those spikes go down to 1ms, but they are still present.
     
  34. Alabatross

    Alabatross

    Joined:
    Dec 15, 2012
    Posts:
    223
    Any word on this?
     
  35. Jaysta

    Jaysta

    Joined:
    Mar 13, 2013
    Posts:
    141
    Getting the same massive spikes too, and I am not even using culling... 5.3.1p1

     
  36. Olafson

    Olafson

    Joined:
    Aug 7, 2012
    Posts:
    251
    I have exactly the same problems as Jaysta.
    We are not using Culling on our cameras, it still happens though. Weirdly enough, I am able to play the game for about 5 minutes before it starts spiking. I have no idea why it starts spiking or how to fix it.


    You can see here that it says that it is the Water Camera, but removing that and having no water, just transfers the culling spikes to the Scene Culling...

    It is also resolution independent, as changing my screen resolution from 4k to HD doesn't change anything.

    I am running this on my Windows7 PC, and all other PCs we tried with W7 have the same problem.
    My laptop is running it in Windows10 and I do not have any spikes there. We also have another computer running it on Windows10 and we also do not see any Spikes.

    But I can not confirm if it is a problem with Windows7... I need to test it on this machine first. I am probably going to have to purchase another Windows10 License and install it.
     
  37. Rycons

    Rycons

    Joined:
    Nov 23, 2013
    Posts:
    31
    Sadly enough we're also seeing the same Culling spikes in our project. We'd be very glad if someone can take a look into this since this is causing major setbacks for us.

    We're currently making use of 5.3.2p1.

     
  38. ruj

    ruj

    Joined:
    Feb 28, 2013
    Posts:
    113
    Just as an addition to the weirdness, I am seeing a huge spike when turning the view, and it appears related to ParticleSystemPrewarm. None of my particle systems are set to "prewarm".
    upload_2016-2-3_17-15-58.png
     
    Rycons likes this.
  39. tcossairt

    tcossairt

    Joined:
    Dec 5, 2013
    Posts:
    129
    Add me to the list seeing this -- in my case I also saw the ParticleSystem.Prewarm spike.
     
    Rycons likes this.
  40. Olafson

    Olafson

    Joined:
    Aug 7, 2012
    Posts:
    251
    Yes, we sometimes also get the ParticleSystem.Prewarm spike, aswell as Physics and Cloth physics spikes.

    BUT! When switching to Windows10 all of these problems go away. Any idea why? I switched to Windows10 on my machine and all of the spikes are gone. Just like on all other Windows10 pcs, there is no spikes.
     
  41. Rycons

    Rycons

    Joined:
    Nov 23, 2013
    Posts:
    31
    It would be great if someone can shed some light on this issue.

    I can confirm what Olafson said about Windows 10 and Culling spikes.
     
  42. Morfeuskiev

    Morfeuskiev

    Joined:
    Oct 10, 2013
    Posts:
    122
    I am not confirm. I have Windows 10 and culling spikes.
     
  43. Olafson

    Olafson

    Joined:
    Aug 7, 2012
    Posts:
    251
    Argh. I wish we could actually get a lead on why this happens...
     
  44. Morfeuskiev

    Morfeuskiev

    Joined:
    Oct 10, 2013
    Posts:
    122
    @Olafson

    I can`t know about your project. But in our project we have 10-30 thousands static and dynamic objects (5-10 thousands is SpeedTree). That frustum and occlusion culling is big impact on performance with stuttering and spikes.
     
  45. Olafson

    Olafson

    Joined:
    Aug 7, 2012
    Posts:
    251
    @Morfeuskiev

    We do not have that many objects, but in worst case, we do have an ocean renderer, two terrains (one for the seabed and one for the coast) A couple of Speedtrees, a bunch of rocks and small objects as well as houses. But everything is very optimized and we mesh combined as many objects as we can.

    In best case, all we have is a few ships and an ocean renderer, and yet we still get those spikes.
     
  46. s141797

    s141797

    Joined:
    Nov 18, 2014
    Posts:
    7
    Bump! Occlusion culling takes more time than everything else on 5.3... Has anyone submitted a bug report?
     
    Rycons likes this.
  47. Rycons

    Rycons

    Joined:
    Nov 23, 2013
    Posts:
    31
    It is not only Occlusion culling for us but even Frustum culling causes huge spikes.
     
  48. Morfeuskiev

    Morfeuskiev

    Joined:
    Oct 10, 2013
    Posts:
    122
    Where is Unity support? They does not reading forum?
     
  49. Rycons

    Rycons

    Joined:
    Nov 23, 2013
    Posts:
    31
    We've upgraded our project to the latest patch release 5.3.2p3 but unfortunately this didn't make too much of a difference.

    We're still seeing massive Culling and Physics.Processing spikes when profiling our build.
     
  50. blunzn

    blunzn

    Joined:
    Jan 14, 2015
    Posts:
    10