Unity 5.2.1p2: Realtime lights not working - Occlusion Problem

Thanks a lot for pushing a fix onto p4 Looking forward to testing it!

 

Following... waiting for this too =)

 

Hi Karl, did you mean 5.2.2p3? Still anxiously awaiting these fixes. Thanks.

Edit: Nvm, I don't see those fixes in the patch notes... .

 

Thanks for update!

 

Is there an official place where I can check when or if the patch is out? I have deadlines this week
Is there a list of patches? Or will it just come up in the What's New page? ( http://unity3d.com/unity/whats-new )

 

here:

http://unity3d.com/unity/qa/patch-releases

 

Is still any chance for patch to come today (Friday) ?

 

Incidentally, do you think a full/final release will be coming soon after 5.2.2p4? Like maybe a 5.2.3f? Just wondering, as you've said full releases receive more comprehensive testing than patches. I imagine the next full release would basically be the same as the most recent patch release before it, but with additional testing and bug fixes?

 

@LeonH 5.3 is planned for 8 December, 3 weeks from now, but if you have latest patch you got all updates.

 

Thanks, Karl. There's exactly a 0% chance that we'll be moving to 5.3, as we'll be starting final submissions around the time of its release. That's why I was asking specifically about the near-term progression of the 5.2 line.

 

@Teo Thank you for trying to help. As has been thoroughly documented in this thread, we are not running the latest patch. We are waiting for a stable patch release, but 5.2.2p4 will include a lot of changes for us; we are currently on the last stable patch, 5.2.1p1. So our main concern is stability. Karl has mentioned previously that patch releases receive less testing than final releases, so that is why I asked about the timeline of the next final release in the 5.2 line. I am well aware of 5.3, but it is not an option for our current project because we are wrapping it up for release.

 

Yes, that's the problem with 5.3 coming on 8 December, is to late for any AppStore vendor. Even you upgrade your project for GGX and rest of stuff on 8 December and everything will work ( !!! ), you still have to wait for approval from Apple, and we all know how overcrowded is AppStore in December, and this can take a week or more. That can put your release at the end of the year or even maybe next year

 

Super, thanks for update Karl and for staying with us

 

Real time lights occlusion culling bug is fixed now in 5.2.2p4, I can confirm that.

But, new bugs:
- Terrain tree colliders are gone, despite the fact that colliders are in SpeedTree model
- Max Mesh Trees have no effect

That's what I've found so far.

 

I was trying to test, but as soon as I installed, now I cannot open any of my projects
Even restoring backups... it just can't open (it stays forever sometimes in "Loading assets", or sometimes in loading Unity window (it starts to open Unity window, but it is all white and... loading forever)
My system is Windows 10 64bits, 16 GB Ram, GTX 980ti
Unity PRO 5.2.2p4

EDIT: yes, confirmed, I installed again version 5.2.2p3, because p4 makes my Unity loading forever when trying to open any project.

 

Worked before 5.2.2p4, but I am sure the numbers of trees are a lot more, initially was around 10k.

I've check that bug report, but I am a bit confused, should not be used any kind of dynamic AABB tree partition for collisions?

 

Hello Karl,
Just to let you know that deleting GI cache worked for us, thank you for your support. We will test the OC asap today and let you know here if it will work. We are not using terrain trees in this project, so, can't help.

 

The OC seems to be now working here too. Although, I was confused at first because of the large yellow camera volume box. I had never seen that before because of issues with the "Visualize" not working before.

 

Yeah, any time you take a new Unity patch or update, I recommend these actions:

1. delete* your Library/ folder before opening the project with the new editor, to let the project completely reimport (*renaming the folder, so you can quickly roll back if necessary, is a much better idea)
   
2. after reimporting, delete your GI cache from the global Preferences menu
3. clear/rebake all lighting in whatever scenes you normally bake it, before trying to run anything
4. clear/rebake all occlusion data in whatever scenes you normally bake it, before trying to run anything

5.2.2p4 is working well in preliminary in-editor tests! Now I need to spend some time with it in builds on target platforms.

Also, it's really great to see the OC vis lines and portals working again...I think it's been about a year since that has functioned! That will be super helpful for perf tuning.

Thanks, Karl!

 

Can you explain a bit more about step 1 (deleting the library). Is this really necessary, or do you find a reason why this should be done? I would get nervous doing this. I agree with all of the other steps (2 through 4).

 

BTW, occlusion data baking also has a cache. I have found that after a long period of time, if that cache is not wiped out, occlusion baking starts taking a lot longer than normal, right at the very end of the bake.

I don't think there is any hook in the editor UI to clear the occlusion cache, but it is located in Library/Occlusion. If you start running into slower than normal occlusion bakes, it could help to delete that. Of course, if you semi-regularly wipe out your Library/ folder and reimport the project, you'll also be cleaning out that cache.

 

First, I would say that the Library/ is not your project, your assets are your project. The Library/ is nothing more than an import cache, where your assets have been analyzed and re-exported into a format that complies with each asset's import settings and the target platform. For example, if you have an asset that is a 4096x4096 png, you might modify its import settings so it is actually 4096x4096 on PC builds, but maybe downsampled to 256x256 on some other platform. The metadata that describes these import settings is stored in an asset's .meta file, which lives along side each asset. However, upon import for the smaller platform, the editor will create a downsampled 256x256 version of that asset and it will be stored in the Library/ folder. Again, note that the new 256x256 png is not actually one of your source assets and you really shouldn't care about it at all. The original 4096x4096 asset is the one you need to care about, and that is somewhere in your Assets/ folder. The new 256x256 version in Library/ can be deleted and recreated any time you want, based off your original asset and the .meta import settings that describe how it should be imported. Apologies if this sort of thing is already well known, I just wanted a simple example to point out that Library/ is just a cache that can be recreated at any time from your assets.

Second, I would point out that in olden times, I think you could set up Unity projects to not use .meta files at all. I wasn't using Unity back then, but I believe that under those circumstances, the Library/ folder may have been essential to the project? Without .meta files, where would custom import settings be stored? Possibly just in the Library/ cache itself? I don't know. So if you are reading this message and using a version of Unity without .meta files, maybe don't delete your Library/ folder.

Third, the actual reasons for clearing the Library-- I don't have any specific examples to name, but we have run into many cases in our projects where we could not convince Unity to import updated assets the same way on all team members' computers. Sometimes it seems that certain import settings are 'sticky' until you just nuke the relevant area in the Library/ cache. If you don't know where to find the relevant area, nuking the whole cache is the only option. Given that we have problems like this without even updating Unity, it seems like a good idea to nuke the cache for a new version of Unity and make absolutely sure the new version of Unity is importing the asset in the way it wants to: **New versions of Unity will sometimes have new or modified default import settings for some assets.** It is very important to us to avoid "surprises" in the future. Like I was saying previously, Library/ is not your project, Assets/ is. We do not store Library/ in source control. When someone needs to set up a new computer, when they get the project, they start with the project assets and project settings and let Unity reimport, they don't get the Library/ folder from anyone else's computer. If we ever need to "send the project" to anyone for any reason, it will not include Library/. So, for reasons like this, it is very important that a complete reimport of the project is reliable and will produce the same result that we are working with every day. If you never do a complete reimport of your project during development, you run a very high risk of there being a) actual problems/errors with a clean import that you have never seen b) some assets appearing in-game differently from the clean import than the version that you have been working with, due to issues like different import defaults in future versions of Unity than in the version that you originally imported your assets in, etc.

Basically, I don't consider an update/patch completely tested until it has had a 100% clean import and everything checks out. You will often find little things that need to be tweaked (because of something that changed in the update or import settings you changed at some point yourself but they didn't "take effect" without a clean reimport) when you do this.

Sorry, I know this is a long message and is probably rambling-- I have to get back to work, so I just dumped a bunch of thoughts out there without really proofreading. Apologies in advance for anything that doesn't make sense!

PS: Caveat: folks from Unity may not agree with or recommend this approach, so do not take this as gospel! Find processes that work for you. Try to understand more about the import process and the role of meta files. I have found that this works for me to solve problems I have encountered and to provide peace of mind regarding any clean reimports that may need to be done in the future.

 

ok, thanks for the info. I should probably get around to doing this, I'll wait until 5.3 comes out.

 

OC working now for me as well.
And thank you LeonH for the tips =)

 

Ok, functionality-wise, 5.2.2p4 seems to address the issues we were hoping for-- the main one being realtime lights seem OK when occlusion culling is enabled! Other benefits coming from 5.2.1p1: hinge joints work correctly again, rendering crashes we were seeing on target platforms randomly within 1-20 mins of game uptime have not been observed after 7 hrs of uptime on one of our targets!

Unfortunately, there is a new showstopper (for us, possibly not for everyone). Perf seems to have taken a major hit since 5.2.1p1 that we have been stuck on. In 5.2.2p4, wth zero content changes from a build using 5.2.1p1, the frame rate on one of our target platforms is literally 30-50% reduced. Rendering (looks like primarily shadow processing) seems to be costing 2-3x as much as before. I have no idea yet if that issue is platform-specific (PS4) or if it's a general problem, but anyone trying this patch may want to keep an eye on their perf.

 

Does anyone know if there is an issue with the occluder static flags needing to be toggled off then on when upgrading to a new version of unity? Right now, when I run the game, pretty much everything which was NOT marked as "occludee static" is being occluded by occluders. I have toggled the occluder flags off and on and rebuilt the occlusion, now it seems to be working fine. I have created a repro and verified this phenomenon.

 

Just grasping at straws, but possibly related to old occlusion cache data from the previous version (mentioned above, in Library/Occlusion)? If so, would have been covered by deleting Library/! *zing*!

 

No, I deleted the occlusion folder and started from scratch. That didn't resolve the issue. Maybe I should delete the entire library folder too?

 

I dunno, I was mostly joking. I wouldn't want you to waste all that time reimporting if it turns out to be unrelated.

 

Are you using dynamic occlusion, btw? The docs aren't 100% clear, but my reading is that if you use OcclusionArea volumes, it allows for dynamic occlusion culling-- i.e. even objects that don't have occludee static can be culled.

 

I'm not sure what you mean. My scene I have an occlusion Area setup.



Toggling the "is view volume" make no difference for me.
What is "dynamic" occlusion?
Thanks.
j

 

When I say dynamic occlusion I mean applying occlusion culling to objects that do not have the "static occludee" flag set (e.g. moving objects). According to the docs (http://docs.unity3d.com/Manual/OcclusionCulling.html), you need to use Occlusion Areas to enable that. In my experience, occlusion culling for objects with the "static occludee" flag set do not require an Occlusion Are to exist in the scene at all.

So yeah, if you are using an Occlusion Area, then you are using what I called "dynamic" occlusion-- which means that it would be normal/expected for objects without the "static occludee" flag to be culled sometimes, according to whatever rules or conditions that OC system uses for that when an Occlusion Area is in effect.

 

This does not sound right at all. It is my understanding OC should not occlude any objects that are not marked occludee static, regardless of whether you have an occlusion area setup.
It is my understanding that you can create an occlusion area to avoid unnecessary areas of the scene from being calculated in the OC process, so to let you override the default of the entire scene being added to the OC.

The docs say:
NOTE: By default if you don’t create any occlusion areas, occlusion culling will be applied to the whole scene.

NOTE: Whenever your camera is outside occlusion areas, occlusion culling will not be applied. It is important to set up your Occlusion Areas to cover the places where the camera can potentially be, but making the areas too large incurs a cost during baking.

But anyways, deleting the Occlusion Area does not fix this issue for me.

 

Yep, you're getting at why I said the docs didn't seem clear to me and require some interpretation. Note that the doc also says this as the very first sentence in the section describing Occlusion Areas:

 

I assume this is because the moving object can move out of the "Default" occlusion area. You could have a scene of 100x100 meters, unity bakes that area. Then if the gameObject moves out 2 miles away, you could run into problems because the default occlusion was not that large to begin with. I think this is why you can create your own occlusion area.

Anyways I'm still having issue. I'm stripping my scene down to a wall and some non-occludee static objects, yet those objects are still being occluded. So I'll have to do another bug report.

 

Ah, you have the opposite problem as me-- I have objects that are not being culled that I think should be culled!

BTW, not sure if you want to keep going with this discussion (if not, that's fine), but I am not really clear about what you described in your first paragraph. In a previous comment, you said you didn't think the OC system would apply to an object that wasn't marked occludee static. However, in your most recent comment you are talking about a "moving object" and its interaction with the OC system. So I'm not sure if we're even talking about the same sort of thing?

 

My issue is that non-moving objects are "disappearing" (being culled). These are NOT marked as "occludee" static objects. So they should not be disappearing. The only time this should happen is if you mark the item as "occludee static". That is what I was trying to say in first paragraph.

For a moving object, it can still be occluded if it is marked "occludee static", but it must be in the occlusion area.
This is my understanding of how it works. I could be wrong here, but if you didn't mark the item as occludee static, how would Unity/Umbra know it is supposed to be culled?
Also, when the manual says "of course it cannot be marked as staic..", I think this might be referring to the fact that it cannot be marked BATCHING static. (Because batching static objects cannot move). I don't think that is referring to "Occludee static"

Maybe someone like Karl could shed some more light on the "occludee static" flag and the Occlusion Area.

My issue is shown here:

1. The door is completely un-marked static. In this view it is visible


.
2. The camera moves over a little, now the door is being culled.


j
THe door is not marked static at all:



I'm 99% sure this is not supposed to happen. Someone correct me if I'm wrong.



j

 

I see. In that case, I think you are mistaken, assuming the sentence from the documentation I quoted is accurate. It describes using Occlusion Areas to apply occlusion culling to objects that, as quoted directly from the doc, "cannot be marked as static." I am not suggesting that is the cause of your problem. I am just saying that if you are using Occlusion Areas, it would be incorrect to assume an object without the "occludee static" flag can never be culled.

That is not what is stated in the doc regarding moving objects within Occlusion Areas, so I think no, that is not how it works. In fact, a moving object that is marked "occludee static" is what is known as a broken object. My understanding and observation is that the word "static" means something very specific in these cases, and checking those static flags is basically signing a contract with the systems that use those flags, promising the object will never be moved. If such an object is moved, what will be observed is bugs. For example, if you move an object that has "occludee static" checked, it will be culled/visible based entirely on whether its original (static) placement is visible to the camera, not based on where its new, actual position is. The same sort of principle applies to all of the static flags and the systems that regard them.

As for how Umbra could know if a non-static-occludee is supposed to be culled, well, how does it know what to cull wrt baked occlusion and static occludees? When baking, it does a S***load of ray traces and/or other calculations to determine if visibility is blocked by an occluder. Some form of that is probably done for dynamic culling, likely with fewer ray traces, even stronger attempts to "err on the side of visibility", and a lot of results caching. Potentially being able to piggyback somewhat off the baked information as well. This would be applied to any active renderers that do not have "occludee static" set, are within an Occlusion Area (based on my reading of the doc), and aren't already culled based on the camera's frustum. Just a guess.

But anyway, none of that helps with your specific problem here. Looks like a bug or something. I understand you may not agree with my interpretation of Occlusion Areas, but if you removed yours and you are still seeing the problem, I don't think there is a valid explanation for what you've shown. After removing Occusion Areas from the equation, Umbra should never cull a renderer that doesn't have the "static occludee" flag set. Maybe it is an issue that would be fixed by wiping out your Library folder, I dunno. Maybe as an experiment, before going to sleep, close your project, rename your Library folder, then reopen the project and let it reimport over night. Check out if the problem still exists. If that didn't fix it and you don't trust the new Library folder, you can then always get rid of the new one and go back to the renamed one you saved.

 

I've made all trees using SpeedTree, and every tree have max one capsule colider, and some of them (bushes,etc) don't have any colliders at all. There are about 10k trees in scene.

The big problem is that some trees don't collide anything now, BUT I get some random invisible collisions where should not be.

 

You are most likely correct regarding the occlusion area. However, I see a potential issue with this. Let's say you have a transparent object moving through your scene. You always want this object to be rendered. Maybe you have a character outline shader. But what you're saying is that as soon as you create an occlusion area, you would then be causing your object to become occluded, regardless of the static flags. This is a problem because then your gameobject gets culled, which is not what you want to happen. So, how to you specify that that particular object should not be culled? This is why I suspected you needed to Clear the Occludee static flag. Otherwise, your outline would constantly be disappearing.

I think you are right that the documentation too is confusing. Because for this line: "NOTE: By default if you don’t create any occlusion areas, occlusion culling will be applied to the whole scene." This is not very clear. I took this mean that if you don't create an occlusion area yourself, then Unity automatically creates the occlusion area for you.

As for my issue, I spent an entire night troubleshooting this and from what I can tell, the issue is the backface threshold value was too low. I had the value set to 10%. This was causing non occludee-static gameobjects to be culled. I since changed the threshold to 100%, now Unity/Umbra is honoring the occludee static flag. So in my scene, the door is not being culled when set to 100%. So I think that is the issue, the backface threshold is set too low, somehow causing occluders to occlude gameobjects regardless of the occludee static flag. But it seems to only happen for things close to the camera. I actually posted a thread before about this: http://forum.unity3d.com/threads/un...atic-objects-very-close-to-the-camera.368359/. I think this might be the same issue.

So for now, I'll leave the backface threshold at 100%, because it seems to be causing issues.

 

Ahhh. Yes, I tried playing with that before. If I remember correctly, the only real benefit for reducing that is a smaller occlusion data size? And I remember seeing almost no change in occlusion data size when I reduced that, so I just leave it at 100% to be safe.

BTW, I did some tests last night and "non-static" occlusion culling is definitely happening. You don't even need an Occlusion Area for it, just like static occlusion. By default it just happens everywhere. So, that is probably related to what you were seeing.

I messed around with Occlusion Areas for a while to try and figure out how exactly they work wrt to the docs, and I really didn't see any difference at all for occlusion behavior inside or outside volumes. The only thing I noticed is if you do not have the 'is view volume' checked, a lot less occlusion data is built within the area, and if you are looking through a camera inside such an area, occlusion culling can get really weird.

As for how you'd implement your example of not culling an object, I think you may just have to use a multiple camera setup for something like that.

 

Cool, it's nice to see someone else has noticed the same thing. So, would you agree that there seems to be problem? Meaning, do you agree that the occlusion should not be happening for non occludee-static objects?

As for creating multiple cameras, I was afraid I would have to do that. I don't really want to do that because then you start running into camera multiple post-effects issues, which is an entirely different discussion.
The way my scene is structured is unusual, it is a large circular area, so it's not really ideal for OC. I might just have to not use OC for this particular scene.
Because right now, the large amount of OC data being needed is resolve this issue might be counter-intuitive.