How to cull gameobject from camera without using layers?

Layers are no use - the 32 limit is too small even for a single project, let alone for code that I'm distributing to different teams (I can't steal their layers, Unity has too few to start with).

Given that, what's the best way to cull gameobjects during rendering?

Things I've tried:

1. Look for something in OnWillRenderObject() that lets you tell Unity "don't render it; cull it" - I expected at least a bool return, but seems no such method exists
   
2. Looking for some form of "ShouldCull()" method, as I've had in other game engines. I can't find anything in the docs
   
3. Replace rendering with manual call to Render(), embedded inside OnWillRenderObject(), and disable the object using SetActive(false) immediately before rendering, then restore it during
   OnRenderObject() - this works in Play mode, but fails in Editor mode
4. Using shader-replacement - this doesn't work because you cannot say "render everything normally EXCEPT FOR object X", instead you have to manually rewrite every single shader that exists on every object everywhere in your scene. Which is effectively impossible.
5. Nuking the mesh bounds, and then restoring it. This feels like a really bad idea from a performance POV, and you have to be a bit careful about choosing bounds that are DEFINITELY outside of Unity's frustum culling (which has some bugs - it will crash on some values - and has a margin-of-error, so simply sticking it a long way away won't work)

This feels like I'm missing something obvious - surely Unity allows games to do their own culling? It's such a huge performance win!

 

In SRP there is additional "rendering layer mask". But afaik its no exposed in standard rendering, nor URP/HDRP. And you only can use it if you write custom render pipeline.

 

I could understand if it's something they decided not to fix for the last few years "because SRP is coming soon and will let people fix this", but it's even more strange if SRP doesn't make this available.

Render performance with the mesh.bounds trick seems to be "not terrible", but it feels very hacky (and I wouldn't be surprised if it costs significant performance in a few places - especially because it's messing-up any caches they have for the PVS stuff)..

 

Hi, you can try stencil buffer in shader.

 

What do you mean?

 

Let me try to explain it.
Using stencil buffer, you can mask some area in the screen.
So you need two shader files, one for camera (whole scene area. You can create a transparent plane gameObject in front of the camera covering whole scene) another for the culled gameObject.
The shader for culled gameobject will set the mask value of the area in the screen, but not repaired stencil buffer value.
The area masked by culled gameObject will be ignored in rendering with the shader for camera.

 

I've used stencial buffers lots for masking, but what you descrive is not culling, it's rendering twice and masking. I guess it's a possibility, but none of the other options increase draw calls, so I would expect this to be worse than them?

 

increase draw call? I don't understand what your mean.
You can try this:
1. In the shader of culled gameobject, you set the stencil buffer value.(e.g. 1)
2. In the sahder of transparent plane, you set a stencil test value and condition to ignore the area which value equal 1.

 

What happens when the culled object is transparent (for instance)? What happens when the culled object's shader is higher in the rendering queue? What happens when you don't want to (or can't!) change the culled object's shader?

I don't think stencilling improves on any of the alternatives here.

Culling is about doing CPU-side changes that affect the whole pipeline, where stencilling is GPU-side (by which time it's too late to do some things, and even if it's not too late, you pay the cost of having pushed stuff all the way through to the GPU when you could have dropped it before it even got there).

 

I've used LODGroups with a single LOD for this. Sounds hacky because it is, but it works.

Alternatively you can disable the renderers when the camera is at a certain distance.

 

Share more info on how you're (ab)using LODgroups for this ? I'm guessing something around: placing items in the last LODGroup, which tricks Unity into culling it?

Also ... Have you compared it vs. null mesh-bounds?

 

I have a component that grabs all child renderers, and put it in the first group. The last group has no renderers. This allows for culling. The upside is that it also handles fading, if you want that. (I did).

I've never tried null mesh bounds.

 

Cull gameObject in shader will be some restrictions. It's certainly.
Do you want perform culling algorithm in script?
That are some limitations too. I think so.

 

As far as I know, CullingGroup doesn't provide any culling at all - I believe it was accidentally given the wrong name (it's really a "QueryVisibilityGroup", but it happens to be calculated during / as a side-effect of the Culling pass, so it got the C word in its name instead).

In most engines, the equivalent to the CullingGroup API would have 2x as many methods, because it would have the methods + callbacks for ShouldCull? WillCull, PreCull, PostCull, etc.

The Unity docs are weak here, but reading between the lines, they expect you to use:

• either: GetComponent<Renderer>().enabled = false
• or: GetComponent<Renderer>().gameObject.setVisible( false )

to do the actual culling. Technically that's a kind of culling - but very coarse-grained and often inefficient, and doesn't work for most interesting/custom situations.

 

PS: the official docs are, as far as I can tell, absolutely wrong in the opening sentence: "CullingGroup offers a way to integrate your own systems into Unity’s culling and LOD pipeline." - it does not do that! Instead, it does the opposite: offers you a way to passively react to Unity's culling pipeline (you don't get to "integrate" with it at all ).

NB: I'd be delighted to be proven wrong here! CullingGroup was the first place I looked when I wanted to cull, but quickly realised that there's nothing in the API that does any culling, and that no-one on the web seems to use it for culling, etc. i.e. it's not a culling API.

EDIT: ... although it's a fine API! You can do some simplistic stuff with it, which is nice because otherwise you'd be implementing it all by hand (although not that hard to write, I think - all the info you need is already exposed publically?), but it doesn't seem to add much that wasn't already in Unity (I'm assuming it's higher performance than running your own visibility calcs per frame - but at this level of simplicity, those calcs are already likely to run so fast I wouldn't expect them to be an issue, and most of the performance cust is in the culling implementation, which CullingGroup gives you no access to / no integration with ).

 

TL;DR: I've committed to using "Very large - but not too large!" mesh.bounds for this. It works (although, as noted, it exposes some obvious bugs inside Unity engine (I think one of them crashes the 2019.x editor, from memory). BUT I cannot think of a situation where someone would deliberately want to make mesh bounds of that size/shape, so I haven't bothered reporting them).

If I identify a performance bottleneck that seems to be due to this hack later, I'll come back and cry for help, but for now it appears to be the unofficial(ly official?) way to do culling in Unity .

 

I believe I may have another approach for you to consider that could be far more performant than enabling/disabling renderers or nuking the render bounds:

• Reserve one layer to never be rendered, eg. name the last layer 'DoNotRender'
• In each camera's OnPreCull set the layer of all objects that this camera should not render to DoNotRender.
• When setting to DoNotRender, store each object's previous layer, and lazily revert only once in OnPreCull of a camera that does need to render the object.
• Since you have to evaluate "Should Object X be rendered by camera Y" quickly, one convenient solution is to use runtime scenes to group objects with the camera(s) that should render them. The check is then as simple as camera.scene == gameobject.scene
• This approach leaves you with 31 layers, and since we change layers before culling happens, each camera will only ever try to frustum cull or render the objects it should.

In my use case I wanted to be able to have multiple 'parallel universes', where each universe has objects with layers (Thing1, Thing2, etc) and cameras that should only render objects in their own universe.

Obviously if I only used static layers, I would need a version of each layer for each universe (Thing1Universe1, Thing2Universe1, Thing1Universe2, etc...), and you run out of layers very quickly then.


Side tangent:
As you know Physics uses the same 32 layers, limiting our options even more, but somewhat recently Unity added the ability to create PhysicsScenes, by passing in a LocalPhysicsMode when you create a scene with SceneManager.CreateScene. Physics scenes are completely isolated regardless of the layers on the objects they contain. If only Unity gave us the ability to create a RenderScene in a similar manner.

Or if gameObject.sceneCullingMask, camera.overrideSceneCullingMask or
EditorSceneManager.SetSceneCullingMask
were writable/usable outside the editor, it would make all of this so much easier...

 

It also disables the game object for all cameras. What happens if multiple cameras from multiple players are looking at the same spot at the same time. How would disabling the gameObject actually work in that case? So, he's right, it's not really culling, it's some half baked work around that doesn't really solve the original problem in many cases.

 

has anyone figured this one out ? the only viable way I found was to either mess around with layers, renderer.enabled, but that's not very satisfying as it may collide with other uses of these features. I agree that it's sad that even in URP we can't just add a filter as an input to specify a list of renderers to even consider for rendering