Unity 4.2 - Stencils for portal rendering

Update: 17.03.17
So its been a while, but finally got round to updating this project and putting the source on github for easier access and modification. The Project is available in versions for Unity 5.2.2.f1 and 5.4.2.f2.

View the Unity-StencilPortalRoomCube Project on Github.

Improved Shader Workflow.
Switched shaders to use Material Property Drawers which both make the stencil states modifyable per material and provides a nice ui in the inspector. This removes the need for many duplicate shaders and the old method of hard-coding values as well as streamlining the project.

Improved Project Layout
Cleaned up the entire project, folder structure, shaders, scripts, materials etc.

Support for Unity DepthNormalsTexture Generation
This means that image effects such as Edge Detection now work correctly with the stencil materials, though in Unity 5.2.2f1 it only works in dx9, the other api's just fail - most likely a bug. Works fine in dx9, dx11 and openglCore in Unity 5.4.2f2.

Improved Example Scenes
The example scenes now have a UI that allow for enabling/disabling various options to observe their effects.

Additional Features

• Stencil Viewer
• Portal Culling
• Revamped Guide Document

Original Post

One of the more interesting features of Unity was the inclusion of access to the stencil buffer, which has a wide range of uses and enables a number of different effects. Two of the most popular being portals and stencil shadows.

In order to explore the new stencil feature I created a little pet project based around portal rendering, with a scene of 9 'room' cubes, with each side of each cube showing a unique view into a different space.

If anyone has played Anti-chamber you'll be familiar with the effect, but its a little hard to explain in words, so I made a quick video of my project to demonstrate the effect.

Its a cool little effect and it can have a variety of uses. I wouldn't be surprised if Anti-chamber made extensive use of stencil portals for many of its mind-bending spatail effects (although its possible to achieve the same thing via Render To Texture portals too).

I'm intending to post the project source here as soon as i've cleaned it up and added some documentation. I'd hoped to provide a compressive guide to the effect and documentation, but time is against me, so i'll likely just have to upload what I already have written.

It should be noted that use of the stencil buffer is only really practical in forward rendering as Unity uses most of it itself in deferred rendering. Its still possible to use it in deferred, but its probably not practical to do so.

Update 22.07.13 WebPlayer Available
Uploaded a webplayer demo which can be found here.

Update 23.07.13: Demo with full source for Unity 4.2
You can download the demo, full source unity project here.

Important!
To run the demo on mobile, you must go into the 'Player Settings -> Resolution Presentation' tab and enable 'Use 24 - bit depth buffer'. This will enable stencils on the device, otherwise the effect doesn't work. However please note this project was designed for standalone (Win Mac) and the demo scene is not optimised for mobile, so performance is quite low.

 

Thats pretty awesome, great effect!

 

Very cool!

 

Neat.

Another thing you can do with the stencil buffer is to fill concave polygons without having to triangulate. I used it in another engine to fill dynamic bezier curves.

You can also use it for dynamic split-screen effects. For example in a two-player split, you can make the split at any angle by splitting along a line perpendicular to the line between the two players. Basically any shape you can render to the stencil buffer can act to stencil in/out the stuff you render to the color buffer. Like so:

​

I'm a bit disappointed this is a pro-only feature, because stencil buffers have been around for over a decade now and I kinda thought they'd stopped being supported because they were so old, better replaced with more modern methods such as render-to-texture. But I guess if Unity gave Free users the stencil buffer they could implement stencil shadows, and some other offects which normally need some kind of render-to-texture, and they don't want that, right? ;-)

 

Damn...That's cool...

Make a tutorial on that.

 

Thanks for the replies, should be uploading the source/tutorial soon.

Yeah there are quite a number of uses for it and its a nice alternative in many of them to using a render-to-texture system and thus save texture memory and improve quality (assuming you're not doing 1:1 renderTexture to screen pixels). Fully expect to see stencil shadows and other effects come to the asset store in the next few weeks. Only wish I had the time to do some myself, but this little portal demo will have to suffice for now.

 

Uploaded a webplayer of the demo and the full Unity project source.

Webplayer demo.

Demo, full source unity project.

 

How are you doing the refractive glass, do you render the insides of the cube to a render texture and then pass it through a refraction shader?

 

Nice work. Will be interested to see shadow solutions for mobile pop up and that sort of thing. Can potentially be faster (and a hell of a lot better looking than low res shadow maps).

 

Really nice!

 

I just check your project, thanks a lot for your work.
There is even a doc !
Good job

 

Awesome!

 

I'm just using the Unity Pro Glass shader (one of the default packages), which uses a grab pass of the entire screen. Obviously its not very accurate doing that, but artefacts aren't really noticeable unless you look for them.

I did make one change to the Unity Glass shader as it used a grab pass per glass plane, which with 36 of them hammered performance. I changed the shader to use a 'named' grab pass, so all glass planes use the same one, which gave 3-4 times the performance. The glass shader also has its queue tag increased to ensure they are rendered and the grab pass is performed after rendering everything else. You'll find all this commented in the glass shader.

Yeah I like to be thorough, though sadly I just don't have the time to write the full documentation I was hoping to do. Hopefully its enough to give an idea about how to accomplish the effect and describe the problems and solutions.


Thanks to everyone else for your kind words. Hope the project is useful.

 

This is really awesome demo. Thank you for sharing.

I tried it quickly on my iPhone 5 and it hovers over 30fps but the stencil itself does not work.
i.e. nothing is being culled.
(I disabled the camera controls to get it to work with no major change to view a specific angle).

Is the stencil feature supposed to work on iOS?

The other interesting thing is that I noticed the anti-aliasing was turned on in the quality settings. I am no expert but when I turned it off, the fps dropped to 10fps. I did not really expect that. am I missing something?

 

Stencil afaik does work on iOS. If it isn't working for you, perhaps a bug report is in order.

 

Hadn't even thought to test it out on iOS and my first thought would be that its not supported, but I don't remember reading that anywhere. Could maybe be an issue with the glass shader, so i'd probably try removing some of those first and rebuilding to see what happens. Its odd really as although the project construction is somewhat complex, the actual features are not, so off the top of my head I can't think of a good reason for this, even less so the problem with AA.

I'll break out the mac and iPad and have a quick look.

I should point out though the sample scene was never designed for mobile, although fundamentally the effect here should work with the stencil shader, the actual scene is loaded with point lights, fancy screen effects (glass shader), forced anisotropic filtering and AA etc. So whilst it will be useful to ensure it does work on iOS, it is not a scene I would recommend for iOS or mobile in general.

 

Thanks to the awesome developers at Unity Technologies, this issue is now resolved.

Its not a bug, just a simple case that in order to use stencils on iOS devices (possibly all mobile devices?) you need to go into the 'Player Settings -> Resolution Presentation' tab and enable 'Use 24 - bit depth buffer'. Doing this will enable the stencil buffer on these devices and I can confirm that the demo project now runs correctly.

However it also runs really slowly (8 fps on iPad 2), though that is not unexpected as the scene was designed on and for desktops, it is heavy in pixel lights ( 1 point light per room and a directional light), has a large number of gameObjects, fancy glass effects using grab pass etc. So although i've not tested, I doubt using stencils is too much of a performance drain, far more likely to just be the unoptimised scene i'm using.

 

Thank you very much, it works on my iPhone now.


I can also confirm that turning off Anti-aliasing stops stencil from working and makes the performance worse.

I removed all the point lightings an changed all the bumped specular shaders to the mobile version. Yes it is not as a nice but it made it work at 60fps on my iPhone5, but if I turn off the anti-aliasing, stencil stops working and the performance drops to 14fps.

Any idea what is the link between stencil and anti-aliasing?

 

No idea, pretty sure in my tests I'd disabled AA early on and everything worked fine. Can't comment on performance changes with AA off, though it sounds very counter-intuitive.

In terms of optimsiations there is much that should be done with the scene. For example Unity has no notion of portals (at least not stencil portals as used here), so even though you might only see the contents of 1 or 2 portals in the same cube room, its still rendering all four portal contents, just that most are being culled by the stencil.

It would probably be advantageous then to add some code to determine which portals are facing the camera, and hide/show or enable/disable the contents within them. This would drastically cut down the number of objects trying to be rendered.

 

I liked the demo so I spent some time to find where exactly the performance issue is.

The glass shader (I also tried the original version) is gaining performance when AA is on. Any Unity developer can help explaining this?

I can also confirm that your version of the glass shader is definitely faster. Nice job there.

 

Thanks for taking the time to work through this. Interesting it turned out to be the glass shader, i'd figure it must be to do with the use of 'grab pass' but maybe there is something else in the shader. However it really doesn't make a whole lot of sense to me that turning AA on would make it perform better, its just counter-intuitive at every level.

I'm busy with client projects atm, but i'll try and have a look into it sometime. In the meantime if you want to pursue this further, perhaps start a thread in the shaderlab forum here. It may be a 'well known' issue or there may be a known issue that certain effects run better with AA on iOS and that will tell you where the issue lies.

 

Does this simple pet sample was like the logic of the game Portal / Portal 2 by Valve? Because I'm planning to make a Portal Effect on my game.

 

No, its not the same as the Portal game which uses RenderToTexture to create a similar effect, that is much more suitable than stencil portals for that case.

I'm just using the term Portal, as a term that has been used graphics programming for decades and just implies a method of masking/occlusion methods, often used to optimise the amount of stuff that is drawn.

 

Thanks this will great for my learning! Looks Great!

 

Awesome stuff, thanks for sharing

 

awesome work man. and thanks for share this stuff

 

Great stuff noisecrime! I have a question though...I am trying to use your stencil shader with Scaleform and no matter what I try to do the shader seems to always stencil out the scaleform as well. Any Ideas to keep the scaleform separate from the stencil?

Thanks!

 

Insted of having multiple shaders for different stencil values. You can use a shader property like here.

Code (csharp):

1. _StencilVal ("stencilVal", Int) = 1
2. ...
3. Pass {
4.     Stencil {
5.         Ref [_StencilVal]
6.         ...
7.     }
8. }

 

hello

This is a great demo, very enlightening and intelligent, ...

Perhaps you can help me, I'm trying to do a series of "stencils" nested, in order to make different cuts, but i been working a lot and , only makes the first or last, cut.

On the next picture, the red box should cut the yellow box and yellow the white one, but it does not happen, I make the red box, cut the yellow and the white, but the yellow do not cut de white, so do you will have one tip for this case ... . ¿?

 

Awsome! Maybe frame buffer fetch will have good performance?but I can not make the fetch work...can u try it for that ?
thanks advanced!

 

Has anyone gotten PNGs with alpha transparency to run properly?

I changed the texture of one of the stencil materials to a PNG with alpha but then the color of the PNG does not come through. When adjusting the maincolor to something else than black it starts to show on the whole quad. The brighter the value of the maincolor, the more it shows in the areas that should be transparent.

 

I hope the answer to this question isn't so obvious nobody asked it yet (I'm have to admit I'm really not too experienced with shaders) but could someone elaborate a bit on what the main pros and cons for doing portals like this with a stencil shader vs. using a render to texture effect are? Why and when would I choose one over the other?

Thanks a lot!

 

Stencil buffer portals will always be cheaper and cleaner than render textured portals, but definitely are harder to implement as they require writing a shader that writes into the stencil buffer. (Not that this is an issue for you as you can download the necessary shaders above).

 

Hi, I'm sorry for bumping this up buy this is too good. Thanks a ton for this. I've learned an awful lot from this.

 

Can we achieve the inverse of this by making changes in the same script?
See pic **
I need to make a window with the Quad. I want the part of the "wall" that is behind the quad to not be rendered but the rest of the wall should be visible as usual. I am awfully new to shader scripting. Any help would be amazing! Thanks!

 

Hey Noisecrime!

Awesome demo, thanks for putting it together.
Would it be ok if I included the shaders from your demo in a package for the asset store?

Thank you,
Kurt

 

If its just the shaders and no other part of the project - i.e. you're not simply repackaging the project or the majority of the project to resell it then yes you may use the shaders with one condition.

You must include the following text either at the head of each of my shaders or as a single 'credits.txt' file in the package somewhere, that states 'Stencil shaders originally developed by NoiseCrime for the PortalRoomCubeViaStencilsExample which can be found at <include link to this thread>'

Same goes for anyone else who wants to use the shaders directly. A credit is required for inclusion/redistribution of the shaders in asset store packages, while a credit is appreciated but not required if simply using them in a personnel project ( commercial or non-commercial). However at no point is permission given to simply repackage the demo in part ( i.e. with minimal modifications) or as a whole to resell or redistribute.

Also the original shader code is pretty old now, I forget off-hand if I uploaded the updated version of project so you could assign the various stencils values through the material, which would be the preferred method these days as it would greatly reduce the number of shaders needed etc.

 

This should be possible, unfortunately I've been and currently still am in crunch mode for two client projects and simply do not have the time to investigate or provide additional guidance. For the time being I would recommend posting your question as a new thread in Shaders sub-forum as i'm sure someone there will be able to point you in the right direction. Sorry about that.

 

First off, thanks for making this, this is awesome! So, RE: the inverse problem, I came up with a super easy, albeit probably inelegant and flawed solution:

Create a second shader, identical to the one for objects you want to show, only change "Comp equal" to "Comp notequal". So, in the example I'm posting, the blue balls have Noisecrime's Normal Diffuse Shader, while the red cube has my modified "notequal" version. Not sure if this will cause issues later on, but it seems to work for now!

 

Hi Noisecrime !

I'm quite interested by this version of the shader, would it be possible for you to post it someday please ?

 

Hi there Noisecrime

I'm trying to use these stencil shaders with particlesystem to kinda create an invisible splat effect that reveals a mesh underneath it, but for reasons, they are not working with particle system, am I missing something?

 

Unfortunately i've not tried with particles, so i'm unaware as to whether there are any issues or not. I would suggest posting some more details, scripts or sample project and perhaps someone can help. Alas currently i'm really pushed for time over the next week or so, so its unlikely i'll be able to look into for a while.

 

Someday is getting closer - I've reworked the whole project, looking to fix some issues. Alas client work is getting in the way of finishing and releasing it. Plus i'd like to release it on github so I can easily make branches for different Unity versions since quite a bit can change between them. Worse case i'd say two weeks before I get a chance to delve back into this.

 

I've updated the project and uploaded the source to github.

Please check the first post in this thread for links and information.

The new version has the shaders streamlined to use modern feature of Unity, support for edge detection image effects, portal culling, updated guide, cleaned up folder structure etc.

 

Wonderful ! Thanks a lot

 

@Noisecrime Fantastic toolkit! Thank you so much for this! This has been so easy to snap in and use.

Do you know of a stencil shader that I can apply to a TextMesh?

 

While you have mentioned that deferred has issues, do you have a way of making it work at least? I'm kinda stuck and I loved the demo so it would be cool thing to have

 

As far as I'm aware stencils with deferred rendering is just not possible.

At one time it looked like it might be, but with further information ( see https://docs.unity3d.com/Manual/SL-Stencil.html ) there is really only 1 stencil bit available ( which really isn't enough to do anything with ) and the complexity of integrating with Unity pipeline, not to mention that the stencil buffer may just simply be ignored for some deferred passes means at this point its probably just not possible at all.

You could maybe create your own stencil system using a separate render texture, but then you'd have to write custom shaders and clip() pixels against it, which I would assume is going to be hugely inefficient especially considering every material will have to do so.

Ultimately for deferred rendering you are most likely to be better of using renderTextures instead.

 

Thanks for the fast reply NoiseCrime, this at least gives me some insight, it is a shame though, a big one, since the SSR for instance only works in deferred lightning, and some other PPE, making this a great barrier.

Also usage of RenderTextures again will effect a lot, rather then having a few shaders like stencil, however, I might be wrong.

Is there any other way other than using stencil, like shaders or any else just faking it to a some decent level?

 

Well even with stencils you tend to need to rewrite a lot of shaders anyway, especially for supporting proper depthNormalTexturemaps, though I guess that is only relevant to forward rendering anyway - see the project and edge detection solution for more info if interested.

I'm not sure that renderTextures are that bad an option. Sure they take up more memory, though these days that isn't much of an issue and they will fit in with Unity's rendering pipeline much easier. Overall they might be more expensive though, depends how tight a fit you can get to the portal shape. Its something i'm dwelling on atm having pursued stencil portals to their logical conclusion and ran into all manor of weird edge cases, particularly with shadows.

Not really. It is possible to do something similar with depth masks, but that is probably more complex than stencils ( far too much to go into details of here ) with plenty of issues of its own.