Unity 4.2 - Stencils for portal rendering

Understandable, well it was worth a shot to ask, it is a shame that Unity doesn't have such things prebuilt, as it would help a lot of people trying to make with something like Stencil, not to mention most of PPE not working in Forward as well, so it's a tricky game we all play.

Again, thank you for your time and the project you ever so give nicely, cheers man!

 

@Noisecrime I finally fixed the stencil thing to work on deferred, it was mere changing how they are rendered in the world.
However the stencil color does not work on the mask?
I'm using the Stencils/Masks/StencilID

Thank you!

 

Not entirely sure what you are asking. Generally the stencil does not draw into the color buffer unless you set the ColorMask to All ( it defaults to none) and that is only really for debugging purposes. However i';m unsure what would happen in deferred since the shader has no deferred passes, so either nothing would happen or it might get rendered via forward rendering afterwards or something.

I'm intrigued to know how you got it 'working' and that might cast some light on why the stencil color isn't showing up.

 

So what I did was have a plane, that is set as a Stencil Standard Shader, and made the Render Queue: Transparent
http://prntscr.com/fbjxom
And disabled the Cast Shadows so it doesn't effect anything.

Next thing I did, was added a another shader to the enemy
And made sure that it is Stencil Operation: Replace
http://prntscr.com/fbjx22

Getting something like this:
http://prntscr.com/fbjyew
*The Hexagon thing is the Stencil Standard

 

Ok, I think what is happening is due to your selection of a transparent stencil shader, all the stencil operations are happening in forward rendering. Its my understanding that in deferred, unity will only render opaque geometry into the deferred buffers, the transparent geometry is rendered on top using normal forward rendering. Its also the case that any shader without explicit deferred passes ( e.g. the stencil mask) will also be rendered using forward rendering after deferred opaque pass.

This should be verifiable by checking out the frame Debugger in unity and noting the names of various camera passes and seeing where the stencil objects are being drawn. If true then there is nothing wrong with this, its obviously not as efficient, but there are many reasons why you might need to have a forward rendering pass when using deferred and this seems like one of them.

What you seem to be doing/need is a way to highlight an object in deferred and the way you are doing it is just falling back to forward rendering for those objects.

When I talk about stencils not working in Deferred rendering i'm talking more about masking out entire areas of the screen and rendering into those area during the deferred pass ( i.e something like real portal system) which to my understanding is not possible - though I keep wondering if there might be a way.

As to the stencil color, in your screenshot the color Mask is coming up as zero, suggesting you've not copied the 'Shader_ColorWriteMask.cs' script into your project as it should be a dropdown with None ( default ), red, green, blue, alpha and all. Setting that to All should draw the stencil color into the colour buffer, but i'm not sure you'll see it since the transparent plane is rendering over it. Maybe I misunderstand you intent here, but the stencilMask shader should not be used to colourise anything, as it should just be writing to the stencil buffer, the fact that you can have it render into the color buffer is just for debugging purposes.

 

Now regarding the render queue I agree, I went with the assumption that it has to render before and thus I've made it transparent as it renders first, making it semi-work, it works good however, like you said, not as efficient as it could be in forward rendering.

I am aware that Deferred is not the ideal for all the situations and while it does not work like you mention aka portal system, it servers its purpose for my game at least, as I am able to mask out it and make it appear as planned to just show the texture where the stencil is. Which is great.

Now the color thingy, the issue with that is that it over renders the color over the stencil (where the stencil isn't it will still render the color like it is there, in other words it is always in some color regardless of stencil and loses its transparency)

 

Also is there a way of making the Stencil Mask use an image to make a mask with a texture?

 

Not directly as there is no relation between the stencil state inputs and textures. However I would assume you could use alpha testing ( not blending ) to achieve something similar, but that depends on where in the pipeline the stencil vs alpha testing comes, as for it to work you'd need to reject ( discard/clip) pixels based on alpha before the stencil operations.

As for your color problems, I'm sorry I'm still finding it difficult to picture what you are trying to do and the results. At this point the only thing I could do to help would be if you prepared a very simple minimal test case unity demo project and I could take a look at it sometime ( but pretty busy atm ).