Help with stencil shader

So I've played around with a stencil shader a bit and would like to create a portal similarly to https://twitter.com/i/status/974431987569217536. I have set up a cube consisting of 6 quads with the stencil mask on. Now objects are rendered behind the mask which is great, but the problem is objects (like the sphere here) between the mask and the camera are also being rendered.
I can't for the love of unity-god figure out how to only render objects which are behind the mask.

I would love some help or suggestions.

 

I've also run into this issue while messing around with "alternate dimension" type portal effects. It looks like you and I are using the same stencil mask approach. I'm not sure that there is a solution that doesn't involve the render texture approach.

The most elegant solution would involve some kind of "stencil depth" test such that pixels would only be rendered if they (a) were in-front of the normal depth buffer, and (b) matched the stencil mask, but (but) were behind the plane of the portal. I think this would involve setting up another buffer, which I'm sure is possible, but I don't know enough to gauge how difficult it would be.

There might be some hacky workarounds that involve the existing depth buffer instructions, the render queue, and maybe even the blending step.

For my project I'm using flat portals, and I'm already planning to cull what is viewed from the other side programmatically, so I might not need to fix this problem at all. Still, please let me know if you run across a solution, and I'll post here again if I find anything.

 

This is unfortunately a limitation of stencils. They just mark pixels on the screen to be a specific value. When an object renders after that there's no way to know at what depth those pixels were rendered at since a stencil buffer doesn't track that. You could write to the depth buffer when you write to the stencil, and flip the ZTest on stencil masked shaders, but then things will draw seemingly in the wrong order, and will break after the first object renders.

There are a few solutions to this.

One is using a shader based clipping plane. That is to define where the stencil's plane is and clip against that in the fragment shader. For shapes more complex than a single plane, like the above, this gets more difficult. You could certainly pass the box's position and size to the shader and do basic in shader raytracing (sounds scary, but isn't) to find out how far away the box is and clip against that.

The other would be to render the stencil mask geometry to its own depth buffer so you can have both the real depth buffer and the stencil's depth to reject against in the fragment shader. Really it's basically the same as the above option, but you can render and clip against arbitrary meshes much more easily.

 

So many concepts here I don't get... Just recently understood how to use stencils - not how they work Well on to shader school and do some learning I guess.

Edit; oh maybe I can just use a clipping plane billboard which follows the camera around a sphere with the same center as the cube? So the billboard is tangent on the circle's surface, like a near clipping plane but stuck to the object instead of the camera.

 

So I've kinda come up with this hacky solution. I use the shader and script from the cross section asset pack. This script allows setting the clipping plane of a material to an actual object instead of world space. So by attaching a quad to the center of my portal cube, which looks at the camera, I can control the clipping. (traditional camera clipping would not work as zooming out would render the objects - this way the clipping stays local to the portal).

The only limitation is that no object can be within the volume of the portal cube, as the clipping will go through those objects making them look weird. But that's no problem for my usage.

Another headache is that every material I use needs a shader which both has the stencil stuff and the clipping stuff, so I have to copy paste these sections - A minor headache as I think the different standard shaders with more complex stuff than just a color and texture, needs to be cut differently - probably not a problem once I figure out exactly how these shaders work.