Need help with Render Queue/Depth of shader

So here is the effect I've been struggling with:
Quick breakdown, the transparent/red-white thing is a glorified cone/funnel shape with all normals facing inward and it's not double sided. Blue cylinder is inside the funnel, green is behind it.
The top picture is with alpha at 0 and bottom is at 255.


What I am looking for is having the red part render in front of the blue cylinder(like it is), but instead of a color like red or alpha, I want it to render the background, or even more ideally, other geometry in order like the green cylinder behind it.
The goal is to make a wormhole effect that shows whatever is inside from the front, but not from behind or the sides. All while being transparent.
So far what I've done is in the first pass I cull all the front faces and set all the back faces to be colored in with alpha. Then in the second pass there is some additive blending and some vertex/fragment animating that happens. Turning ZWrite Off doesn't visually change anything.

Here's the shader code....

Code (CSharp):

1. Shader "Custom/Wormhole" {
2.         Properties {
3.         _Color ("Color", Color) = (1.0, 1.0, 1.0, 1.0)
4.         _MainTex ("Base (RGB)", 2D) = "white" { }
5.         _NoiseTex("Noise Texture", 2D) = "white" {}  
6.  
7.  
8.         _IntensityAndScrolling("Intensity (XY), Scrolling (ZW)", Vector) = (0.1,0.1,0.1,0.1)
9.     }
10.     SubShader {
11.     Tags { "RenderType" = "Transparent" "Queue" = "Transparent" }
12.  
13.         ZWrite On
14.         Blend SrcAlpha OneMinusSrcAlpha  
15.         Pass {
16.         Cull Front
17.         Lighting Off
18.         Color [_Color]
19.         }
20.  
21. //Tags { "RenderType" = "Transparent" "Queue" = "Transparent" }
22.         Pass {
23.             //ZWrite Off
24.             Blend One One      
25.             CGPROGRAM
26.             #pragma vertex vert
27.             #pragma fragment frag
28.             #include "UnityCG.cginc"
29.             uniform sampler2D _MainTex;
30.             uniform sampler2D _NoiseTex;
31.             uniform float4 _MainTex_ST;
32.             uniform float4 _NoiseTex_ST;
33.             uniform float4 _IntensityAndScrolling;
34.             struct VertexInput {
35.                 float4 vertex : POSITION;
36.                 float2 texcoord0 : TEXCOORD0;
37.                 float2 texcoord1 : TEXCOORD1;
38.                 float4 vertexColor : COLOR;
39.             };
40.             struct VertexOutput {
41.                 float4 pos : SV_POSITION;
42.                 float2 uv0 : TEXCOORD0;
43.                 float2 uv1 : TEXCOORD1;
44.                 float4 vertexColor : COLOR;
45.             };
46.             VertexOutput vert(VertexInput v) {
47.                 VertexOutput o;
48.                 o.uv0 = TRANSFORM_TEX(v.texcoord0, _MainTex);
49.                 o.uv1 = TRANSFORM_TEX(v.texcoord1, _NoiseTex);
50.                 o.uv1 += _Time.yy * _IntensityAndScrolling.zw;
51.                 o.vertexColor = v.vertexColor;
52.                 o.pos = UnityObjectToClipPos(v.vertex);
53.                 return o;
54.             }
55.             float4 frag(VertexOutput i) : COLOR {
56.                 float4 noiseTex = tex2D(_NoiseTex, i.uv1);
57.                 float2 offset = (noiseTex.rg * 2 - 1) * _IntensityAndScrolling.rg;
58.                 float2 uvNoise = i.uv0 + offset;
59.                 float4 mainTex = tex2D(_MainTex, uvNoise);
60.                 float3 emissive = (mainTex.rgb * i.vertexColor.rgb) * (mainTex.a * i.vertexColor.a);
61.                 return fixed4(emissive, 1);
62.             }
63.             ENDCG
64.         }
65.     }
66. }

What does "work" is if I give every single material in my game a render queue of 3000. Effectively everything is sorted by depth on the transparent layer as opposed to geometry. Like this:


The only issue with this, besides using the wrong render queue for everything, is that when the camera view of blue cylinder passes to the side it pops back in to view. I am not certain, but I feel like it has to do with the origin of the cylinder model passing in front of the origin of the wormhole model.

While all of that isn't the worst thing, I would certainly like a better solution/alternative if anyone has any ideas. I did also try a stencil buffer approach at one point, but the cylinders were hidden both while inside the wormhole(intended) and hidden again after they left the edge of the model and I couldn't get it to behave how I wanted either.

 

This is an extremely difficult problem to solve well. As you’ve discovered, the sorting order is extremely important and has to essentially be perfect for the illusion to succeed. You are also mostly correct about needing to be using Queue 3000 and that the sorting order depends on on the origin of the model. More accurately the sorting order needs to be back to front, which all render queues above 2500 use, and the order is sorted mostly based on the mesh bounds’ center distance.

Some thoughts which may help.

First, the real problem is you only need to worry about objects that are entering / exiting the wormhole, and the red wormhole “mask” itself. All other objects can, and should, be rendered normally. I would recommend detecting when an object is going to enter the wormhole and only change their render queue then.

Secondly, you will want the mask geometry to render after all opaque objects in the scene, and the skybox. I would recommend using queue 2501. The skybox renders between queues 2500 (the last opaque queue) and 2501 (the first transparent queue) anywhere the depth has not been written to. Since your mask is writing to the depth, but not rendering any color, it will otherwise leave a big visible hole where the skybox should be if renderer during the opaque queue range.

Now, for objects entering or exiting the wormhole, change their queue to 2502. This will ensure the wormhole mask renders before these objects, thus allowing them to be masked by the depth buffer. It also means most objects in the scene aren’t being rendered in the wrong queue, which is good for performance reasons, and allows them to receive shadows if that’s a concern.

You could also use opaque queues as long as you render your skybox manually at the start of the frame (queue 0) rather than relying on the skybox setting. It’s not super hard to do, especially if you’re using a cubemap sky, but it does require some additional work and finesse.


Some Alternatives...

Second camera & shader based clipping plane:
The idea here is the center opening of the wormhole is actually two parts. In one it’s just the outer edge and a “blank” flat plane, and the other is the tube going in. In one camera you render everything that’s inside the wormhole, including the wormhole tube. In the other camera, the main one rendering the rest of the scene, and in the same position as the first, you render the outside edge of the wormhole and fill the “blank” flat area at the center with the other camera’s view. This is basically how portals are done in games like, well, Portal, as well as mirrors in many modern game’s. This can be done with the “interior” camera rendering to a render texture, or some tricky use of clear flags and invisible depth only objects (remember the skybox hole comment?). But how to deal with the object that’s sticking through?

For that you’ll need a special shader, only for objects that are entering or exiting the wormhole, and only for the main scene camera. That shader will define a plane in the world that the shader will clip off if the pixels intersect with. Search for cross section shaders.


Stencils:
You can use stencils, but you’ll need to loose the use of the depth buffer to clip objects. You still need to be mindful of render queues though. Basically draw the mask only to the stencil buffer, no depth, no color, with a low queue like 1000. Then when objects get close to the wormhole swap their shader to one that will discard where ever the stencil has been drawn to. This requires you be far more careful with when you swap the shader, but is potentially a little more efficient.

 

That should be very easy to do, it's mainly an effect that will accompany instantiating a new object, so I can just do it then.

"Queue" = "AlphaTest+51"(2501) is working exactly as you describe for the mask pass. However setting AlphaTest+52 all the way to Transparent at 3000 has no effect on the front face pass.
Update: Did some googleing, it appears the queue is part of the subshader, not the pass(this was not apparent at first glance). So it seems like I can't have multiple render queues between passes. And being that after the first working subshader is found, it ignores any others ones, I can't do different queues between two subshader's either.

So here's where it's at now, the front and back faces must be on the same queue, which isn't that bad. However it's still not working in a narrow ~10 degree arc where it appears like things behind the wormhole are actually inside.

Update2: In the most seemingly arbitrary way possible, it appears that different distances and angles away from center of the wormhole appear to cause different render orders. 1/1/1 cube for scale. The red one and the bottom two are all working as intended.

Slightly better example: If I move the camera slightly left they all begin working as intended, and if I move it slightly right they all stop working. Except the red one which works all of the time(He's my favorite).

 

Correct. Also, you shouldn't have multiple SubShaders unless you're setting up shader LODs or #pragma target levels. You should be using multiple passes. A shader with multiple subshaders will only render the first valid subshader, where as a subshader with multiple passes, all* valid passes will render. I suspect that's what you're already doing, but the terminology is important.

* all unlit and forward passes will render. Only the first deferred or shadowcaster pass will be used.

The visible wormhole should be a completely separate shader and material from the mask. You want this part to render during the default transparency queue of 3000 for the best results. You can do this by making two materials, one for the mask and one for the visible wormhole, and assigning them both to the same renderer component as two material indices. Unity will pop up a warning about a potential performance impact of using multiple materials and that you should use a multi pass shader, but you can ignore it as in this case you can't use a multi pass shader and get the correct result. The alternative is to use two mesh renderers, which is worse for performance.

Basically, anything actually transparent you want to have render as late as possible as it can't use the depth buffer to help sort stuff rendered after. But you do want all of the objects that you want to have render "inside" the portal to render to the depth buffer as they are actually opaque, even though they're rendering during the early transparent queues. Because they render to the depth buffer, when the visible part of the wormhole renders later, it'll properly sort with them.

Also, what queue are the tubes in the background? If they're not supposed to be entering the inside of the wormhole, they should be using the default Geometry queue of 2000.

 

Yup, that did the trick. Separating each pass into two separate shaders and using two materials did it. It's fully working now. Thank you!

Cylinders were on 2502 just to make sure it was working. It's possible that one or more wormholes might be on screen at once and I wanted to make sure they were working properly with eachother.