Postprocessing seamless portals

Hi all!

The game I've tinkering with relies quite heavily on seamless portals. The game is an MC Escher meets Mario64 kind of labyrinth, where the player gets lost finding the way out. I use portals heavily for my "dimensional engineering", to make connections between rooms that just don't add up geometrically. For instance some rooms look infinitely large by having a portal on each side looping back to the other side.

The portal rendering is based on this reflection script, but with a custom matrix to allow for translations, rotations and other transformations. Basically an extra camera is created when rendering the portal, it writes its output to a render texture and the custom shader apples that render texture onto the portal in screen space.

However, when starting looking into the post processing stack it seems the portals don't work very well with it. The current portal shader only renders the color, not the depthmap, tangents or other maps that post processing may rely on.

Is the best approach here to try to draw these additional maps from the portal camera onto separate render textures, and try to get the portal shader to read these textures, and then write them to the camera output.

Or is it perhaps better to chose an art style that doesn't rely on post processing?

 

It's certainly possible to output the depth to a separate render texture and output that, but I can't help but wonder if stencils might be a better approach for you to take.

Generally though, it's going to be a huge pain to get useful depth and normal information that works with the built in post processing effects when doing portals. You may end up needing to write your own from scratch no matter the path you take. The normal buffer for one is output in world space, which means for each portal you'll have to rotate the stored normals into the player's current space. Same with the depth, which is a per-camera depth between the near and far planes, and would have to be converted to match the player's main camera. There are some examples of that floating around that people have done to get fog working in stuff like water shaders.

However because of the way Unity generates the normal and depth textures for those post process effects, it can very difficult to inject new objects into them or modify them, and outright impossible in some cases.

 

How do you mean stencils could be used here? (I think I heard of it for the first time yesterday)

I've read up a bit more on writing to the depth buffer and it appears to be impossible, at least to write to the right one. I'm starting to think this is beyond my level for my first ever game. It would be quite cool though to combine some highly stylized textures with some very realistic post-processing effects, to form some kind of contrast.

What buffers are needed for what post processing? I suppose bloom and motion blur don't need the depth buffer? Fog, which I originally intended to use to obscure the end of infinite rooms, absolutely need the depth buffer. Depth of Field would be rather sweet too from a visual perspective, but that can't be done either without depth info. However depth of field is maybe a bit more forgiving than fog when the depth is slightly incorrect. In some cases I could place a copy of the portal destination behind the portals, but in other cases there wont be the space (like when something that is bigger on the inside).





I can use other techniques than post processing to hide the end of infinite rooms though. My bridge room uses portals with 8 or so recursions. After the last recursion the room continue for some distance and just ends with a flat floor(or is it a ceiling?). When seen in the distance through the gaps between the bridges you can't really tell it's a full floor and not another bridge you are seeing.





It's of course a little tricker in other rooms that don't have obstacles like the bridge, but maybe I could use a forced perspective for that.

 

https://www.alanzucconi.com/2015/12/09/3873/
http://ludumdare.com/compo/2016/12/23/non-euclidean-room-postmortem/

It's best to assume you have access to color, and nothing else. That means you get bloom. No per object motion blur, no edge detection, no depth of field. All of those require a depth texture.

Fog doesn't need the depth buffer as it's done as part of the shader drawing the object rather than a post process, assuming you're using forward rendering and not deferred. Even when using the deferred path, any non-opaque object will be rendered using the forward rendering path anyway.

As mentioned in the link you posted about writing to the "right depth", the depth texture is what Unity uses for this and for forward rendering that's generated in a totally different pass of the entire scene before rendering it proper. Same with the normals and motion vectors. Injecting into the depth texture itself is possible using command buffers, but the normal and motion vector passes cannot be injected into. Unity simply lacks the hooks to do so. You can use the frame debugger to step through how Unity renders a scene to see this all happening.

Technically Unity's deferred rendering path outputs the depth and normals as part of generating the gbuffers, but adds other complications.

With some work (ie: not using any built in Unity shaders) you could use a forward rendered MRT and have your shaders output shaded color, depth and normals all in one pass rather than using Unity's additional depth and normal passes, but that's a lot more work. You'd also have to modify or write your own post process effects to utilize those outputs as you can't override the built in ones directly. Though, if you go the stencil route, you'd already have to go with 100% custom shaders so it might not be that much more.

 

Hello bgolus

Thanks again for sharing your knowledge!

Is per object motion blur when another object moves in the scene? I'll have very little of that, and would mostly use blur for fast camera movement.

I've looked a bit into stencils and it seems like a possible alternative to a render texture being rendered onto the portal mesh. However I have no idea how to implement it in practice. Currently I use the OnWillRenderObject hook for the portal object to render a separate camera and apply the output to the portal. This works recursively too as OnWillRenderObject gets called when this new camera sees a portal. With stencils I don't know how this can be implemented and what order I should render what in based on what.

Also I'm not sure what benefits it has. Would it allow for drawing to a depth and normal buffer when looking through a portal? Or are stencils taken into account only after this has already rendered? I can imagine it would use less memory though as more things can be drawn to the same buffer, rather than be temporarily stored in a render texture, and maybe be a generally "cleaner" solution codewise.

I'm using an oblique projection for portals to clip what's shown at the portal's plane, and I suspect that could interfere with depth values. One of the pages you linked mentioned the oblique camera projection as a bit troublesome, and I've already had some problems with shadows fading away when getting close to the near clipping plane.



I have however started experimenting with having no shadows, and make that a part of my retro art style.

This scene is far from finished - there should be some stuffed low pony horses in the center of the room, some swords, shields and possibly paintings between the windows, some stylized stone blocks in the walls and tiles on the floor - but I think I'm getting somewhat closer s a visual style.

Maybe I should bump up my ambient lighting and nerf the directional light to make that effect more subtle.

I'm also planning on making my sky shader dependent on the lighting direction, to have it match the general light direction in the scene (my portal rendering and teleportation both transforms the directional light already, so this would make windows match when viewed through a portal).



For an art style like this post processing becomes less of an issue, but could perhaps still be useful to add that extra juicyness. Some color correction could perhaps set the mood and very subtle motion blur could perhaps make it feel more like a modern retro game and less than an actually old game, I don't know.

 

Unity's current Post Processing Stack motion blur doesn't differentiate between per object and camera only. The only difference is an object set to render "per object motion" includes the current object's velocity when rendering to the velocity buffer and not just the camera motion. Older versions of motion blur still used the depth texture. You can do really naive directional blurs based on camera rotation or panning speed like old PS2 era games did, but they're not great and tend not to look like the camera is moving but rather that the screen is being blurred (because it is).

Yep. It get complicated. The render texture approach is certainly more straightforward in terms of getting it working quickly and keeping all of Unity's built in rendering features mostly still functional.

The benefit is not having to have recursive render textures. A lot of render textures means a lot of memory which gets expensive quickly. Stencils use a lot less memory, as you said. For this reason any game with significant non-euclidean space rendering likely use portals rather than render textures as you can have nearly unlimited recursion with only the cost of the vertex counts really being a major factor (assuming no or limited realtime shadows). A game with an occasional mirror or single "portal" will likely use a render texture as they can inject more easily into an existing pipeline.

Also, if you're using a deferred pipeline, or MRT, or at the very least resolving a depth buffer from the actual scene, you only have to render "once" and you get all the stuff you need. Unity's pre-pass depth and normal buffers kill this if you use the built in features, and Unity's deferred rendering path disables stencils, so it makes things a bit harder.

Yep. The depth values being weird is one of the major complications with using oblique projections. For example you cannot use Unity's built in fog as it's based on the projection's Z plane, which with an oblique projection is aligned to the "reflection"/"portal" plane. Instead you need to use custom fog using distance from the viewer's position. For forward rendering this necessitates custom shaders, for deferred you're sh** out of luck. I'm not even sure Unity will allow you to use an oblique projection and deferred, it may just fallback to forward rendering anyway.

The shadow fading you're seeing is quite possibly a bug with Unity's systems mistakenly thinking the near plane is the far plane. The reasons why are to do with reversed Z, and I'll skip that for now.

I think this is a valid path to take. An alternative would be to use baked lighting with no directional lights at all. You could also try disabling shadow cascades in the quality settings.

 

I think I have to refrain from stencils, simply because it's too much to learn at once. Maybe in the future though!

Regarding depth and other buffers I think nearly all my portals will have enough space behind them to contain a copy of the next room. If the portal aren't drawn to these buffers I think I could get a quite decent result.

Anyhow, I feel any post processing will have to wait until I get other things done.

I have at least decided on using some sort of low poly/abstract art style rather than going for realism.

Thanks for all the help!