Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

What are the options to mask a camera to a non-rectangle viewport?

Discussion in 'General Graphics' started by Appleguysnake, Mar 30, 2021.

  1. Appleguysnake

    Appleguysnake

    Joined:
    Jan 9, 2015
    Posts:
    19
    I'm working with the built-in render pipeline, but I'm happy to hear if there are more options in SRP!

    I'm trying to create a dynamic splitscreen effect, so that's adding some factors I'm considering, but I'm asking this question to try and get a more general overview of the options regardless of application.

    The goal is:
    1. Mask a cameras viewport/rendering area to a non-rectangular area...
    2. so that it can be overlaid on another camera
    3. and reduce rendering in masked areas as much as possible
    4. and optionally modify the masked area at runtime
    Here are the options I'm aware of (please correct me if any are wrong)
    1. Render camera to a RenderTexture with a mask texture and custom shader
      • pro: Fairly simple to implement
      • cons: Doesn't reduce objects rendered by the camera, can't easily modify the mask
    2. Create a mesh in front of the camera with a cut out area and a stencil shader
      • pro: Skips some rendering in masked areas
      • cons: Have to modify all other shaders to use the stencil
    3. Create a mesh mask with a shader that blocks objects behind it, but doesn't render anything itself
      • pros: Skips some rendering. Doesn't require updating other shaders
      • cons: Possibly based on me completely misunderstanding ztest, complicated mesh updating at runtime
    Are there other options I haven't considered, or easier ways to do this?
     
    Last edited: Apr 1, 2021
    mandisaw and asimdeyaf like this.
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    You've kind of got the 3 viable options down. There are other options, but they're insane things like modifying all of the geometry in real time.

    Some random thoughts before we get further in.

    Dynamically modifying masked area is a completely separate topic, and kind of doesn't matter for which method you use. All 3 options you list can use a dynamic mesh or texture mask constructed on the CPU, or use a shader to create the shape on the surface of a static mesh. What you do and how you do it really comes down to the specific question of what exactly do you want to do. "Modify the masked area at runtime" is a very open ended phrase that could mean a lot of things.

    An advantage of the render texture based approach is you can do soft masks. If that's something you're interested in doing, then it's the only option for doing that.

    An advantage of the ZTest & stencil based approaches is you don't need a render texture, which can make some things a little cheaper when rendering.

    You can use the ZTest or stencil based approach to help speed up the render texture approach, they're not mutually exclusive.

    You can use stencils and ZTest together to make some parts of rendering the Z mask easier, or allow for more complex interactions with the "mask" and the world, like for portals. But that's probably not applicable here.

    None of these techniques will prevent objects within the camera view frustum, but not visible due to the mask, from rendering. For the most part this shouldn't be a concern, but this does mean you're still paying the CPU cost of sending those objects to the GPU to render, and the GPU cost of calculating the vertex positions. There are ways to optimize this, but this is again another topic entirely.


    You mentioned split screen as your end goal. If you're thinking about doing something along the lines of a dynamic split screen like TT Games' Lego games, then all 3 options are perfectly viable.

    Ignoring the complications of figuring out the best position for each camera, or the angle to split the view along, all you need for the effect to work is an otherwise boring quad mesh that's aligned to the camera(s) that you either move around in view space to create the "slice" or use an alpha tested shader.

    Generally speaking I would avoid the "basic" stencil based approach as, like you said, you'd need to use custom shaders for everything as the only way to clip something with a stencil is for that shader to be stencil aware. At least for the built in rendering path. It might be possible to force all objects in the scene to use a stencil comparison regardless of if the shader is doing them. But there's really no reason to go that route since a ZTest based approach can do everything you need without using stencils.


    For the ZTest approach, really what you're doing is rendering an object to cover the screen as close to the camera as possible. You can do this by just putting an invisible object as close to the camera as possible, or you can use a custom shader that always outputs its depth at the camera's near plane.
    Code (csharp):
    1. Shader "Near Plane Depth Mask" {
    2.   SubShader {
    3.     Tags { "Queue" = "Background-999" }
    4.     Pass {
    5.       ZTest Always
    6.       Cull Off
    7.       ColorMask 0 // only render to depth or stencil
    8. CGPROGRAM
    9. #pragma vertex vert
    10. #pragma fragment frag
    11. #include "UnityCG.cginc"
    12.  
    13. float4 vert(float4 vertex : POSITION) : SV_Position
    14. {
    15.     float4 clipPos = UnityObjectToClipPos(vertex);
    16.     // most graphics APIs, the near clip space plane is at w
    17.     // so we need to be just slightly less than that
    18.     clipPos.z = clipPos.w * 0.999999;
    19.     #if !defined(UNITY_REVERSED_Z)
    20.     // openGL the near plane is at -w
    21.     clipPos.z = -abs(clipPos.z);
    22.     #endif
    23.     return clipPos;
    24. }
    25.  
    26. // frag doesn't need to do anything, so it doesn't
    27. void frag() {}
    28. ENDCG
    29.     }
    30.   }
    31. }
    Slap that on any mesh and it'll stop anything that doesn't use
    ZTest Always
    from rendering where that mesh is on screen. For split screen you'd use a second camera that has its Clear Flags set to Depth Only and then render a mesh that covers the parts of the screen that weren't covered in the first camera. Render UI on top of that and you're done. To get the TT Games look you'd also draw a thick black line across the split screen edge.

    For more complex shaped split screens you could use stencils, but only to mask the depth mask shader. You'd render an invisible object that writes to the stencil buffer, then render the depth mask either only where the stencil was written to only where the stencil wasn't written to. Otherwise the setup is the same.
     
    mandisaw and asimdeyaf like this.
  3. Appleguysnake

    Appleguysnake

    Joined:
    Jan 9, 2015
    Posts:
    19
    Wow, thanks for the detailed answer! I'll see if I can clarify at all where things were unclear, but your answer is already a huge help.

    Two main issues I was confused about were that I thought stencils only worked with geometry and didn't care what the fragment shader did, and how to update rendertextures without doing a bunch of expensive setpixel() type stuff.

    For the first, it seems I was wrong, and clipped pixels aren't stenciled? And for the second, it seems like the answer is to just use a camera/shader to update the texture.

    I am very much interested in soft masks! I wasn't sure if it was too much to hope for, and it's part of what I was hoping to understand better here. My naive thought was (for a pie-shaped splitscreen mask on each camera) to make a texture with a gradient and modify either the mesh or the UVs to match the mask area, then use the gradient to set the alpha of whatever pixel was rendered. Not that I actually know how to do those things, they just seemed possible, hence me asking for an idea of what's possible/sane. :confused:

    Sorry for the open-ended question, but what I'm trying to do is perhaps overly ambitious so I thought I should limit the scope of the question. I've wondered about this before so I wanted to understand the general options for masking on their own and proceed from there.

    I'll explain a bit more. Feel free to focus on the parts you think are relevant and I'll break out the remainder into a separate post.

    I'd like to do a voronoi style splitscreen as you mentioned, but with 6-10 cameras. (It's not a normal multiplayer game, I realize players would hate it if it was even playable). Here's an example of it working with four players (should start at 30:30). Ideally with fuzzy edges between cameras, but I consider that part a bonus. Obviously figuring out the geometry is a whole separate issue, I think I mostly have that part figured out. It relates to the rendering in that what geometry information I need would vary depending on how I was creating the masks.

    (Side note about insane ideas: I was going to generate the points for players and boundaries, create a delaunay triangulation, use that to make a mesh for each camera each frame, use the mesh as a stencil, create a voronoi diagram from that... that's when I came to my senses and decided to ask for help)

    One of the things I don't fully understand in your answer is the second camera set to Depth Only. The second camera has its own mask mesh blocking everything not in its viewable area, right? What is it rendering in the viewable area if it's set to depth only?

    Possibly related: I found a project that implements this splitscreen for two players, with incomplete code for up to 8, but it only works in orthographic. In this one, there's just one camera for all players and it's moved to each player's position and rendered manually. I was planning on having a separate camera for each player (to use Cinemachine), but the second camera having different settings (or not existing) is confusing me.

    Again, thanks for the help, and congrats on breaking 10,000 posts!
     
  4. Appleguysnake

    Appleguysnake

    Joined:
    Jan 9, 2015
    Posts:
    19
    I instantly realized that I had confused what clear flags does, but the forum isn't letting me edit my post. :(
    Please ignore the bit about the Depth Only camera and replace the last paragraph with this one:

    I found a project that implements this splitscreen for two players, with incomplete code for up to 8, but it only works in orthographic. In this one, there's just one camera for all players and it's moved to each player's position and rendered manually. I was planning on having a separate camera for each player (to use Cinemachine), but this implementation is confusing me and I'm not sure if it's necessary or just the creator's preference. Their shader creates the mask by measuring the distance to every camera for every pixel and that struck me as potentially slow, but I realize that I'm pre-optimizing and perhaps that can be addressed by the mixed approach you mentioned.
     
  5. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    By soft masks I mean where the split screen overlap and blend between each other. Something like this (bad) mockup.
    upload_2021-4-1_11-22-17.png
    If that kind of fade from one "screen" to another is something you want, then it's render textures or nothing.

    If by "soft mask" you mean something more like this:
    upload_2021-4-1_11-24-15.png
    There's no fade between the two "screens", only a soft dark edge between, then this doesn't require render textures. This is just a soft dark line drawn on top afterwards.

    Yep. It's funny because you have that completely backwards. Stencils don't actually apply to geometry at all and only affect the fragment.

    The stencil buffer is a 2D screen space thing. Basically it's another render buffer next to depth that holds an 8 bit value per pixel. You can set the per pixel bit values by rendering geometry with shaders that writes to it, and compare against the bit values to mask fragment shader rendering at those pixel positions. The shape that's written to the stencil is either the shape of the geometry being rendered, or you can use an alpha test shader and only those pixels still visible after alpha testing will render to the stencil.

    You can't even use
    SetPixel()
    on a
    RenderTexture
    ! The easiest way to use render textures in Unity is to assign a render texture to a camera as its target, and then render the camera normally. The render texture now holds the output of that camera instead of rendering to the frame buffer (ie: effectively directly to your screen, or at least some internal render texture Unity created). You then need to
    Blit()
    the render texture to the frame buffer, or put it on some geometry you render in another camera, etc.


    The easiest solution I can think of for what you want to do with a fuzzy voronoi is to render all 6 of your camera views to individual render textures. Then have a image effect shader that calculates the soft voronoi from the n points you pass to it and render the appropriate render texture in each cell. Maybe one camera at a time with multiple passes, or all 6 possible cameras in one pass.
     
    mandisaw likes this.
  6. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    That's using the stencil method. It's using a voronoi shader to set the stencil masks for the up to 8 cells to a render texture. Then rendering each player's camera to that same render texture using a modified standard shader that gets masked by the stencil. Then it draws lines & UI on top of that. Could be done with a voronoi shader that uses something like the depth write I posted above and without render textures and not require a custom shader used on everything in the scene.
     
  7. Appleguysnake

    Appleguysnake

    Joined:
    Jan 9, 2015
    Posts:
    19
    Haha of course I have it exactly backwards! Now maybe you can see why I wanted a general overview to clear up my understanding of the fundamentals.

    I think the stencil confusion is because every example I've found for stencil buffers was some variation on how to cut a hole in a mesh with another mesh. The fact that it's screen-space should have been a clue!

    Anyway, the first image you posted is what I had in mind for fuzzy, so it sounds like rendertextures are the textures for me, especially since I have that example to start from. If I can get that working then I can start worrying about how performant it is and if there's a better way.

    Prototyping sure is tough when you've misunderstood the basic principles of the tech you're using!