Decoupling visibility POV from render POV

I'm currently working on a project where I need to render the view rendered by a camera, from a different point-of-view.

I previously wrote a program (in raw OpenGL) that would take information from an optical and depth camera both mounted on a robot (using an Azure Kinect), and then display it from an alternative point of view. Imagine having a webcam that is sitting on top of your monitor, but it rotates your head (in 3d space) so that even though you're facing the monitor, it looks like you're facing the camera. Or if a robot has an impression of the depths of what it's seeing, and you're trying to debug it by moving around in the space it thinks it's seeing.

Here, I'm trying to accomplish something like that, but simulated. There's a robot-camera object that represents the color+depth camera, and then a number of objects that represent objects in the world; some of their polys are visible to the camera, and some aren't. I want the user to be able to move through the world (with regular WASD+mouse) and see a reconstructed view of what the robot-camera object can directly perceive. The robot-camera isn't trying to reconstruct what actually exists here; all that the player can view is what the robot-camera can directly perceive.

Essentially, if you're looking just at a cube in worldspace, then you should only see the faces that are facing the camera (easy to compute from normals). I could do that in a geometry shader, or even in the vertex shader by denormalizing the triangle, and might still do that as an optimization. But if there's a smaller cube behind the big cube (from the robot's POV), then that shouldn't be visible at all, and a pure normal-based computation can't do that.

The obvious way to do this would be to use what amounts to a shadow mapping: compute a depth map from the POV of the robot-camera, and store that in a 1-channel texture. Then, when in the user-camera's fragment shader, reproject the position to the robot-camera, and compare the Z value. (Presumably, I'd also have the user-camera's frag shader pass both the user-camera's MVP transformation and the robot-camera's MVP transformation; can I pass two SV_POSITION values?) If there's a mismatch in the Z value, then discard the fragment. Otherwise, apply the usual Lit Shader (or, if needed, I'm fine with simpler models like Phong for this application).

However, I have no clue how to do this in Unity. I'm open to using the built-in shader pipeline or URP or whatever, based on what's needed. To do this with the shadow mapping technique, I would need to override every material's shader to get the "shadow map" in the robot-camera's pass, and then pass the resulting render texture to the object's normal render shader. I might be able to do this by putting a light at the same position of the camera with a cookie that's the shader, but... well, now we're well beyond what I can do.

It's a low-poly system, so I could use shadow volumes, but that's even more foreign for me to do in Unity.

Here's a different idea I might try: render the robot-camera to a 3D render texture (which I haven't experimented with yet), and then use a cube (or truncated pyramid from the robot's POV based on its frustrum) object with a raymarching renderer to render that to the screen. I'd hope that rendering to a 3D render texture would only fill in the voxels that correspond to places that are visible to the camera. Of course, that uses a lot of memory, so I'm not sure if that's a good plan. I also haven't thought too much about the best way to interpolate the texture.

Any thoughts about the best way to do this in Unity?

 

The shadow mapping approach is likely the "easiest" option. A "dumb" approach would be to put an actual Unity camera where the robot camera is, have it render a depth texture (using

camera.depthTextureMode = DepthTextureMode.Depth

) and "copying" that texture to a global texture using a command buffer. Really just assigning the existing texture to be a global texture should be good enough.

Code (csharp):

1. // just need to do this once
2. var cmd = new CommandBuffer();
3. cmd.name = "Set Depth Texture As Global";
4. cmd.SetGlobalTexture("_RobotDepth", BuiltinRenderTextureType.Depth);
5. robotCam.AddCommandBuffer(CameraEvent.AfterDepthTexture, cmd);

Then you need to also get the robot cam's view projection matrix and assign that as a

Shader.SetGlobalMatrix()

each time the robot cam moves.

In the shader you'd then be using the robot cam's view projection matrix to transform stuff from world space to that camera's projection space, divide the xyz by w, rescale the xy range from -1.0 .. 1.0 to 0.0 .. 1.0, and use that as the UV to sample the depth texture with. Then compare the z with the depth with a little bias and you're done.

So the funny thing with shadow volumes is to use them you have to render all the geometry first. Then you render the shadow volume mesh to intersect with the already rendered geometry, so you can't use shadow volumes to hide geometry. So this is a dead end.

Technically totally plausible, and yes it'd take a ton of memory for any reasonably sharp facsimile of the geometry you're trying to match. And it'll be super slow as you then need to raymarch through that 3D texture. A lower resolution SDF might work a bit better, but you'll still need to do the raymarching and you'll get blobby "shadows" unless you keep the 3D texture high resolution.

 

Thank you so much for your detailed and clear response! This is exactly the information I needed. Thank you for taking the time to write this up!