Retro game with prerender image as background

I used to play games like dino crisis and resident evil back in the days when PC hardwares were not as powerful. these games use prerender still images as background, with characters and objects rendering in realtime. I wonder how it can be created using unity.

There are some difficulties I can think of:
1. How can I tell the game to draw objects that are partially blocked by foreground? I know when rendering in 3D software I can create depth maps, which use black and white pixels to tell the distance from camera, but can unity use these data for rendering? Like in photoshop the layers on top covers the ones on bottom, the camera should be able to tell whether a prerendered object is in front of a realtime rendered object using depth map.

2. How to set objects that only receive dynamic shadow and lighting, cast reflections, but do not rendered in realtime? It's to be used on geometries like floors and walls. Any character/object should be able to cast shadow on it, as well as gunflare reflection.
I think I can import models and camera positions from other 3D software to unity, and align unity camera with the rendering perspective. If I turn off the mesh rendering of imported objects, it will help with game performance and graphics.
I can deactive the mesh renderer for floor object, but then the shadow of objects do not cast on it. If I active the mesh renderer, I can set it to not cast shadow, only receive shadow, but the floor object will be blocking the rendered image in the background.

Maybe these problems have been solve previously in unity, but I'm new to game dev. I hope someone can show me how to do.

 

If you make your backgrounds with depth buffer, all this get pretty easy.

 

yes,I have depth maps,but how to use them?

 

The "easy" solution is to still use a basic mesh, but have the prerendered version of the scene projected onto that mesh and have it use an unlit shader that samples from the shadow map still. The other option would be to use

SV_Depth

to output the depth you have from the prerendered texture from the fragment shader.

The problem is to get from "new to game dev" to "inject prerendered scene into Unity in a way that interacts with Unity's shadows and depth" is seriously jumping into the deep end. It's one of those things where if you don't already know what

SV_Depth

is, you probably don't have enough knowledge to do what you want any time soon.

Simple stuff like "a depth map" is more complicated than it seems.
What kind of depth do you have? What range? What image format / bit depth and how do you plan on over coming Unity's lack of proper 16 or 32 bit texture support (exr, packed 32 bit, custom importer, live with quantized depth)?
How are you going to convert the depth you have into the depth Unity is rendering?

If you don't have answers for all of those questions, then you're going to need to start doing a lot of reading.

 

Thanks for reply, good to know what to expect down this path. I just thought there would be a template or something for such type of game.

 

Hi, I made some progress
1. I have found someone who has done similar project and published it online. It sort of work, but the distance of objects are incorrect, maybe the equation using to convert depth value to clip space is flawed due to the depth map value being nonlinear. I'm not sure how to make it right.
My project files:
https://github.com/dzft3w/PrerenderBackground_S

ref: https://mehm.net/blog/?p=1218

2. I have found shader file from your post to make invisible objects receive shadow. I've test reflections and SSAO etc, works well.

 

I'm not a blender user, so I have no idea what kind of depth data you're getting out of that. But I can tell you the depth values aren't anything like what the shader is expecting. The shader is expecting a linear depth value, re-scaled so the near and far of the depth texture have the same near and far as the in game camera. If Unity's camera and the Blender rendered depth don't match, it won't work.

But that depth texture is not linear. I have no idea what it is honestly since I don't know the depth range you used to render it with was, or if you re-scaled it, etc.

Also you should never, ever use .jpg files for game source content. Use something uncompressed, or at least losslessly compressed like .png if you really insist on using a compressed format. Plus if you're going to stick with an 8 bit depth texture, you probably also want to disable compression on the texture importer. DXT1 is going to do a pretty poor job of compressing a depth texture too.

 

Hi, thank you for your advices.
I have solved the distance matching problem by using Linear01Depth on image value.
Now I have a shadow blending problem.


a). Do you know a way to make the realtime shadow blend into rendered lighting map, like in photoshop there is a darkend mode.
"The Darken Blending Mode looks at the luminance values in each of the RGB channels and selects either the base color or blend color depending on which is darker."
It reduce shadow darkness by comparing it to the lighting map, so it won't simply multiply the shadow in the scene.

b). If possible, the difference made to the realtime shadow can be cast onto the player. Like when player under the shade of trees, he will get nice soft shadow of leafs on him.

 

If that's the case, then the depth already matched the output depth, in which case you shouldn't need to do any conversion at all and just output the sampled texture's depth value straight (or specifically whatever value you were passing into

Linear01Depth

).

Photoshop's Darken blend mode is a

min

comparison. You can look up the hlsl equivalents for any Photoshop blend mode if you want. That'll do an okay job if your end goal is scene is like what you have above with no detail or color in the floor. Old games that used this technique generally did a multiply or alpha blend, which would have the same issue as you have above, but it doesn't seem like you're trying to mimic that effect exactly.

The best option is to render out two image of your scene, one with just the main light's diffuse direct output, and a second that is the diffuse indirect of that same light and all additional lighting. Then in your shader apply the shadows only to the diffuse direct, and add that with the other texture. That's basically what real time lighting, or even some of Unity's more advanced mixed lighting setups do.

 

SV_DEPTH? I sometimes forget about that thing.
Time for me to read up on it again.

 

Hi, how did you solved the distance matching ?, im using unity 2021, and the same ref as you (https://mehm.net/blog/?p=1218), but is too old, that example only working on unity 4.5