Retro shader bugs

I'm having a couple of problems with a shader and woulld appreciate some help. The first problem is related to converting vertex positions in the shadow caster. I currently perform some vertex snapping (taken from keijiro's retro3d shader) but the shadows don't match. I think I need to convert back into a different space in shadow caster but don't know how.

Code (CSharp):

1.             float3 wp = UnityObjectToViewPos (v.vertex);
2.             wp = floor(wp * _Distortion) / _Distortion;
3.             //
4.             float4 sp = mul(UNITY_MATRIX_P, fixed4(wp.xyz,1));
5.             o.pos = sp;
6.             //
7.             float2 uv = TRANSFORM_TEX(v.uv, _MainTex);
8.             o.uv = float3(uv * sp.w, sp.w);
9.             o.screenPos = ComputeScreenPos(o.pos);

The second problem seems to be related to the camera, the shadows don't show in the game window while not running and pop in and out of existence when running.

unity package: https://mega.nz/#!LYUnBQrZ!YgEqMz58sXPTLdc-Yv6nMircpz26UA8d9Gh-o5BUKMU
InputManger: https://mega.nz/#!nIEHGKCB!9zN5ClOFXsw_ZlSa4EvwNCNIX0Hbz9lgLoYsep8sTiA
gif + pic: https://imgur.com/a/44sZd9u

 

Keijiro’s shaders are quantizing the positions of objects in camera’s view space. The problem is in the shadow map generation the camera’s view space isn’t something those shaders are given, they have their own “view space” that’s relative to the light, so things won’t line up. You’d have to set the world to view space matrix of the camera as a custom global shader property and use that instead.
https://docs.unity3d.com/ScriptReference/Camera-worldToCameraMatrix.html
https://docs.unity3d.com/ScriptReference/Camera-cameraToWorldMatrix.html

You’ll the also need to transform back to world space after you’ve done the floor() on the position, the apply the UNITY_MATRIX_VP to the world position.

Alternatively you can just do the floor in world space. Technically neither Keijiro’s version or flooring the position in world space accurately recreate what happened in old console rendering, but they’re close enough to kind of get the feeling of it.

 

I don't completely understand how to implement what you've explained. For example how would I set the new view matrix for the calculations after creating a global variable, what do is set to it? By transforming back to world space would that be wp = unity_ObjectToWorld(wp) and then v.vertex = mul(unity_matrix_vp, wp)?

I've attempted to floor in world space instead of ViewPos but that didn't work like I expected, I'm not sure what sp does apart from it being somehow related to the position on the screen due to the result of mul().

What I've done so far is a far cry from being anything close to older consoles but I'm considering simulating model subdivision with LOD, pop in with layercull distances and importing models with detached faces as separate meshes then moving vertices on meshes that arent culled to the bound of the cameras frustrum.

 

On second thought, I'm guessing you have some other issues in your shader causing the flickering. You also don't have a custom shadow caster pass which is another issue. The flickering might be because you don't have a light mode tag on your base pass. My original suggestion may not really be fully relevant, but I honestly am not really sure what you're trying to go for with your look.

Can you post your entire shader rather than just a snippet?

To get from a mesh asset to being drawn on screen, each individual vertex needs to be transformed in a number of ways, through several different spaces.

A single vertex position is initially stored in object space, aka local space. This is a 3D position relative to the mesh's pivot position and orientation. This is the position that is stored in the v.vertex value that comes into the vertex shader.

This is then transformed into world space using an object to world matrix, sometimes called the "model" matrix. In Unity shaders this is the unity_ObjectToWorld matrix, or UNITY_MATRIX_M matrix. M for Model.

The world space position is then transformed into view space, also called eye or camera space. This is the UNITY_MATRIX_V matrix. V for View. Technically it's only "camera relative" when rendering from the point of view of a camera, like what you see in the scene or game views. Things like shadow maps are rendered from the view of the light rather than from a "camera", so in those cases the UNITY_MATRIX_V is that light's position and orientation.

From view space the position is transformed into clip space, also called projection space and more accurately called homogeneous clip space. This is what the UNITY_MATRIX_P is for, and what a vertex shader should be outputting. This one is the hardest to understand as it's no longer a "regular" 3D space position where only the first three components matter. It's now a 4D position of sorts. Really it's a 3D position specially encoded for interpolation in screen space so the original positions can be reconstructed. I'll leave that to you to research more if you want, but the easiest way to think about it is, for a perspective camera, the range of x and y are from -w to +w for stuff that's within view, where w is the view space depth. I don't mention z because the range of z depends on the graphics API, with most being from w to 0.0, but OpenGL is -w to +w like x and y. I won't go into why now, but know OpenGL was first, and everyone else realized this was a mistake to do.

After that things get transformed by the GPU into normalized device space coordinates, or "NDC", which is basiscally just the xyz / w, and from there into viewport space, aka window, or screen space, which is the screen pixel positions and a 0.0 to 1.0 depth range. But, again, that's all done by the GPU on the value output by the vertex shader before it runs the fragment shader and you can ignore.

Here's a graphic for which I don't know the original author of to give proper attribution to.


Unity also has a couple of extra macros for matrices like UNITY_MATRIX_MV, UNITY_MATRIX_VP, and UNITY_MATRIX_MVP, which are the model and view matrix, view and projection matrix, and model, view, and projection matrix combined into a single matrix.

So what Keijiro's shader is doing is converting the position into view space, snapping those positions to a view space grid of sorts, then multiplying by the projection matrix to put it into the coordinate system the vertex shader should be outputting.


Back to my original suggestion, I'm guessing at some point you're going to need a custom shadow caster pass so the camera depth texture, screen space shadows from the main directional light, and all other shadows, actually line up with the geometry rather than shadowing as if they're normal geometry.

UNITY_MATRIX_V, and the c# camera.worldToCameraMatrix are the same matrix when rendering to the camera view. Both convert from a world space position to the same camera relative view space position. However when rendering the shadow maps, they're not the same since the UNITY_MATRIX_V is the view from the light's orientation rather than the camera. Hence why I recommended passing that matrix as a shader global, though I forgot to link to the appropriate script function to do so:
https://docs.unity3d.com/ScriptReference/Shader.SetGlobalMatrix.html

However, I think Unity may already have access to those matrices in the shadow maps in the form of the unity_WorldToCamera and unity_CameraToWorld matrices. So try this:

Code (csharp):

1. float4 worldPos = mul(unity_ObjectToWorld, float4(v.vertex.xyz, 1));
2. float4 cameraPos = mul(unity_WorldToCamera, worldPos);
3. cameraPos.xyz = floor(cameraPos.xyz * _Distortion) / _Distortion;
4. worldPos = mul(unity_CameraToWorld, cameraPos);
5. v.vertex = mul(unity_WorldToObject, worldPos);

All together it would look something like this:

Code (csharp):

1. Shader "Custom/Keijiro Style Retro With Shadows"
2. {
3.     Properties
4.     {
5.         _MainTex ("Texture", 2D) = "white" {}
6.         _Distortion ("Distortion", Float) = 1
7.     }
8.     SubShader
9.     {
10.         Tags { "RenderType"="Opaque" }
11.         LOD 100
12.  
13.         Pass
14.         {
15.             Tags { "LightMode" = "ForwardBase" }
16.  
17.             CGPROGRAM
18.             #pragma vertex vert
19.             #pragma fragment frag
20.             #pragma multi_compile_fwdbase
21.             #pragma multi_compile_fog
22.  
23.             #include "UnityCG.cginc"
24.             #include "AutoLight.cginc"
25.  
26.             struct v2f
27.             {
28.                 float4 pos : SV_POSITION;
29.                 float2 uv : TEXCOORD0;
30.                 float3 diff : TEXCOORD1;
31.                 float3 ambient : TEXCOORD2;
32.                 SHADOW_COORDS(3)
33.                 UNITY_FOG_COORDS(4)
34.             };
35.  
36.             sampler2D _MainTex;
37.             float4 _MainTex_ST;
38.             float _Distortion;
39.             half3 _LightColor0;
40.  
41.             v2f vert (appdata_full v)
42.             {
43.                 v2f o;
44.  
45.                 float4 worldPos = mul(unity_ObjectToWorld, float4(v.vertex.xyz, 1));
46.                 float4 cameraPos = mul(unity_WorldToCamera, worldPos);
47.                 cameraPos.xyz = round(cameraPos.xyz * _Distortion) / _Distortion;
48.                 worldPos = mul(unity_CameraToWorld, cameraPos);
49.                 v.vertex = mul(unity_WorldToObject, worldPos);
50.  
51.                 o.pos = UnityObjectToClipPos(v.vertex);
52.                 o.uv = TRANSFORM_TEX(v.texcoord, _MainTex);
53.                 float3 worldNormal = UnityObjectToWorldNormal(v.normal);
54.                 half diff = saturate(dot(worldNormal, _WorldSpaceLightPos0.xyz));
55.                 o.diff = diff * _LightColor0.rgb;
56.                 o.ambient = ShadeSH9(float4(worldNormal, 1));
57.  
58.                 UNITY_TRANSFER_FOG(o,o.vertex);
59.                 TRANSFER_SHADOW(o)
60.                 return o;
61.             }
62.  
63.             fixed4 frag (v2f i) : SV_Target
64.             {
65.                 fixed4 col = tex2D(_MainTex, i.uv);
66.                 half shadow = SHADOW_ATTENUATION(i);
67.                 col.rgb *= i.diff * shadow + i.ambient;
68.                 UNITY_APPLY_FOG(i.fogCoord, col);
69.                 return col;
70.             }
71.             ENDCG
72.         }
73.  
74.         // Pass to render object as a shadow caster
75.         Pass {
76.             Name "ShadowCaster"
77.             Tags { "LightMode" = "ShadowCaster" }
78.  
79.             CGPROGRAM
80.             #pragma vertex vert
81.             #pragma fragment frag
82.             #pragma target 2.0
83.             #pragma multi_compile_shadowcaster
84.             #pragma multi_compile_instancing // allow instanced shadow pass for most of the shaders
85.             #include "UnityCG.cginc"
86.  
87.             struct v2f {
88.                 V2F_SHADOW_CASTER;
89.                 UNITY_VERTEX_OUTPUT_STEREO
90.             };
91.  
92.             float _Distortion;
93.  
94.             v2f vert( appdata_base v )
95.             {
96.                 v2f o;
97.                 UNITY_SETUP_INSTANCE_ID(v);
98.                 UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
99.  
100.                 float4 worldPos = mul(unity_ObjectToWorld, float4(v.vertex.xyz, 1));
101.                 float4 cameraPos = mul(unity_WorldToCamera, worldPos);
102.                 cameraPos.xyz = round(cameraPos.xyz * _Distortion) / _Distortion;
103.                 worldPos = mul(unity_CameraToWorld, cameraPos);
104.                 v.vertex = mul(unity_WorldToObject, worldPos);
105.  
106.                 TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
107.                 return o;
108.             }
109.  
110.             float4 frag( v2f i ) : SV_Target
111.             {
112.                 SHADOW_CASTER_FRAGMENT(i)
113.             }
114.             ENDCG
115.  
116.         }
117.     }
118. }

 

Well you described it well in your first response, I'm just trying to recreate the feeling of a playstation title. It boils down to imitating anything that devs at the time tried to hide and adding some features, like lighting to "improve" the visuals. For example I've added cubemap reflections because I want metallic objects to resemble the magazine covers where game characters where renedered at a higher quality. Not because that's realistic to the games released at the time but because those effects are part of that era. This is what I currently have, its mostly pasted together:

https://pastebin.com/v9fkruHJ

All passes are in the lightmode=vertex, I struggled getting vertex lights to work with forwardbase then instead of ditching shadows I used mobile and legacy shaders to create what I currently have. At first the directional lighting was fine in lightmode=vertex but after some time I broke the lighting, somehow. I have no idea how, why it broke or how to fix it. The project can be accessed through the megalink in my post. Thanks for the explanation its staterted to clear some things up for me. I'll try to add it now.

 

If you want to mimic the original playstation closer, you want to be quantizing the positions in screen space rather than in view space. Something like what this project does.
https://github.com/dsoft20/psx_retroshader/blob/master/Assets/Shaders/psx-unlit.shader

 

When I try to multiply cameraPos by the projection matrix the pass diseapears from both views. From what I understand both your cameraPos and sp (in keijiro's script) are in view space. Why does that happen?

I'll copy that instead, it also seems like its written in a way that's easier for me to understand. Thanks for the help, would doing shadows in lightmode=vertex cause the shadow popping problem?

 

Because unity_WorldToCamera and UNITY_MATRIX_V are both transforming from world to view space, but they’re not the “same” view space, even when rendering from the camera.

Unity’s rendering view space, represented by UNITY_MATRIX_V, and in c# as camera.worldToCameraMatrix, are in a form where -Z is forward. I believe unity_WorldToCamera matches the in-editor +Z forward, like the camera.transform.worldToLocalMatrix has. So while they’re both view space, they’re in different coordinate systems. That’s why I transformed back to the local space first. Really you could transform back to world space and apply UNITY_MATRIX_VP to optimize it a bit, but since UNITY_MATRIX_V and unity_WorldToCamera don’t match, it’s not necessarily safe to only use UNITY_MATRIX_V in the forward pass.

 

Okay, that makes sense I think. So the shadows where moved behind the camera?

I've added logic from the shader you linked and it works fine, except for the shadow pop.

https://imgur.com/a/7RB0qIP

How do I go from UNITY_MATRIX_MVP or UnityObjectToClipPos() (clip/projection space?) back to object space in the shadow caster? I can't find anything online about this but did find stuff about the projection matrix. Are vertices on models that aren't culled but also not drawn accessible by shader? I'd also like to find a way to clamp those vertices inside the projection space or should I do that in an image in screen space?

 

More than the shadows, the camera depth texture too, which is what the main directional light’s shadows are cast on.

I still think it’s the LightMode that’s the problem, since the expectation of that light mode is to not have any shadows. Does the problem happen if you use the example shader I posted as is?

 

Your're correct, #pragma multi_compile_fwdbase was causing the shadows to intermittently show up. I think I read in one of your previous posts that vertex lights don't work in forwardbase and I've tried to add them in with ShadeVertexLightsFull and that seems to be the case. Point lights also don't iluminate when turned to not important. Unless theres a way to add them in the shader I'll have to do it in script somehow. I've decided to keep shadows regardless. Any tips or resources on going from projection to object before shadow caster?