Bump offset/parallax mapping window shader

Hi, I'm trying to write a window shader that offsets an interior texture based on viewing angle.
Here is an example of how to do this in unreal:

The offset is done by a node called bump offset which is given the viewing direction and the texture coordinates. This is what is called parallax mapping, but with a static height instead of a hightmap.

How would I go about writing this function in cg?
I've tried this to translate an open gl function I found here: http://www.learnopengl.com/#!Advanced-Lighting/Parallax-Mapping

like this:

Code (CSharp):

1.  float2 ParallaxMapping(float2 texCoords, float3 viewDir)
2.  {
3.  float height = -1;
4.  float2 p = viewDir.xy * (height * 0.1);
5.  return texCoords - p;
6.  }

and then in the vertex program set the view dir to:

Code (CSharp):

1.  
2.  mul(modelMatrix, input.vertex).xyz - _WorldSpaceCameraPos;
3.  

and then in the fragment program do this to the uv:

Code (CSharp):

1.  
2.  uv = ParallaxMapping(input.tex.xy, input.viewDir);
3.  

and it kind of works, looks great when looking up and down but moves inverted to how I want it when looking from left to right, and it moves unpredictably when the window is rotated. I think I need to take the normal of the windowpane into account somehow.

If you have done this before I'd love to get some tips to point me in the right direction!

 

This is an interesting technique! It kept me thinking the whole Sunday, I hate when that happens

Now about 6 hours later, I put together a shader with three different approaches which I came up with. You can find a download link to the project at the bottom of this post.

Here is how the result looks:

Spoiler: Shader Code

Code (CSharp):

1. //
2. // download from http://www.console-dev.de/bin/Unity_Shader_Fake_Interior.zip
3. //
4. // this shader is part of an answer to the following forum post:
5. // http://forum.unity3d.com/threads/bump-offset-parallax-mapping-window-shader.407091/
6. //
7. // I believe this approach is rather unconventional.
8. // My idea was that I could use the clip space coordinates to offset the mesh texture coordinates.
9. // clip space is in range -1 to +1
10. // if the vertex is at the left screen edge, x would be -1 in clip space
11. // if the vertex is at the right screen edge, x would be +1 in clip space
12. // the sample principle applies for the y coordinate.
13. //
14. // if the interior quad is also mapped in this fashion (top-left = 0,0 and bottom-right=1,1)
15. // it should be possible to simply add the clip-space coordinates to the texture coordinates
16. // and we have the parallax effect already already working.
17. //
18. // however, if we look to the left we actually want to reveal more of the left part of the texture.
19. // just adding the clip-space coordinates would simple move the texture to the left, revealing more of
20. // the right texture part. so I simply changed the sign (subtract rather than add) to scroll it into
21. // the opposite direction.
22. //
23. // since we just modify texture coordinates, we can also do just everything in the vertex shader \o/
24. //
25. // in this file you can find 3 different tests, showing my hackery.
26. //
27.  
28. Shader "Custom/Interior"
29. {
30. Properties {
31.     _Color ("Main Color", Color) = (1,1,1,1)
32.     _MainTex ("Base (RGB)", 2D) = "white" {}
33.  
34.     // The maximum parallax value should not be greater than
35.     // the uv border in the mesh. Usually you would uv map a quad from:
36.     // top-left=0,0 to bottom-right=1,1
37.     // however, the shader moves the uv's by the specified _Parallax value.
38.     // in order to never go outside the 0,0 .. 1,1 range, you "add a border to the uvs in the mesh".
39.     // To properly support a maximum parallax value of 0.25, you would uv map the quad as:
40.     // top-left=0.25,0.25 to bottom-right=0.75,0.75
41.     // if the shader adds the _Parallax value to the uvs, we have the full uv range again.
42.     _Parallax ("Parallax", Range (0, 1)) = 0.25
43. }
44.  
45. CGINCLUDE
46. sampler2D _MainTex;
47. float4 _MainTex_ST;
48. fixed4 _Color;
49. float _Parallax;
50.  
51. struct Input
52. {
53.     float2 st_MainTex;
54. };
55.  
56. // test 2 seems to work well and I understand why it is working
57. #define TEST_2
58.  
59. #if defined(TEST_1)
60. inline float2 InteriorUV(float4 vertex, float3 normal, float2 texcoord)
61. {
62.     float4 clipSpace = mul(UNITY_MATRIX_MVP, vertex);
63.     clipSpace.y = -clipSpace.y;
64.     clipSpace = normalize(clipSpace);
65.  
66.     // applys texture scaling or displacement using the screen center as origin.
67.     // this results in undesired scretching at the screen borders.
68.     // you can still compensate by adjusting the texture tiling and offset in the material.
69.     // to get a _Parallax value of 0.25 to apply the scaling from the quad center, you would need
70.     // to modify the material settings as followed:
71.     //   Tiling X = 1.5    Y = 1.5
72.     //   Offset X = -0.25  Y = -0.25
73.     float2 offset = clipSpace.xy * _Parallax;
74.     return texcoord - offset;
75. }
76. #endif
77.  
78. #if defined(TEST_2)
79. inline float2 InteriorUV(float4 vertex, float3 normal, float2 texcoord)
80. {
81.     float4 clipSpace = mul(UNITY_MATRIX_MVP, vertex);
82.     clipSpace.y = -clipSpace.y;
83.     clipSpace = normalize(clipSpace);
84.  
85.     // This is the same as TEST_1, but we modify the texcoord so that the
86.     // displacement occurs not using the screen center anymore, but the
87.     // quads center. this seems to look nice at all angles and positions.
88.     float2 offset = clipSpace.xy * _Parallax;
89.     return (texcoord + texcoord * _Parallax * 2) - _Parallax - offset;
90. }
91. #endif
92.  
93. #if defined(TEST_3)
94. // another approach that seems to work, figured out through trial&error.
95. // I don't understand entirely why it works though :)
96. // it seems to work really nice though.
97. inline float2 InteriorUV(float4 vertex, float3 normal, float2 texcoord)
98. {
99.     float3 distance = normalize(mul((float3x3)UNITY_MATRIX_MV, reflect(ObjSpaceViewDir(vertex), normal)));
100.  
101.     float2 offset = distance.xy * _Parallax;
102.     return texcoord - offset;
103. }
104. #endif
105.  
106.  
107. void vert(inout appdata_full v, out Input o)
108. {
109.     UNITY_INITIALIZE_OUTPUT(Input, o);
110.  
111.     float2 texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
112.     o.st_MainTex = InteriorUV(v.vertex, v.normal, texcoord);
113. }
114.  
115. void surf (Input IN, inout SurfaceOutput o)
116. {
117.     fixed4 c = tex2D(_MainTex, IN.st_MainTex) * _Color;
118.     o.Albedo = c.rgb;
119.     o.Alpha = c.a;
120. }
121. ENDCG
122.  
123.  
124. SubShader
125. {
126.     Tags { "RenderType"="Opaque" }
127.     LOD 500
128.  
129.     CGPROGRAM
130.     #pragma surface surf Lambert vertex:vert
131.     #pragma target 3.0
132.     ENDCG
133. }
134.  
135.  
136.  
137. FallBack "Legacy Shaders/Diffuse"
138. }
139.  

I believe my approach is rather unconventional, but I also know nothing about interior mapping except for the video you posted.

My idea was that I could use the clip space coordinates to offset the mesh texture coordinates.
Clip space is in range -1 to +1. If the vertex is at the left screen edge, x would be -1 in clip space. If the vertex is at the right screen edge, x would be +1 in clip space. The same principle applies for the y coordinate.

If the interior quad is also mapped in this fashion (top-left = 0,0 and bottom-right=1,1 (but you need a border, see shader comments)) it should be possible to simply add the clip-space coordinates to the texture coordinates and we have the parallax effect already working.

However, if we look to the left we actually want to reveal more of the left part of the texture. Just adding the clip-space coordinates would move the texture to the left, moving in more of the right texture part. So I simply changed the sign (subtract rather than add) to scroll it into the opposite direction.

Since we just modify texture coordinates, we can also do everything in the vertex shader \o/

You can find three different approaches to this problem in in "Assets/Interior.shader", basically 3 different tests showing my hackery.

Here is the Unity 5.3 compatible project with all the assets shown in the video.
http://www.console-dev.de/bin/Unity_Shader_Fake_Interior.zip

 

It's a nice idea, it's not interior mapping as seen in bioshock infinite and friends, but it does seem much lighter on the gpu and not computationally expensive. I can see this being sort of ideal for mid and far lods.

Check some of the tricks out on this site, here's one of them: https://simonschreibt.de/gat/windows-ac-row-ininite/

 

Hello again!
Yours is an intresting solution, but it doesn't really take the position of the camera into account which makes it break when you strafe.

I recently had time to start thinking about this again (got my degree in 3d art, officially unemployed!), and here is what I've come up with:



If you take the normal direction of the verticies of the window pane and their position and the camera position all in world space shouldn't you be able to convert the camera position into a kind of vertex space that is relative to that vertex's position and rotation? Then you could simply take the x and y values of this new point and use them as offset values of the uv after setting the min and max values of the offset and a scaling factor.

The problem is that my vector math is both rusty, and basic. I figured that the math needed would be close to when you calculate view space coordinates for verticies, so I tried to converting some code I found online:



This is the method I use to create the vertex space matrix

Code (CSharp):

1. float4x4 viewFrom(float3 nor,float3 up, float3 vpos){
2.  
3.             //normal of the vertex, the way the matrix should "look"
4.             float3 n = normalize(nor);
5.             //up direction of the camera, I assume (0,1,0) when I send it
6.             float3 v = normalize(up);
7.  
8.             //third angle, looking to the side, cross product of the two
9.             float3 u = cross(nor,up);
10.  
11.             float4x4 m = float4x4(u.x, u.y, u.z, 0.0,
12.                                   v.x, v.y, v.z, 0.0,
13.                                   n.x, n.y, n.z, 0.0,
14.                                   0.0, 0.0, 0.0, 1.0);
15.  
16.             float4x4 mpos = float4x4(1.0, 0.0, 0.0, vpos.x,
17.                                        0.0, 1.0, 0.0, vpos.y,
18.                                       0.0, 0.0, 1.0, -vpos.z,
19.                                        0.0, 0.0, 0.0, 1.0);
20.  
21.                m = mul(m,mpos);
22.            
23.             return m;
24.         }

and then in my vertex shader I do this:

Code (CSharp):

1. //get the matrix
2. float4x4 vertexMatrix = viewFrom(normalDirection.xyz,float3(0.0,1.0,0.0),vertexWorldPos.xyz);
3.  
4. //convert the position of the camera to this "vertex space"
5.              float3 relPos =    mul(vertexMatrix,_WorldSpaceCameraPos).xyz;

And finally I scale and clamp the value to a better scrolling offset that can be used to offset the uv:

Code (CSharp):

1. float2 offsetUV = ((relPos).xy*_OffsetScale);
2.              offsetUV.x = clamp(offsetUV.x,-_MaxOffset,_MaxOffset);
3.              offsetUV.y = clamp(offsetUV.y,-_MaxOffset,_MaxOffset);

This works much better than my first try, but it still doesn't work when rotated, and behaves differently when moved further away. Am I missing something obvious, or have I completely misjudged the problem?

 

What you want for this kind of effect is the tangent space view direction.

If you're using surface shaders I think you can just add viewDir to your input struct and you'll get the tangent space view direction in the surf function. Then you just need something like this:

float2 interiorUV = IN.uv_MainTex + (IN.viewDir.xy / IN.viewDir.z) * _OffsetScale;

 

Thank you! That worked out perfectly! I knew I was overcomplicating things!
Just one question, what is the point of the devision by the z-value? It seems to work without it but acts a little differently..

Oh and if somebody finds this and wonders how to do it in cg, here is how I got the tangent space view direction:

Code (CSharp):

1. TANGENT_SPACE_ROTATION;
2. float3 viewDir  = mul(rotation,  ObjSpaceViewDir(v.vertex));

You just need to make sure you've included the tangent in your vertex input.

 

If you use a normalized view direction vector, what you're likely starting with, it is going to be essentially moving the offset around in a circle / hemisphere. Dividing by the z component projects the vector onto a unit plane. The normalized vector is fine too use as it gets you most of the parallax look, and gets you a built in "max" offset, but the other way is more accurate. You may want to fade out the effect at extreme angles.

 

Any chance we can see the final code? I love the effect, but I'm not a shader coder... I doubt I can get it to work by myself...