Billboard Reflection

Hi, fellow Unity devs
I've been trying to implement the "billboard reflection" technique that you can see in the Unreal Samaritan Demo and now known as "Image-based reflection" in UDK.
It's basically a textured quad that represents an object and is only rendered in reflections. It is less accurate but cheaper than the MirrorReflection shader since, unlike the latter, it doesn't require a RenderTexture and is directly rendered from a texture like the built-in decal shader (it also means that you have to place a quad for everything that you want to reflect).
It is used in conjunction with the Box Projected Cubemap shader (I've written the shader on top of it).
Since I'm not too good with maths and I'm pretty much a newbie in shader language, I've been struggling with this but I've finally managed to implement the core of this feature. Here's the result so far:


And here's the code for the shader:

Code (csharp):

1.  
2. Shader "Custom/BPCEM with billboard reflection" {
3. Properties {
4.     _Color ("Main Color", Color) = (1,1,1,1)
5.     _ReflectColor ("Reflection Color", Color) = (1,1,1,0.5)
6.     _MainTex ("Base (RGB) RefStrength (A)", 2D) = "white" {}
7.     _Cube ("Reflection Cubemap", Cube) = "_Skybox" { TexGen CubeReflect }
8.     _BumpMap ("Normalmap", 2D) = "bump" {}
9.     _BoxPosition ("Bounding Box Position", Vector) = (0, 0, 0)
10.     _BoxSize ("Bounding Box Size", Vector) = (10, 10, 10)
11. }
12.  
13. SubShader {
14.     Tags { "RenderType"="Opaque" }
15.     LOD 300
16.    
17. CGPROGRAM
18. #pragma target 3.0
19. #pragma surface surf Lambert
20.  
21. sampler2D _MainTex;
22. sampler2D _BumpMap;
23. sampler2D _Billboard;
24. samplerCUBE _Cube;
25.  
26. fixed4 _Color;
27. fixed4 _ReflectColor;
28. float3 _BoxSize;
29. float3 _BoxPosition;
30. float3 _QuadLLPos;
31. fixed3 _QuadX;
32. fixed3 _QuadY;
33. float2 _QuadScale;
34. fixed _Culling;
35.  
36. struct Input {
37.     float2 uv_MainTex;
38.     float2 uv_BumpMap;
39.     fixed3 worldPos;
40.     float3 worldNormal;
41.     INTERNAL_DATA
42. };
43.  
44. void surf (Input IN, inout SurfaceOutput o) {
45.     // Base diffuse texture
46.     fixed4 tex = tex2D(_MainTex, IN.uv_MainTex);
47.     fixed4 c = tex * _Color;
48.     o.Albedo = c.rgb;
49.    
50.     fixed3 n = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));
51.    
52.     // Reflection-ray
53.     float3 viewDir = IN.worldPos - _WorldSpaceCameraPos;
54.     float3 worldNorm = IN.worldNormal;
55.     worldNorm.xy -= n;
56.     float3 reflectDir = reflect (viewDir, worldNorm);
57.     fixed3 nReflDirection = normalize(reflectDir);
58.    
59.     // Parallax correction
60.     float3 boxStart = _BoxPosition - _BoxSize / 2.0;
61.     float3 firstPlaneIntersect = (boxStart + _BoxSize - IN.worldPos) / nReflDirection;
62.     float3 secondPlaneIntersect = (boxStart - IN.worldPos) / nReflDirection;
63.     float3 furthestPlane = (nReflDirection > 0.0) ? firstPlaneIntersect : secondPlaneIntersect;
64.     float3 intersectDistance = min(min(furthestPlane.x, furthestPlane.y), furthestPlane.z);
65.     float3 intersectPosition = IN.worldPos + nReflDirection * intersectDistance;
66.     fixed4 reflcol = texCUBE(_Cube, intersectPosition - _BoxPosition);
67.    
68.     // Ray-Plane intersection
69.     fixed3 quadNormal = cross(_QuadX, _QuadY);
70.     float planeIntersectDistance = (dot(IN.worldPos - _QuadLLPos, quadNormal) / dot(nReflDirection, quadNormal));
71.     intersectPosition = IN.worldPos - nReflDirection * planeIntersectDistance;
72.     float3 localPlaneIntersectPosition = intersectPosition - _QuadLLPos;
73.     float2 billboardUV = float2(dot(_QuadX, localPlaneIntersectPosition) / _QuadScale.x, dot(_QuadY, localPlaneIntersectPosition) / _QuadScale.y);
74.     fixed4 billboardCol = tex2D(_Billboard, billboardUV);
75.     float reflectDot = dot(nReflDirection, quadNormal);
76.    
77.     fixed w = billboardCol.a;
78.     w = billboardUV.x > 1.0 ? 0.0 : w;
79.     w = billboardUV.y > 1.0 ? 0.0 : w;
80.     w = billboardUV.x < 0.0 ? 0.0 : w;
81.     w = billboardUV.y < 0.0 ? 0.0 : w;
82.     w = reflectDot <= 0.0  _Culling == 1.0 ? 0.0 : w;
83.    
84.     reflcol.rgb = lerp(reflcol.rgb, billboardCol.rgb, w);
85.    
86.     // Reflection display
87.     reflcol *= tex.a;
88.     o.Emission = reflcol.rgb * _ReflectColor.rgb;
89.     o.Alpha = reflcol.a * _ReflectColor.a;
90. }
91. ENDCG
92. }
93.  
94. FallBack "Reflective/VertexLit"
95. }
96.  

You need to provide the shader (with SetVector, SetFloat, etc.) the following parameters:

• "_QuadLLPos" the quad's lower-left vertex position (in World Coordinates)
• "_QuadX" the quad's "transform.right"
• "_QuadY" the quad's "transform.up"
• "_QuadScale" the quad's localScale (x and y) in Vector2
• "_Culling" should the reflection quad's backface be culled ? (1.0 for cull, any other value for 2-sided)

Please refer to the Box Projected Cubemapping shader topic (referenced at the end of this post) for parameters related to it (although I just changed one of them, from "EnvBoxStart" to "BoxPosition" which is the bounding box's center pos in world space).

The shader currently supports translation, rotation scaling, movietextures, backface culling and alpha testing.

However, it works only with one billboard for now. So I have to make it work with multiple quads, as well as taking into account the occlusion between them.

There's still a lot to do and the code could use some optimization but I wanted to share this since I found nothing about how to implement it in Unity or concrete cg code when I started working on it.

References:
http://udn.epicgames.com/Three/rsrc/Three/DirectX11Rendering/MartinM_GDC11_DX11_presentation.pdf
http://seblagarde.wordpress.com/201...ng-approaches-and-parallax-corrected-cubemap/
http://forum.unity3d.com/threads/11...Box-Projection-Correction-Environment-Mapping
https://www.cs.purdue.edu/cgvlab/papers/popescu/popescuGemEG06.pdf

 

Very promising and thank you for sharing code !

Would you mind sharing more info about how you send your parameters ? Maybe a scene setup ?

So far this is my attempt to make it work :





I've changed the way you did the ray to have a more physical approach.

I'm using a plane as a source for the billboard ( just the size,scale etc no image on it atm ) but to get the reflection facing me I need to rotate the plane in the wrong way also the size is wrong.

And here is my C# to send the data.

Code (csharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. [ExecuteInEditMode]
5. public class Billboard : MonoBehaviour
6. {
7.    
8.     // Billboard
9.    
10.     protected Vector3 m_right = new Vector3(1, 0, 0);
11.     protected Vector3 m_up = new Vector3(0, 1, 0);
12.    
13.     //public Mesh plane;
14.     public GameObject vertice;
15.     public Vector3 quadLLPos;
16.     public Vector3 quadX;
17.     public Vector3 quadY;
18.     public Vector2 quadScale;
19.     public Texture2D billboard;
20.  
21.     void goVariable()
22.     {
23.        
24.         Shader.SetGlobalVector("_QuadLLPos",quadLLPos);
25.         Shader.SetGlobalVector("_QuadX",quadX);
26.         Shader.SetGlobalVector("_QuadY",quadY);
27.         Shader.SetGlobalVector("_QuadScale",quadScale);
28.         Shader.SetGlobalTexture("_Billboard",billboard);
29.  
30.     }
31.  
32.     // Use this for initialization
33.     void Start ()
34.     {
35.    
36.     }
37.    
38.     // Update is called once per frame
39.     void Update ()
40.     {
41.         quadLLPos = vertice.transform.position;
42.        
43.         quadX = this.transform.localToWorldMatrix.MultiplyVector(m_right);
44.         quadX.Normalize();
45.        
46.         quadY = this.transform.localToWorldMatrix.MultiplyVector(m_up);
47.         quadY.Normalize();
48.        
49.         quadScale = this.transform.localScale;
50.         goVariable();
51.     }
52. }
53.  

Thanks

 

Ok using a cube as shape works better :



But having a sample scene would help a lot .

 

Sorry I didn't respond earlier. I forgot about this thread and I kind of gave up on this effect for now after knowing that you can't send arrays to shaders in Unity. I tried to used this for multiple billboards on my projects by stacking planes using this shader with transparency (a pretty dirty workaround I'll give you that but for now, with 5 billboards, there is no noticable performance cost).

Thanks for your interst anyway For the billboard source, I used Unity's default "quad" and not the "plane", and you should use that instead since I've based my "QuadX" and "QuadY" axis on their axis (the Quad's normal is on -z while the plane's is on +y) . Also it has less faces than the plane. I understand it's quite an arbitrary decision so I'll try to make it adapt to the normal of any planes...
Huh, forget about it actually, I now recall that I've changed my shader code so now you decide which Vector is your plane's normal and you just give it the inverse matrix as well:

Code (csharp):

1.  
2. Shader "Custom/BPCEM with billboard reflection" {
3. Properties {
4.     _Color ("Main Color", Color) = (1,1,1,1)
5.     _ReflectColor ("Reflection Color", Color) = (1,1,1,0.5)
6.     _MainTex ("Base (RGB) RefStrength (A)", 2D) = "white" {}
7.     _Cube ("Reflection Cubemap", Cube) = "_Skybox" { TexGen CubeReflect }
8.     _BumpMap ("Normalmap", 2D) = "bump" {}
9.     _BoxPosition ("Bounding Box Position", Vector) = (0, 0, 0)
10.     _BoxSize ("Bounding Box Size", Vector) = (10, 10, 10)
11. }
12.  
13. SubShader {
14.     Tags { "RenderType"="Opaque" }
15.     LOD 300
16.    
17. CGPROGRAM
18. #pragma target 3.0
19. #pragma surface surf Lambert
20.  
21. sampler2D _MainTex;
22. sampler2D _BumpMap;
23. sampler2D _Billboard;
24. samplerCUBE _Cube;
25.  
26. fixed4 _Color;
27. fixed4 _ReflectColor;
28. float3 _BoxSize;
29. float3 _BoxPosition;
30. float3 _QuadPosition;
31. fixed4 _QuadNormal;
32. float4x4 _QuadInverseMatrix;
33.  
34. struct Input {
35.     float2 uv_MainTex;
36.     float2 uv_BumpMap;
37.     fixed3 worldPos;
38.     float3 worldNormal;
39.     INTERNAL_DATA
40. };
41.  
42. void surf (Input IN, inout SurfaceOutput o) {
43.     // Base diffuse texture
44.     fixed4 tex = tex2D(_MainTex, IN.uv_MainTex);
45.     fixed4 c = tex * _Color;
46.     o.Albedo = c.rgb;
47.    
48.     fixed3 n = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));
49.    
50.     // Reflection-ray
51.     float3 viewDir = IN.worldPos - _WorldSpaceCameraPos;
52.     float3 worldNorm = IN.worldNormal;
53.     worldNorm.xy -= n;
54.     float3 reflectDir = reflect (viewDir, worldNorm);
55.     fixed3 nReflDirection = normalize(reflectDir);
56.    
57.     // Parallax correction
58.     half3 boxStart = _BoxPosition - _BoxSize / 2.0;
59.     half3 firstPlaneIntersect = (boxStart + _BoxSize - IN.worldPos) / nReflDirection;
60.     half3 secondPlaneIntersect = (boxStart - IN.worldPos) / nReflDirection;
61.     half3 furthestPlane = (nReflDirection > 0.0) ? firstPlaneIntersect : secondPlaneIntersect;
62.     half3 intersectDistance = min(min(furthestPlane.x, furthestPlane.y), furthestPlane.z);
63.     half3 intersectPosition = IN.worldPos + nReflDirection * intersectDistance;
64.     fixed4 reflcol = texCUBE(_Cube, intersectPosition - _BoxPosition);
65.    
66.     // Ray-quad intersection
67.     half planeIntersectDistance = (dot(IN.worldPos - _QuadPosition, _QuadNormal.xyz) / dot(nReflDirection, _QuadNormal.xyz));
68.     intersectPosition = IN.worldPos - nReflDirection * planeIntersectDistance;
69.     half4 localPlaneIntersectPosition = mul(_QuadInverseMatrix, float4(intersectPosition, 1.0));
70.     half2 billboardUV = half2(localPlaneIntersectPosition.x, localPlaneIntersectPosition.y);
71.     fixed4 billboardCol = tex2D(_Billboard, billboardUV);
72.    
73.     half reflectDot = dot(nReflDirection, _QuadNormal.xyz);
74.     fixed w = billboardCol.a;
75.     w = billboardUV.x > 1.0 ? 0.0 : w;
76.     w = billboardUV.y > 1.0 ? 0.0 : w;
77.     w = billboardUV.x < 0.0 ? 0.0 : w;
78.     w = billboardUV.y < 0.0 ? 0.0 : w;
79.     w = reflectDot >= 0.0  _QuadNormal.w == 1.0 ? 0.0 : w;
80.    
81.     reflcol.rgb = lerp(reflcol.rgb, billboardCol.rgb, w);
82.    
83.     // Reflection display
84.     reflcol *= tex.a;
85.     o.Emission = reflcol.rgb * _ReflectColor.rgb;
86.     o.Alpha = reflcol.a * _ReflectColor.a;
87. }
88. ENDCG
89. }
90.  
91. FallBack "Reflective/VertexLit"
92. }
93.  

And this is the C# script that I attach to the source billboard:

Code (csharp):

1.  
2. using UnityEngine;
3. using System.Collections;
4.  
5. [ExecuteInEditMode]
6. public class ReflectionBillboardScript : MonoBehaviour {
7.  
8.     public Material[] materials;
9.     public bool backfaceCulling = true;
10.  
11.     private Mesh mesh;
12.     private Transform myTransform;
13.     private Matrix4x4 transformMatrix;
14.     private Vector3 quadNormal;
15.  
16.     void Update () {
17.         myTransform = transform;
18.         mesh = GetComponent<MeshFilter>().sharedMesh;
19.  
20.         quadNormal = -1.0F * myTransform.forward;
21.  
22.         transformMatrix.SetTRS(myTransform.TransformPoint(mesh.vertices[0]), myTransform.rotation, myTransform.localScale);
23.  
24.         for (int i = 0; i < materials.Length; i++) {
25.             materials[i].SetTexture("_Billboard", renderer.sharedMaterial.mainTexture);
26.         }
27.  
28.         for (int i = 0; i < materials.Length; i++) {
29.             materials[i].SetVector("_QuadPosition", myTransform.TransformPoint(mesh.vertices[0]));
30.             materials[i].SetVector("_QuadNormal", new Vector4(quadNormal.x, quadNormal.y, quadNormal.z, backfaceCulling ? 1.0F : 0.0F));
31.             materials[i].SetMatrix("_QuadInverseMatrix", transformMatrix.inverse);
32.         }
33.     }
34. }
35.  

"Material[] materials" are all the materials in which this billboard is reflected.
Also i'm still using Unity's quad here so I've assigned my "quadNormal" value to the opposite of the quad's forward (you can change it to "transform.up" if you're using a Unity plane). The code should execute in edit mode but you might need to hit play or reload your scene for some updates.

I'll try to upload a simple example scene.

 

Hey!

Thanks for the details !

I've myself switch to quad so it's perfect.

For arrays it's true but there's work around : http://forum.unity3d.com/threads/223100-Image-Based-Reflection

I've tested Dolkar's idea and it worked but how I did it was really tricky, with a better script I'm sur it can be done.

 

Yeah I've seen that you can use arrays that way but I think it is still too limited... Maybe for now we can make a fixed sized array which sets the limit to the number of billboards per reflective material and only assign through script what we need.

Also I've been wondering: is your reflection distored because of a normal map ?

PS : We should also add an "actual array length" property to the shader so that it doesn't loop more than necessary.

 

I've changed my ray to match GGX D term so I can have different roughness I also use the mip map of the image. I will share my code.

Yeah that would be nice to set array length ( 8 ? ). With my test I've tested two billboard and works really fine but still I got an issue with the masking ( only one mask working ), did you run into that issue ?

 

That's really awesome!

Just wondering, can it be used to get less expensive reflections on a waterplane for example?

 

Would be nice also to setup a kind of capture gameobject that can save to disk the image reflection you want to capture ( with an Ortho camera ) in RGBM.

indiegamemodels : That would work yes and combined with a cubemap that will give you some nice effect.

 

Thanks for your quick answer! Definately something I will look into for my game (but I don't use Unity)

Just thinking, but being able to use a static cubemap for static objects and this method for dynamic objects should give a great result for a very low performance cost!

 

Does this use unity pro?(didn't see any use render texture in the script)

 

It should work with unity free.