Projective Mesh Decals

Several months ago, I posted on this topic as well, but I now realize that I only got one response. I never looked back at it, because I got entirely tangled up in school. However, I'm now trying to implement it once again and I'm not sure that I'm going along the right path at all.

I get the concept, mostly, but I'm not entirely sure that I'm doing it right.

So, here I am trying to implement something similar to: http://blog.wolfire.com/2009/06/how-to-project-decals/

I'm sure many of you have happened upon this and although I have googled "Mesh Project Decals" and a number of other terminologies, I haven't happened to come upon any concrete help.

Anyway, the general theory I've come up with is this:

Using a box collider (the projector), check which meshrenderer's bounds intersect it.

From there, test for each triangle that is inside the box collider and discard the ones that are not. (Not sure about this part at all, but positive there is a widely available math formula for it)

Continuing on, here is the part I'm not sure about: Convert the MeshRenderer's to "Projective Space", I'm assuming this is the space based on the bounding box of the box collider(projector).

Is using gameObject.transform.TransformPoint handling that correctly? I know that TransformPoint will be in 3D based on the projector's space in the world. However, the article shows "Projector" space as 2D, so could I just drop the Z component considering that the 3D space is correct? or atleast I think it is correct.

Note: Please do not direct me to an already available mesh decal system unless it is freely open source for me to read through. As, I am doing this entirely for learning.

Thank you,
The0xStandard

 

Testing whether a triangle is inside of the projection box consists of several steps and as you later on need to cut the triangles, it is far easier to implement it differently.
What I am doing in my Decal System is to take the mesh and cut it with each of the six sides of the projection box one after another. This simplifies the implementation to cutting a mesh using a plane. At this stage, I am computing for each vertex whether it is outside of the plane or not. Afterwards, there is an iteration through all the triangle. Now it is straight forward to find out whether the triangle is outside, inside or whether there is an intersection that needs to be handled in a special way.

 

I see, so mesh slicing using the 6 planes of the projector. Lemme see what I can get working here real quick in that area. If you don't mind me asking, how did you convert the models to projector space? That is honestly my main confusion. As I said, I'm assuming transformpoint works correctly, but I'm not all that positive.

I appreciate it.

 

I also made a decal system based on that blog post. You just transform points from source mesh renderer coordinates to world coordinates and then back to local points in your projector mesh renderer.
You can combine local to world and world to local matrices to do this in one step, or you can use TransformPoint and InverseTransformPoint.

 

I was originally going to combine the matrices, but I'm honestly not sure how. Would I just get the inverse matrix of the model and multiply it by the matrix of the projector? That sounds right in my head.

 

Here's how I do it:

Code (csharp):

1.  
2.         Mesh tempMesh;
3.         TFDecalPolygon tempPoly;
4.         Matrix4x4 matrix;
5.         // For each triangle in the mesh, add its vertices and normals into a TFDecalPolygon and add it to the list.
6.         for (int m = 0; m < sourceMeshes.Length; m++) {
7.             tempMesh = sourceMeshes[m].sharedMesh;
8.             matrix = localTransform.worldToLocalMatrix * sourceMeshes[m].transform.localToWorldMatrix;
9.             for (int t = 0; t < tempMesh.triangles.Length; t+=3) {
10.                 tempPoly = new TFDecalPolygon();
11.                 for (int k = 0; k < 3; k++) {
12.                     tempPoly.vertex[k] = matrix.MultiplyPoint( tempMesh.vertices[tempMesh.triangles[t+k]]);              
13.                     tempPoly.normal[k] = matrix.MultiplyVector(tempMesh.normals[tempMesh.triangles[t+k]]).normalized;
14.                 }
15.                 sourcePolygons.Add(tempPoly);
16.             }
17.         }
18.  

The interesting line here is :

Code (csharp):

1.  
2.             matrix = localTransform.worldToLocalMatrix * sourceMeshes[m].transform.localToWorldMatrix;
3.  

where localTransform is the transform of my projector object, where this component also is attatched.
sourceMeshes is an array of MeshRenderers that should be affected by the decal.

Code (csharp):

1.  
2.             matrix = projectorGameObject.transform.worldToLocalMatrix * affectedMeshRenderer.transform.localToWorldMatrix;
3.  

Then using this vector you just feed the original vertice points/normals through the MultiplyPoint and MultiplyVector to get the resulting vectors.

 

Opposite of what I said, appreciate it man that is a real help.

 

Alright, so I'm finally down to the last hall.
I've gotten the projector to split the mesh based on the 6 planes of the projector.
I've also implemented ThermalFusion's transformation matrix.
However, I'm not getting correct UV coordinates at all.

For some reason I'm getting vertices in the range of insanely weird values (all the way up to -7e) instead of the usual [-1,-1] to [1,1] where I could multiply the bias matrix. Wondering if I did something specific wrong or is that what I should be getting?

I'm expecting the error to be in:

Code (CSharp):

1.  
2.     //Convert to Projector Space
3.     void ConvertToProjectorSpaceToCalculateUVs(GameObject in_MeshGameObject, Mesh in_Mesh)
4.     {
5.         Matrix4x4 Mat = gameObject.transform.worldToLocalMatrix * in_MeshGameObject.transform.localToWorldMatrix;
6.  
7.         List<Vector3> Vertices = new List<Vector3>(in_Mesh.vertices);
8.         List<Vector2> Uvs = new List<Vector2>();
9.  
10.  
11.         for(int I = 0; I < Vertices.Count; I++)
12.         {
13.             Vertices[I] = Mat.MultiplyPoint(Vertices[I]);
14.  
15.             float X = Vertices[I].x;
16.             float Y = Vertices[I].y;
17.  
18.             Debug.Log (X);
19.             Debug.Log (Y);;
20.             Uvs.Add (new Vector2(Vertices[I].x, Vertices[I].y));
21.         }
22.  
23.  
24.         in_Mesh.uv = Uvs.ToArray();
25.     }
26.  

However, I'm not positive. So, here is the whole thing.

Code (CSharp):

1. using UnityEngine;
2. using System.Collections.Generic;
3.  
4. //Decal Projector that does the Slicing in World Space rather than local space.
5. [RequireComponent(typeof(BoxCollider))]
6. public class DecalProjectorWSS : MonoBehaviour
7. {
8.  
9.     void Start()
10.     {
11.         List<MeshFilter> TempMeshList = GetMeshesInBounds();
12.         List<GameObject> GameObjectList;
13.  
14.         if(TempMeshList != null && TempMeshList.Count != 0)
15.         {
16.             GameObjectList = GenerateNewMeshesAndGameObjects(TempMeshList);
17.      
18.             if(GameObjectList != null && GameObjectList.Count != 0)
19.             {
20.                 for(int I = 0; I < GameObjectList.Count; I++)
21.                 {
22.                     MeshFilter MF = GameObjectList[I].GetComponent<MeshFilter>();
23.                     Vector3[] PointsOnPlane = new Vector3[6];
24.                     SetPointsOnPlane(ref PointsOnPlane);
25.  
26.                     Debug.Log ("Slicing..");
27.                     SliceMesh(GameObjectList[I], MF.sharedMesh, PointsOnPlane[0], gameObject.transform.up, MF.sharedMesh.triangles, MF.sharedMesh.vertices, MF.sharedMesh.uv, MF.sharedMesh.normals);
28.                     SliceMesh(GameObjectList[I], MF.sharedMesh, PointsOnPlane[1], -gameObject.transform.up, MF.sharedMesh.triangles, MF.sharedMesh.vertices, MF.sharedMesh.uv, MF.sharedMesh.normals);
29.  
30.                     SliceMesh(GameObjectList[I], MF.sharedMesh, PointsOnPlane[2], gameObject.transform.forward, MF.sharedMesh.triangles, MF.sharedMesh.vertices, MF.sharedMesh.uv, MF.sharedMesh.normals);
31.                     SliceMesh(GameObjectList[I], MF.sharedMesh, PointsOnPlane[3], -gameObject.transform.forward, MF.sharedMesh.triangles, MF.sharedMesh.vertices, MF.sharedMesh.uv, MF.sharedMesh.normals);
32.  
33.                     SliceMesh(GameObjectList[I], MF.sharedMesh, PointsOnPlane[4], gameObject.transform.right, MF.sharedMesh.triangles, MF.sharedMesh.vertices, MF.sharedMesh.uv, MF.sharedMesh.normals);
34.                     SliceMesh(GameObjectList[I], MF.sharedMesh, PointsOnPlane[5], -gameObject.transform.right, MF.sharedMesh.triangles, MF.sharedMesh.vertices, MF.sharedMesh.uv, MF.sharedMesh.normals);
35.                     Debug.Log ("After-Slice Tri Count: " + MF.sharedMesh.triangles.Length);
36.                     Debug.Log ("After-Slice Vert Count: " + MF.sharedMesh.vertexCount);
37.  
38.                     ConvertToProjectorSpaceToCalculateUVs(GameObjectList[I], MF.sharedMesh);
39.  
40.  
41.                 }
42.             }
43.             else
44.             {
45.                 Debug.Log ("No Models in Projector.");
46.             }
47.         }
48.         else
49.         {
50.             Debug.Log ("No Models in Projector.");
51.         }
52.     }
53.  
54.     //Gets Every Mesh that this projector intersects.
55.     List<MeshFilter> GetMeshesInBounds()
56.     {
57.         List<MeshFilter> MeshesList = new List<MeshFilter>(GameObject.FindObjectsOfType<MeshFilter>());
58.  
59.         for(int I = MeshesList.Count - 1; I >= 0; I--)
60.         {
61.             if(!MeshesList[I].gameObject.renderer.bounds.Intersects(gameObject.collider.bounds))
62.             {
63.                 MeshesList.RemoveAt(I);
64.             }
65.         }
66.  
67.         return MeshesList;
68.     }
69.  
70.     //Produces the Decal List
71.     List<GameObject> GenerateNewMeshesAndGameObjects(List<MeshFilter> in_MeshFilters)
72.     {
73.         List<GameObject> TempList = new List<GameObject>();
74.  
75.         for(int I = 0; I < in_MeshFilters.Count; I++)
76.         {
77.             GameObject NewGameObject = new GameObject("Decal");
78.             NewGameObject.transform.position = in_MeshFilters[I].gameObject.transform.position;
79.             NewGameObject.transform.rotation = in_MeshFilters[I].gameObject.transform.rotation;
80.             //NewGameObject.transform.parent = in_MeshFilters[I].transform;
81.      
82.             MeshFilter NewGameObjectMeshFilter = NewGameObject.AddComponent<MeshFilter>();
83.             Mesh NewGameObjectMesh = (Mesh)Mesh.Instantiate(in_MeshFilters[I].sharedMesh);
84.             NewGameObjectMeshFilter.mesh = NewGameObjectMesh;
85.      
86.             NewGameObject.AddComponent<MeshRenderer>();
87.      
88.             TempList.Add (NewGameObject);
89.         }
90.  
91.         return TempList;
92.     }
93.  
94.     //Creates the center points of the planes.
95.     //This version only works in world space.
96.     void SetPointsOnPlane(ref Vector3[] in_PointsOnPlaneArray)
97.     {
98.         Matrix4x4 LTWMat = this.transform.localToWorldMatrix;
99.         Quaternion Rotation = this.transform.rotation;
100.         this.transform.rotation = Quaternion.identity;
101.  
102.         Vector3 extents = collider.bounds.extents;
103.  
104.         //Up Center
105.         in_PointsOnPlaneArray[0] = LTWMat.MultiplyPoint3x4(new Vector3(0, extents.y, 0));
106.  
107.         //Down Center
108.         in_PointsOnPlaneArray[1] = LTWMat.MultiplyPoint3x4(new Vector3(0, -extents.y, 0));
109.  
110.         //Forward Center
111.         in_PointsOnPlaneArray[2] = LTWMat.MultiplyPoint3x4(new Vector3(0, 0, extents.z));
112.  
113.         //Backward Center
114.         in_PointsOnPlaneArray[3] = LTWMat.MultiplyPoint3x4(new Vector3(0, 0, -extents.z));
115.  
116.         //Right Center
117.         in_PointsOnPlaneArray[4] = LTWMat.MultiplyPoint3x4(new Vector3(extents.x, 0, 0));
118.  
119.         //Left Center
120.         in_PointsOnPlaneArray[5] = LTWMat.MultiplyPoint3x4(new Vector3(-extents.x, 0, 0));
121.  
122.         this.transform.rotation = Rotation;
123.  
124.     }
125.  
126.     //Test which side a point is on. If true, then the point is opposite of the Normal and thus it might be contained by the box.
127.     //A total of 6 PointSide test must be made per-vertex that hasn't already been discarded.
128.     //If it passes all 6, then it is inside of the projector box.
129.     bool PointSide(Vector3 Normal, Vector3 Point, Vector3 PointOnPlane)
130.     {
131.         Vector3 V = new Vector3(Point.x - PointOnPlane.x, Point.y - PointOnPlane.y, Point.z - PointOnPlane.z);
132.         float SignedDistance = Vector3.Dot(V, Normal);
133.  
134.         return (SignedDistance <= 0); //If True, Point is inside projector
135.     }
136.  
137.     void SliceMesh(GameObject MeshGameObject, Mesh in_Mesh, Vector3 PointOnPlane, Vector3 PlaneNormal, int[] in_TempTriangleArray, Vector3[] in_TempVertexArray, Vector2[] in_TempVertexUVArray, Vector3[] in_TempVertexNormalArray)
138.     {
139.         List<Vector3> ConvertedVertices = new List<Vector3>(in_TempVertexArray);
140.  
141.         List<int> TriangleList = new List<int>(in_TempTriangleArray);
142.         List<Vector3> NonConvertedVertices = new List<Vector3>(in_TempVertexArray);
143.         List<Vector3> VertexNormalList = new List<Vector3>(in_TempVertexNormalArray);
144.         List<Vector2> VertexUVList = new List<Vector2>(in_TempVertexUVArray);
145.         List<int> VertexRefCount = new List<int>();
146.  
147.         //Reference Counting for a vertex, it shall be removed if its ref is zero.
148.         for(int I = 0; I < ConvertedVertices.Count; I++)
149.         {
150.             VertexRefCount.Add (3);
151.         }
152.  
153.         //Convert the vertices to world space to be tested.
154.         for(int I = 0; I < ConvertedVertices.Count; I++)
155.         {
156.             ConvertedVertices[I] = MeshGameObject.transform.TransformPoint(ConvertedVertices[I]);
157.         }
158.  
159.         //Slice Mesh
160.         for(int I = TriangleList.Count-1; I-2 >= 0; I-=3)
161.         {
162.             int Index1 = TriangleList[I];
163.             int Index2 = TriangleList[I-1];
164.             int Index3 = TriangleList[I-2];
165.      
166.             bool Point1InsidePlane = PointSide(PlaneNormal, ConvertedVertices[Index1], PointOnPlane);
167.             bool Point2InsidePlane = PointSide(PlaneNormal, ConvertedVertices[Index2], PointOnPlane);
168.             bool Point3InsidePlane = PointSide(PlaneNormal, ConvertedVertices[Index3], PointOnPlane);
169.      
170.             if(!Point1InsidePlane || !Point2InsidePlane || !Point3InsidePlane)
171.             {
172.                 TriangleList.RemoveAt(I);
173.                 VertexRefCount[Index1] -= 1;
174.                 TriangleList.RemoveAt(I-1);
175.                 VertexRefCount[Index2] -= 1;
176.                 TriangleList.RemoveAt(I-2);
177.                 VertexRefCount[Index3] -= 1;
178.             }
179.         }
180.  
181.         //Handle Reference Count
182.         for(int I = VertexRefCount.Count-1; I > 0; I--)
183.         {
184.             if(VertexRefCount[I] == 0)
185.             {
186.                 for(int J = 0; J < TriangleList.Count; J++)
187.                 {
188.                     if(TriangleList[J] > I)
189.                     {
190.                         TriangleList[J] -= 1;
191.                     }
192.                 }
193.  
194.                 NonConvertedVertices.RemoveAt(I);
195.                 VertexNormalList.RemoveAt(I);
196.                 VertexUVList.RemoveAt(I);
197.             }
198.         }
199.  
200.         in_Mesh.Clear ();
201.         in_Mesh.vertices = NonConvertedVertices.ToArray ();
202.         in_Mesh.normals = VertexNormalList.ToArray ();
203.         in_Mesh.uv = VertexUVList.ToArray ();
204.         in_Mesh.triangles = TriangleList.ToArray ();
205.     }
206.  
207.     //Convert to Projector Space
208.     void ConvertToProjectorSpaceToCalculateUVs(GameObject in_MeshGameObject, Mesh in_Mesh)
209.     {
210.         Matrix4x4 Mat = gameObject.transform.worldToLocalMatrix * in_MeshGameObject.transform.localToWorldMatrix;
211.  
212.  
213.         List<Vector3> Vertices = new List<Vector3>(in_Mesh.vertices);
214.         List<Vector2> Uvs = new List<Vector2>();
215.  
216.  
217.         for(int I = 0; I < Vertices.Count; I++)
218.         {
219.             Vertices[I] = Mat.MultiplyPoint(Vertices[I]);
220.  
221.             Debug.Log (Vertices[I].x);
222.             Debug.Log (Vertices[I].y);;
223.             Uvs.Add (new Vector2(Vertices[I].x, Vertices[I].y));
224.         }
225.  
226.  
227.         in_Mesh.uv = Uvs.ToArray();
228.     }
229. }

 

So I gave it a quick look, and besides the coordinates being inside the -1 to 1 range instead of 0 to 1, it looked just fine uv-wise.
Also I did not see any actual slicing being done yet, only discarding of polys with vertices outside.

 

That is what I was about to correct, I totally worded that wrong.

I'm going to implement "actual" slicing now. As for the UVs, weird I'm getting nothing when I apply a material to the decals.

I have some vertices that output -1.9 X/Y, but I'm assuming those are the vertices that are actually outside of the projector. Guess I'll just have to implement slicing and then find out.

Noticed it after I stopped having a retard moment, haha.

 

@The0xStandard, could you by any chance share your source after adding the slicing bit as well?