[SOLVED] Vertex Color Painting (Flat shading)

I'm having issues with the following code; it makes the triangles 'smooth' but I want them to paint 'sharp' (without blending on the edges)

Is there a way I can find the 'surrounding' triangles of my hitPoint and paint them 'fully' ?

Thanks !

Code (CSharp):

1.      void PaintVertexColor () {
2.        
3.             Vector3[] verts = curMesh.vertices;
4.  
5.             Color[] colors = new Color[0];
6.  
7.             if (curMesh.colors.Length > 0) {
8.                 colors = curMesh.colors;
9.             } else {
10.                 colors = new Color[verts.Length];
11.             }
12.  
13.             // Go all thru the vertices of the object
14.             for (int i = 0; i < verts.Length; i++) {
15.                 Vector3 vertPos = curGO.transform.TransformPoint (verts[i]);
16.                 float sqrMag = (vertPos - curHit.point).sqrMagnitude;
17.                 if (sqrMag > brushSize) {
18.                     continue;
19.                 } else {
20.                     if (i % 3 == 0) {
21.                         currentColor = fgColor;
22.                     }
23.                     colors[i] = fgColor;
24.                 }
25.             }
26.             curMesh.colors = colors;
27.          
28.         }
29.  

 

For each vertex you want to color, to find the face it’s attached to you’ll have to iterate over the triangle list to find a reference to the vertex index. The triangle list is actually a list of vertex indices where each 3 vertex indices is one triangle. So once you find a match it’ll either be the previous two indices, next two indices, or surrounding indices that make up the triangle depending on the current triangle list index.

Code (csharp):

1. for (int t=0; t<tris.Length; t++)
2. {
3.   if (tris[t] == i) // closest vertex i
4.   {
5.     int subIndex = t % 3;
6.     if (subIndex == 0)
7.       // apply color to t+1 & t+2
8.     else if (subIndex == 1)
9.       // apply color to t-1 and t+1
10.     else
11.       // apply color to t-2 & t-1
12.   }
13. }

 

Thanks for the reply much appreciated, I used your code and implemented but ... It doesn't work.
It paints the wrong ones. Gosh I'm missing something obvious here but I don't get why.

Code (CSharp):

1.  
2.  
3. if (Physics.Raycast (ray, out curHit)) {
4.     if (isPainting) {
5.         PaintNeighbours (curMesh, curHit.triangleIndex, fgColor);
6.     }
7. }
8.  
9. void PaintNeighbours (Mesh mesh, int i, Color color) {
10.             Vector3[] vertices = mesh.vertices;
11.             Color[] colors = new Color[0];
12.  
13.             if (mesh.colors.Length > 0) {
14.                 colors = mesh.colors;
15.             } else {
16.                 colors = new Color[vertices.Length];
17.             }
18.          
19.             int[] triangles = mesh.triangles;
20.             for (int t = 0; t < triangles.Length; t++) {
21.                 if (triangles[t] == i) // closest vertex i
22.                 {
23.                     int subIndex = t % 3;
24.                     if (subIndex == 0) {
25.                         // apply color to t+1 & t+2
26.                         colors[t + 1] = color;
27.                         colors[t + 2] = color;
28.                     } else if (subIndex == 1) {
29.                         // apply color to t-1 and t+1
30.                         colors[t - 1] = color;
31.                         colors[t + 1] = color;
32.                     } else {
33.                         // apply color to t-2 & t-1
34.                         colors[t - 2] = color;
35.                         colors[t - 1] = color;
36.                     }
37.                 }
38.             }
39.             mesh.colors = colors;
40.         }
41. }

 

Your original example code was getting a vertex index. My example code took that vertex index, and finds the appropriate range in the tri array.

The new code you posted is getting the triangle index from a raycast, not the vertex index. To get the vertex indices for that triangle it's just that tri index value * 3, +1, +2. No need to search though anything at that point.

 

Yes you're right ... then I want to paint the triangles that are within a radius of my brush; the question is then : "how to find the neighbors from that triangle index" ?

 

If you just want all triangles within the radius, then you’ll want to iterate over all of the vertex positions like you were before, then iterate over the triangle list to find the vertices of those triangles.

If you want to only color triangles that are both in range, and "connected" to the first triangle you touched, that's a lot harder. That requires iterating over all of the triangles to find those that have shared edges, or at least shared vertex positions. Since your mesh is using hard edged normals, each triangle has unique vertices after being imported into Unity, so you have to compare each vertex's position against all other vertices and match ones that are approximately the same. (Note, the built in == comparison for Vector3 values is already an "approximate equal" which should be good enough).

 

wow; i see; this is where my brain explodes
Do you have an example ? Even pseudo-code ?

 

Ok I did move forward now I can paint the way I want; except I want to make sure I'm not having this 'triangle' effect.

So how to always have the 'right' triangle to form the quad ?

Code (CSharp):

1.  void PaintNeighbors (Mesh mesh, int i, Color color) {
2.             Vector3[] vertices = mesh.vertices;
3.             Color[] colors = new Color[0];
4.             int[] triangles = mesh.triangles;
5.  
6.             if (mesh.colors.Length > 0) {
7.                 colors = mesh.colors;
8.             } else {
9.                 colors = new Color[vertices.Length];
10.             }
11.  
12.             colors[i * 3] = color;
13.             colors[i * 3 + 1] = color;
14.             colors[i * 3 + 2] = color;
15.  
16.             Vector3 p0 = vertices[triangles[i * 3 + 0]];
17.             Vector3 p1 = vertices[triangles[i * 3 + 1]];
18.             Vector3 p2 = vertices[triangles[i * 3 + 2]];
19.             float rad = brushSize;
20.             for (int z = 0; z < triangles.Length / 3; z++) {
21.                 Vector3 f0 = vertices[triangles[z * 3 + 0]];
22.                 Vector3 f1 = vertices[triangles[z * 3 + 1]];
23.                 Vector3 f2 = vertices[triangles[z * 3 + 2]];
24.                 if (((f0 - p0).sqrMagnitude < (brushSize * brushSize)) && ((f1 - p1).sqrMagnitude < (brushSize * brushSize)) && ((f2 - p2).sqrMagnitude < (brushSize * brushSizef))) {
25.                     colors[z * 3] = color;
26.                     colors[z * 3 + 1] = color;
27.                     colors[z * 3 + 2] = color;
28.                 }
29.             }
30.             mesh.colors = colors;
31.         }

 

For the connected triangles? It's a lot of recursive nested loops. I would suggest not trying to do the "connected" test at all. No point if your mesh is terrain-like mesh like you have as all of the triangles are guaranteed to be connected, so the test would be a very expensive function that would always return true.

If you want to find all triangles that have a vertex in range, and are connected to the raycasted triangle in a continuous way, then you could go about it in this way.

Get the vertices of the triangle you hit:
v0 = tris[triangleIndex * 3]
v1 = tris[triangleIndex * 3 +1]
v2 = tris[triangleIndex * 3 + 2]

For each vertex, iterate over the vertex positions to find other vertices with the same position:
for (i=0; i<vertexCount; i++)
if (i != v0 && i != v1) && i != v2) && (vertices == vertices[v0] || vertices == vertices[v1] || vertices[v2])
// is connected vertex

Then find all of the vertex indices for that triangle using the first example I gave you, and set their color. Then, for all of those vertices that aren't the original triangle's, if they're in range, find the vertices & triangles that are attached to that, color, repeat until you run out.

 

I guess you didn't see my reply; 3 mins before yours; so I got it almost right; just missing the last triangles; if you have a trick for that I owe you one beer

 

Why are you comparing each vertex to another arbitrary vertex? Shouldn’t you be comparing each vertex to the brush’s hit position?

 

I compare the vertex from where the brush hit (index "i") to the one I parse index "z" but at the end I'm always missing the last triangles that form a quad; if we could figure out this that would be super.

 

You’re comparing the 3 vertices from triangle “i” to the 3 vertices in triangle “z”, but that’s completely arbitrary. You have no idea how the first vertex of triangle “i” relates to the first vertex of triangle “z”, so that comparison is likely causing some of saw toothed the weirdness you’re seeing.

Here’s a basic example. Let’s take 4 triangles in a strip like this:

Now, you have a brush with a radius that’s slightly less than a single triangle wide, and your brush’s center hits the red triangle.

The red triangle’s vertices are in this order: EBC.
Your code now iterates over the triangles one by one. Let’s say the blue triangle is triangle 0, and its vertices are DAB. Let’s compare each vertex in that order, D-E fails, as does A-B, as does B-C. Blue is not colored.
Now let’s say green is triangle 1, with the order DBE. D-E fails again, but the next vertex is B-B, and that succeeds! So green is colored.
Triangle 2 is the red triangle, that is already colored, but it gets colored again since obviously each vertex is close enough to itself. Triangle 3 is the yellow triangle, with the order ECF. The first vertex succeeds, so it too is colored.

So now we have this:

Notice the saw tooth shape forming? This is why you shouldn’t arbitrarily compare vertices in a specific order. You do not know how that order relates to other meshes, or how that might impact the results. Do the distance comparison against the hit position, not the hit triangle or its vertices.

The quad isn’t really a thing though. It’s all just triangles by the time you’re looking at it from code. There are no quads though, only triangles. Any concept of quads innate to the mesh was lost when you imported or exported, or whenever the mesh was triangulated. Sure there’s an implicit quad there, but you’d have to figure out which edge is the “diagonal” edge of the triangle, which two vertices make up that edge, and then find another triangle that has vertices in both of those locations.

 

Thank you very much for taking the time to draw and explain; I understood why I should compare arbitrary vertices but I still don't understand what distance you compare to what

Code (CSharp):

1. for (i=0; i<vertexCount; i++)
2. if (i != v0 && i != v1) && i != v2) && (vertices == vertices[v0] || vertices == vertices[v1] || vertices[v2])
3.  

What is the 'vertices' variable here ?

 

If that's helping I have written some code that turn any mesh into a half edge structure by collapsing split vertex, it might help you iterate through vertices more easily (it keep a reference to the triangle index of the mesh)?

https://github.com/Neoshaman/Mesg-to-half-edge

 

If you're looking to color all triangles within range of the brush, then you need to be doing a distance check between each triangles' vertex positions and the brush's center position to see if it is within the brush size. The curHit.point value in the function that does the raycast would be that brush position, so you should be passing that along too.

If you're looking to find triangles that are "next" to each other, you need to compare each of the 3 vertices of a triangle against all 3 vertices of the next, individually. And you need to do it recursively, so once you check if a triangle is attached, you need to find triangles that attach to that triangle, and triangles that attach to that triangle, etc. But if you do it naively, you'll run into infinite loops, so you'd probably want to keep a list of triangles you've already found in a list and skip those triangles when they're found as a neighbor. You might also only want to look for neighbors of one or two vertices of a triangle at a time rather than all 3 after the first triangle depending on how many matched the previous triangle to limit the search space since the previous search for "that vertex" presumably already found all its neighbors.

But it's not clear to me that you need to do neighbor searches since all of the example meshes you've shown have been terrain-like. Hence my previous suggestion you skip this entirely and only use distance to the brush position.

Finding "quads" is similar, but since it's not recursive you can be a little dirtier there. Like I said before, you need to find the "diagonal" edge. Assuming your meshes are nicely axis aligned, and y is up, you can compare a triangles 3 vertices to find the two that are not aligned along an axis. Those will be the two vertices that make the "diagonal" edge. Then you'd search through all of the triangles to find another that also has two vertices that match those positions. You could also only do this check on triangles where only that right angle corner vertex is the one within the brush range, as if any other vertex is in range the matching triangle to make the quad will have already been found.

pseudo code:

Code (csharp):

1. foreach triangle
2. {
3.   int inRangeCount = 0;
4.   bool triangleInRange = false;
5.  
6.   foreach triangle vertex
7.   {
8.     if ((vertexPos - brushPos).sqrMag < sqrBrushRadius)
9.     {
10.       triangleInRange = true;
11.       inRangeCount++;
12.     }
13.   }
14.  
15.   if triangleInRange
16.     foreach triangle vertex
17.       colors[vertex index] = brushColor;
18.  
19.   if (inRangeCount == 1)
20.   {
21.     Vector3 diagVertPos0, diagVertPos1;
22.     // find "corner" vertex, if in range, set positions from other two vertices
23.     // otherwise continue loop, skipping the rest of this code
24.     foreach triangle
25.     {
26.       if (triangle == current triangle) // don't check itself
27.         continue
28.  
29.       foreach triangle vertex
30.         if (vertexPos == diagVertPos0)
31.           foreach triangle vertex
32.             if (vertexPos == diagVertPos1)
33.               // this is the matching tri to make the "quad", color all vertices, and break out of loop
34.     }
35.   }
36. }
37.  

 

Thanks.

So i've tried understanding and translating your pseudocode into code.
But i get endless loop. Can't understand where i messed up.

Code (CSharp):

1. public void Bgolus (Mesh mesh, int i, Color color) {
2.             Vector3[] vertices = mesh.vertices;
3.             Color[] colors = new Color[0];
4.             int[] triangles = mesh.triangles;
5.  
6.             if (mesh.colors.Length > 0) {
7.                 colors = mesh.colors;
8.             } else {
9.                 colors = new Color[vertices.Length];
10.             }
11.             for (int z = 0; z < triangles.Length; z++) {
12.                 int inRangeCount = 0;
13.                 bool triangleInRange = false;
14.  
15.                 for (int v = 0; v < 3; v++) {
16.                     Vector3 vertexPos = vertices[triangles[z * 3 + v]];
17.                     if ((vertexPos - curHit.point).sqrMagnitude < brushSize) {
18.                         // This triangle is within range
19.                         triangleInRange = true;
20.                         // We add to the count
21.                         inRangeCount++;
22.                     }
23.                 }
24.                 // If this triangle is in Range
25.                 if (triangleInRange) {
26.                     for (int v = 0; v < 3; v++) {
27.                         // Color that triangle
28.                         colors[z * 3 + v] = color;
29.                     }
30.                 }
31.                 // if we have only one triangle in range
32.                 if (inRangeCount == 1) {
33.                     Vector3 f0 = vertices[triangles[z * 3 + 0]];
34.                     Vector3 f1 = vertices[triangles[z * 3 + 1]];
35.                     Vector3 f2 = vertices[triangles[z * 3 + 2]];
36.  
37.                     // Calculer
38.                     float f0f1 = Vector3.Distance (f0, f1);
39.                     float f0f2 = Vector3.Distance (f0, f2);
40.                     float f1f2 = Vector3.Distance (f2, f1);
41.  
42.                     if (f0f1 > f0f2 && f0f1 > f1f2) {
43.                         diagVertPos0 = f0;
44.                         diagVertPos1 = f1;
45.                     }
46.  
47.                     if (f0f2 > f0f1 && f0f2 > f1f2) {
48.                         diagVertPos0 = f0;
49.                         diagVertPos1 = f2;
50.                     }
51.  
52.                     if (f1f2 > f0f1 && f1f2 > f0f2) {
53.                         diagVertPos0 = f1;
54.                         diagVertPos1 = f2;
55.                     }
56.  
57.                     for (int x = 0; x < triangles.Length; x++) {
58.                         if (x == z) continue;
59.                         for (int v = 0; v < 3; v++) {
60.                             if (vertices[triangles[x * 3 + v]] == diagVertPos0) {
61.                                 for (int w = 0; w < 3; w++) {
62.                                     if (vertices[triangles[x * 3 + v]] == diagVertPos1) {
63.                                         colors[x * 3] = color;
64.                                         colors[x * 3 + 1] = color;
65.                                         colors[x * 3 + 2] = color;
66.                                         break;
67.                                     }
68.                                 }
69.                             }
70.                         }
71.                     }
72.                 }
73.             }
74.         }

 

The triangles array is in sets of 3 vertices, so it should be < triangles.Length / 3 for both triangle loops. With out that you're going to be going outside the bounds of the triangles array.

The "diagonal" edge is not guaranteed to be the longest edge in 3D space. If you have a decently steep slope, the right angle's edges can be longer than the diagonal edge. You need to compare the distance (or square magnitude) in 2D space ignoring the height offset.

float f0f1 = (f0.x - f1.x) * (f0.x - f1.x) + (f0.z - f1.z) * (f0.z - f1.z); // assuming your mesh is y up

I'd also strongly recommend checking if the one vertex that is in range was the right angle corner vertex and skipping the second iteration if it is not.

Code (csharp):

1. bool cornerInRange = false;
2. if (f0f1 > f0f2 && f0f1 > f1f2) {
3.   diagVertPos0 = f0;
4.   diagVertPos1 = f1;
5.   cornerInRange = (f2 - curHit.point).sqrMagnitude < brushSize;
6. }
7.  
8. // other two checks
9.  
10. if (!cornerInRange)
11.   continue;
12.  
13. for (int x = 0; x < triangles.Length; x++) {
14. // etc

Also, I'm assuming the "brushSize" value is actually brushRadius * brushRadius?

 

Thanks we're almost there !

Now the only issue is that the painting doesn't exactly occur on the hit point; moving the brush around i see it only paints on the 'edges' of the brush; i can't figure out why ...

This shape is when i put the brush to the edge of the mesh.

[edit: nevermind it doesn't work, something is f***ed up]

Code (CSharp):

1.    public void UseTriangulation (Mesh mesh, int i, Color brushColor) {
2.  
3.             Vector3[] vertices = mesh.vertices;
4.             Color[] colors = new Color[0];
5.             int[] triangles = mesh.triangles;
6.  
7.             if (mesh.colors.Length > 0) {
8.                 colors = mesh.colors;
9.             } else {
10.                 colors = new Color[vertices.Length];
11.             }
12.  
13.             float brushRadius = brushSize * brushSize;
14.  
15.             for (int z = 0; z < triangles.Length / 3; z++) {
16.                 int inRangeCount = 0;
17.                 bool triangleInRange = false;
18.  
19.                 for (int v = 0; v < 3; v++) {
20.                     Vector3 vertexPos = vertices[triangles[z * 3 + v]];
21.                     if ((vertexPos - curHit.point).sqrMagnitude < (brushRadius)) {
22.                         triangleInRange = true;
23.                         inRangeCount++;
24.                     }
25.                 }
26.          
27.                 if (triangleInRange) {
28.                     for (int v = 0; v < 3; v++) {
29.                         colors[z * 3 + v] = VertexPainter_Utils.VertexColorLerp (colors[z * 3 + v], brushColor, brushRate);
30.                     }
31.                 }
32.  
33.                 // if we have only one triangle in range
34.                 if (inRangeCount == 1) {
35.                     Vector3 f0 = vertices[triangles[z * 3]];
36.                     Vector3 f1 = vertices[triangles[z * 3 + 1]];
37.                     Vector3 f2 = vertices[triangles[z * 3 + 2]];
38.            
39.                     float f0f1 = (f0.x - f1.x) * (f0.x - f1.x) + (f0.z - f1.z) * (f0.z - f1.z);
40.                     float f0f2 = (f0.x - f2.x) * (f0.x - f2.x) + (f0.z - f2.z) * (f0.z - f2.z);
41.                     float f1f2 = (f1.x - f2.x) * (f1.x - f2.x) + (f1.z - f2.z) * (f1.z - f2.z);
42.                
43.                     bool cornerInRange = false;
44.                     if (f0f1 > f0f2 && f0f1 > f1f2) {
45.                         diagVertPos0 = f0;
46.                         diagVertPos1 = f1;
47.                         cornerInRange = (f2 - curHit.point).sqrMagnitude < brushRadius;
48.                     }
49.  
50.                     if (f0f2 > f0f1 && f0f2 > f1f2) {
51.                         diagVertPos0 = f0;
52.                         diagVertPos1 = f2;
53.                         cornerInRange = (f1 - curHit.point).sqrMagnitude < brushRadius;
54.                     }
55.  
56.                     if (f1f2 > f0f1 && f1f2 > f0f2) {
57.                         diagVertPos0 = f1;
58.                         diagVertPos1 = f2;
59.                         cornerInRange = (f0 - curHit.point).sqrMagnitude < brushRadius;
60.                     }
61.  
62.                     if (!cornerInRange)
63.                         continue;
64.                 }
65.             }
66.  
67.             for (int x = 0; x < triangles.Length / 3; x++) {
68.                 for (int v = 0; v < 3; v++) {
69.                     if (vertices[triangles[x * 3 + v]] == diagVertPos0) {
70.                         for (int w = 0; w < 3; w++) {
71.                             if (vertices[triangles[x * 3 + w]] == diagVertPos1) {
72.                                 for (int t = 0; t < 3; t++) {
73.                                     //colors[x * 3 + t] = Color.Lerp (colors[x * 3 + t], brushColor, brushRate);
74.                                     colors[x * 3 + t] = VertexPainter_Utils.VertexColorLerp (colors[x * 3 + t], brushColor, brushRate);
75.                                 }
76.                                 break;
77.                             }
78.                         }
79.                     }
80.                 }
81.             }
82.             mesh.colors = colors;
83.         }
84.  

 

Here's a hint. The diagVertPos0 and diagVertPos1 values should only be needed inside the scope of the if (inRangeCount == 1) condition. If you move the lines where they're defined to be just after that if statement, nothing should break (but it will with how you have your code).

I also don't know if your mesh is Y up or not, so the diagonal & "right angle" vertices might not be correct.

 

Gosh that was it ... too many hours ... too many hours !

Works as expected thank you so much !