Question Convert Points of Point Cloud to Mesh

I'm trying to generate a mesh based on point cloud coordinates I'm retrieving from the Replicate API, I'm not too familiar with building a mesh on the fly, but this is the method I got to build the mesh...

Code (CSharp):

1.  
2.     private void CreateMesh(Vector3[] coordinates)
3.     {
4.         Mesh mesh = new Mesh();
5.  
6.         Vector3[] vertices = new Vector3[coordinates.Length];
7.         for (int i = 0; i < coordinates.Length; i++)
8.         {
9.             vertices[i] = coordinates[i];
10.         }
11.         mesh.vertices = vertices;
12.  
13.         int[] triangles = new int[(coordinates.Length - 2) * 3];
14.         int vertexIndex = 1;
15.         int triangleIndex = 0;
16.  
17.         for (int i = 0; i < coordinates.Length - 2; i++)
18.         {
19.             triangles[triangleIndex] = 0;
20.             triangles[triangleIndex + 1] = vertexIndex;
21.             triangles[triangleIndex + 2] = vertexIndex + 1;
22.  
23.             vertexIndex++;
24.             triangleIndex += 3;
25.         }
26.         mesh.triangles = triangles;
27.  
28.         Vector3[] normals = new Vector3[vertices.Length];
29.         for (int i = 0; i < normals.Length; i++)
30.         {
31.             normals[i] = Vector3.up;
32.         }
33.         mesh.normals = normals;
34.  
35.         meshFilter.mesh = mesh;
36.     }
37. }
38.  

It's generating a mesh, but it's spiky and weird. I'm assuming the coordinates of the point cloud aren't alligned in a manner to be constructed from triangles like this... I read somewhere you can change the MeshTopology to "point" but i don't really understand how to do that, the SetIndices method confuses me

I tried generating a faux point cloud with CreatePrimitive and made a bunch of spheres to represent points, it worked just fine and was generating a decent mesh each time... But when i tried generating the tris it all fell apart lol

Sorry for the short question, i think that's all the info i can really give haha. Thanks in advance

 

Conversion of a point cloud to a mesh is a deep topic and is usually solved by a category of algorithms called Convex Hulls. You can't approach this naively and expect any kind of result, because triangles demand an exquisite ordering of their vertices, and yet all you can provide is a soup. Trying to shuffle them to satisfy some ad-hoc rules is both super-slow and prone to errors. So you need to be incredibly systematic about this, which is why this is a whole discipline of computational geometry.

 

Well, i'm not sure if it matters, but i'm not converting a point cloud as in a ply file or anything... Just generating a mesh based on a point cloud's coordinates. The API returns nothing but coordinates and colours.

 

do you want to reconstruct mesh (create surfaces from those points)
or just display points as is (but using Mesh gameobject, instead of say, particle system, VFX graph, graphics.drawprocedural..)

 

I am currently displaying the points via primitives(spheres), i would like to reconstruct a mesh if that's doable

 

that is quite difficult, i haven't seen anything suitable for that inside unity..

you could pass points into external tool, that does it in the background (or cloud)..
but even still, requires fiddling with settings to get ok results for each cloud, if they are different..
(and wouldn't be realtime anyway, if your data is streamed in?)

 

Realtime? Basically it's just going to be used in the editor for generating models based on a text prompt. What sort of tools is there for this?

 

some free tools: Meshlab, Cloudcomplare can do basic reconstruction,
but usually (more expensive) tools are used, like Recap, RealityCapture, Pointools.. although these are more for laser scanning work, not sure if they would even work well with small objects (that you generate..)

one other option is marching cubes, can look into that also, but often the quality is not too great.

if the object is convex shape, could just connect vertices from the middle.. but maybe wont look nice.

search for: unstructured point cloud to mesh, to see some efforts.

maybe you can post sample images that you are generating, to see how complex they are.

 

I saw meshlab, but the issue is i need to make the mesh within the editor, as it's not being saved to a file, i'm simply getting coordinates. I guess I said it's fine if I don't do it realtime, which is true, but i do need to generate it within Unity haha.

Well, here's one I made, it's a dog(with antlers?) I'm not sure why it's returning sideways... The weird white thing beside it was my attempt at generating a mesh

Marching cubes might be a solution, but I have no idea how to do that haha

 

point-e has other tools to convert into mesh, seems to use marching cubes
https://github.com/openai/point-e/blob/main/point_e/examples/pointcloud2mesh.ipynb

 

Can i run this with Python in Unity? Or do I have to make my own script for marching cubes?

No triangle data, Replicate returns 2 arrays, colours and coords.

Well, my script is creating triangles based on each point, i'm not sure what I could do different, it's waiting until the whole mesh is generated before generating the mesh.

 

if you are already calling the generation in the cloud, then could do that conversion at the same time there, and receive mesh file (or just the mesh data to be able to build triangles yourself)
(or have python run in the background to do it)

or could try that marching cubes alone inside unity, there's plenty of marching cube examples for unity.

this is one idea
https://hhoppe.com/proj/recon/

 

I'm calling the Replicate API. I don't know how to call it from the cloud through Unity. Is there a way i could get a server and host it and call it myself or is that a huge amount of work? haha

As for the Marching cubes... What am I looking at? This looks way beyond my programming level... but that doesn't necessarily mean i don't want to learn, if it's not crazy complex lol

 

yeah probably difficult to setup own server, and most likely needs gpu too..
where is that point-e running? does it have any extra options or params that you might be able to use?

look for unity marching cubes, there's lots of tutorials,
one example: https://polycoding.net/marching-cubes/part-2/

 

It's much easier than what you've asked for originally. Almost trivial, and I hope that doesn't sound too arrogant.

 

Well, I've some experience with Lightsail, but probably would be too hard.

I'm using https://replicate.com/docs/reference/http I don't think there's too many options, i just finally figured out how to change the output type to JSON(it was originally giving a link to an animated gif of a rotating point cloud lol) and the dev finally responded today when i asked them if there was any options, telling me to pretty much use the method i tried in the first post haha, so i assume there's no way to get a ply or obj or anything out of it.

I'll see about Marching Cubes, it all looks rather confusing to me, I'm looking at the scripts and confused as to where i'll be incorporating in the Vector3 array? lol, maybe I just lack experience in this field

 

Ah, after hours of trying to figure out how to convert a point cloud to a mesh, i found out Replicate has a text-to-3D generator as well... Now i just need to figure out how to change the output type lol

Thanks for all the help, i still wanna learn marching cubes one day...

 

I love Sebastian Lague I watched that video in the past, and just watched it again today lol