Scripting brushes for mesh manipulation

Hello, I've been endlessly looking for a basic tutorial onto how I could script a custom brush to use in the unity editor, but I couldn't find it, either here on the forums or on any other website.

Can someone point me out to some Unity classes that would allow me to do that? I was looking at the GridBrush class but I don't think that's quite what I'm looking for.

I'm creating my own terrain editor because unity's deafult terrain editor isn't good enough for very large maps, and my current goal is to be able to sculpt detail onto a Mesh (yes a Unity Mesh class instance) that i have procedurally generated, by creating a brush that modifies the height of the mesh's vertices.

I hope this is the correct section for this because I am starting to suspect that it isn't even possible through scripting in Unity.

Even just a couple pointers on how i could achieve my goal would be great, thank you very much for your attention.

 

Modifying a mesh instance (or even the terrain heightmap) in Unity is totally possible. I've done it myself many times!

The former gets a bit tricky to do in the editor because it's up to you to get the mesh data serialized right. There's lots of examples, but it's still a bit fiddly.

On the other hand you can modify the heightmap of a terrain all day long and Unity will take care of getting it out into the TerrainData asset for you, so that's slick.

If you want some tips on fiddling with meshes, google up procedural mesh generation, and google up editor scripts for how to edit and serialize assets confidently so you don't lose work done in your editor.

Same goes for terrain. The trickiest math in the terrain example is determining where in the heightmap you actually modify based on where the user clicked/dragged. There's examples but you kinda gotta take your time and go in small steps to make sure you have taken all the scaling into account. Who knows, there's probably complete examples out there for that too, take a good google around.

 

Thank you very much for replying, but I really can't find what I'm looking for anywhere, and it's weird cause I think it's something that comes to mind very easily when dealing with terrain.

I ran into some unity forum threads where someone was asking how to create custom brushes, and of course the answers were: make your own in photoshop and save them as .png, then import them and use them in the terrain brushes.

My goal is not to create an alpha to use on a unity terrain brush, my goal is to actually script a brush (not an alpha) that i can use on any mesh to adjust vertex positions.

I already have my procedurally made mesh, but now I want to be able to click on my mesh in the editor at a specific position and have the vertices that are close to the mouse pointer move, just as I was scuplting terrain with the default terrain editor or in Zbrush for example.

I was now getting a bit more creative and I found that it's possible to convert the mouse pointer's position on the object, from the editor camera, to a world coordinate by using a combination of ScreenPointToRay and Raycast (https://docs.unity3d.com/Manual/CameraRays.html). If I could do that, I could get the hit's world coordinates, and affect all vertices on the mesh in a certain radius.

But the problem is that Raycasts use colliders, and in my case I need a mesh collider because my mesh would be constantly changing due to sculpting, and I know that mesh colliders are static (calculated once) and are very taxing for the cpu to calculate. If I calculated a mesh collider for each frame that would probably send my pc straight to the grave...

So that means that I would have to implement my own raycast function so that it works without colliders (when encountering a mesh triangle or something), but I have no idea where to start from.

Again, my question seems to be very common, yet very uncommon based on the results I get when searching for it on the Internet.

I really have spent countless hours looking for at least something similar, but couldn't find anyone that has a tutorial for it or a thread that talks about it.

If you could link me to some examples you know about that would be great help.
All I need is a spark, I've got plenty of willpower to turn it into a wildfire.

Thanks again for the reply, but I think I haven't made my goal clear enough in my first post.

 

Not sure where you got this information but you may want to check your sources. You can change collider meshes all day long. You just have to be sure to assign the newly-changed mesh into the MeshCollider after you change it.

Another interesting assumption... Have you actually tested it?

Raycast at the mesh collider, get the point in world space, use the objects' transform to find your point in local space, collect up the closest points, do what you must deformation wise, update the meshes. I cannot imagine that being a performance bottleneck on any PC made in the past decade.

 

First of all, thank you for reading my reply, it was pretty long.

In this case I am the source, and yes I am no Unity veteran user as you can see
I just meant that each time a mesh changes you need to update the mesh collider and let unity recalculate the collider from the mesh, so that basically means that the collider is calculated once per mesh and that's why I said static, I said that the wrong way around.

Actually I have just tested it and I wasn't wrong unfortunately... takes about 2 to 3 minutes per collider mesh update for a 2049x2049 vertex mesh. Of course I didn't test the recalculation in the Update() method otherwise my pc would have died lol. My cpu is an i5-7400 (not a good neither a bad cpu, an average cpu).
Tested it on a 129x129 vertex mesh and still runs fairly bad.

Yes that's exactly what I had in mind, but I don't think that's an option unfortunately, due to the performance issues I had with just 1 mesh collider recalculation.
I did a quick search and it looks like others are tying to achieve this goal in the exact same way, but I don't think it's possible unless you have like 2000 cpus running at the same time lol.
This is where I found the answer: https://forum.unity.com/threads/how-to-update-a-mesh-collider.32467/

So I guess I either have to script my own collision classes/methods, or dynamic mesh sculpting is not possible with the tools Unity currently offers.

I'll keep thinking, if anyone comes up with an idea, I would be extremely grateful.