Question a complicated mesh cut ?

Hello everyone,

I hope this message finds you well. I am working on an academic project and I could use some assistance from you guys.
Specifically, I am creating my first Unity project which is a , and I am struggling with a particular issue.

I need to create that includes:

-A curved incision only applied on human flesh surface(mesh surface)
-Variable depth based on user interaction
-The ability to insert objects into the incision

I found a video demonstrating exactly what I am trying to achieve, but unfortunately, the source code is not available and the owner has not responded to my requests for assistance.

I am hoping that someone in my network might be able to help me with either the source code or a roadmap on how to achieve this result. Any advice, suggestions or insights would be greatly appreciated. If you are able to help, please feel free to share your thoughts in the comments or send me a private message.

Thank you in advance for your help, and please note that I have a deadline for this project.

 

Do you have any solid C# programing, or game dev experience?

If not, this will be a tough problem to tackle for you.

Being realistic, this maybe not something you want even dive in. There is various subjects to be addressed here and is pointless discussing it, withouth knowing first, what you know.

How much time you are willing to put into the project?

You should describe, which part of the problem you are struggle specifically.

Regarding road map, you pretty have that described on the vid. Write it down and split it in smallest problems components.

You may also consider hire someone to write such project. But then you need to consider, what you expect of the project.
So you are not left with bunch of code, which not knowing what to do with it.

 

It seems that you're trying to build a surgery sim from scratch.

If you have no previous SERIOUS experience with programming with focus on geometry, implementation of something like that is an extremely difficult thing to do.

Basically you'll be implementing something similar to booleans while simultaneously dealing with soft-body sim. It is a royal pain and people do not normally tackle that.

A rough idea of implementing that would be:

* Given mesh, subdivide it into tetrahedras . I think CGAL can do something like that, but free version is GPL only meaning you cannot use it in unity project. Tetrahedras are one way to keep the mesh soft-body compatible.

* Store previous and current position of the knife, based on them implement a boolean cut, and retriangulate the mesh if neceessary.

It is an extremely annoying thing to do, and likely it will be running slowly in VR.

----

For a practice, try implementing this:

* Procedurally generate a primitive.
* Cut it into two halves with an infinite plane.
* Allow repeated cuts of cut parts.

This was implemented in Cooking Simulator VR, by the way.

* Now make the cut parts soft bodies and allow player to slice them while pressing them with a rigidbody brick or something like that.

This will get you halfway to the goal, but won't allow you to implement a surgery sim. Because a knife is not an infinite plane.

 

I am actually pretty experienced with programing with multiple languages, C# I know all the basics and what I need to know to get me going, I have no game dev experience so I struggle with its principales, laws and constraints. I m willing to put on effort and time to achieve my goal and i do have a plan and expectation for this project .
In my case time is kind of crucial so if paying someone a reasonable price is an option then let's talk business. would be a great experience to understand this feature s needs and break its difficulties but if it's as hard as you re saying then time is ticking.

 

What's the deadline?

 

couldnt you just create a texture mask which adds opacity and exposes an inner mesh? Thats beginner friendly and would look the same as whats shown in video.



-A curved incision only applied on human flesh surface(mesh surface)

• human flesh = identify material types
• curved incision - what does this mean exactly? That we can freeform draw the incision?

-Variable depth based on user interaction

• if you have parts of the inner model identified as layers you can decide how many layers to expose with the mask

-The ability to insert objects into the incision

• this can simply be faked by intersecting the model. You can figure out where the surface of the model is and decide how far to intersect the object based on the incision depth

edit: err, probably not beginner friendly. But I bet you can find youtube tutorials for creating a texture mask at runtime.
The rest of the stuff will still take some work but any competent unity programmer ought to be able to do stuff like that in reasonable time frame. You can simulate the universe but games are usually made to be simple because requirements change often so you don't want to waste time building something more complicated than necessary to trick the viewer into believing whatever your making. It just has to look convincing, not actually simulate anything.

 

First of all, thank you for your answer it was really informative and helpful.
I am building a surgery sim and I don't have a SERIOUS previous experience with geometry tbh, but i believe that Im capable of breaking into it with the right guidance because i didn't find any good explanation for what i need yet .
I will look into implementing booleans, soft-body, CGAL and retriangulating the mesh.
Concerning the Cooking Simulation it's basically cutting things with no complications which i already have such a result but I'm struggling with how can I make it a curved cut , show it's depth and objects can pass through the cut .
In case this would make my VR sim run slow what do you recommend as solution or alternative principle.

 

Deadline is about two weeks from now tho. I might squeeze an additional week if I was close to a complete prototype so it depends.

 

Can you procedurally generate a polygonal torus, turn it into a unity mesh, slice it with a plane, separate the resulting parts into two separate meshes, and then ensure resulting parts have no holes and are watertight?

That's the level of this task you want to implement. Actually, the level you want is higher.

The cut is an extruded curve (catmull rom spline would do), and you'd remove it from the mesh using a boolean.

Demonstrating depth is a shader problem.

In the video you posted, the mesh most likely is not being cut and instead the person is employing some sort of stencil buffer trickery. It is likely unusable in your case.

Booleans on polygonal geometry in general are implemented t his way:
* You intersect two meshes.
* Find intersection CONTOURS.
* Use those contours split meshes into four parts. For both bodies you get a part that's outside of the other mesh, and the part that is inside.
* From that you assemble the resulting mesh. You might need to flip the normals for "inside" parts.

For example, if you combine A and B, then if you're implementing a union, resulting mesh will contain OUTER part of A and OUTER part of B, welded together.
If you want to carve A using B, then resulting mesh will contain outer part of A combined with inner part of B (meaning it is inside of A) with normal flipped.

Here's the problem. This thing will absolutely not be a stable reliable operation. It can generate degenerate t riangles and t hat will break the algorithm.

And because you have a soft body, it'll be hard for you to implement it using voxels or something similar.

Voxels with dual contouring are easier approximation of booleans, but you will not get scalpel level precision you need.

That's just booleans alone. However, because you want to make a surgical sim, you'd need to turn your meshes into soft bodies. And for them to become soft bodies, you can't quite represent them with triangles, and want to use simplexes/tetrahedra instead. That will allow you to make a basic soft body sim, where you can treat the body as a collection of particles connected with springs.

Here's another fun part to think over.

I have never seen software that implements the thing you're asking for. Meaning interactive free form cut of a piece of flesh with a knife with no restrictions. Cooking simulator does not have that. Metal Gear Rising does not have that.

It is also possible to simplify things If whatever it is you're trying to cut can be generalized as an uv surface, the task becomes easier. Because in this case cuts can be processed in two dimensions. It'll be still quite annoying to deal with,though.

 

It is very short deadline for such thing like this, even for an experienced Unity dev. Specially you need VR and testing.
If you never really touched of cutting mesh effect before, doing it by yourself in such short time will be unrealistic.

Specially you may have technicalities, which has nothing to do with initial OP problem.

Is this your course assignment by any chance?

We have commercial forum section, you can post your project offer there.

 

Yikes. You might be able to hack something together with an asset like Dynamic Mesh Cutter. I don't think it can make variable depth cuts but I don't see why you couldn't have multiple meshes layered on top of each other to give the illusion of depth.

https://assetstore.unity.com/packages/tools/modeling/dynamic-mesh-cutter-208384

 

I didn't really get what you mean by the texture mask but is it like I have to prepare a pre-cutted mesh so an incision is already on it but render it on action by replacing the original mesh with pre cutted one.
that would be a fake script which i do not want to implement since I am aiming for a realistic surgery.
I will explain it more by answering each point .

human flesh i meant a soft body mesh and yes you are right it would be a freeform draw incision since i want a precise and realistic sim.

I basically need to make an incision just on the skin and it's small so it won't expose any inner model then i need to insert a surgical tool in that incision . so depth is needed to validate if the surgeon (user) didn't touch any vital organs (means not too deep) .

I totally agree with you on this one, you are correct.

I really appreciate your help and I hope i understood what you are saying about the texture mask, but It's a POC so I'm trying my best here to find out if this is doable and can really be made with a reasonable time frame as you said . in case i couldn't find a way i would probably try some tricks to make it seem as convincing as possible.

 

I do understand the situation I'm in thats why Im considering paying someone who's capable of delivering this feature, everything is negotiable (time, constraints and technicalities ) so if you are interested and can offer help sir. please feel free to DM me.

 

Two weeks is not a "reasonable time frame" given some of the questions you're asking. Boolean operations are not exactly simple (you'll find many professional pieces of software still struggle with them) and you're really presenting yourself as at a beginner to intermediate skill level. You especially aren't in a good position to make a surgery sim.

What in the world has put you in a situation where you have two weeks to implement something like this?

 

Since I dont intend to visually show the depth rather I just need it as a check on the interaction made by the surgeon if it was deeper that it should be, it's like a correction system to precisely evaluate the move .
So depth Im wondering how can I get the depth of the cut ?

 

You do make a point on how i got myself into this, which i do wonder the same, but this project was a 2 months duration but I got lost on searching about this matter i did not know how complicated it's until weeks of searching with no results, there are many other features that i spent time on too, looking for assets, also i was hesitant to post about it at first than time was tightening up so i had no other choice.
this project is a POC as i said so it s what it is.
Im working with a group of my friends on this , we are all new, i think we are doing pretty good as beginners in this but we kind of lack some guidance which what made me reach out the community so all your comments are really appreciated.

 

The normal way would be to compute the geometric (volume) intersection between your cutting tool and your body/object, not an easy task. An approximate way could be if you can represent the cutting tool as a thin box or cylinder (think of a laser cutter) and your 3d object as an ensemble of boxes (you can voxelize it with some algo or maybe in blender and under certain conditions, i.e. mesh is static, this can ease your algo), then you can compute the volume intersection between each box and the thin box representing your tool (each in the same space that is rotated and traslated according to local to world matrix), the sum of these volumes would be proportional to the depth you are looking for.

 

This is what I was thinking from watching the actual video posted, except I was initially thinking of using stencils to achieve the same result, though there might be issues with that approach.

In the video it states

• 'The geometry of the object being cut is not modified'
• 'Instead a 'cut mesh' is procedurally generated, with the points and depth of the cut'
• 'A ceiling mesh is generated to allow the cut to be visible underneath the object using passthrough rendering'

I'm not sure what 'passthrough rendering' means in this context in terms of the method being used, hence my initial thought of using stencils, though a texture mask would also fit that description.

In fact this whole approach reminds me of the shader cross-section clipping technique so maybe that is a potential avenue to explore. Though whenever i've used it in the past it was always limited to just a couple of planes or basic primitives at once. Looking at the end of the video this approach may not be suitable to achieve the overall complexity in the number of cuts shown.

These sort of approaches greatly simplify aspects of the system as there is no need for complex boolean operations, which can lead to all manor of complications once you aren't dealing with basic primitives due to floating point inaccuracies.

If it were me, i'd probably start with a stencil geometry approach as I feel it might give a bit more control and precision over a texture mask and simpler than shader clipping, especially after viewing the complexity of the final model being 'cut' ( human body ). The only issue is I'm unsure what level of support VR systems have for stencils.

 

@Noisecrime

that is interesting, I am not familiar at all with stencils or the shader cross-section clipping.

I looked at the video again and I think they are doing similar to what I suggested but for creating the "cut geometry" I think its just adding some angled planes along a curve (or spline, not sure what standard term is).

I could be wrong of course but that would be my go to because its all pretty easy stuff you can find youtube tutorials for, it would look the same as the video, and there is a deadline. Though for a beginner I don't see even the easiest version of this completed in two weeks. Who knows though, would be great to see a follow up!


@saiflayouni

sorry, just saw your response.

First, you aren't going to even be able to do what I am suggesting in two weeks. I doubt you could do 50%. I could be wrong though!

What I am suggesting I feel pretty confident is what is being done in the video. There is no boolean operations, runtime geometry manipulation, or anything "complicated". You've mentioned a realistic simulation and soft body physics but there is nothing like that in the video, and there is virtually no chance of doing anything like that for you in the next five years so it seems like just pie-in-the-sky dreaming to me. If there is any chance of finishing anything (on a deadline!) it would be best to focus on how to actually do that instead of dreaming of something that isn't going to happen.

First you have to identify what reality actually is. So the facts are:

• you are not the elite student who has already been doing this non-stop for years before class and will impress with the professional grade simulation
• you are not a large studio with several experts who could actually realistically think of doing something like that
• you are a few beginners in an academic setting with no time and you spent most of it already unproductively
• you don't know basic game dev terminology

So that's what the situation is, right? So then you need to focus on solutions that actually fit the situation.

It's a few task:

• identify one material over others
• create a texture mask at runtime (google it) with parameters for the brushes diameter and depth of application
  
• create a static mesh that is organized into layers
• create procedural mesh along a runtime generated spline (this is same technique used to make roads in city-gen games. very youtube-able)

the most complicated part would be reading data back about the "depth" of an incision for putting objects in it, though if you have the procedural mesh you could probably just look for the opposite side of extents.


It is all stuff you can learn from youtube in a few weeks, however putting it all together and ironing out bugs can stack up to months. For a beginner I'd really expect more like a year if you were super diligent.

If you procrastinated this far and left yourself with a fraction of the available time I don't know what to tell you. You wont even be able to hire an expert who could create this in two weeks using the simplest version possible.

 

The simple solution is to use heightmap and tesselation, and use pixel shader blending of texture for "reveal of interior". It doesn't seem they actually need segmentation.

https://docs.unity3d.com/Manual/SL-SurfaceShaderTessellation.html

He draws to a grayscale texture using UV every frame and use that as tesselation. You can fake interaction the same way, detect the hit position, then offset contact point by the heightmap values.

 

@neginfinity
That is a very interesting and well explained approach tho. I did looked into it and I found these two libraries that I belive it can help me a lot throught this but I'm wondering if this can work with soft-body or is it possible to make it compatible.
Github repo 1
Github repo 2 (this one looks more hellpfull in my case)

@Noisecrime
Stencil seems to be very complicated and risky to be honest and I dont think it would actualy perform well on VR ,it is also very low on ressources to learn more about it (on VR) so I will skip that one . but thank you so much you comment was very Informative.


@neoshaman
That actualy does look like the simplest way to do this but can you take a look at this asset I found it has some pretty cool stuff i believe I can implement and customize it so it fit my needs, such as :
* Raycast Event ( 16:50 )
* Surface Tracking ( 17:47 )
please take a look at it and give me your feedback thanks a lot .

@BIGTIMEMASTER
I think you are right we did underestimate the matter but that doesn't make it impractical so we are getting there anyways but I do appreciate your honesty ,I actually understand the topic a lot more so thank you .

this does make a lot of sense

IMPORTANT
I'am looking into that but guys can you actually take a look on this good stuff i found on this post . I think this is so close to what i need it has forceps , scalpel and even mesh sewing .
it has like this soft body effect kind of slimy flesh and i tried it on a seperate project and it appears to work fine . (not on VR yet)
I still have to deal with :
* depth: I still dont really have a complete understanding to it's factors. most likely it would be dropped as a feature.
* inserting a surgical tool : object intersection which can be faked i guess.

The Algorithms & Prototype used on this project :

• Creating the initial skin mesh (a set of triangles)
• Adding an additional point to the skin mesh (additional triangles) to allow grabbing and dragging an arbitrary point.
• The actual movement behavior of the skin after being pulled – treating the triangle’s edges as springs using Hooke’s Law.
• Cutting the skin when using the scalpel tool.
• Tearing the skin when it’s pulled hard.
• Removing redundant points.

for more info : https://previewlabs.com/surgical-simulation-prototype-for-skysurgery/
https://www.precisionostech.com/
There is also this github repository which is mentioned in this repo but I didn't find a good documentation on how to use it but it seems pretty usefull.

 

You should narrow scope to very basics.
Basics to the level like, can you "draw" on mesh, using mouse. Nothing fancy. Your time is short, plus depending where are you living, there may be easter holidays coming.

Instead you are looking into even more complex solutions, for which you have no time. They are interesting nevertheless, but it is not about what the assignment should be. You meant learn things, while producing it, not copy paste ready project.

I advise you don't even touch DOTS, as this will consume you weeks alone, to understand how DOTS works. Not to mention manipulation of the mesh vertices etc. and adapt it to your needs.

Also don't do your work yourself. Ask other team mates to help you out. As far I can read, you spending time researching, and you want to do all by yourself? Question is, where your teammates fit in this project? What they doing and what are theirs responsibilities?

 

You can probably fake it per pixel (displacement map) or in vertex shader, look for interactive grass. That will save you time. The beauty is that it would make you handle everything with texture, which is magnitude of order easier to use than mesh. Especially on a tight schedule.

For depth, it's a simple matter of mapping the max depth/255 to interval, for example, from 0 to 5% that's the epidermis, from 5% to 20% that's the dermis, then you have subdermis tissues past that point. For each interval, you simply sample the right texture. if you want bone to poke out, do max(cut map, bone heightmap value).

I'm not sure tesselation is performant enough for VR, no idea, so it's better to be safe, if you have time, try a simple but arbitrary tesselation shader first to evaluate, it's probable that some hardware don't even support it. But if you follow that path, having a texture only fallback using the same idea (still having displacement and heightmap to reveal tissues) can help a lot not having retooling.

Yagni:
https://forum.unity.com/threads/get-uv-coordinate-from-collision.12720/

I feel this is a case where asset might hinder you because not specific enough.