Assets Procedural Hand Grasping of Arbitrary Mesh

Hey guys,
For a while now Ive been working on a way to procedurally generate a hand pose that approximates a human's grip when grasping arbitrary objects. Basically, given a palm position and rotation in space along with a mesh, the goal is to determine the finger bone's orientations to best grasp that mesh. Think Lone Echo's hand tech but extended to grasping small objects which the fingers can wrap fully around.

Heres what Ive got so far:

As you may notice, there are still some small bugs to work out but in general, its working quite well and at very reasonable speeds.

The main guts of the algorithm is a combination of a custom IK solver for 3 joint chains that runs in constant-time, and a method for searching the mesh for "ideal" finger end-effector points to feed to the IK solver.

I figured this might be very useful for hand presence in VR when using tracked motion controllers, but it has applications in a few other domains as well. If this is of interest to anyone, please let me know as Im considering releasing it as a Unity Asset if so.

 

Updated video with rigged mesh hands

 

Rig Setup Video Demo

 

hello!
This really interests me! can you provide a scene/github repository so i can reproduce what you have built? Thanks!

 

+1 nice!

would buy this from asset store. (and hopefully it would be pretty much plug and play for regular steamvr controllers?)

 

Glad to hear youre interested!
The set up would be fairly simple:

-One script would need to be attached to any object that youd like to be able to grab.
-A demo prefab hand (left and right) will be provided, hopefully with a rigged hand mesh. This should be attached to the VR controllers
-What would be required on your part would be to call a single function on the hand prefab when the grab button of your choice is pressed, passing in the gameobject being grabbed.

This works well with systems like VRTK but I will also be providing a simple interaction system with the asset.
Im also planning on including a demo scene for SteamVR but I still need to work out how to resolve the dependencies on an external plugin as I cant include the SteamVR plugin in my asset.

Let me know if you have any questions or feature requests!

 

Hello!

I have a couple of questions for you:

1. Do you apply a shader to highlight the grabable object when the hand is getting close to it?
2. Do you add an offset when picking up an object (and parenting it to the hand I guess) to allow a realistic grabbing motion?
3. Basically when the hand gets close to an object you set the object as the IK's target for each fingertip? I don't understand how to set multiple targets since each fingertip needs to have a different target on the object's mesh right?

Thank you!

Alex

 

1. The mesh highlight shader is separate from the core asset and is similar to the functionality in VRTK. I do plan to include it as part of my demo interaction system bundled with this Hand IK asset, but those systems are functionally unrelated.

2. No offset is applied when grabbing: the IK solver takes the current hand position and rotation relative to the grabbable object and wraps the fingers around the object. The parenting or attaching of the object to the hand is handled by the interaction system and in my case, it maintains the relative transform between the hand and object at the moment the grab is initiated. The IK solver is able to update in real time so it would be possible to animate the grabbed object towards the hand and have the fingers adjust during that process.

3. The IK targets for each finger is automatically calculated for the object by my custom solver based on the mesh and wont require you to do anything manually. Simply attach a script to any grabbable object with a mesh and call a function on the hand solver when the object is in grabbing range and the finger poses are calculated and applied.

As for posting the source publicly, im planning on releasing this as a Unity store asset so I wont be open sourcing the code. I can post a demo .exe if anyone is interested in trying it.

Hope that answers your questions!

 

Great thanks! so the only remaining question is when you are planning on releasing it!

 

This Looks excellent, thank you for using my First Person Hands Asset in this demo/package. I agree, would love to see this on that Asset store. Amazing solver especially for VR!

 

This looks amazing and can be used for many VR applications and games! Are you still planning to release this as an asset to the store?

 

Glad your interested! Its still a plan to release but my timeline has been delayed lately due to this whole global pandemic stuff... I will definitely update this thread when I have more information on release.

 

Will gladly buy this.

 

We need this man this is super cool! Any updates on the asset?

 

Sorry guys, its going to be a while before I can devote any time to this.
The current prototype is functional and most of the code is final. The last steps needed are full documentation (the part I dread...) and making as many optimizations as possible since Ive had a lot of questions about its viability on mobile platforms like Oculus Quest.
Nows the time for feature requests since once I dive back into this Ill be willing to add whatever makes it as flexible and ubiquitous as it can be!
Thanks again for staying interested in my work

 

Sweet! Definitely can't wait for this it looks amazing!

 

Hi,
Great work! I just wonder how is it going with release? I was about to develop my own physical grasp using leap motion (ultraleap) and now I found this. This is amazing! Will you include source code as well? What HMD are you targeting? Will it be extensible and open for modifications? Thanks and good luck with development!