Oculus HandTracking with custom hand model

So I've been working on improving full body presence in VR

I added hand-tracking today and so far it looks like this:

I know this is super new so far but I'm wondering if anyone else has experience with connecting one mesh to another's?

I don't necessarily expect many people on the forums have experience with the oculus stuff specifically, but really, if I can find a way to connect my BASE MODEL's bones and joints to the OCULUS bones and joints, I imagine I'd be smooth sailing.

Does anyone have recommendations on how I might access a model's underlying rigging? Is it possible to modify this stuff at runtime? Hook up the "OnAnimatorIK" event to include just over 2 dozen additional points of rotation and position?

Maybe I should be posting on the animator subforum instead here.


If any of this is useful to anyone else for learning purposes the repo is sitting here for your enjoyment (or laughter):

https://github.com/AlexChesser/VR_ARMS_DEMO

 

if anyone is interested - I made some progress

Thumbs are a little off, but other than that it's in (IMO) pretty good shape.

Since this is built to support a MIXAMO character rig, you could pretty easily swap out any character model you want in this space. My next step is to do a bit of finesse on the thumb-tuning - maybe a tool to adjust the offset digits while in the app, since the work to fine tune rigging adjustments is pretty significant.

 

Hi alexchesser,

Do you plane to explain how to integrate finger IK with hand tracking ? just test the VR arms demo and work fine like Humanoid VR but Hand tracking very interesting.

Best,

 

@ibompuis sure! I'll try. Let me know if you want any clarification after too.

First of all, recall that this is working on the quest. So the hand tracking you see in the video above with "no finger rigging" is the code basically as it comes out of the box from Oculus after you follow their docs.

https://developer.oculus.com/documentation/unity/unity-handtracking/

For the NEXT part which is moving my custom-model's fingers in time with the OCULUS fingers, I had to put a function in LateUpdate. LateUpdate as you may or may not know happens within the Unity frame's lifecycle AFTER animation has happend but before the frame gets written to the screen. By moving the custom model's finger transforms in LateUpdate, the changes you make don't get overwritten by your model's AnimationController.

Now -
Conceptually what I've done is I've hard-coded an object called an HTFinger (ht for hand-tracking) it has a transform and an offset. It's actually a bit of a bad name since I should probably call it a HandTrackingFingerBone but we'll live.
https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/HTFinger.cs

Then I have an object called a HandTrackingFingerMap
https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/HandTrackingFingerMap.cs
this is a hard-coded list of all of the fingers that you can find in an OVRSkeleton (the oculus finger bones object).

Within the editor, I add a HandTrackingFingerMap to each of my custom-model's HAND transforms. (strictly speaking they could be empty game objects outside of my model, which would probably be slightly better organization, but I digress.

Each BONE in the custom model, gets mapped within the hand tracking finger map.

THEN I put my FingerIK on my character model as a sibling of their Animator
https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/FingerIK.cs

You'll notice FingerIK in the editor takes a reference to the Oculus object for left and right hand as well as the HandTrackingFingerMap for the left and right hand of my custom model. *ACTUAL* IK arm position is handled elsewhere as if it was regular unity Mechanim IK.

Finger IK (in late update) then traverses each bone in the OVRSkeleton and positions the appropriately mapped HTFinger at the same position. *FINALLY* because your custom model's transforms are not likely to have the exact same forward direction as the OVR transforms, you have to ROTATE your custom fingers by an additional offset/vector3

If you're like *made of money* and have access to modelers and riggers to do custom 3d you could actually work backwards from the default provided oculus hands and build custom meshes to swap out (which is recommended by the real pros) but the offset step works OK.

Hopefully that's useful to you?



If you use the technique I'd love to know what you eventually built.

 

Sorry @alexchesser , missed your post... thanks for sharing tips and code, definitivly trying this tomorrow ! and keep in touch when I'v done something interesting

 

Your Demo looks quite amazing. I quickly tested it, but unfortunately, here is what I get in the HandTest_Chesser scene:

I'm able to move the fingers, but as you can see they are all messed up...
I didn't change anything on your project, except maybe I'm not using the same version of Oculus Integration as you do... I had to update it, otherwise my hands weren't tracked at all...

Any Idea on what may be happening ?

Thanks in advance for your answer !

 

Got to say ... I'm loving these screenshots/videos . It's like you're making "alien hand simulators" ... impressively creepy

 

xD
This is not intended of course. So my question remains...

 

Hello, try to modify rotation on HandTrackingFingerMap script attached to LeftHand and RightHand

Very nice demo alex ! (thumbs are little off yes but maybe rotation modification ?)

 

I think I have made some progress by adjusting the values of each finger offset based on the project from @alexchesser.

As the screenshot shown here, the nails on the thumb are facing "up" instead of forward (towards the camera). I will list the rotations for each finger on each hand below, they are based on the latest version of Oculus Integration (25.0).

Left Hand
thumb x: -180 y:0 z:90
index x: -90 y:90 z:0
middle x: -90 y:90 z:0
ring x: -90 y:90 z:0
pinky x: -90 y:90 z:0

Right Hand
thumb x: 0y:0 z:-90
index x: 90 y:90 z:0
middle x: 90 y:90 z:0
ring x: 90 y:90 z:0
pinky x: 90 y:90 z:0

The base of the thumb still bulge outward, which I think can be tweaked slightly by adjusting the rotation on thumb1.
Lastly, thank you alexchesser for providing the project on github.

 

Awesome! How to do this with touch controller hands and not actual hands?

 

I've been trying all day to get my XR rig to accept the OVR oculus prefabs but cannot get the hands to animate. I have a turn based controller set up and I can change the meshes and the build is clean and both trigger and grip work on both controllers with interactables however, I cannot get the finger to point or the fingers to cup in when I go to grab the objects in my scene. The raycasts work as intended but I would really like to get the hands moving along so i can move on with a fully functional character "rig" Any ideas? I'm using the XR input toolkit with the OVR handmeshes for quest 2 but can't get them to move.

 

i am at the same spot that you are in ... but the docu seems outdated, as the Unity 2021 or OVR prefabs differ in content and scope/function. It is astonishing that they do not updat the docu for 3 years.... hmmmm

 

@alexchesser Hello, I am traying to implement hand tracking with custom hands and using this package https://github.com/AlexChesser/VR_ARMS_DEMO, after adjusting the fingers offset angles everything works fine. Now the problem is that I tried to change avatar and use a female one downloaded from mixamo as well. I applied the same scripts and animators of the previous avatar but I am able just to move the avatar's arms but not the fingers..
Do you have an idea about how to solve this issue?

Thank you very much!

 

Ok so, I'm trying to mix this solution with the FinalIK plugin on the asset store to get a full-body IK avatar with finger movement using the HandTracking feature on the Oculus Quest 2.

Everything seems to be working ok, fingers are following HandTracking values coming from the Quest2, and the avatar's body and legs are moving accordingly to the FinalIK plugin.

Now, the hands rotations are completely offset from the orginial position. I tried everything to make them look better but it's a mess.

Idk what else to do what I guess is something related to the IK Target_LH and IK Target_RH targets rotational values.
Still did not get even closer by replacing the values manually. Any clue about this? @alexchesser

See image below: Black hand is the one coming from the Oculus2 HandTracking. The messed one is the one I'm trying to match.

 

Hi everyone, I just realized that the weird position of my hands is starting just after the foreArm, around the palm. In the example from @alexchesser the hands are at least ok but the fingers are the problem, while with my hands it seems to be a problem with the hand itself, not only the fingers. Any clue about this? I attached the avatar I'm using in case a hero dev wants to test it.

 

The Oculus hand tracking system has unusual bone orientations. On the right hand the "right" axis points forward (towards the fingertips) and on the left hand the "right" axis points backwards (towards the wrist). So if you're trying to set the position of your bones based on the local position or local rotation of the bones in the Oculus hand mesh you're going to get weird results if your bone orientations have the forward axis pointing toward the fingertips.

 

Sorry to necro, but I had to address this for a project and want to help anyone visiting in the future.

The hands use a different orientation system to Unity. From memory they use the one OpenGL uses (right hand?). In order to rotate the fingers properly, you need to translate the coordinate system of the hand using a rotation matrix.

How? No idea. We had a math dude do it, but that was the explanation.

 

A problem seems to be that if any of the bones in OVRCustomSkeleton is not assigned, the fingers don't work at all. I assigned dummy gameobjects to the empty bones and it suddenly worked. (Though I'm still experiencing other issues...)