Search Unity

  1. The 2022.1 beta is now available for testing. To find out what's new, have a look at our 2022.1 beta blog post.
    Dismiss Notice
  2. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Oculus HandTracking with custom hand model

Discussion in 'AR/VR (XR) Discussion' started by alexchesser, Jan 16, 2020.

  1. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    So I've been working on improving full body presence in VR

    I added hand-tracking today and so far it looks like this:



    I know this is super new so far but I'm wondering if anyone else has experience with connecting one mesh to another's?

    I don't necessarily expect many people on the forums have experience with the oculus stuff specifically, but really, if I can find a way to connect my BASE MODEL's bones and joints to the OCULUS bones and joints, I imagine I'd be smooth sailing.

    Does anyone have recommendations on how I might access a model's underlying rigging? Is it possible to modify this stuff at runtime? Hook up the "OnAnimatorIK" event to include just over 2 dozen additional points of rotation and position?

    Maybe I should be posting on the animator subforum instead here. :)


    If any of this is useful to anyone else for learning purposes the repo is sitting here for your enjoyment (or laughter):

    https://github.com/AlexChesser/VR_ARMS_DEMO
     
    shumido likes this.
  2. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    if anyone is interested - I made some progress

    Thumbs are a little off, but other than that it's in (IMO) pretty good shape.

    Since this is built to support a MIXAMO character rig, you could pretty easily swap out any character model you want in this space. My next step is to do a bit of finesse on the thumb-tuning - maybe a tool to adjust the offset digits while in the app, since the work to fine tune rigging adjustments is pretty significant.

     
  3. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    76
    Hi alexchesser,

    Do you plane to explain how to integrate finger IK with hand tracking ? just test the VR arms demo and work fine like Humanoid VR but Hand tracking very interesting.

    Best,
     
    xmanlch likes this.
  4. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    @ibompuis sure! I'll try. Let me know if you want any clarification after too.

    First of all, recall that this is working on the quest. So the hand tracking you see in the video above with "no finger rigging" is the code basically as it comes out of the box from Oculus after you follow their docs.

    https://developer.oculus.com/documentation/unity/unity-handtracking/

    For the NEXT part which is moving my custom-model's fingers in time with the OCULUS fingers, I had to put a function in LateUpdate. LateUpdate as you may or may not know happens within the Unity frame's lifecycle AFTER animation has happend but before the frame gets written to the screen. By moving the custom model's finger transforms in LateUpdate, the changes you make don't get overwritten by your model's AnimationController.

    Now -
    Conceptually what I've done is I've hard-coded an object called an HTFinger (ht for hand-tracking) it has a transform and an offset. It's actually a bit of a bad name since I should probably call it a HandTrackingFingerBone but we'll live.
    https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/HTFinger.cs

    Then I have an object called a HandTrackingFingerMap
    https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/HandTrackingFingerMap.cs
    this is a hard-coded list of all of the fingers that you can find in an OVRSkeleton (the oculus finger bones object).

    Within the editor, I add a HandTrackingFingerMap to each of my custom-model's HAND transforms. (strictly speaking they could be empty game objects outside of my model, which would probably be slightly better organization, but I digress.

    Each BONE in the custom model, gets mapped within the hand tracking finger map.

    THEN I put my FingerIK on my character model as a sibling of their Animator
    https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/FingerIK.cs

    You'll notice FingerIK in the editor takes a reference to the Oculus object for left and right hand as well as the HandTrackingFingerMap for the left and right hand of my custom model. *ACTUAL* IK arm position is handled elsewhere as if it was regular unity Mechanim IK.

    Finger IK (in late update) then traverses each bone in the OVRSkeleton and positions the appropriately mapped HTFinger at the same position. *FINALLY* because your custom model's transforms are not likely to have the exact same forward direction as the OVR transforms, you have to ROTATE your custom fingers by an additional offset/vector3

    If you're like *made of money* and have access to modelers and riggers to do custom 3d you could actually work backwards from the default provided oculus hands and build custom meshes to swap out (which is recommended by the real pros) but the offset step works OK.

    Hopefully that's useful to you?

    :)

    If you use the technique I'd love to know what you eventually built.
     
    TigerHix, shumido and ibompuis like this.
  5. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    76
    Sorry @alexchesser , missed your post... thanks for sharing tips and code, definitivly trying this tomorrow ! and keep in touch when I'v done something interesting
     
    alexchesser likes this.
  6. remiC3D

    remiC3D

    Joined:
    Dec 15, 2017
    Posts:
    30
    Your Demo looks quite amazing. I quickly tested it, but unfortunately, here is what I get in the HandTest_Chesser scene:
    upload_2020-5-24_19-35-14.png
    I'm able to move the fingers, but as you can see they are all messed up...
    I didn't change anything on your project, except maybe I'm not using the same version of Oculus Integration as you do... I had to update it, otherwise my hands weren't tracked at all...

    Any Idea on what may be happening ?

    Thanks in advance for your answer !
     
  7. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    1,389
    Got to say ... I'm loving these screenshots/videos :D. It's like you're making "alien hand simulators" ... impressively creepy :)
     
    harleydk and jipsen like this.
  8. remiC3D

    remiC3D

    Joined:
    Dec 15, 2017
    Posts:
    30
    xD
    This is not intended of course. So my question remains...
     
  9. SIgmar_Storm

    SIgmar_Storm

    Joined:
    Jul 3, 2020
    Posts:
    1
    Hello, try to modify rotation on HandTrackingFingerMap script attached to LeftHand and RightHand :)

    Very nice demo alex ! (thumbs are little off yes but maybe rotation modification ?)
     
  10. Imbazephyr

    Imbazephyr

    Joined:
    Jul 2, 2015
    Posts:
    8
    I think I have made some progress by adjusting the values of each finger offset based on the project from @alexchesser.
    upload_2021-2-27_22-15-57.png
    As the screenshot shown here, the nails on the thumb are facing "up" instead of forward (towards the camera). I will list the rotations for each finger on each hand below, they are based on the latest version of Oculus Integration (25.0).

    Left Hand
    thumb x: -180 y:0 z:90
    index x: -90 y:90 z:0
    middle x: -90 y:90 z:0
    ring x: -90 y:90 z:0
    pinky x: -90 y:90 z:0

    Right Hand
    thumb x: 0y:0 z:-90
    index x: 90 y:90 z:0
    middle x: 90 y:90 z:0
    ring x: 90 y:90 z:0
    pinky x: 90 y:90 z:0

    The base of the thumb still bulge outward, which I think can be tweaked slightly by adjusting the rotation on thumb1.
    Lastly, thank you alexchesser for providing the project on github.
     
    Ferdowsur likes this.
  11. Proto-G

    Proto-G

    Joined:
    Nov 22, 2014
    Posts:
    178
    Awesome! How to do this with touch controller hands and not actual hands?
     
  12. MarkSharpe

    MarkSharpe

    Joined:
    Feb 3, 2021
    Posts:
    18
    I've been trying all day to get my XR rig to accept the OVR oculus prefabs but cannot get the hands to animate. I have a turn based controller set up and I can change the meshes and the build is clean and both trigger and grip work on both controllers with interactables however, I cannot get the finger to point or the fingers to cup in when I go to grab the objects in my scene. The raycasts work as intended but I would really like to get the hands moving along so i can move on with a fully functional character "rig" Any ideas? I'm using the XR input toolkit with the OVR handmeshes for quest 2 but can't get them to move.
     
  13. ulthien

    ulthien

    Joined:
    Apr 25, 2020
    Posts:
    12
    i am at the same spot that you are in ... but the docu seems outdated, as the Unity 2021 or OVR prefabs differ in content and scope/function. It is astonishing that they do not updat the docu for 3 years.... hmmmm
     
    AldeRoberge likes this.
  14. Fabiana96

    Fabiana96

    Joined:
    Mar 8, 2021
    Posts:
    1
    @alexchesser Hello, I am traying to implement hand tracking with custom hands and using this package https://github.com/AlexChesser/VR_ARMS_DEMO, after adjusting the fingers offset angles everything works fine. Now the problem is that I tried to change avatar and use a female one downloaded from mixamo as well. I applied the same scripts and animators of the previous avatar but I am able just to move the avatar's arms but not the fingers..
    Do you have an idea about how to solve this issue?

    Thank you very much!
     
    Last edited: Sep 16, 2021
  15. Matiasbru

    Matiasbru

    Joined:
    Sep 1, 2015
    Posts:
    20
    Ok so, I'm trying to mix this solution with the FinalIK plugin on the asset store to get a full-body IK avatar with finger movement using the HandTracking feature on the Oculus Quest 2.

    Everything seems to be working ok, fingers are following HandTracking values coming from the Quest2, and the avatar's body and legs are moving accordingly to the FinalIK plugin.

    Now, the hands rotations are completely offset from the orginial position. I tried everything to make them look better but it's a mess.

    Idk what else to do what I guess is something related to the IK Target_LH and IK Target_RH targets rotational values.
    Still did not get even closer by replacing the values manually. Any clue about this? @alexchesser

    See image below: Black hand is the one coming from the Oculus2 HandTracking. The messed one is the one I'm trying to match.
     

    Attached Files:

  16. Matiasbru

    Matiasbru

    Joined:
    Sep 1, 2015
    Posts:
    20
    Hi everyone, I just realized that the weird position of my hands is starting just after the foreArm, around the palm. In the example from @alexchesser the hands are at least ok but the fingers are the problem, while with my hands it seems to be a problem with the hand itself, not only the fingers. Any clue about this? I attached the avatar I'm using in case a hero dev wants to test it.
     

    Attached Files:

  17. Dark-Table

    Dark-Table

    Joined:
    Nov 25, 2008
    Posts:
    312
    The Oculus hand tracking system has unusual bone orientations. On the right hand the "right" axis points forward (towards the fingertips) and on the left hand the "right" axis points backwards (towards the wrist). So if you're trying to set the position of your bones based on the local position or local rotation of the bones in the Oculus hand mesh you're going to get weird results if your bone orientations have the forward axis pointing toward the fingertips.
     
unityunity