Search Unity

  1. Dismiss Notice
  2. Unity wants to learn about your experiences working on a Unity project today. We'd like to hear from you via this survey.
    Dismiss Notice
  3. All Pro and Enterprise subscribers: find helpful & inspiring creative, tech, and business know-how in the new Unity Success Hub. Sign in to stay up to date.
    Dismiss Notice
  4. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Oculus HandTracking with custom hand model

Discussion in 'AR/VR (XR) Discussion' started by alexchesser, Jan 16, 2020.

  1. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    So I've been working on improving full body presence in VR

    I added hand-tracking today and so far it looks like this:



    I know this is super new so far but I'm wondering if anyone else has experience with connecting one mesh to another's?

    I don't necessarily expect many people on the forums have experience with the oculus stuff specifically, but really, if I can find a way to connect my BASE MODEL's bones and joints to the OCULUS bones and joints, I imagine I'd be smooth sailing.

    Does anyone have recommendations on how I might access a model's underlying rigging? Is it possible to modify this stuff at runtime? Hook up the "OnAnimatorIK" event to include just over 2 dozen additional points of rotation and position?

    Maybe I should be posting on the animator subforum instead here. :)


    If any of this is useful to anyone else for learning purposes the repo is sitting here for your enjoyment (or laughter):

    https://github.com/AlexChesser/VR_ARMS_DEMO
     
    shumido likes this.
  2. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    if anyone is interested - I made some progress

    Thumbs are a little off, but other than that it's in (IMO) pretty good shape.

    Since this is built to support a MIXAMO character rig, you could pretty easily swap out any character model you want in this space. My next step is to do a bit of finesse on the thumb-tuning - maybe a tool to adjust the offset digits while in the app, since the work to fine tune rigging adjustments is pretty significant.

     
  3. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    71
    Hi alexchesser,

    Do you plane to explain how to integrate finger IK with hand tracking ? just test the VR arms demo and work fine like Humanoid VR but Hand tracking very interesting.

    Best,
     
    xmanlch likes this.
  4. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    @ibompuis sure! I'll try. Let me know if you want any clarification after too.

    First of all, recall that this is working on the quest. So the hand tracking you see in the video above with "no finger rigging" is the code basically as it comes out of the box from Oculus after you follow their docs.

    https://developer.oculus.com/documentation/unity/unity-handtracking/

    For the NEXT part which is moving my custom-model's fingers in time with the OCULUS fingers, I had to put a function in LateUpdate. LateUpdate as you may or may not know happens within the Unity frame's lifecycle AFTER animation has happend but before the frame gets written to the screen. By moving the custom model's finger transforms in LateUpdate, the changes you make don't get overwritten by your model's AnimationController.

    Now -
    Conceptually what I've done is I've hard-coded an object called an HTFinger (ht for hand-tracking) it has a transform and an offset. It's actually a bit of a bad name since I should probably call it a HandTrackingFingerBone but we'll live.
    https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/HTFinger.cs

    Then I have an object called a HandTrackingFingerMap
    https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/HandTrackingFingerMap.cs
    this is a hard-coded list of all of the fingers that you can find in an OVRSkeleton (the oculus finger bones object).

    Within the editor, I add a HandTrackingFingerMap to each of my custom-model's HAND transforms. (strictly speaking they could be empty game objects outside of my model, which would probably be slightly better organization, but I digress.

    Each BONE in the custom model, gets mapped within the hand tracking finger map.

    THEN I put my FingerIK on my character model as a sibling of their Animator
    https://github.com/AlexChesser/VR_ARMS_DEMO/blob/master/Assets/Scripts/FingerIK.cs

    You'll notice FingerIK in the editor takes a reference to the Oculus object for left and right hand as well as the HandTrackingFingerMap for the left and right hand of my custom model. *ACTUAL* IK arm position is handled elsewhere as if it was regular unity Mechanim IK.

    Finger IK (in late update) then traverses each bone in the OVRSkeleton and positions the appropriately mapped HTFinger at the same position. *FINALLY* because your custom model's transforms are not likely to have the exact same forward direction as the OVR transforms, you have to ROTATE your custom fingers by an additional offset/vector3

    If you're like *made of money* and have access to modelers and riggers to do custom 3d you could actually work backwards from the default provided oculus hands and build custom meshes to swap out (which is recommended by the real pros) but the offset step works OK.

    Hopefully that's useful to you?

    :)

    If you use the technique I'd love to know what you eventually built.
     
    TigerHix, shumido and ibompuis like this.
  5. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    71
    Sorry @alexchesser , missed your post... thanks for sharing tips and code, definitivly trying this tomorrow ! and keep in touch when I'v done something interesting
     
    alexchesser likes this.
  6. remiC3D

    remiC3D

    Joined:
    Dec 15, 2017
    Posts:
    30
    Your Demo looks quite amazing. I quickly tested it, but unfortunately, here is what I get in the HandTest_Chesser scene:
    upload_2020-5-24_19-35-14.png
    I'm able to move the fingers, but as you can see they are all messed up...
    I didn't change anything on your project, except maybe I'm not using the same version of Oculus Integration as you do... I had to update it, otherwise my hands weren't tracked at all...

    Any Idea on what may be happening ?

    Thanks in advance for your answer !
     
  7. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    1,251
    Got to say ... I'm loving these screenshots/videos :D. It's like you're making "alien hand simulators" ... impressively creepy :)
     
    jipsen likes this.
  8. remiC3D

    remiC3D

    Joined:
    Dec 15, 2017
    Posts:
    30
    xD
    This is not intended of course. So my question remains...
     
  9. SIgmar_Storm

    SIgmar_Storm

    Joined:
    Jul 3, 2020
    Posts:
    1
    Hello, try to modify rotation on HandTrackingFingerMap script attached to LeftHand and RightHand :)

    Very nice demo alex ! (thumbs are little off yes but maybe rotation modification ?)
     
  10. Imbazephyr

    Imbazephyr

    Joined:
    Jul 2, 2015
    Posts:
    8
    I think I have made some progress by adjusting the values of each finger offset based on the project from @alexchesser.
    upload_2021-2-27_22-15-57.png
    As the screenshot shown here, the nails on the thumb are facing "up" instead of forward (towards the camera). I will list the rotations for each finger on each hand below, they are based on the latest version of Oculus Integration (25.0).

    Left Hand
    thumb x: -180 y:0 z:90
    index x: -90 y:90 z:0
    middle x: -90 y:90 z:0
    ring x: -90 y:90 z:0
    pinky x: -90 y:90 z:0

    Right Hand
    thumb x: 0y:0 z:-90
    index x: 90 y:90 z:0
    middle x: 90 y:90 z:0
    ring x: 90 y:90 z:0
    pinky x: 90 y:90 z:0

    The base of the thumb still bulge outward, which I think can be tweaked slightly by adjusting the rotation on thumb1.
    Lastly, thank you alexchesser for providing the project on github.
     
    Ferdowsur likes this.
  11. Proto-G

    Proto-G

    Joined:
    Nov 22, 2014
    Posts:
    178
    Awesome! How to do this with touch controller hands and not actual hands?
     
  12. MarkSharpe

    MarkSharpe

    Joined:
    Feb 3, 2021
    Posts:
    5
    I've been trying all day to get my XR rig to accept the OVR oculus prefabs but cannot get the hands to animate. I have a turn based controller set up and I can change the meshes and the build is clean and both trigger and grip work on both controllers with interactables however, I cannot get the finger to point or the fingers to cup in when I go to grab the objects in my scene. The raycasts work as intended but I would really like to get the hands moving along so i can move on with a fully functional character "rig" Any ideas? I'm using the XR input toolkit with the OVR handmeshes for quest 2 but can't get them to move.
     
  13. ulthien

    ulthien

    Joined:
    Apr 25, 2020
    Posts:
    7
    i am at the same spot that you are in ... but the docu seems outdated, as the Unity 2021 or OVR prefabs differ in content and scope/function. It is astonishing that they do not updat the docu for 3 years.... hmmmm
     
unityunity