Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Resolved Human Body Tracking Model

Discussion in 'AR' started by mlesterdampios, Aug 31, 2020.

  1. mlesterdampios

    mlesterdampios

    Joined:
    Aug 29, 2020
    Posts:
    3
    Hi, does anyone successfully replaced the default robot model on AR Foundation 4.x?

    I've been trying to download the biped-robot.zip provided by Apple from this page https://developer.apple.com/documentation/arkit/rigging_a_model_for_motion_capture
    Then I compared .fbx file into the prefab on AR Foundation.

    I tried directly importing the .fbx file without any changes on it.
    Load the BoneController component into it, but it produce weird view.

    I read on some post and articles that I must account the rotation and position of my rig with the prefab's.
    Has anyone got successfully made changes to the default robot?

    As I read post and comments from here https://forum.unity.com/threads/example-rig-for-3d-human-skeleton.696512/ @distastee has made successful rigging. I tried his code but I cant seem to make it work.
    Maybe I am missing something?

    Please I need your advice.

    Thank you so much.
     
  2. mdurand

    mdurand

    Joined:
    Aug 1, 2016
    Posts:
    45
    Have you taken a look at the AR Foundation sample for 3D human tracking? Please be aware that the model used here is not a MechAnim model. It has a bone and joint structure that adheres to Apple's spec.

    https://github.com/Unity-Technologies/arfoundation-samples#humanbodytracking3d
     
  3. mlesterdampios

    mlesterdampios

    Joined:
    Aug 29, 2020
    Posts:
    3
    Yes I already looked and analyzed it.
    From what I observed, the prefab model in the AR Foundation (right) has different joint directions from the fbx file released by the apple (https://developer.apple.com/documentation/arkit/rigging_a_model_for_motion_capture , left)



    As far as what I understand from his post, he said that i must compare my T-pose to T-pose of default prefab and use the offset when updating the transform later.

    Here is my code.

    Code (CSharp):
    1. using UnityEngine.XR.ARFoundation;
    2. using UnityEngine;
    3. using UnityEngine.XR.ARSubsystems;
    4. using System.Collections.Generic;
    5. using System;
    6.  
    7. namespace UnityEngine.XR.ARFoundation.Samples
    8. {
    9.     public class CustomBoneController : MonoBehaviour
    10.     {
    11.         // 3D joint skeleton
    12.         enum JointIndices
    13.         {
    14.             Invalid = -1,
    15.             root = 0, // parent: <none> [-1]
    16.             hips_joint = 1, // parent: Root [0]
    17.             left_upLeg_joint = 2, // parent: Hips [1]
    18.             left_leg_joint = 3, // parent: LeftUpLeg [2]
    19.             left_foot_joint = 4, // parent: LeftLeg [3]
    20.             left_toes_joint = 5, // parent: LeftFoot [4]
    21.             left_toesEnd_joint = 6, // parent: LeftToes [5]
    22.             right_upLeg_joint = 7, // parent: Hips [1]
    23.             right_leg_joint = 8, // parent: RightUpLeg [7]
    24.             right_foot_joint = 9, // parent: RightLeg [8]
    25.             right_toes_joint = 10, // parent: RightFoot [9]
    26.             right_toesEnd_joint = 11, // parent: RightToes [10]
    27.             spine_1_joint = 12, // parent: Hips [1]
    28.             spine_2_joint = 13, // parent: Spine1 [12]
    29.             spine_3_joint = 14, // parent: Spine2 [13]
    30.             spine_4_joint = 15, // parent: Spine3 [14]
    31.             spine_5_joint = 16, // parent: Spine4 [15]
    32.             spine_6_joint = 17, // parent: Spine5 [16]
    33.             spine_7_joint = 18, // parent: Spine6 [17]
    34.             left_shoulder_1_joint = 19, // parent: Spine7 [18]
    35.             left_arm_joint = 20, // parent: LeftShoulder1 [19]
    36.             left_forearm_joint = 21, // parent: LeftArm [20]
    37.             left_hand_joint = 22, // parent: LeftForearm [21]
    38.             left_handIndexStart_joint = 23, // parent: LeftHand [22]
    39.             left_handIndex_1_joint = 24, // parent: LeftHandIndexStart [23]
    40.             left_handIndex_2_joint = 25, // parent: LeftHandIndex1 [24]
    41.             left_handIndex_3_joint = 26, // parent: LeftHandIndex2 [25]
    42.             left_handIndexEnd_joint = 27, // parent: LeftHandIndex3 [26]
    43.             left_handMidStart_joint = 28, // parent: LeftHand [22]
    44.             left_handMid_1_joint = 29, // parent: LeftHandMidStart [28]
    45.             left_handMid_2_joint = 30, // parent: LeftHandMid1 [29]
    46.             left_handMid_3_joint = 31, // parent: LeftHandMid2 [30]
    47.             left_handMidEnd_joint = 32, // parent: LeftHandMid3 [31]
    48.             left_handPinkyStart_joint = 33, // parent: LeftHand [22]
    49.             left_handPinky_1_joint = 34, // parent: LeftHandPinkyStart [33]
    50.             left_handPinky_2_joint = 35, // parent: LeftHandPinky1 [34]
    51.             left_handPinky_3_joint = 36, // parent: LeftHandPinky2 [35]
    52.             left_handPinkyEnd_joint = 37, // parent: LeftHandPinky3 [36]
    53.             left_handRingStart_joint = 38, // parent: LeftHand [22]
    54.             left_handRing_1_joint = 39, // parent: LeftHandRingStart [38]
    55.             left_handRing_2_joint = 40, // parent: LeftHandRing1 [39]
    56.             left_handRing_3_joint = 41, // parent: LeftHandRing2 [40]
    57.             left_handRingEnd_joint = 42, // parent: LeftHandRing3 [41]
    58.             left_handThumbStart_joint = 43, // parent: LeftHand [22]
    59.             left_handThumb_1_joint = 44, // parent: LeftHandThumbStart [43]
    60.             left_handThumb_2_joint = 45, // parent: LeftHandThumb1 [44]
    61.             left_handThumbEnd_joint = 46, // parent: LeftHandThumb2 [45]
    62.             neck_1_joint = 47, // parent: Spine7 [18]
    63.             neck_2_joint = 48, // parent: Neck1 [47]
    64.             neck_3_joint = 49, // parent: Neck2 [48]
    65.             neck_4_joint = 50, // parent: Neck3 [49]
    66.             head_joint = 51, // parent: Neck4 [50]
    67.             jaw_joint = 52, // parent: Head [51]
    68.             chin_joint = 53, // parent: Jaw [52]
    69.             left_eye_joint = 54, // parent: Head [51]
    70.             left_eyeLowerLid_joint = 55, // parent: LeftEye [54]
    71.             left_eyeUpperLid_joint = 56, // parent: LeftEye [54]
    72.             left_eyeball_joint = 57, // parent: LeftEye [54]
    73.             nose_joint = 58, // parent: Head [51]
    74.             right_eye_joint = 59, // parent: Head [51]
    75.             right_eyeLowerLid_joint = 60, // parent: RightEye [59]
    76.             right_eyeUpperLid_joint = 61, // parent: RightEye [59]
    77.             right_eyeball_joint = 62, // parent: RightEye [59]
    78.             right_shoulder_1_joint = 63, // parent: Spine7 [18]
    79.             right_arm_joint = 64, // parent: RightShoulder1 [63]
    80.             right_forearm_joint = 65, // parent: RightArm [64]
    81.             right_hand_joint = 66, // parent: RightForearm [65]
    82.             right_handIndexStart_joint = 67, // parent: RightHand [66]
    83.             right_handIndex_1_joint = 68, // parent: RightHandIndexStart [67]
    84.             right_handIndex_2_joint = 69, // parent: RightHandIndex1 [68]
    85.             right_handIndex_3_joint = 70, // parent: RightHandIndex2 [69]
    86.             right_handIndexEnd_joint = 71, // parent: RightHandIndex3 [70]
    87.             right_handMidStart_joint = 72, // parent: RightHand [66]
    88.             right_handMid_1_joint = 73, // parent: RightHandMidStart [72]
    89.             right_handMid_2_joint = 74, // parent: RightHandMid1 [73]
    90.             right_handMid_3_joint = 75, // parent: RightHandMid2 [74]
    91.             right_handMidEnd_joint = 76, // parent: RightHandMid3 [75]
    92.             right_handPinkyStart_joint = 77, // parent: RightHand [66]
    93.             right_handPinky_1_joint = 78, // parent: RightHandPinkyStart [77]
    94.             right_handPinky_2_joint = 79, // parent: RightHandPinky1 [78]
    95.             right_handPinky_3_joint = 80, // parent: RightHandPinky2 [79]
    96.             right_handPinkyEnd_joint = 81, // parent: RightHandPinky3 [80]
    97.             right_handRingStart_joint = 82, // parent: RightHand [66]
    98.             right_handRing_1_joint = 83, // parent: RightHandRingStart [82]
    99.             right_handRing_2_joint = 84, // parent: RightHandRing1 [83]
    100.             right_handRing_3_joint = 85, // parent: RightHandRing2 [84]
    101.             right_handRingEnd_joint = 86, // parent: RightHandRing3 [85]
    102.             right_handThumbStart_joint = 87, // parent: RightHand [66]
    103.             right_handThumb_1_joint = 88, // parent: RightHandThumbStart [87]
    104.             right_handThumb_2_joint = 89, // parent: RightHandThumb1 [88]
    105.             right_handThumbEnd_joint = 90, // parent: RightHandThumb2 [89]
    106.         }
    107.         const int k_NumSkeletonJoints = 91;
    108.  
    109.         [SerializeField]
    110.         Transform[] m_BoneMapping = new Transform[k_NumSkeletonJoints];
    111.  
    112.         [SerializeField]
    113.         private const float jointScaleModifier = 1f;
    114.  
    115.         [SerializeField]
    116.         private Dictionary<JointIndices, Quaternion> ROBOTPOSE = new Dictionary<JointIndices, Quaternion>
    117.         {
    118.             //rotation
    119.            
    120.             { JointIndices.root, new Quaternion(0f, 0f, 0f, 1f) },
    121.             { JointIndices.hips_joint, new Quaternion(0f, 0f, 0f, 1f) },
    122.             { JointIndices.left_upLeg_joint, new Quaternion(-0.5251914f, 0.4729119f, 0.5257655f, -0.4733909f) },
    123.             { JointIndices.left_leg_joint, new Quaternion(-0.4589739f, 0.5374146f, 0.4594831f, -0.5379627f) },
    124.             { JointIndices.left_foot_joint, new Quaternion(-0.6927235f, 0.1400152f, 0.6934642f, -0.1401326f) },
    125.             { JointIndices.left_toes_joint, new Quaternion(-0.7067051f, 0.006181896f, 0.7074544f, -0.006156165f) },
    126.             { JointIndices.left_toesEnd_joint, new Quaternion(-0.7067052f, 0.006181717f, 0.7074544f, -0.006156351f) },
    127.             { JointIndices.right_upLeg_joint, new Quaternion(-0.4731499f, -0.5254809f, 0.4731455f, 0.5254829f) },
    128.             { JointIndices.right_leg_joint, new Quaternion(-0.5376937f, -0.4592238f, 0.5376896f, 0.4592263f) },
    129.             { JointIndices.right_foot_joint, new Quaternion(-0.1400807f, -0.6930928f, 0.1400763f, 0.6930935f) },
    130.             { JointIndices.right_toes_joint, new Quaternion(-0.006176293f, -0.7070798f, 0.00617099f, 0.7070799f) },
    131.             { JointIndices.right_toesEnd_joint, new Quaternion(-0.00617677f, -0.7070798f, 0.006170507f, 0.7070799f) },
    132.             { JointIndices.spine_1_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
    133.             { JointIndices.spine_2_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
    134.             { JointIndices.spine_3_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
    135.             { JointIndices.spine_4_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
    136.             { JointIndices.spine_5_joint, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },
    137.             { JointIndices.spine_6_joint, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },
    138.             { JointIndices.spine_7_joint, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },
    139.             { JointIndices.left_shoulder_1_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
    140.             { JointIndices.left_arm_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
    141.             { JointIndices.left_forearm_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
    142.             { JointIndices.left_hand_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
    143.             { JointIndices.left_handIndexStart_joint, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },
    144.             { JointIndices.left_handIndex_1_joint, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },
    145.             { JointIndices.left_handIndex_2_joint, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },
    146.             { JointIndices.left_handIndex_3_joint, new Quaternion(-0.5000001f, -0.5000001f, 0.5000001f, 0.5000001f) },
    147.             { JointIndices.left_handIndexEnd_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    148.             { JointIndices.left_handMidStart_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    149.             { JointIndices.left_handMid_1_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    150.             { JointIndices.left_handMid_2_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    151.             { JointIndices.left_handMid_3_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    152.             { JointIndices.left_handMidEnd_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    153.             { JointIndices.left_handPinkyStart_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    154.             { JointIndices.left_handPinky_1_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
    155.             { JointIndices.left_handPinky_2_joint, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },
    156.             { JointIndices.left_handPinky_3_joint, new Quaternion(-0.7072248f, -0.0002385676f, -0.0009994805f, 0.706988f) },
    157.             { JointIndices.left_handPinkyEnd_joint, new Quaternion(-0.7065828f, -0.03009519f, -0.03084728f, 0.7063168f) },
    158.             { JointIndices.left_handRingStart_joint, new Quaternion(-0.0001880527f, -0.04309286f, -0.0005318075f, 0.9990709f) },
    159.             { JointIndices.left_handRing_1_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
    160.             { JointIndices.left_handRing_2_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
    161.             { JointIndices.left_handRing_3_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
    162.             { JointIndices.left_handRingEnd_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
    163.             { JointIndices.left_handThumbStart_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
    164.             { JointIndices.left_handThumb_1_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
    165.             { JointIndices.left_handThumb_2_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
    166.             { JointIndices.left_handThumbEnd_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
    167.             { JointIndices.neck_1_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
    168.             { JointIndices.neck_2_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
    169.             { JointIndices.neck_3_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
    170.             { JointIndices.neck_4_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
    171.             { JointIndices.head_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
    172.             { JointIndices.jaw_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
    173.             { JointIndices.chin_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
    174.             { JointIndices.left_eye_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
    175.             { JointIndices.left_eyeball_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
    176.             { JointIndices.left_eyeLowerLid_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
    177.             { JointIndices.left_eyeUpperLid_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
    178.             { JointIndices.nose_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
    179.             { JointIndices.right_eye_joint, new Quaternion(-0.6163417f, -0.3677837f, -0.1319845f, 0.6836949f) },
    180.             { JointIndices.right_eyeball_joint, new Quaternion(-0.6163417f, -0.3677837f, -0.1319845f, 0.6836949f) },
    181.             { JointIndices.right_eyeLowerLid_joint, new Quaternion(-0.6163417f, -0.3677837f, -0.1319845f, 0.6836949f) },
    182.             { JointIndices.right_eyeUpperLid_joint, new Quaternion(-0.6163418f, -0.3677837f, -0.1319845f, 0.6836948f) },
    183.             { JointIndices.right_shoulder_1_joint, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },
    184.             { JointIndices.right_arm_joint, new Quaternion(-0.7069917f, -0.0005104197f, -0.0002469169f, -0.7072219f) },
    185.             { JointIndices.right_forearm_joint, new Quaternion(-0.7062967f, -0.03134356f, 0.0305951f, -0.7065598f) },
    186.             { JointIndices.right_hand_joint, new Quaternion(-0.9990404f, -0.0005291998f, 0.04379734f, -0.0001842082f) },
    187.             { JointIndices.right_handIndexStart_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
    188.             { JointIndices.right_handIndex_1_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
    189.             { JointIndices.right_handIndex_2_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
    190.             { JointIndices.right_handIndex_3_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
    191.             { JointIndices.right_handIndexEnd_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
    192.             { JointIndices.right_handMidStart_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
    193.             { JointIndices.right_handMid_1_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
    194.             { JointIndices.right_handMid_2_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
    195.             { JointIndices.right_handMid_3_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
    196.             { JointIndices.right_handMidEnd_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
    197.             { JointIndices.right_handPinkyStart_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
    198.             { JointIndices.right_handPinky_1_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
    199.             { JointIndices.right_handPinky_2_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
    200.             { JointIndices.right_handPinky_3_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
    201.             { JointIndices.right_handPinkyEnd_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
    202.             { JointIndices.right_handRingStart_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
    203.             { JointIndices.right_handRing_1_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
    204.             { JointIndices.right_handRing_2_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
    205.             { JointIndices.right_handRing_3_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
    206.             { JointIndices.right_handRingEnd_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
    207.             { JointIndices.right_handThumbStart_joint, new Quaternion(-0.6848241f, -0.1300335f, 0.3655986f, -0.6168025f) },
    208.             { JointIndices.right_handThumb_1_joint, new Quaternion(-0.6848241f, -0.1300335f, 0.3655986f, -0.6168025f) },
    209.             { JointIndices.right_handThumb_2_joint, new Quaternion(-0.6848241f, -0.1300335f, 0.3655986f, -0.6168026f) },
    210.             { JointIndices.right_handThumbEnd_joint, new Quaternion(-0.684824f, -0.1300335f, 0.3655986f, -0.6168026f) }
    211.            
    212.         };
    213.  
    214.         [SerializeField]
    215.         Dictionary<JointIndices, Transform> MyRigJoints = new Dictionary<JointIndices, Transform>
    216.         {
    217.             {JointIndices.root, null},
    218.             {JointIndices.hips_joint, null},
    219.             {JointIndices.left_upLeg_joint, null},
    220.             {JointIndices.left_leg_joint, null},
    221.             {JointIndices.left_foot_joint, null},
    222.             {JointIndices.left_toes_joint, null},
    223.             {JointIndices.left_toesEnd_joint, null},
    224.             {JointIndices.right_upLeg_joint, null},
    225.             {JointIndices.right_leg_joint, null},
    226.             {JointIndices.right_foot_joint, null},
    227.             {JointIndices.right_toes_joint, null},
    228.             {JointIndices.right_toesEnd_joint, null},
    229.             {JointIndices.spine_1_joint, null},
    230.             {JointIndices.spine_2_joint, null},
    231.             {JointIndices.spine_3_joint, null},
    232.             {JointIndices.spine_4_joint, null},
    233.             {JointIndices.spine_5_joint, null},
    234.             {JointIndices.spine_6_joint, null},
    235.             {JointIndices.spine_7_joint, null},
    236.             {JointIndices.left_shoulder_1_joint, null},
    237.             {JointIndices.left_arm_joint, null},
    238.             {JointIndices.left_forearm_joint, null},
    239.             {JointIndices.left_hand_joint, null},
    240.             {JointIndices.left_handIndexStart_joint, null},
    241.             {JointIndices.left_handIndex_1_joint, null},
    242.             {JointIndices.left_handIndex_2_joint, null},
    243.             {JointIndices.left_handIndex_3_joint, null},
    244.             {JointIndices.left_handIndexEnd_joint, null},
    245.             {JointIndices.left_handMidStart_joint, null},
    246.             {JointIndices.left_handMid_1_joint, null},
    247.             {JointIndices.left_handMid_2_joint, null},
    248.             {JointIndices.left_handMid_3_joint, null},
    249.             {JointIndices.left_handMidEnd_joint, null},
    250.             {JointIndices.left_handPinkyStart_joint, null},
    251.             {JointIndices.left_handPinky_1_joint, null},
    252.             {JointIndices.left_handPinky_2_joint, null},
    253.             {JointIndices.left_handPinky_3_joint, null},
    254.             {JointIndices.left_handPinkyEnd_joint, null},
    255.             {JointIndices.left_handRingStart_joint, null},
    256.             {JointIndices.left_handRing_1_joint, null},
    257.             {JointIndices.left_handRing_2_joint, null},
    258.             {JointIndices.left_handRing_3_joint, null},
    259.             {JointIndices.left_handRingEnd_joint, null},
    260.             {JointIndices.left_handThumbStart_joint, null},
    261.             {JointIndices.left_handThumb_1_joint, null},
    262.             {JointIndices.left_handThumb_2_joint, null},
    263.             {JointIndices.left_handThumbEnd_joint, null},
    264.             {JointIndices.neck_1_joint, null},
    265.             {JointIndices.neck_2_joint, null},
    266.             {JointIndices.neck_3_joint, null},
    267.             {JointIndices.neck_4_joint, null},
    268.             {JointIndices.head_joint, null},
    269.             {JointIndices.jaw_joint, null},
    270.             {JointIndices.chin_joint, null},
    271.             {JointIndices.left_eye_joint, null},
    272.             {JointIndices.left_eyeball_joint, null},
    273.             {JointIndices.left_eyeLowerLid_joint, null},
    274.             {JointIndices.left_eyeUpperLid_joint, null},
    275.             {JointIndices.nose_joint, null},
    276.             {JointIndices.right_eye_joint, null},
    277.             {JointIndices.right_eyeball_joint, null},
    278.             {JointIndices.right_eyeLowerLid_joint, null},
    279.             {JointIndices.right_eyeUpperLid_joint, null},
    280.             {JointIndices.right_shoulder_1_joint, null},
    281.             {JointIndices.right_arm_joint, null},
    282.             {JointIndices.right_forearm_joint, null},
    283.             {JointIndices.right_hand_joint, null},
    284.             {JointIndices.right_handIndexStart_joint, null},
    285.             {JointIndices.right_handIndex_1_joint, null},
    286.             {JointIndices.right_handIndex_2_joint, null},
    287.             {JointIndices.right_handIndex_3_joint, null},
    288.             {JointIndices.right_handIndexEnd_joint, null},
    289.             {JointIndices.right_handMidStart_joint, null},
    290.             {JointIndices.right_handMid_1_joint, null},
    291.             {JointIndices.right_handMid_2_joint, null},
    292.             {JointIndices.right_handMid_3_joint, null},
    293.             {JointIndices.right_handMidEnd_joint, null},
    294.             {JointIndices.right_handPinkyStart_joint, null},
    295.             {JointIndices.right_handPinky_1_joint, null},
    296.             {JointIndices.right_handPinky_2_joint, null},
    297.             {JointIndices.right_handPinky_3_joint, null},
    298.             {JointIndices.right_handPinkyEnd_joint, null},
    299.             {JointIndices.right_handRingStart_joint, null},
    300.             {JointIndices.right_handRing_1_joint, null},
    301.             {JointIndices.right_handRing_2_joint, null},
    302.             {JointIndices.right_handRing_3_joint, null},
    303.             {JointIndices.right_handRingEnd_joint, null},
    304.             {JointIndices.right_handThumbStart_joint, null},
    305.             {JointIndices.right_handThumb_1_joint, null},
    306.             {JointIndices.right_handThumb_2_joint, null},
    307.             {JointIndices.right_handThumbEnd_joint, null}
    308.         };
    309.  
    310.         [SerializeField]
    311.         private Quaternion[] RotationCompensations = new Quaternion[k_NumSkeletonJoints];
    312.    
    313.  
    314.         [SerializeField]
    315.         [Tooltip("The root bone of the skeleton.")]
    316.         Transform m_SkeletonRoot;
    317.  
    318.         /// <summary>
    319.         /// Get/Set the root bone of the skeleton.
    320.         /// </summary>
    321.         public Transform skeletonRoot
    322.         {
    323.             get
    324.             {
    325.                 return m_SkeletonRoot;
    326.             }
    327.             set
    328.             {
    329.                 m_SkeletonRoot = value;
    330.             }
    331.         }
    332.  
    333.         void Start()
    334.         {
    335.             //InitializeSkeletonJoints();
    336.         }
    337.        
    338.  
    339.         //Breadth-first search
    340.         Transform FindDeepChild(Transform aParent, string aName)
    341.         {
    342.             Queue<Transform> queue = new Queue<Transform>();
    343.             queue.Enqueue(aParent);
    344.             while (queue.Count > 0)
    345.             {
    346.                 var c = queue.Dequeue();
    347.                 if (c.name == aName)
    348.                 {
    349.                     return c;
    350.                 }
    351.                 foreach (Transform t in c)
    352.                     queue.Enqueue(t);
    353.             }
    354.             return null;
    355.         }
    356.  
    357.         public void InitializeSkeletonJoints()
    358.         {
    359.             // Walk through all the child joints in the skeleton and
    360.             // store the skeleton joints at the corresponding index in the m_BoneMapping array.
    361.             // This assumes that the bones in the skeleton are named as per the
    362.             // JointIndices enum above.
    363.             Queue<Transform> nodes = new Queue<Transform>();
    364.             nodes.Enqueue(m_SkeletonRoot);
    365.  
    366.             while (nodes.Count > 0)
    367.             {
    368.                 Transform next = nodes.Dequeue();
    369.                 for (int i = 0; i < next.childCount; ++i)
    370.                 {
    371.                     nodes.Enqueue(next.GetChild(i));
    372.                 }
    373.                 ProcessJoint(next);
    374.             }
    375.  
    376.            
    377.             foreach (var jointKeyValuePair in ROBOTPOSE)
    378.             {
    379.                 MyRigJoints[jointKeyValuePair.Key] = FindDeepChild(m_SkeletonRoot, jointKeyValuePair.Key.ToString());
    380.             }
    381.  
    382.             int j = 0;
    383.             foreach (var jointKeyValuePair in MyRigJoints)
    384.             {
    385.                 //current world rotation of the joint in my rig
    386.                 Quaternion startingRot = jointKeyValuePair.Value.rotation;
    387.                 //arkit t pose
    388.                 Quaternion robotJointRotation = Quaternion.identity;
    389.                 JointIndices index = jointKeyValuePair.Key;
    390.                 if (!ROBOTPOSE.TryGetValue(index, out robotJointRotation))
    391.                 {
    392.                     Debug.LogError("key missing is " + jointKeyValuePair.Key.ToString());
    393.                     continue;
    394.                 }
    395.                 //conversionRotation can convert ARKit rotations to somewthing we can apply to our rig
    396.                 var conversionRotation = Quaternion.Inverse(robotJointRotation) * startingRot;
    397.  
    398.                 RotationCompensations[j] = conversionRotation;
    399.                 j++;
    400.             }
    401.  
    402.         }
    403.  
    404.         public void ApplyBodyPose(ARHumanBody body)
    405.         {
    406.             var joints = body.joints;
    407.             if (!joints.IsCreated)
    408.                 return;
    409.  
    410.             for (int i = 0; i < k_NumSkeletonJoints; ++i)
    411.             {
    412.                 XRHumanBodyJoint joint = joints[i];
    413.                 var bone = m_BoneMapping[i];
    414.                 if (bone != null)
    415.                 {
    416.                     bone.transform.localPosition = joint.localPose.position;
    417.                     bone.transform.rotation = joint.anchorPose.rotation * RotationCompensations[i];
    418.                 }
    419.             }
    420.         }
    421.  
    422.         void ProcessJoint(Transform joint)
    423.         {
    424.             int index = GetJointIndex(joint.name);
    425.             if (index >= 0 && index < k_NumSkeletonJoints)
    426.             {
    427.                 m_BoneMapping[index] = joint;
    428.             }
    429.             else
    430.             {
    431.                 Debug.LogWarning($"{joint.name} was not found.");
    432.             }
    433.         }
    434.  
    435.         // Returns the integer value corresponding to the JointIndices enum value
    436.         // passed in as a string.
    437.         int GetJointIndex(string jointName)
    438.         {
    439.             JointIndices val;
    440.             if (Enum.TryParse(jointName, out val))
    441.             {
    442.                 return (int)val;
    443.             }
    444.             return -1;
    445.         }
    446.     }
    447. }
    But I cant seem to make it work.
    It just produce weird view of the model when running the app.

    I hope someone can help me, im willing to pay for the tutorial/source code just to make this work.
    Thank you so much.
     
    khanhabib likes this.
  4. EdgarasArt

    EdgarasArt

    Joined:
    May 21, 2015
    Posts:
    11
    I've been tracking these issues for over 9 months now... The issues are still not solved. I mean KEY feature of ARFoundation + ARKit is UNUSABLE "as is", so many inconsistencies that it amazes me.
     
    Last edited: Sep 14, 2020
    macgregger and khanhabib like this.
  5. mlesterdampios

    mlesterdampios

    Joined:
    Aug 29, 2020
    Posts:
    3
    do you have a working solution for this? im willing to pay some bucks for the solution
     
    khanhabib and Blarp like this.
  6. brendanluu

    brendanluu

    Joined:
    Jan 5, 2017
    Posts:
    7
    Has anyone found a fix for this?
     
    Blarp likes this.
  7. EdgarasArt

    EdgarasArt

    Joined:
    May 21, 2015
    Posts:
    11
    Well, kinda... Funny thing, the robot rig provided in APPLE documentation for this purpose is WRONG! DO NOT USE IT. You should export robot rig from the arfoundation unity3D project, that's right, export it, because the original model is hidden. It seems there is no shurtcuts at the moment - you need to take that rig from that robot and apply it to your outfit/character model.
     
  8. rayansoban

    rayansoban

    Joined:
    Oct 19, 2020
    Posts:
    5
    How to export the robot rig (one from arfoundation unity3D project) from unity so that it can be imported in e.g. blender?
     
  9. floatingfactory

    floatingfactory

    Joined:
    Aug 18, 2016
    Posts:
    18
    I was able to do this, sort of. I did not replace the ControlledRobot, instead I use it to control a separate model but hide the ControlledRobot as needed.

    Images are from mapping the ControlledRobot to a separate avatar with a fairly standard rig (e.g. 3 spine joints, 1 neck vs the ControlledRobot with 7 spine joints and 4 neck).

    The solution I came up with is a bit complicated, so I'm sure there is a better way to do this, but the essentials are:

    a) I map each joint from the new model to the closest corresponding ControlledRobot joint, ignoring extras (something like avatar:Spine1 => robot:Spine, avatar:Spine2 => robot:Spine3, avatar:Spine3 => robot:Spine7, etc...).

    b) I record the initial rotation for each joint on both models

    c) I get and store the initial delta between their rotations:


    robotLocalRotation = controlledRobotJoint.bone.worldToLocalMatrix.rotation
    avatarLocalRotation = avatarJoint.bone.worldToLocalMatrix.rotation

    avatarJoint.originalLookRotation = avatarLocalRotation* Quaternion.Inverse(robotLocalRotation);


    d) On each LateUpdate I loop through all the joints of the avatar and find the mapped controlledRobot joint. I rotate the avatar joint to point to the same currentLookRotation (so same forward and up) as the matched controlledRobot joint. Than I rotate that back by the starting delta rotation (the originalLookRotation above).



    controlledRobotJoint.currentLookRotation = Quaternion.LookRotation(bone.transform.forward, bone.transform.up);

    avatarBone.rotation = controlledRobotJoint.currentLookRotation * Quaternion.Inverse(avatarJoint.originalLookRotation)


    It tracks pretty well though I should be doing some additional trig to improve the rotations when there are intervening joints, but it has been decent for my purposes.
     

    Attached Files:

    Last edited: Jun 6, 2021
    khanhabib likes this.
  10. khanhabib

    khanhabib

    Joined:
    Oct 11, 2018
    Posts:
    29
    thanks for sharing! sir can you please make a short tutorial about this ?
     
  11. floatingfactory

    floatingfactory

    Joined:
    Aug 18, 2016
    Posts:
    18
    Dmitry-Kuz likes this.
  12. jhuh3226

    jhuh3226

    Joined:
    Oct 24, 2018
    Posts:
    9
    I appreciate you a lot for sharing your code. I was having a problem with this, as my character looks totally different from the ControlledRobot model. It was making weird stretch distortion to match the bone position/ location in ARkit. I'm planning to play with different numbers and equations to make this more precise. Also looking forward to your updated version.
     
  13. shibi2017

    shibi2017

    Joined:
    Jan 18, 2018
    Posts:
    153
    How to export a prefab?
     
  14. Sheeks

    Sheeks

    Joined:
    Jun 9, 2017
    Posts:
    12
    Hello there, i am still facing the problem with a custom Avatar for the Human Body tracking. I export the ControlledRobot with the FBX Exporter to Maya, unbind the skin, rebind my custom mesh to it, export as FBX and the whole model is broken (kind of rotated wrong?!). Are there any hints on this?!

    @floatingfactory the example scene is also not working for me :(
     
  15. kimvasquez17

    kimvasquez17

    Joined:
    Dec 19, 2017
    Posts:
    22
    Is this already solved? I need a fix this also.

    I tried exporting the prefab to FBX but when I import it in the blender, the model is messed up.

    @floatingfactory same with @Sheeks it's not working for me when I tried it, Is there a specific way for this to work?