Search Unity

Example Rig for 3D Human Skeleton

Discussion in 'Handheld AR' started by Magno-D-Turpin, Jun 17, 2019.

  1. Magno-D-Turpin

    Magno-D-Turpin

    Joined:
    Nov 3, 2016
    Posts:
    6
    Hello,

    I'm attempting to use the new ARKit 3 "motion capture" feature for realtime 3D human skeleton creation and retargeting. I couldn't find any documentation on this, and the ARFoundation sample project on git only superficially touched on it by mapping a sphere onto a human "head" joint.

    Based on my sleuthing, however, I was able to create a system that should theoretically map a the AR rig to a character rig from Maya. Unfortunately, this approach quite literally blew up in my face, with random triangles morphing everywhere in a cacophony of geometry. Something is clearly wrong here.

    Is there by chance an example rig or prefab I can take a look at, so our artist and I can make the necessary adjustments to get our rig to work?

    Thanks!
    David
     
    ina likes this.
  2. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    Dude. Came here to post exactly this. By using position only I can at least get the joints into the right location - but the mesh is all torn up (because obviously rigs are built for rotation only). Using the rotations directly....is a bad idea.

    I've been thinking that using the world space forward vector of each joint translated into model space might be the best idea. That way if you have a different rig hierarchy (or move the model around) you won't have weird issues.

    Still trying to get it to work - but hoping to maybe crowdsource some solutions.

    Edit: totally doesn't work. It gives you the same value as the world space quaternion. Makes sense but I was really hoping that there was a bug somewhere in the rotations.
     
    Last edited: Jun 18, 2019
  3. Magno-D-Turpin

    Magno-D-Turpin

    Joined:
    Nov 3, 2016
    Posts:
    6
    Thanks for posting this! Good to know we have some help :)

    We created a flat hierarchy (i.e. sibling relationship) of cubes for each joint and placed them under the root joint. Then we updated both the local positions and rotations of each cube to match the joint from ARFoundation.

    What we discovered surprised us. The rotations of the cubes looked correct, down to the finger joints. This means that the data we get from ARFoundation assumes our rig joints are siblings. This is pretty non standard for a rig, I know.

    Tomorrow we're going to make a very simple rig made of cylinders to see if we can at least get a humanoid-looking figure that way. We suspect we simply aren't correctly weighting the joints to the skinned mesh (seriously, what are you supposed to do with 7 spine joints??). That's why having an example from Apple or ARFoundation would be just peachy ;)
     
  4. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    That's super surprising. Yes please let me know how that cylinder test goes.

    I'd been working under the assumption that there was a bug in rotations. As such - I tested out having the joints "look at" the location of the next point in the rig. This causes a ton of issues if your rig isn't the same size as the recording of the human - and I was in the middle of trying to fix that part too.

    If the rotations coming out of ARFoundation are correct - then what could be the root problem is that our joints aren't calibrated to the same "zero" rotation. Essentially we would have to convert the ARFoundation rotation into "joint space" and then do the rotation.

    What would be super helpful in your test is coloring one end of the cylinder to ensure that the joints aren't flipping around 180 degrees or something weird like that....as that would also explain the busted behavior I've been seeing.
     
  5. drewhong

    drewhong

    Joined:
    Jun 21, 2017
    Posts:
    5
    Here's a script that can generate the neutral rig used by ARKit 3. I got the information from a native iOS app. Apple provided an API to access "neturalPose" but Unity's ARFoundation didn't implement it.

    Please drag a small sphere as the prefab to the "Sphere Prefab" slot.

    https://paste.ubuntu.com/p/nvQ7QyQbhS/
     
    Last edited: Jun 18, 2019
    KLWN likes this.
  6. drewhong

    drewhong

    Joined:
    Jun 21, 2017
    Posts:
    5
    This is probably because you are using
    ARHumanBody.joints[index].anchorPose instead of ARHumanBody.joints[index].localPose
    The former transform is relative the root, the later is the local transform
     
  7. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    Thanks for posting that Drew. I used that to figure out that what they consider "neutral pose" and what my rig considers "neutral pose" are wildly different. It looks like those values are in world space?

    The closest I've gotten to something that looks correct is creating a rotation per-joint that converts ARKit neutral pose to my rig's neutral pose:
    Quaternion conversionRotation = Quaternion.Inverse(arkitNeutralRot) * myRigNeutralRot;

    I then take the ARKit world rotation and apply the conversionRotation to it:
    tform.rotation = ARHumanBody.joints[index].anchorPose * conversionRotation;

    Unfortunately - the root of each new hierarchy (arms, legs, spine_0) looks like it has a 180 turn applied to it in the y direction.

    I'm starting to wonder if there's an issue with coordinate systems or something going on. Though honestly - it feels near impossible to fix this without access to source or a working example.
     
  8. drewhong

    drewhong

    Joined:
    Jun 21, 2017
    Posts:
    5
    Yeah I'm having the same issue. The neutral rig I got from Apple's API and the tracking rig information I got from Unity doesn't match exactly. Some joints has a 180 turn and some don't.
    Either Apple made a mistake in their API, or Unity did some remapping under the hood.
    I spent a day to manually change their rotation to make the mapping barely work. It's not worth it. I suggest to wait for updates from Apple/Unity.
     
    distastee likes this.
  9. Magno-D-Turpin

    Magno-D-Turpin

    Joined:
    Nov 3, 2016
    Posts:
    6
    Thanks guys, this has been super helpful.

    @drewhong, we used the rig generated from the code you provided to create our own rig. We then used

    Code (CSharp):
    1. jointTransform.localRotation = joint.localPose.rotation;
    ...iteratively on each joint in the rig.

    We now have a skinned mesh that looks mostly humanoid in AR, made out of cylinders and a half-sphere for the head. However, there is still a lot of twisting at the joints so we're looking into what's going on there.
     
  10. Magno-D-Turpin

    Magno-D-Turpin

    Joined:
    Nov 3, 2016
    Posts:
    6
    Forgot to mention, we are totally ignoring positional data for simplicity. Also, we discovered that localPose.position was returning a zero-vector anyway. The only way for us to apply positions was to use anchoredPose.position and then re-convert it into the correct coordinate space. It didn't look much better on our rig anyway, so we left it off.
     
  11. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    Happy (but sad) to hear I'm not alone on this one. Sounds like bug territory if we're all running into exactly the same thing.

    I went digging around in the package source - but its basically a thin shim to a native dll....so there wasn't much to learn. Maybe we can get @jimmya for some wisdom?
     
    ina likes this.
  12. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
  13. nishantkumar2004

    nishantkumar2004

    Joined:
    Jun 20, 2019
    Posts:
    1
    The HumanBodyTracking3D scene, in arfoundation-samples, has been updated to control the ARKit3 Robot model's skeleton, using the ARHumanBody pose detected by ARFoundation.
    The HumanBodyTracker MonoBehavior takes care of detecting human bodies and passing on the detected body pose to the Robot's skeleton, via BoneController. The BoneController is a component on the Robot model, that maps the detected Human body pose to the Robot's skeleton bones.
     
    ina and distastee like this.
  14. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    Thanks for publishing and posting a response! I think this may have helped me figure out the problem.

    Background: To quickly iterate - I had recorded some data from my iPhone and hacked up a quick body-tracking simulator. For simplicity I set world rotation of my rig to equal the world rotation of the pre-recorded body data.

    When hooking this up to my character - I get total garbage. When I hooked it up to the robot rig - it worked perfectly. WTF. So clearly there's some assumption baked into the rigs that's different (which I THOUGHT that world rotation would overcome. Seems not). I've attached two images showing the knee joint selected (and the move gizmo set to local space).
    knee.JPG knee2.JPG

    I THINK this is the root of the problem. The robot rig has the X axis pointing to the next joint. My rig has negative-z pointing to the next joint. I get very different results when setting the same rotation on both rigs. I'll be honest - I wasn't expecting that result when I'm setting the world rotation of the joint....but I don't have extensive experience with character rigs so maybe this should have been obvious.

    In any case - I think I was close with the idea of "defining a rest pose - calibrating against it to create an offset - applying that offset when reading in tracking data" I just need to figure out why my initial attempt failed.
     
  15. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    GOT IT

    Turns out the neutral pose that drew pasted has some bad rotations in it. This is all super hacked together but is a good starting point. Using this index map:

    Code (CSharp):
    1.  
    2.     enum JointIndices
    3.     {
    4.         Invalid = -1,
    5.         Root = 0, // parent: Invalid
    6.         Hips = 1, // parent: Root
    7.         LeftUpLeg = 2, // parent: Hips
    8.         LeftLeg = 3, // parent: LeftUpLeg
    9.         LeftFoot = 4, // parent: LeftLeg
    10.         LeftToes = 5, // parent: LeftFoot
    11.         LeftToesEnd = 6, // parent: LeftToes
    12.         RightUpLeg = 7, // parent: Hips
    13.         RightLeg = 8, // parent: RightUpLeg
    14.         RightFoot = 9, // parent: RightLeg
    15.         RightToes = 10, // parent: RightFoot
    16.         RightToesEnd = 11, // parent: RightToes
    17.         Spine1 = 12, // parent: Hips
    18.         Spine2 = 13, // parent: Spine1
    19.         Spine3 = 14, // parent: Spine2
    20.         Spine4 = 15, // parent: Spine3
    21.         Spine5 = 16, // parent: Spine4
    22.         Spine6 = 17, // parent: Spine5
    23.         Spine7 = 18, // parent: Spine6
    24.         RightShoulder1 = 19, // parent: Spine7
    25.         RightShoulder2 = 20, // parent: RightShoulder1
    26.         RightArm = 21, // parent: RightShoulder2
    27.         RightForearm = 22, // parent: RightArm
    28.         RightHand = 23, // parent: RightForearm
    29.         RightHandThumbStart = 24, // parent: RightHand
    30.         RightHandThumb1 = 25, // parent: RightHandThumbStart
    31.         RightHandThumb2 = 26, // parent: RightHandThumb1
    32.         RightHandThumbEnd = 27, // parent: RightHandThumb2
    33.         RightHandIndexStart = 28, // parent: RightHand
    34.         RightHandIndex1 = 29, // parent: RightHandIndexStart
    35.         RightHandIndex2 = 30, // parent: RightHandIndex1
    36.         RightHandIndex3 = 31, // parent: RightHandIndex2
    37.         RightHandIndexEnd = 32, // parent: RightHandIndex3
    38.         RightHandMidStart = 33, // parent: RightHand
    39.         RightHandMid1 = 34, // parent: RightHandMidStart
    40.         RightHandMid2 = 35, // parent: RightHandMid1
    41.         RightHandMid3 = 36, // parent: RightHandMid2
    42.         RightHandMidEnd = 37, // parent: RightHandMid3
    43.         RightHandRingStart = 38, // parent: RightHand
    44.         RightHandRing1 = 39, // parent: RightHandRingStart
    45.         RightHandRing2 = 40, // parent: RightHandRing1
    46.         RightHandRing3 = 41, // parent: RightHandRing2
    47.         RightHandRingEnd = 42, // parent: RightHandRing3
    48.         RightHandPinkyStart = 43, // parent: RightHand
    49.         RightHandPinky1 = 44, // parent: RightHandPinkyStart
    50.         RightHandPinky2 = 45, // parent: RightHandPinky1
    51.         RightHandPinky3 = 46, // parent: RightHandPinky2
    52.         RightHandPinkyEnd = 47, // parent: RightHandPinky3
    53.         LeftShoulder1 = 48, // parent: Spine7
    54.         LeftShoulder2 = 49, // parent: LeftShoulder1
    55.         LeftArm = 50, // parent: LeftShoulder2
    56.         LeftForearm = 51, // parent: LeftArm
    57.         LeftHand = 52, // parent: LeftForearm
    58.         LeftHandThumbStart = 53, // parent: LeftHand
    59.         LeftHandThumb1 = 54, // parent: LeftHandThumbStart
    60.         LeftHandThumb2 = 55, // parent: LeftHandThumb1
    61.         LeftHandThumbEnd = 56, // parent: LeftHandThumb2
    62.         LeftHandIndexStart = 57, // parent: LeftHand
    63.         LeftHandIndex1 = 58, // parent: LeftHandIndexStart
    64.         LeftHandIndex2 = 59, // parent: LeftHandIndex1
    65.         LeftHandIndex3 = 60, // parent: LeftHandIndex2
    66.         LeftHandIndexEnd = 61, // parent: LeftHandIndex3
    67.         LeftHandMidStart = 62, // parent: LeftHand
    68.         LeftHandMid1 = 63, // parent: LeftHandMidStart
    69.         LeftHandMid2 = 64, // parent: LeftHandMid1
    70.         LeftHandMid3 = 65, // parent: LeftHandMid2
    71.         LeftHandMidEnd = 66, // parent: LeftHandMid3
    72.         LeftHandRingStart = 67, // parent: LeftHand
    73.         LeftHandRing1 = 68, // parent: LeftHandRingStart
    74.         LeftHandRing2 = 69, // parent: LeftHandRing1
    75.         LeftHandRing3 = 70, // parent: LeftHandRing2
    76.         LeftHandRingEnd = 71, // parent: LeftHandRing3
    77.         LeftHandPinkyStart = 72, // parent: LeftHand
    78.         LeftHandPinky1 = 73, // parent: LeftHandPinkyStart
    79.         LeftHandPinky2 = 74, // parent: LeftHandPinky1
    80.         LeftHandPinky3 = 75, // parent: LeftHandPinky2
    81.         LeftHandPinkyEnd = 76, // parent: LeftHandPinky3
    82.         Neck1 = 77, // parent: Spine7
    83.         Neck2 = 78, // parent: Neck1
    84.         Neck3 = 79, // parent: Neck2
    85.         Neck4 = 80, // parent: Neck3
    86.         Head = 81, // parent: Neck4
    87.         Jaw = 82, // parent: Head
    88.         Chin = 83, // parent: Jaw
    89.         Nose = 84, // parent: Head
    90.         RightEye = 85, // parent: Head
    91.         RightEyeUpperLid = 86, // parent: RightEye
    92.         RightEyeLowerLid = 87, // parent: RightEye
    93.         RightEyeBall = 88, // parent: RightEye
    94.         LeftEye = 89, // parent: Head
    95.         LeftEyeUpperLid = 90, // parent: LeftEye
    96.         LeftEyeLowerLid = 91, // parent: LeftEye
    97.         LeftEyeBall = 92, // parent: LeftEye
    98.     }
    I dumped the T pose of the robot into a dictionary IN WORLD COORDINATES:

    Code (CSharp):
    1.         private Dictionary<JointIndices, Quaternion> ROBOTPOSE = new Dictionary<JointIndices, Quaternion>
    2.         {
    3.             { JointIndices.Root, new Quaternion(0f, 0f, 0f, 1f) },{ JointIndices.Hips, new Quaternion(0f, 0f, 0f, 1f) },{ JointIndices.LeftUpLeg, new Quaternion(-0.5251914f, 0.4729119f, 0.5257655f, -0.4733909f) },{ JointIndices.RightUpLeg, new Quaternion(-0.4731499f, -0.5254809f, 0.4731455f, 0.5254829f) },{ JointIndices.Spine1, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftLeg, new Quaternion(-0.4589739f, 0.5374146f, 0.4594831f, -0.5379627f) },{ JointIndices.RightLeg, new Quaternion(-0.5376937f, -0.4592238f, 0.5376896f, 0.4592263f) },{ JointIndices.Spine2, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftFoot, new Quaternion(-0.6927235f, 0.1400152f, 0.6934642f, -0.1401326f) },{ JointIndices.RightFoot, new Quaternion(-0.1400807f, -0.6930928f, 0.1400763f, 0.6930935f) },{ JointIndices.Spine3, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftToes, new Quaternion(-0.7067051f, 0.006181896f, 0.7074544f, -0.006156165f) },{ JointIndices.RightToes, new Quaternion(-0.006176293f, -0.7070798f, 0.006170991f, 0.7070799f) },{ JointIndices.Spine4, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftToesEnd, new Quaternion(-0.7067052f, 0.006181717f, 0.7074544f, -0.006156351f) },{ JointIndices.RightToesEnd, new Quaternion(-0.00617677f, -0.7070798f, 0.006170507f, 0.7070799f) },{ JointIndices.Spine5, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.Spine6, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.Spine7, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.RightShoulder1, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },{ JointIndices.LeftShoulder1, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },{ JointIndices.Neck1, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightShoulder2, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },{ JointIndices.LeftShoulder2, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },{ JointIndices.Neck2, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightArm, new Quaternion(-0.7072251f, -4.330277E-05f, -0.0008040965f, 0.706988f) },{ JointIndices.LeftArm, new Quaternion(-0.7069915f, -0.0008285469f, 7.144874E-05f, -0.7072218f) },{ JointIndices.Neck3, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightForearm, new Quaternion(-0.706572f, -0.03035438f, -0.03110623f, 0.7063052f) },{ JointIndices.LeftForearm, new Quaternion(-0.7063079f, -0.03109138f, 0.03034304f, -0.7065705f) },{ JointIndices.Neck4, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightHand, new Quaternion(-0.0001886487f, -0.04345921f, -0.0005316734f, 0.9990551f) },{ JointIndices.LeftHand, new Quaternion(-0.999056f, -0.0005290508f, 0.04344079f, -0.0001838654f) },{ JointIndices.Head, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.RightHandThumbStart, new Quaternion(-0.563176f, -0.450308f, -0.1651125f, 0.6728995f) },{ JointIndices.RightHandIndexStart, new Quaternion(-0.001846135f, -0.1037805f, -0.0003555566f, 0.9945986f) },{ JointIndices.RightHandMidStart, new Quaternion(-0.001781464f, -0.04345801f, -0.0004611611f, 0.9990537f) },{ JointIndices.RightHandRingStart, new Quaternion(-0.001758099f, 0.006566688f, -0.0005474091f, 0.9999769f) },{ JointIndices.RightHandPinkyStart, new Quaternion(-0.00176309f, 0.05727234f, -0.0006380975f, 0.9983569f) },{ JointIndices.LeftHandThumbStart, new Quaternion(-0.6729128f, -0.16509f, 0.4503065f, -0.5631677f) },{ JointIndices.LeftHandIndexStart, new Quaternion(-0.9946009f, -0.0003358126f, 0.1037586f, -0.00184305f) },{ JointIndices.LeftHandMidStart, new Quaternion(-0.9990543f, -0.000755012f, 0.04344012f, -0.001763821f) },{ JointIndices.LeftHandRingStart, new Quaternion(-0.9999767f, -0.0005270988f, -0.006585166f, -0.001753271f) },{ JointIndices.LeftHandPinkyStart, new Quaternion(-0.9983559f, -0.0006175488f, -0.05729078f, -0.001757562f) },{ JointIndices.Jaw, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.Nose, new Quaternion(-0.5000001f, -0.5000001f, 0.5000001f, 0.5000001f) },{ JointIndices.RightEye, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEye, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightHandThumb1, new Quaternion(-0.6383249f, -0.3353892f, -0.0345653f, 0.691998f) },{ JointIndices.RightHandIndex1, new Quaternion(-0.141001f, -0.06067762f, 0.1492173f, 0.976817f) },{ JointIndices.RightHandMid1, new Quaternion(-0.001823545f, -0.01739544f, 0.08677565f, 0.9960744f) },{ JointIndices.RightHandRing1, new Quaternion(0.0863688f, 0.043634f, 0.0978535f, 0.9904854f) },{ JointIndices.RightHandPinky1, new Quaternion(0.1280032f, 0.08536577f, 0.1233153f, 0.980368f) },{ JointIndices.LeftHandThumb1, new Quaternion(-0.6920068f, -0.03454062f, 0.3353892f, -0.6383165f) },{ JointIndices.LeftHandIndex1, new Quaternion(-0.9768156f, 0.1492398f, 0.06065904f, -0.1409955f) },{ JointIndices.LeftHandMid1, new Quaternion(-0.9961001f, 0.08648346f, 0.01738015f, -0.001796693f) },{ JointIndices.LeftHandRing1, new Quaternion(-0.9904823f, 0.09787226f, -0.04365361f, 0.08637467f) },{ JointIndices.LeftHandPinky1, new Quaternion(-0.980363f, 0.1233332f, -0.08538568f, 0.1280103f) },{ JointIndices.Chin, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.RightEyeUpperLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightEyeLowerLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightEyeBall, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeUpperLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeLowerLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeBall, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightHandThumb2, new Quaternion(-0.6677005f, -0.2722507f, 0.03234765f, 0.692105f) },{ JointIndices.RightHandIndex2, new Quaternion(-0.1477825f, -0.0415144f, 0.2769809f, 0.9485351f) },{ JointIndices.RightHandMid2, new Quaternion(-0.004873842f, -0.01679799f, 0.2617181f, 0.964986f) },{ JointIndices.RightHandRing2, new Quaternion(0.0938583f, 0.02353925f, 0.3136097f, 0.9446088f) },{ JointIndices.RightHandPinky2, new Quaternion(0.1357787f, 0.07236205f, 0.2191423f, 0.9634857f) },{ JointIndices.LeftHandThumb2, new Quaternion(-0.6921116f, 0.03237311f, 0.2722515f, -0.6676922f) },{ JointIndices.LeftHandIndex2, new Quaternion(-0.9485307f, 0.277003f, 0.04149672f, -0.1477744f) },{ JointIndices.LeftHandMid2, new Quaternion(-0.965063f, 0.2614351f, 0.01678771f, -0.004844725f) },{ JointIndices.LeftHandRing2, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinky2, new Quaternion(-0.963479f, 0.2191598f, -0.07238111f, 0.1357878f) },{ JointIndices.RightHandThumbEnd, new Quaternion(-0.6677005f, -0.2722507f, 0.03234762f, 0.692105f) },{ JointIndices.RightHandIndex3, new Quaternion(-0.1494033f, -0.03523868f, 0.3167771f, 0.9359966f) },{ JointIndices.RightHandMid3, new Quaternion(-0.005977005f, -0.01643786f, 0.325129f, 0.945508f) },{ JointIndices.RightHandRing3, new Quaternion(0.09519702f, 0.01735026f, 0.3747187f, 0.9220752f) },{ JointIndices.RightHandPinky3, new Quaternion(0.1402616f, 0.06323615f, 0.2822996f, 0.9469081f) },{ JointIndices.LeftHandThumbEnd, new Quaternion(-0.6921116f, 0.03237309f, 0.2722515f, -0.6676922f) },{ JointIndices.LeftHandIndex3, new Quaternion(-0.9358417f, 0.3172399f, 0.03515086f, -0.1494111f) },{ JointIndices.LeftHandMid3, new Quaternion(-0.9456036f, 0.3248518f, 0.01642954f, -0.005947292f) },{ JointIndices.LeftHandRing3, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinky3, new Quaternion(-0.9469005f, 0.2823165f, -0.06325462f, 0.1402719f) },{ JointIndices.RightHandIndexEnd, new Quaternion(-0.1494033f, -0.03523868f, 0.3167771f, 0.9359965f) },{ JointIndices.RightHandMidEnd, new Quaternion(-0.005977005f, -0.01643786f, 0.325129f, 0.945508f) },{ JointIndices.RightHandRingEnd, new Quaternion(0.09519702f, 0.01735026f, 0.3747187f, 0.9220752f) },{ JointIndices.RightHandPinkyEnd, new Quaternion(0.1402616f, 0.06323615f, 0.2822996f, 0.9469081f) },{ JointIndices.LeftHandIndexEnd, new Quaternion(-0.9358417f, 0.3172399f, 0.03515095f, -0.1494111f) },{ JointIndices.LeftHandMidEnd, new Quaternion(-0.9456036f, 0.3248518f, 0.01642954f, -0.005947292f) },{ JointIndices.LeftHandRingEnd, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinkyEnd, new Quaternion(-0.9469005f, 0.2823165f, -0.06325462f, 0.1402719f) },
    4.  
    5.         };
    At initialization - I compare my character's T pose to the robot's T pose - create a rotation that will convert from "robot joint space" to "my joint's space" - and store that value for later:
    Code (CSharp):
    1.             foreach (var jointKeyValuePair in MyRigJoints)
    2.             {
    3.                 //current world rotation of the joint in my rig
    4.                 Quaternion startingRot = jointKeyValuePair.Value.rotation;
    5.                 //arkit t pose
    6.                 Quaternion robotJointRotation = Quaternion.identity;
    7.                 JointIndices index = jointKeyValuePair.Key;
    8.                 if (!ROBOTPOSE.TryGetValue(index, out robotJointRotation))
    9.                 {
    10.                     Debug.LogError("key missing is " + jointKeyValuePair.Key.ToString());
    11.                     continue;
    12.                 }
    13.                 //conversionRotation can convert ARKit rotations to somewthing we can apply to our rig
    14.                 var conversionRotation = Quaternion.Inverse(robotJointRotation) * startingRot;
    15.  
    16.                 RotationCompensations[index] = compensate;
    17.             }
    Then later when you get the updated human body pose from ARKit, you apply that stored compensation to the arkit rotations:

    Code (CSharp):
    1. myjointTransform.rotation = ARHumanBody.joints[index].anchorPose.rotation * RotationCompensations[index];
    The closer your character's T pose is to the reference T pose - the more accurate your animations will be.

    Since all of this is in world space - you'll still need to account for any parent transforms. Eventually I'd like to get this working in local space.....but really I just want to move on at this point.

    Hope that helps and fixes everyone else's issues. Thanks again for all the help folks.
     
    drewhong likes this.
  16. drewhong

    drewhong

    Joined:
    Jun 21, 2017
    Posts:
    5

    Sorry for the bad rotations... I really have no idea where it went wrong. I'll look in to it. Sorry for the confusion.
     
  17. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    No worries. It was impossible to tell since nothing at all was working. In all honesty - it still helped me significantly. Once I had correct data I was able to quickly fashion a solution. Wouldn't have been able to do that without your post.
     
    Last edited: Jun 21, 2019
    drewhong likes this.
  18. RickPix

    RickPix

    Joined:
    Nov 27, 2017
    Posts:
    1
    @distastee Don't suppose you've made one of these for the changed mappings and robot in beta 4 yet?
     
  19. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    Not yet. Just upgraded yesterday and all my S*** is completely hosed. I'm working on fixing it today - which means I'll post here once I figure out wtf went wrong.
     
  20. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    31
    I don't think any changes are necessary. The indicies all switched place - but it was thankfully built to handle that. From what I can tell - all the rotations + tracking seem to remain accurate.
     
  21. roshan_unity825

    roshan_unity825

    Joined:
    Jun 14, 2019
    Posts:
    4
    @distastee I'm trying to implement your suggestion in my project. I kinda struck. can you please share any demo scene if you have created with your modified script for body tracking. Please help. Thanks
     
  22. logikben

    logikben

    Joined:
    Jun 6, 2017
    Posts:
    1
    Hello everyone. I'm trying to implement my own model on the robot rig without success. All rotations are reversed or do not follow well. Could someone send a test scene that works with our own models? I am not an experienced programmer so it would be super appreciated. Thank you in advance!