Example Rig for 3D Human Skeleton

Hello,

I'm attempting to use the new ARKit 3 "motion capture" feature for realtime 3D human skeleton creation and retargeting. I couldn't find any documentation on this, and the ARFoundation sample project on git only superficially touched on it by mapping a sphere onto a human "head" joint.

Based on my sleuthing, however, I was able to create a system that should theoretically map a the AR rig to a character rig from Maya. Unfortunately, this approach quite literally blew up in my face, with random triangles morphing everywhere in a cacophony of geometry. Something is clearly wrong here.

Is there by chance an example rig or prefab I can take a look at, so our artist and I can make the necessary adjustments to get our rig to work?

Thanks!
David

 

Dude. Came here to post exactly this. By using position only I can at least get the joints into the right location - but the mesh is all torn up (because obviously rigs are built for rotation only). Using the rotations directly....is a bad idea.

I've been thinking that using the world space forward vector of each joint translated into model space might be the best idea. That way if you have a different rig hierarchy (or move the model around) you won't have weird issues.

Still trying to get it to work - but hoping to maybe crowdsource some solutions.

Edit: totally doesn't work. It gives you the same value as the world space quaternion. Makes sense but I was really hoping that there was a bug somewhere in the rotations.

 

Thanks for posting this! Good to know we have some help

We created a flat hierarchy (i.e. sibling relationship) of cubes for each joint and placed them under the root joint. Then we updated both the local positions and rotations of each cube to match the joint from ARFoundation.

What we discovered surprised us. The rotations of the cubes looked correct, down to the finger joints. This means that the data we get from ARFoundation assumes our rig joints are siblings. This is pretty non standard for a rig, I know.

Tomorrow we're going to make a very simple rig made of cylinders to see if we can at least get a humanoid-looking figure that way. We suspect we simply aren't correctly weighting the joints to the skinned mesh (seriously, what are you supposed to do with 7 spine joints??). That's why having an example from Apple or ARFoundation would be just peachy

 

That's super surprising. Yes please let me know how that cylinder test goes.

I'd been working under the assumption that there was a bug in rotations. As such - I tested out having the joints "look at" the location of the next point in the rig. This causes a ton of issues if your rig isn't the same size as the recording of the human - and I was in the middle of trying to fix that part too.

If the rotations coming out of ARFoundation are correct - then what could be the root problem is that our joints aren't calibrated to the same "zero" rotation. Essentially we would have to convert the ARFoundation rotation into "joint space" and then do the rotation.

What would be super helpful in your test is coloring one end of the cylinder to ensure that the joints aren't flipping around 180 degrees or something weird like that....as that would also explain the busted behavior I've been seeing.

 

Here's a script that can generate the neutral rig used by ARKit 3. I got the information from a native iOS app. Apple provided an API to access "neturalPose" but Unity's ARFoundation didn't implement it.

Please drag a small sphere as the prefab to the "Sphere Prefab" slot.

https://paste.ubuntu.com/p/nvQ7QyQbhS/

 

This is probably because you are using
ARHumanBody.joints[index].anchorPose instead of ARHumanBody.joints[index].localPose
The former transform is relative the root, the later is the local transform

 

Thanks for posting that Drew. I used that to figure out that what they consider "neutral pose" and what my rig considers "neutral pose" are wildly different. It looks like those values are in world space?

The closest I've gotten to something that looks correct is creating a rotation per-joint that converts ARKit neutral pose to my rig's neutral pose:
Quaternion conversionRotation = Quaternion.Inverse(arkitNeutralRot) * myRigNeutralRot;

I then take the ARKit world rotation and apply the conversionRotation to it:
tform.rotation = ARHumanBody.joints[index].anchorPose * conversionRotation;

Unfortunately - the root of each new hierarchy (arms, legs, spine_0) looks like it has a 180 turn applied to it in the y direction.

I'm starting to wonder if there's an issue with coordinate systems or something going on. Though honestly - it feels near impossible to fix this without access to source or a working example.

 

Yeah I'm having the same issue. The neutral rig I got from Apple's API and the tracking rig information I got from Unity doesn't match exactly. Some joints has a 180 turn and some don't.
Either Apple made a mistake in their API, or Unity did some remapping under the hood.
I spent a day to manually change their rotation to make the mapping barely work. It's not worth it. I suggest to wait for updates from Apple/Unity.

 

Thanks guys, this has been super helpful.

@drewhong, we used the rig generated from the code you provided to create our own rig. We then used

Code (CSharp):

1. jointTransform.localRotation = joint.localPose.rotation;

...iteratively on each joint in the rig.

We now have a skinned mesh that looks mostly humanoid in AR, made out of cylinders and a half-sphere for the head. However, there is still a lot of twisting at the joints so we're looking into what's going on there.

 

Forgot to mention, we are totally ignoring positional data for simplicity. Also, we discovered that localPose.position was returning a zero-vector anyway. The only way for us to apply positions was to use anchoredPose.position and then re-convert it into the correct coordinate space. It didn't look much better on our rig anyway, so we left it off.

 

Happy (but sad) to hear I'm not alone on this one. Sounds like bug territory if we're all running into exactly the same thing.

I went digging around in the package source - but its basically a thin shim to a native dll....so there wasn't much to learn. Maybe we can get @jimmya for some wisdom?

 

Had a convo with Jimmy yesterday. He poked some people aaaaaaaaaaaand https://github.com/Unity-Technologies/arfoundation-samples was updated with a 3d rig example. It looks pretty thin though. I'll be digging into it today.

 

The HumanBodyTracking3D scene, in arfoundation-samples, has been updated to control the ARKit3 Robot model's skeleton, using the ARHumanBody pose detected by ARFoundation.
The HumanBodyTracker MonoBehavior takes care of detecting human bodies and passing on the detected body pose to the Robot's skeleton, via BoneController. The BoneController is a component on the Robot model, that maps the detected Human body pose to the Robot's skeleton bones.

 

Thanks for publishing and posting a response! I think this may have helped me figure out the problem.

Background: To quickly iterate - I had recorded some data from my iPhone and hacked up a quick body-tracking simulator. For simplicity I set world rotation of my rig to equal the world rotation of the pre-recorded body data.

When hooking this up to my character - I get total garbage. When I hooked it up to the robot rig - it worked perfectly. WTF. So clearly there's some assumption baked into the rigs that's different (which I THOUGHT that world rotation would overcome. Seems not). I've attached two images showing the knee joint selected (and the move gizmo set to local space).


I THINK this is the root of the problem. The robot rig has the X axis pointing to the next joint. My rig has negative-z pointing to the next joint. I get very different results when setting the same rotation on both rigs. I'll be honest - I wasn't expecting that result when I'm setting the world rotation of the joint....but I don't have extensive experience with character rigs so maybe this should have been obvious.

In any case - I think I was close with the idea of "defining a rest pose - calibrating against it to create an offset - applying that offset when reading in tracking data" I just need to figure out why my initial attempt failed.

 

GOT IT

Turns out the neutral pose that drew pasted has some bad rotations in it. This is all super hacked together but is a good starting point. Using this index map:

Code (CSharp):

1.  
2.     enum JointIndices
3.     {
4.         Invalid = -1,
5.         Root = 0, // parent: Invalid
6.         Hips = 1, // parent: Root
7.         LeftUpLeg = 2, // parent: Hips
8.         LeftLeg = 3, // parent: LeftUpLeg
9.         LeftFoot = 4, // parent: LeftLeg
10.         LeftToes = 5, // parent: LeftFoot
11.         LeftToesEnd = 6, // parent: LeftToes
12.         RightUpLeg = 7, // parent: Hips
13.         RightLeg = 8, // parent: RightUpLeg
14.         RightFoot = 9, // parent: RightLeg
15.         RightToes = 10, // parent: RightFoot
16.         RightToesEnd = 11, // parent: RightToes
17.         Spine1 = 12, // parent: Hips
18.         Spine2 = 13, // parent: Spine1
19.         Spine3 = 14, // parent: Spine2
20.         Spine4 = 15, // parent: Spine3
21.         Spine5 = 16, // parent: Spine4
22.         Spine6 = 17, // parent: Spine5
23.         Spine7 = 18, // parent: Spine6
24.         RightShoulder1 = 19, // parent: Spine7
25.         RightShoulder2 = 20, // parent: RightShoulder1
26.         RightArm = 21, // parent: RightShoulder2
27.         RightForearm = 22, // parent: RightArm
28.         RightHand = 23, // parent: RightForearm
29.         RightHandThumbStart = 24, // parent: RightHand
30.         RightHandThumb1 = 25, // parent: RightHandThumbStart
31.         RightHandThumb2 = 26, // parent: RightHandThumb1
32.         RightHandThumbEnd = 27, // parent: RightHandThumb2
33.         RightHandIndexStart = 28, // parent: RightHand
34.         RightHandIndex1 = 29, // parent: RightHandIndexStart
35.         RightHandIndex2 = 30, // parent: RightHandIndex1
36.         RightHandIndex3 = 31, // parent: RightHandIndex2
37.         RightHandIndexEnd = 32, // parent: RightHandIndex3
38.         RightHandMidStart = 33, // parent: RightHand
39.         RightHandMid1 = 34, // parent: RightHandMidStart
40.         RightHandMid2 = 35, // parent: RightHandMid1
41.         RightHandMid3 = 36, // parent: RightHandMid2
42.         RightHandMidEnd = 37, // parent: RightHandMid3
43.         RightHandRingStart = 38, // parent: RightHand
44.         RightHandRing1 = 39, // parent: RightHandRingStart
45.         RightHandRing2 = 40, // parent: RightHandRing1
46.         RightHandRing3 = 41, // parent: RightHandRing2
47.         RightHandRingEnd = 42, // parent: RightHandRing3
48.         RightHandPinkyStart = 43, // parent: RightHand
49.         RightHandPinky1 = 44, // parent: RightHandPinkyStart
50.         RightHandPinky2 = 45, // parent: RightHandPinky1
51.         RightHandPinky3 = 46, // parent: RightHandPinky2
52.         RightHandPinkyEnd = 47, // parent: RightHandPinky3
53.         LeftShoulder1 = 48, // parent: Spine7
54.         LeftShoulder2 = 49, // parent: LeftShoulder1
55.         LeftArm = 50, // parent: LeftShoulder2
56.         LeftForearm = 51, // parent: LeftArm
57.         LeftHand = 52, // parent: LeftForearm
58.         LeftHandThumbStart = 53, // parent: LeftHand
59.         LeftHandThumb1 = 54, // parent: LeftHandThumbStart
60.         LeftHandThumb2 = 55, // parent: LeftHandThumb1
61.         LeftHandThumbEnd = 56, // parent: LeftHandThumb2
62.         LeftHandIndexStart = 57, // parent: LeftHand
63.         LeftHandIndex1 = 58, // parent: LeftHandIndexStart
64.         LeftHandIndex2 = 59, // parent: LeftHandIndex1
65.         LeftHandIndex3 = 60, // parent: LeftHandIndex2
66.         LeftHandIndexEnd = 61, // parent: LeftHandIndex3
67.         LeftHandMidStart = 62, // parent: LeftHand
68.         LeftHandMid1 = 63, // parent: LeftHandMidStart
69.         LeftHandMid2 = 64, // parent: LeftHandMid1
70.         LeftHandMid3 = 65, // parent: LeftHandMid2
71.         LeftHandMidEnd = 66, // parent: LeftHandMid3
72.         LeftHandRingStart = 67, // parent: LeftHand
73.         LeftHandRing1 = 68, // parent: LeftHandRingStart
74.         LeftHandRing2 = 69, // parent: LeftHandRing1
75.         LeftHandRing3 = 70, // parent: LeftHandRing2
76.         LeftHandRingEnd = 71, // parent: LeftHandRing3
77.         LeftHandPinkyStart = 72, // parent: LeftHand
78.         LeftHandPinky1 = 73, // parent: LeftHandPinkyStart
79.         LeftHandPinky2 = 74, // parent: LeftHandPinky1
80.         LeftHandPinky3 = 75, // parent: LeftHandPinky2
81.         LeftHandPinkyEnd = 76, // parent: LeftHandPinky3
82.         Neck1 = 77, // parent: Spine7
83.         Neck2 = 78, // parent: Neck1
84.         Neck3 = 79, // parent: Neck2
85.         Neck4 = 80, // parent: Neck3
86.         Head = 81, // parent: Neck4
87.         Jaw = 82, // parent: Head
88.         Chin = 83, // parent: Jaw
89.         Nose = 84, // parent: Head
90.         RightEye = 85, // parent: Head
91.         RightEyeUpperLid = 86, // parent: RightEye
92.         RightEyeLowerLid = 87, // parent: RightEye
93.         RightEyeBall = 88, // parent: RightEye
94.         LeftEye = 89, // parent: Head
95.         LeftEyeUpperLid = 90, // parent: LeftEye
96.         LeftEyeLowerLid = 91, // parent: LeftEye
97.         LeftEyeBall = 92, // parent: LeftEye
98.     }

I dumped the T pose of the robot into a dictionary IN WORLD COORDINATES:

Code (CSharp):

1.         private Dictionary<JointIndices, Quaternion> ROBOTPOSE = new Dictionary<JointIndices, Quaternion>
2.         {
3.             { JointIndices.Root, new Quaternion(0f, 0f, 0f, 1f) },{ JointIndices.Hips, new Quaternion(0f, 0f, 0f, 1f) },{ JointIndices.LeftUpLeg, new Quaternion(-0.5251914f, 0.4729119f, 0.5257655f, -0.4733909f) },{ JointIndices.RightUpLeg, new Quaternion(-0.4731499f, -0.5254809f, 0.4731455f, 0.5254829f) },{ JointIndices.Spine1, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftLeg, new Quaternion(-0.4589739f, 0.5374146f, 0.4594831f, -0.5379627f) },{ JointIndices.RightLeg, new Quaternion(-0.5376937f, -0.4592238f, 0.5376896f, 0.4592263f) },{ JointIndices.Spine2, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftFoot, new Quaternion(-0.6927235f, 0.1400152f, 0.6934642f, -0.1401326f) },{ JointIndices.RightFoot, new Quaternion(-0.1400807f, -0.6930928f, 0.1400763f, 0.6930935f) },{ JointIndices.Spine3, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftToes, new Quaternion(-0.7067051f, 0.006181896f, 0.7074544f, -0.006156165f) },{ JointIndices.RightToes, new Quaternion(-0.006176293f, -0.7070798f, 0.006170991f, 0.7070799f) },{ JointIndices.Spine4, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftToesEnd, new Quaternion(-0.7067052f, 0.006181717f, 0.7074544f, -0.006156351f) },{ JointIndices.RightToesEnd, new Quaternion(-0.00617677f, -0.7070798f, 0.006170507f, 0.7070799f) },{ JointIndices.Spine5, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.Spine6, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.Spine7, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.RightShoulder1, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },{ JointIndices.LeftShoulder1, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },{ JointIndices.Neck1, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightShoulder2, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },{ JointIndices.LeftShoulder2, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },{ JointIndices.Neck2, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightArm, new Quaternion(-0.7072251f, -4.330277E-05f, -0.0008040965f, 0.706988f) },{ JointIndices.LeftArm, new Quaternion(-0.7069915f, -0.0008285469f, 7.144874E-05f, -0.7072218f) },{ JointIndices.Neck3, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightForearm, new Quaternion(-0.706572f, -0.03035438f, -0.03110623f, 0.7063052f) },{ JointIndices.LeftForearm, new Quaternion(-0.7063079f, -0.03109138f, 0.03034304f, -0.7065705f) },{ JointIndices.Neck4, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightHand, new Quaternion(-0.0001886487f, -0.04345921f, -0.0005316734f, 0.9990551f) },{ JointIndices.LeftHand, new Quaternion(-0.999056f, -0.0005290508f, 0.04344079f, -0.0001838654f) },{ JointIndices.Head, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.RightHandThumbStart, new Quaternion(-0.563176f, -0.450308f, -0.1651125f, 0.6728995f) },{ JointIndices.RightHandIndexStart, new Quaternion(-0.001846135f, -0.1037805f, -0.0003555566f, 0.9945986f) },{ JointIndices.RightHandMidStart, new Quaternion(-0.001781464f, -0.04345801f, -0.0004611611f, 0.9990537f) },{ JointIndices.RightHandRingStart, new Quaternion(-0.001758099f, 0.006566688f, -0.0005474091f, 0.9999769f) },{ JointIndices.RightHandPinkyStart, new Quaternion(-0.00176309f, 0.05727234f, -0.0006380975f, 0.9983569f) },{ JointIndices.LeftHandThumbStart, new Quaternion(-0.6729128f, -0.16509f, 0.4503065f, -0.5631677f) },{ JointIndices.LeftHandIndexStart, new Quaternion(-0.9946009f, -0.0003358126f, 0.1037586f, -0.00184305f) },{ JointIndices.LeftHandMidStart, new Quaternion(-0.9990543f, -0.000755012f, 0.04344012f, -0.001763821f) },{ JointIndices.LeftHandRingStart, new Quaternion(-0.9999767f, -0.0005270988f, -0.006585166f, -0.001753271f) },{ JointIndices.LeftHandPinkyStart, new Quaternion(-0.9983559f, -0.0006175488f, -0.05729078f, -0.001757562f) },{ JointIndices.Jaw, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.Nose, new Quaternion(-0.5000001f, -0.5000001f, 0.5000001f, 0.5000001f) },{ JointIndices.RightEye, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEye, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightHandThumb1, new Quaternion(-0.6383249f, -0.3353892f, -0.0345653f, 0.691998f) },{ JointIndices.RightHandIndex1, new Quaternion(-0.141001f, -0.06067762f, 0.1492173f, 0.976817f) },{ JointIndices.RightHandMid1, new Quaternion(-0.001823545f, -0.01739544f, 0.08677565f, 0.9960744f) },{ JointIndices.RightHandRing1, new Quaternion(0.0863688f, 0.043634f, 0.0978535f, 0.9904854f) },{ JointIndices.RightHandPinky1, new Quaternion(0.1280032f, 0.08536577f, 0.1233153f, 0.980368f) },{ JointIndices.LeftHandThumb1, new Quaternion(-0.6920068f, -0.03454062f, 0.3353892f, -0.6383165f) },{ JointIndices.LeftHandIndex1, new Quaternion(-0.9768156f, 0.1492398f, 0.06065904f, -0.1409955f) },{ JointIndices.LeftHandMid1, new Quaternion(-0.9961001f, 0.08648346f, 0.01738015f, -0.001796693f) },{ JointIndices.LeftHandRing1, new Quaternion(-0.9904823f, 0.09787226f, -0.04365361f, 0.08637467f) },{ JointIndices.LeftHandPinky1, new Quaternion(-0.980363f, 0.1233332f, -0.08538568f, 0.1280103f) },{ JointIndices.Chin, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.RightEyeUpperLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightEyeLowerLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightEyeBall, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeUpperLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeLowerLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeBall, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightHandThumb2, new Quaternion(-0.6677005f, -0.2722507f, 0.03234765f, 0.692105f) },{ JointIndices.RightHandIndex2, new Quaternion(-0.1477825f, -0.0415144f, 0.2769809f, 0.9485351f) },{ JointIndices.RightHandMid2, new Quaternion(-0.004873842f, -0.01679799f, 0.2617181f, 0.964986f) },{ JointIndices.RightHandRing2, new Quaternion(0.0938583f, 0.02353925f, 0.3136097f, 0.9446088f) },{ JointIndices.RightHandPinky2, new Quaternion(0.1357787f, 0.07236205f, 0.2191423f, 0.9634857f) },{ JointIndices.LeftHandThumb2, new Quaternion(-0.6921116f, 0.03237311f, 0.2722515f, -0.6676922f) },{ JointIndices.LeftHandIndex2, new Quaternion(-0.9485307f, 0.277003f, 0.04149672f, -0.1477744f) },{ JointIndices.LeftHandMid2, new Quaternion(-0.965063f, 0.2614351f, 0.01678771f, -0.004844725f) },{ JointIndices.LeftHandRing2, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinky2, new Quaternion(-0.963479f, 0.2191598f, -0.07238111f, 0.1357878f) },{ JointIndices.RightHandThumbEnd, new Quaternion(-0.6677005f, -0.2722507f, 0.03234762f, 0.692105f) },{ JointIndices.RightHandIndex3, new Quaternion(-0.1494033f, -0.03523868f, 0.3167771f, 0.9359966f) },{ JointIndices.RightHandMid3, new Quaternion(-0.005977005f, -0.01643786f, 0.325129f, 0.945508f) },{ JointIndices.RightHandRing3, new Quaternion(0.09519702f, 0.01735026f, 0.3747187f, 0.9220752f) },{ JointIndices.RightHandPinky3, new Quaternion(0.1402616f, 0.06323615f, 0.2822996f, 0.9469081f) },{ JointIndices.LeftHandThumbEnd, new Quaternion(-0.6921116f, 0.03237309f, 0.2722515f, -0.6676922f) },{ JointIndices.LeftHandIndex3, new Quaternion(-0.9358417f, 0.3172399f, 0.03515086f, -0.1494111f) },{ JointIndices.LeftHandMid3, new Quaternion(-0.9456036f, 0.3248518f, 0.01642954f, -0.005947292f) },{ JointIndices.LeftHandRing3, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinky3, new Quaternion(-0.9469005f, 0.2823165f, -0.06325462f, 0.1402719f) },{ JointIndices.RightHandIndexEnd, new Quaternion(-0.1494033f, -0.03523868f, 0.3167771f, 0.9359965f) },{ JointIndices.RightHandMidEnd, new Quaternion(-0.005977005f, -0.01643786f, 0.325129f, 0.945508f) },{ JointIndices.RightHandRingEnd, new Quaternion(0.09519702f, 0.01735026f, 0.3747187f, 0.9220752f) },{ JointIndices.RightHandPinkyEnd, new Quaternion(0.1402616f, 0.06323615f, 0.2822996f, 0.9469081f) },{ JointIndices.LeftHandIndexEnd, new Quaternion(-0.9358417f, 0.3172399f, 0.03515095f, -0.1494111f) },{ JointIndices.LeftHandMidEnd, new Quaternion(-0.9456036f, 0.3248518f, 0.01642954f, -0.005947292f) },{ JointIndices.LeftHandRingEnd, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinkyEnd, new Quaternion(-0.9469005f, 0.2823165f, -0.06325462f, 0.1402719f) },
4.  
5.         };

At initialization - I compare my character's T pose to the robot's T pose - create a rotation that will convert from "robot joint space" to "my joint's space" - and store that value for later:

Code (CSharp):

1.             foreach (var jointKeyValuePair in MyRigJoints)
2.             {
3.                 //current world rotation of the joint in my rig
4.                 Quaternion startingRot = jointKeyValuePair.Value.rotation;
5.                 //arkit t pose
6.                 Quaternion robotJointRotation = Quaternion.identity;
7.                 JointIndices index = jointKeyValuePair.Key;
8.                 if (!ROBOTPOSE.TryGetValue(index, out robotJointRotation))
9.                 {
10.                     Debug.LogError("key missing is " + jointKeyValuePair.Key.ToString());
11.                     continue;
12.                 }
13.                 //conversionRotation can convert ARKit rotations to somewthing we can apply to our rig
14.                 var conversionRotation = Quaternion.Inverse(robotJointRotation) * startingRot;
15.  
16.                 RotationCompensations[index] = compensate;
17.             }

Then later when you get the updated human body pose from ARKit, you apply that stored compensation to the arkit rotations:

Code (CSharp):

1. myjointTransform.rotation = ARHumanBody.joints[index].anchorPose.rotation * RotationCompensations[index];

The closer your character's T pose is to the reference T pose - the more accurate your animations will be.

Since all of this is in world space - you'll still need to account for any parent transforms. Eventually I'd like to get this working in local space.....but really I just want to move on at this point.

Hope that helps and fixes everyone else's issues. Thanks again for all the help folks.

 

Sorry for the bad rotations... I really have no idea where it went wrong. I'll look in to it. Sorry for the confusion.

 

No worries. It was impossible to tell since nothing at all was working. In all honesty - it still helped me significantly. Once I had correct data I was able to quickly fashion a solution. Wouldn't have been able to do that without your post.

 

@distastee Don't suppose you've made one of these for the changed mappings and robot in beta 4 yet?

 

Not yet. Just upgraded yesterday and all my S*** is completely hosed. I'm working on fixing it today - which means I'll post here once I figure out wtf went wrong.

 

I don't think any changes are necessary. The indicies all switched place - but it was thankfully built to handle that. From what I can tell - all the rotations + tracking seem to remain accurate.

 

@distastee I'm trying to implement your suggestion in my project. I kinda struck. can you please share any demo scene if you have created with your modified script for body tracking. Please help. Thanks

 

Hello everyone. I'm trying to implement my own model on the robot rig without success. All rotations are reversed or do not follow well. Could someone send a test scene that works with our own models? I am not an experienced programmer so it would be super appreciated. Thank you in advance!

 

Has anyone been able to substitute their own rigged model? is there a way (Ican't figure) to export/ "unpack" the provided robot skeleton in "Tracking Bodies in Motion" ARFoundation 3.0 /ARKit 3.0 to see just how it is weighted and rigged?

 

Can't share any of the work I've done so far (sorry!) but I've mostly gotten our own rig working in the app. I have a very strange bug that occasionally tilts the characters off-axis and haven't been able to figure out why yet.

I can give some tips, though. There are a bunch of assumptions baked into the API. They are:
1) Your hips are at the origin. Not the feet. If you want correct body rotations and your rig's hips aren't at the origin - you have to correct for this by doing a RotateAround(). This can cause a host of issues elsewhere. It's not ideal.
2) The hips are (Y or Z I forget) up. You will get funky rotations if your up vector doesn't match.
3) The distance from your hips to your feet is something like 0.8954496. If the distance doesn't match - you get some funky positional stuff happening. You have to correct for this by finding the correct ratio and multiplying by that.
4) If you don't have the exact number of bones - using local rotation will give you the wrong pose. You either need to account for the skipped bones by concatenating the local space rotations together OR pose your character with world rotations and then rotate the body into place. I haven't tried the former - and I constantly run into issues with the latter.

Suffice to say - I would LOVE LOVE LOVE another level of abstraction between this API and driving our characters. The current method is extremely bug prone and not very flexible.

 

thanks for sharing this...reminds me (unfortunately) of very early experiments w/ kinect camera..

 

You're welcome. Sorry I can't be of a bigger help.

Yeah - it's super early days with this stuff so it's very rough around the edges. The lack of a robust community using it (currently) also makes things tough. It's been a few months of me slamming my face against the keyboard getting everything to work. My co-founder just looks at me like "Can't you work any faster?"

 

ugh
i think apple is making this very difficult--

 

I'm struggling with this too. I'm trying to make a FBX creator, but I always end up with really weird rotations.

Who in the world makes the T pose like this?! Apple



Tried some weeks ago making a post on StackOverflow, but no luck Seems no-one has managed to control a puppet different than the demo project by Apple.

https://stackoverflow.com/questions...from-3d-skeleton-in-arkit-3/58476267#58476267

I've done it before with Kinects, but Apple is making it so weeeeeird

 

I have it working with 3 different rigs. Mine looked like yours at one point. That pose is not the T-Pose that Apple uses - it actually means that the rig you're applying it to has different up/forward/left vectors than Apple is expecting. Load up the example rig and select a few joints with the LOCAL gizmo active to see what Apple/Unity EXPECT your rig to look like.

To fix this - you need to figure out the rotation you would need to get from THEIR joints to YOUR joints. I do this by comparing the tpose of every joint of my character to the tpose of their character - and saving the rotation:

correctiveRotation = Quaternion.Inverse(theirTPoseRotation) * myTposeRotation;

then I apply that corrective rotation when reading in from the body tracking data:

finalJointRotation = arkitBodyTrackedJointRotation * correctiveRotation;

You need to keep in mind that if you are setting the LOCAL rotation for every joint - then you need to have literally the same hierarchy and number of joints as ARKit does. Your pose will be incorrect by varying amounts if you don't. If you set the GLOBAL rotation for your joints this doesn't matter. The downside of global rotations is that you have to add a ton of complexity in if you want to also move and rotate the character around.

Hope that helps. I wish it were a better API that plugged directly into their human body rig system so that we could seamlessly handle retargeting / different orientations / scales / joint orders / joint directions / etc. I'm sure it will happen at some point. Until then.....gotta roll our own implementations of that stuff.

 

So, how exactly should we make a rig for new character for ARKit body feature to work properly?

 

Hello. In which script(s) have you added your own?

 

Wish there was a rig creator for fbx humanoids for arkit! All my rigs are bunk,

thx for the enum tho!