Question ARFoundation/ARKit tracking quality: drifting objects

In our current project, we are overlapping real objects with their virtual representation. The objects are pretty small (2 by 15cm). We not only overlay the objects for visual representation, but also for measuring distances between the objects and try to reach an accuracy within millimetres.

While this works quite well in most cases, we observe that sometimes the objects start to drift off when moving the device. The virtual objects are offset of the real objects in the same direction as the movement of the device, also going back to the original position when moving the device back. Until now, we did not find a way to detect if the tracking is stable or not.

We are using Unity 2020.3, ARFoundation 4.1.7, ARKit XR Plugin 4.1.7 running on iOS 14.7.1 (iPad Pro 11’ 2. and 3. Generation). Strangely, we also find that some devices work better than others, see https://forum.unity.com/threads/ark...ences-between-ios-devices-and-models.1165631/

To nail this down we created a simple ARFoundation sample project. Using a Raycast we find a position (e.g. plane or feature point) to anchor a simple test object. In this simple sample project, we feel the stability of the tracking (moving the device and expecting the objects to be at a fixed position) to be working well for some objects, while observing the same ‘drifts’ we see in our main project.

What we did try so far:

• making sure the AR session is properly initialised (moving/orbiting the device around the scene)
• make sure the environment is good regarding lighting, avoiding glass or other reflecting materials, etc)
• adding ARAnchor to the objects
  • using gameobject.AddComponent<ARAnchor>()
  • using arAnchorManager.AddAnchor(position)
  • parenting the objects to ARSessionOrigin
• testing on different devices
• checking for drops in framerate: constant at 60fps

Generally, we feel that adding ARAnchors to the object does not help a lot, or only in extreme situations (e.g. when moving the device completely out to of the scene, shaking it etc.). Then we notice that the trackingState of the anchors are .NONE when the anchors are created using the gameobject.AddComponent<ARAnchor>() method, but .TRACKING when created using the ARAnchorManager instance’s AddAnchor() method. No difference between tracked and non-tracked objects/anchors, though.

The demo video (link below) shows the placed objects to slightly drift with the device movement. Red spheres (anchored) are at the same position as the slightly smaller grey cubes (no anchors).

https://drive.google.com/file/d/17PnI6nJX0LaJ0ypZrvUDAZTvAds5v5nM/view?usp=sharing

Code (CSharp):

1.  
2. void Update()
3. {
4.     if (Input.touchCount > 0)
5.     {
6.         Touch touch = Input.GetTouch(0);
7.         if (touch.phase == TouchPhase.Began)
8.         {
9.             if (arRaycastManager.Raycast(touch.position, _raycastHits, TrackableType.All))
10.             {
11.                
12.                 //adding a sphere at the first raycast hit and adding an ARAnchor to it
13.                 GameObject placedObject = GameObject.CreatePrimitive(PrimitiveType.Sphere);
14.                 placedObject.transform.localScale = Scale;
15.                 placedObject.transform.position = _raycastHits[0].pose.position;
16.                 placedObject.transform.SetParent(aRSessionOrigin.transform);
17.  
18.                 ARAnchor anchor = placedObject.GetComponentInParent<ARAnchor>();
19.                 if (anchor == null)
20.                 {
21.                     placedObject.AddComponent<ARAnchor>();
22.                     Debug.Log($"New anchor added. ");
23.                 }
24.                 _placedObjects.Add(placedObject);
25.                
26.                 //adding a slightly smaller cube without an anchor at the same position
27.                 GameObject placedObject2 = GameObject.CreatePrimitive(PrimitiveType.Cube);
28.                 placedObject2.transform.localScale = Scale * 0.99f;
29.                 var cameraTransform = aRSessionOrigin.camera.transform;
30.                 placedObject2.transform.position = _raycastHits[0].pose.position;
31.                 _placedObjects.Add(placedObject2);
32.             }
33.         }
34.     }
35.    
36.     UpdateTrackingStates();
37. }
38.  
39.  

We are aware of the fact that current tracking technology in these devices is limited and intended to be used to place chairs, tables, pictures on the wall etc. We are pushing this to the limit. However, we saw that it works for our use case sometimes, but not all the time. We would like to have an indicator of the tracking quality to detect the drifting. The trackingState of trackable objects does not help here, unfortunately.

Are we missing something?

 

We had similar problems in the beginning, but noticed that it was due to the lack of occlusion. We realized that the behaviour similar to your video was not drifting at all. When we activated occlusion, we could see that the objects were not drifting. It just looks like they are drifting due to the objects not being occluded by the real world.

That being said, we still have problems with actual drifting. Did you ever solve this/improve stability? We have also tried the points stated by you, with no improvements. Would be interesting to know if you ever came across a solution to drifting.

 

I'm not sure what you refer to with 'lack of occlusion'. I think this could be a perspective issue showing.

In our use case (ArUco marker detection), we improved the accuracy and precision by averaging* different poses we have for the same object (marker). *The trick was not to average the actual pose/position but to intersect the camera rays. This led to a position that showed less drifting. Still, drifting sometimes is still there.

 

I can explain this apparent drifting: you anchored those cubes with their CENTER aligned with the surface of the keyboard, while you expect the cubes to be aligned with their BASE with the surface. So the drift that you see is because the base of the cubes is actually UNDER the surface of the keyboard, making you actually see a parallax, and not a drift.

If you have instead an object with its pivot point on its base, you will not notice it anymore.

If you had occlusions, you would see the bottom half part of the cube UNDER the surface occluded, so you will see only the top half part of these cubes perfectly aligned on the surface.

Imagine like a necklace with pearls in it: the string is the keyboard surface, the pearls are the cubes, with the string passing it at its center. The behaviour in the video is perfectly expected as geometrically correct

 

Thanks a lot for the explanation - yes, it makes complete sense. The pivot should be on the surface.

This sample project was a quick showcase to demo the effect of the drifting and I missed that point. However, we are still seeing some drifts now and then (in the actual project). We found that this usually comes from bad lighting conditions or sometimes you just can be unlucky. The drifts are minor but still relevant when trying to achieve millimeter precision and accuracy.

Anyway, thanks a lot for the input!