Bug ARFoundation: ImageTracking with WorldMap

Hi,
I'm writing an AR program that uses world map and image tracking.
Then I ran into a problem.

What I want to do is simple.
1. Load the world map.
2. Add the image to the image reference library. (at runtime)

Then image tracking does not work properly.
The "trackedImagesChanged" event of "ARTrackedImageManager" is fired, but the ARTrackedImage passed as an argument is messed up. "ARTrackedImage.referenceImage.name" should normally be set to the name you specified during registration, but in the above situation it is set to "null". It is not possible to determine which image was detected.

Is there a way to fix the problem?
Best Regard,

[Environment]
- Unity 2019.4.16f1
- ARFoundation 4.1.1
- ARKit XR Plugin 4.1.1
- iPad Pro 2020
- iOS 14.3

Code (CSharp):

1.  
2. using System.Collections;
3. using System.IO;
4. using TMPro;
5. using Unity.Collections;
6. using UnityEngine;
7. using UnityEngine.UI;
8. using UnityEngine.XR.ARFoundation;
9. using UnityEngine.XR.ARKit;
10. using UnityEngine.XR.ARSubsystems;
11.  
12. namespace SoftCube.Domain
13. {
14.     public class Manager : MonoBehaviour
15.     {
16.         [SerializeField]
17.         private ARSession arSession = default;
18.  
19.         [SerializeField]
20.         private ARTrackedImageManager arTrackedImageManager = default;
21.  
22.         [SerializeField]
23.         private Button worldMapSaveButton = default;
24.  
25.         [SerializeField]
26.         private TMP_Text messageText = default;
27.  
28.         private IEnumerator Start()
29.         {
30.             arTrackedImageManager.trackedImagesChanged += ArTrackedImageManager_trackedImagesChanged;
31.             worldMapSaveButton.onClick.AddListener(OnWorldMapSaveButtonClick);
32.  
33.             var worldMapFilePath = Path.Combine(Application.persistentDataPath, "ImageTracking.worldmap");
34.             if (File.Exists(worldMapFilePath))
35.             {
36.                 yield return StartCoroutine(LoadWorldMapCoroutine(worldMapFilePath));
37.             }
38.  
39.             LoadImage();
40.         }
41.  
42.         private void ArTrackedImageManager_trackedImagesChanged(ARTrackedImagesChangedEventArgs e)
43.         {
44.             foreach (var image in e.added)
45.             {
46.                 Debug.Log(image.referenceImage.name);
47.             }
48.             foreach (var image in e.updated)
49.             {
50.                 Debug.Log(image.referenceImage.name);
51.             }
52.         }
53.  
54.         private void OnWorldMapSaveButtonClick()
55.         {
56.             var worldMapFilePath = Path.Combine(Application.persistentDataPath, "ImageTracking.worldmap");
57.             StartCoroutine(SaveWorldMapCoroutine(worldMapFilePath));
58.         }
59.  
60.         public IEnumerator LoadWorldMapCoroutine(string worldMapFilePath)
61.         {
62.             messageText.text = $"Load WorldMap.";
63.  
64.             var arSessionSubsystem = (ARKitSessionSubsystem)arSession.subsystem;
65.             if (arSessionSubsystem == null)
66.             {
67.                 messageText.text = "No session subsystem available. Could not load.";
68.                 yield return new WaitForSeconds(5.0f);
69.                 yield break;
70.             }
71.             if (!File.Exists(worldMapFilePath))
72.             {
73.                 messageText.text = $"File {worldMapFilePath} does not exist.";
74.                 yield return new WaitForSeconds(5.0f);
75.                 yield break;
76.             }
77.  
78.             var allBytes = File.ReadAllBytes(worldMapFilePath);
79.             var data = new NativeArray<byte>(allBytes.Length, Allocator.Temp);
80.             data.CopyFrom(allBytes);
81.  
82.             ARWorldMap worldMap;
83.             if (ARWorldMap.TryDeserialize(data, out worldMap))
84.             {
85.                 data.Dispose();
86.             }
87.  
88.             if (!worldMap.valid)
89.             {
90.                 messageText.text = "Data is not a valid ARWorldMap.";
91.                 yield return new WaitForSeconds(5.0f);
92.                 yield break;
93.             }
94.  
95.             messageText.text = $"Apply WorldMap.";
96.             arSessionSubsystem.ApplyWorldMap(worldMap);
97.  
98.             yield return new WaitForSeconds(1.0f);
99.             while (ARSession.state != ARSessionState.SessionTracking)
100.             {
101.                 yield return null;
102.             }
103.             messageText.text = string.Empty;
104.         }
105.  
106.         public IEnumerator SaveWorldMapCoroutine(string worldMapFilePath)
107.         {
108.             messageText.text = $"Save WorldMap.";
109.  
110.             var arSessionSubsystem = (ARKitSessionSubsystem)arSession.subsystem;
111.             if (arSessionSubsystem == null)
112.             {
113.                 messageText.text = "No session subsystem available. Could not save.";
114.                 yield return new WaitForSeconds(5.0f);
115.                 yield break;
116.             }
117.  
118.             var request = arSessionSubsystem.GetARWorldMapAsync();
119.             while (!request.status.IsDone())
120.             {
121.                 yield return null;
122.             }
123.             if (request.status.IsError())
124.             {
125.                 messageText.text = $"Session serialization failed with status {request.status}";
126.                 yield return new WaitForSeconds(5.0f);
127.                 yield break;
128.             }
129.  
130.             var worldMap = request.GetWorldMap();
131.             request.Dispose();
132.  
133.             var data = worldMap.Serialize(Allocator.Temp);
134.             var file = File.Open(worldMapFilePath, FileMode.Create);
135.             var writer = new BinaryWriter(file);
136.             writer.Write(data.ToArray());
137.             writer.Close();
138.             data.Dispose();
139.             worldMap.Dispose();
140.  
141.             yield return new WaitForSeconds(3.0f);
142.             messageText.text = string.Empty;
143.         }
144.  
145.         private void LoadImage()
146.         {
147.             if (arTrackedImageManager.CreateRuntimeLibrary() is MutableRuntimeReferenceImageLibrary referenceLibrary)
148.             {
149.                 arTrackedImageManager.referenceLibrary = referenceLibrary;
150.  
151.                 foreach (var imageFilePath in Directory.GetFiles(Application.persistentDataPath, "*.png"))
152.                 {
153.                     var texture = new Texture2D(1, 1);
154.                     texture.LoadImage(File.ReadAllBytes(imageFilePath));
155.  
156.                     var job = referenceLibrary.ScheduleAddImageWithValidationJob(texture, Path.GetFileName(imageFilePath), 0.1f);
157.                     job.jobHandle.Complete();
158.                     Debug.Log(job.status);
159.                 }
160.                 arTrackedImageManager.enabled = true;
161.             }
162.         }
163.     }
164. }
165.  

 

Does the textureGuid in the referenceImage match?

 

Thank you for reply.

referenceImage.name is null.
referenceImage.guid is 00000000-0000-0000-0000-000000000000.
referenceImage.textureGuid is 00000000-0000-0000-0000-000000000000.
referenceImage.size is (0.0, 0.0).
referenceImage.texture is null.

I think there are three possibilities.

1. There is a mistake in my usage of the API.
   
   
2. Bug of ARFoundation/ARKit
   
   
3. Specification of ARFoundation/ARKit.
   In other words, WorldMap is not designed to be used with image tracking.

Which case is this time?

 

I can tell you for certain that it's not ARFoundation. ARFoundation is simply a wrapper around ARKit.
MutableRuntimeReferenceImageLibrary is an abstract class in ARFoundation, that is implemented by the Provider.

ARKit SHOULD support mutable referencelibraries (it would return null for that library if it didn't).
Your usage also seems to be correct.
My guess is a bug in the ARKit XRPlugin, but I'm not sure.

Have you tried using one of the other Add-Methods on the MutableRuntimeReferenceImageLibrary?

 

I tried the following code. This is what you call "the other Add-Methods", right?
The result did not change (it does not work).

Code (CSharp):

1.  
2. //var job = referenceLibrary.ScheduleAddImageWithValidationJob(texture, Path.GetFileName(imageFilePath), 0.05f);
3. var guid = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] SerializableGuid();
4. var textureGuid = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] SerializableGuid();
5. var size = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] Vector2(0.05f, 0.05f);
6. var name = Path.GetFileName(imageFilePath);
7. var referenceImage = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] XRReferenceImage(guid, textureGuid, size, name, texture);
8. var job = referenceLibrary.ScheduleAddImageWithValidationJob(texture.GetRawTextureData<byte>(), [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] Vector2Int(texture.width, texture.height), texture.format, referenceImage);
9. job.jobHandle.Complete();
10.  

I think this is a common use case.
If it's a bug in the ARKit XR Plugin, it should be investigated and fix in a future release.

 

Thank you for reply.
I will make a report.

 

I received the following reply from Unity's QA team regarding this matter.
```
I think I found the cause of this issue: It looks like you haven't created an Image Tracking subsystem.
Can you verify that creating one fixes the problem that you're experiencing?
```
But I didn't know how to verify it.
Is there anyone who knows how to create an Image Tracking subsystem?

 

Adding ARTrackedImageManager to the ARSessionOrigin game object should create the Image Tracking subsystem automatically.
Please ensure that you've also enabled the ARKit provider in the 'XR Plugin Management' window.



Does this scene from the official AR Foundation Samples work for you?
https://github.com/Unity-Technologi...g/BasicImageTracking/BasicImageTracking.unity

 

Thank you for reply.

I checked the ARKit provider in the XR Plugin Management window, and it was enabled.
And AR Foundation Samples work for me.

My perception is the same as yours. In other words, I believe that the Image Tracking subsystem is generated automatically.
So, I am having a hard time understanding how to fix the bug regarding the reply from QA team.

 

hello there. Do you have any update on this regard? I tried to write this exact same code, but my mutableLibrary.count is equal to 0 (so no images have been loaded in the mutable library). I have this problem with both ARCore and ARKiit. I'm sure that the application with a "standard" image reference library build in edit mode in Unity works fine.

 

Please check the AddReferenceImageJobState.status after AddReferenceImageJobState is finished (only available in AR Foudatuon >= 4.1). If the status is ErrorInvalidImage, it means your image is not suitable for Image Tracking.

 

Also, there is a tool to validate images for ARCore Image Tracking:
https://forum.unity.com/threads/can...oints-from-target-image.1137007/#post-7383557