Dynamic XRReferenceImageLibrary in AR Foundation

Interesting. I'm using AR Foundation to hopefully keep my app cross-platform, but I have not started iOS testing yet. I'm only at the very start of this project and trying out different APIs and solutions. Is there a place to submit bugs for this?

 

Wow. This actually worked for me. I was able to load an image from a web url, add it to my reference library, and then start tracking that image immediately after it finished loading. I didn't need to reload the scene or anything.

The problem is that this code is extremely similar to my own, so I have no idea why this worked and why my code did not. I will need to do some more testing.

Thank you for the link!

 

damn really, this code worked without issue? I hadn't thought to actually run this code for testing, only referred to it while writing my own code. I guess I better try this now...

EDIT: hm well I'm seeing "myRuntimeReferenceImageLibrary: 0" in the console, which is the debug message for library count. I'm guessing there is something subtle you accidentally got right, like maybe it matters which object this code is attached to.

EDIT2: I just noticed something that may be related in the logcat for my application. I see this OpenGL error the first time I load the scene, but not the second:

OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_ENUM: enum argument out of range
(Filename: ./Runtime/GfxDevice/opengles/GfxDeviceGLES.cpp Line: 348)

 

This is working for me. Check if your setup is similar to mine and then try the code below.

Game Objects in the Scene:

• AR Session Origin (with AR Session Origin component)
  • AR Camera (with AR Pose Driver, AR Camera Manager, and AR Camera Background components)
• AR Session (with AR Session and AR Input Manager components)
• EventSystem (with Event System and Standalone Input Module components)
• Directional Light

Then let's create a new script. I called mine DynamicImageLibrary and I attached it to the AR Session Origin object.

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using System.IO;
4. using UnityEngine;
5. using UnityEngine.Networking;
6. using UnityEngine.XR.ARFoundation;
7. using UnityEngine.XR.ARSubsystems;
8.  
9. public class DynamicImageLibrary : MonoBehaviour
10. {
11.     public ARTrackedImageManager mARTrackedImageManager;
12.     public GameObject mTrackedImagePrefab;
13.     public Texture2D imageToTexture2d;
14.     public MutableRuntimeReferenceImageLibrary myRuntimeReferenceImageLibrary;
15.  
16.     private void Awake()
17.     {
18.         Screen.sleepTimeout = SleepTimeout.NeverSleep;
19.         mARTrackedImageManager = gameObject.AddComponent<ARTrackedImageManager>();
20.         mARTrackedImageManager.enabled = false;
21.         myRuntimeReferenceImageLibrary = mARTrackedImageManager.CreateRuntimeLibrary() as MutableRuntimeReferenceImageLibrary;
22.         mARTrackedImageManager.maxNumberOfMovingImages = 1;
23.         mARTrackedImageManager.trackedImagePrefab = mTrackedImagePrefab;
24.         mARTrackedImageManager.trackedImagesChanged += OnImageChanged;
25.     }
26.  
27.     void Start()
28.     {      
29.         StartCoroutine(AddImageTrackerByUrl("https://upload.wikimedia.org/wikipedia/commons/9/97/The_Earth_seen_from_Apollo_17.jpg[/URL]"));
30.     }
31.  
32.     public void OnImageChanged(ARTrackedImagesChangedEventArgs args)
33.     {
34.         foreach (var trackedImage in args.added)
35.         {
36.             Debug.Log(trackedImage.referenceImage.name);
37.         }
38.     }
39.  
40.     IEnumerator AddImageTrackerByUrl(string url)
41.     {
42.         mARTrackedImageManager.enabled = false;
43.         if (mARTrackedImageManager.descriptor.supportsMutableLibrary)
44.         {
45.             UnityWebRequest webRequest = UnityWebRequestTexture.GetTexture(url);
46.             yield return webRequest.SendWebRequest();
47.             imageToTexture2d = DownloadHandlerTexture.GetContent(webRequest);
48.  
49.             Unity.Jobs.JobHandle jobHandle = myRuntimeReferenceImageLibrary.ScheduleAddImageJob(imageToTexture2d, Path.GetFileName(url), 0.2f);
50.             jobHandle.Complete();
51.  
52.             if (myRuntimeReferenceImageLibrary != null)
53.             {
54.                 Debug.Log("Image Library Count: " + myRuntimeReferenceImageLibrary.count);
55.  
56.                 mARTrackedImageManager.referenceLibrary = myRuntimeReferenceImageLibrary;
57.             }
58.         }
59.         mARTrackedImageManager.enabled = true;
60.     }
61. }

I have only tested this on Android so far. If you get it to work on either iOS or Android please let me know!

 

I've been trying to get this working for a while now, and currently have something very similar to yours (except I'm loading images locally, not from web). Still doesn't seem to be working for me though.
Do you mind specifying what the versions of everything you're using is? ARFoundation, ARCore, and Unity itself. Might be something which only works in specific version.

 

Great idea, and thanks for putting together this minimal test. When I build that for Android it doesn't work, with these messages in logcat:

12-02 21:22:42.124 5597 5662 E Unity : OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_ENUM: enum argument out of range
12-02 21:22:42.124 5597 5662 E Unity : (Filename: ./Runtime/GfxDevice/opengles/GfxDeviceGLES.cpp Line: 348)

12-02 21:22:43.868 5597 5618 I Unity : Image Library Count: 0
12-02 21:22:43.868 5597 5618 I Unity : <AddImageTrackerByUrl>d__7:MoveNext()
12-02 21:22:43.868 5597 5618 I Unity : UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
12-02 21:22:43.868 5597 5618 I Unity : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

That's the same OpenGL error I was seeing before, so I'm gonna try googling that error message to see if I come up with anything. And yeah I second specifying which version of stuff you are using. I've got Unity 2019.2.13 with preview.3 3.0.0 for both AR Foundation and ARCore; I know there's a newer preview package available, but I've tried upgrading and that didn't change anything.

In the meantime, I'm going to build this for iOS to let you know if it works there.

 

Confirmed, it is working on iOS. Incidentally, I added more logging so that I can tell in the console when it's tracking:

Code (CSharp):

1.     public void OnImageChanged(ARTrackedImagesChangedEventArgs args)
2.     {
3.         foreach (var trackedImage in args.added)
4.         {
5.             Debug.Log($"added: {trackedImage.referenceImage.name}");
6.         }
7.         foreach (var trackedImage in args.updated)
8.         {
9.             Debug.Log($"updated: {trackedImage.transform.position.x}, {trackedImage.transform.position.y}, {trackedImage.transform.position.z}");
10.         }
11.     }

EDIT: whoah now it seems to be working on Android! I built it again cause I didn't have this additional logging before and was curious. Incidentally, I still see that OpenGL error, but the library count was 1 this time, so I guess that OpenGL error is a red herring.

 

Interesting. I am using AR Foundation 3.1.0 which I believe is the latest preview release. Installed via the Unity Package Manager. Unity is 2019.2.8f1.

This seems good enough for testing and early development prototyping. Hopefully as the updates for AR Foundation move out of preview they will become more reliable and have better documentation. This would make it so that I don't have to rely on Vuforia.

 

yeah replacing Vuforia in our application is why I started messing with the AR Foundation preview package. The prototyping has been going great, except that it is pretty annoying that I can't test the AR tracking within the editor, and have to keep doing builds every time.

 

Yes, that is a major downside. At one point they had a remote option where you could use your phone's video feed (connected via usb) in the editor play mode, but they removed that for some reason. I hope they bring it back soon.

 

oh hey I just got it working immediately (ie. I don't need to reload the scene for tracking) in my big app! While looking at the minimal sample you put together, it occurred to me that there is a pause for downloading the image between when the tracking manager is created and when the image is added. On a hunch I figured maybe on Android the AR system needs more initialization time, so I did things in a coroutine and paused for a bit before starting to add images to the created image library.

 

Does seem to be something to do with timing. I changed my code back to being done in a co routine (which hadn't worked previously) and added

yield return null

just after where it loads a list of all the images as byte[] and it works now!
For testing purposes I just tried moving it to the end of the method (where I'd had it before just so it'd work as a co routine) and it didn't work. So it definitely depends on where it is and needs to be after you load/get your images, and before you start adding them to the library.

 

Just did a fresh install of my app and it didn't work when it first started up. However closing and reopening it did work. So the different between the files already being in persistent storage and having to load them in there from the jar first also slows things down enough that it doesn't work.
So for anyone that's developing an app for Android using ARFoundation, be aware of timing and that having an extra `yield return` can be the difference between having count = 0 and count = n.

 

ah, it did seem like my code needed two 'yield return null;' lines (ie. pause two frames) not just one, I guess that's why.

 

New challenge. Creating the library dynamically seem to not work as well for actual image tracking. Looking at the statuses on trackedImagesChanged, for those in args.updated the only TrackingStatus they ever have is TrackingStatus.None, it no longer shows up with Tracking or Limited whilst it did previously. This is an issue for me as previously this is how I was getting around Android's limitation that it never removes images from tracking in when they're no longer visible.

Anyone else had this? Or any thoughts about why this might be happening?

 

I don't have any thoughts why that's happening to you, I'll think about that since the tracking status is working for me with the library created dynamically.

I also have a new issue that just started happening after this change to the AR initialization. The camera view has become really choppy. Several odd details:

This only affects one of the two Android phones I've tested on. It happens on a Pixel 2, but this Galaxy S9 is fine.

The camera view is choppy but tracking is still smooth, implying that the issue is solely with the camera texture in the scene, not the camera input itself.

The camera view is actually fine at first, but then turns choppy when I reload the scene. It's almost as if the previous camera feed persists between loads, and thus interferes with a newly created camera background.

 

I confirm for android, the two 'yield return null;' are absolutly needed to add the texture to the library. Otherwise it won't work.
Thanks a lot for your work there !
Using Unity 2019.3.0f3 with ARFondation, ARCore and ARKit 3.1.0

 

Anyone encounter this error ever: NotSupportedException: No image tracking subsystem found. This usually means image tracking is not supported.

I have both the ARKit Package and ARCore Package installed, so I'm not sure why it wouldn't find the tracking subsystem. Using the code from above on Unity 2019.3.0f3 and get this error with ARF, ARC, and ARK preview 3 3.0.0 as well as 3.1.0... Am I missing another package somewhere?

EDIT: Turns out I was testing in Editor, and because your computer doesn't have ARKit or ARCore, it doesn't work DUH. Code runs on device (iPhone 11) great!

 

You can use Tasks rather than 2 "yield return null" on a coroutine.

 

update regarding the choppy camera issue I mentioned:

I upgraded my AR Foundation (and of course both ARKit and ARCore with it) package and this seems fixed. I had been using 3.0.0-preview3 all along, but I just noticed there is 3.0.1 not preview, so I upgraded to that newer stable build.

 

I am using 2019.3.0f6 and AR Foundation (and the rest) 3.1.0-preview 4 (also tried 3.0.1) but most of the times when I build I get a black background. Sometimes though it builds and I can see the camera feed just find. EDIT : The only time I see camera feed is the first time after I do a fresh install of the app.
On top of that the scanning works just fine and method described above works pretty good.

The whole project is fresh installation with a clean scene with only the objects described as above.

Edit (2): I did managed to fix the camera issue by adding URP.

 

First of all thanks for all the feedback in this thread. I don't believe I would have continued along this path left alone with the documentation. This code has been stable on AR foundation 3.0.1 / 2019.3.41f on a Huawei P20 pro for some time now and avoids the "yield return null" guessing:

Code (CSharp):

1. ARTrackedImageManager TrackedImageManager;
2.  
3. private void Start()
4.         {
5.            StartCoroutine(CreateTrackedImageManager());
6.         }
7.  
8.  
9. IEnumerator CreateTrackedImageManager()
10.         {
11.             TrackedImageManager = transform.gameObject.AddComponent<ARTrackedImageManager>();
12.             var markerIndicator = Resources.Load<GameObject>("MarkerIndicator");
13.             TrackedImageManager.trackedImagePrefab = markerIndicator;
14.  
15.             var referenceImageLibrary = (MutableRuntimeReferenceImageLibrary)TrackedImageManager.CreateRuntimeLibrary();
16.             foreach (var marker in Configuration.Markers)// However you want
17.             {
18.                 var image = Resources.Load<Texture2D>($"Markers/{marker.Key}");
19.                 var handle = referenceImageLibrary.ScheduleAddImageJob(image, marker.Key, marker.Value.Size);
20.                 yield return new WaitUntil(() => handle.IsCompleted);
21.             }
22.             TrackedImageManager.referenceLibrary = referenceImageLibrary;
23.  
24.             TrackedImageManager.enabled = true;
25.             TrackedImageManager.trackedImagesChanged += OnTrackedImagesChanged;
26.  
27.             transform.parent.gameObject.GetComponent<ARSession>().Reset();
28.         }

UPDATE: Tested working on 4.0 preview, but I went back to 3.0.1 as 4.0 started to degenerate other parts of my app

 

You can try this one
https://github.com/codemaker2015/ARFoundation-Dynamic-Image-Database-Creation

 

You can try this one
https://github.com/codemaker2015/ARFoundation-Dynamic-Image-Database-Creation

 

Hi,

you think it also possible to add image at runtime to the library and upload it from your app to a remote server location so other users of the app can download it and scan the images you added first to the library?

 

Hi all,
The RuntimeReferenceImageLibrary (MutableRuntimeReferenceImageLibrary ) works very well on iOS
but on android, the tracking is not as fast as local referenceimagelibrary, the tracking state will be none and it takes around 10s to change the state to tracked.

I have tried many ways to figure out, but with no luck. Has anyone runs into the same problem?

 

https://github.com/Unity-Technologies/arfoundation-samples/issues/359

 

We're seeing the same thing, we're also trying to get the unity guys at the github samples to see our issues:
https://github.com/Unity-Technologies/arfoundation-samples/issues/586

If anyone find a fix, please do post it here, currently a showstopper for us!

 

This is long post and seems people are still struggling with RuntimeReferenceImageLibrary. I was able to get this running with following code snippet. And if you want to see it running here is a link

<iframe width="1024" height="576" src="

" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

You can see I have loaded multiple images and was able to detect this.
I have used 2020.1.1f1, ARFoundation 3.1.3 and ARCore 3.1.5
Here is OnImgaeChanged() that we enable in OnEnable() mehtod.

Code (CSharp):

1. public void OnImageChanged(ARTrackedImagesChangedEventArgs args)
2.     {
3.         // for each tracked image that has been added
4.         foreach (var addedImage in args.added)
5.         {
6.             //update the current Tracked Image as soon as args.added list is triggered
7.             currentTrackedImage = addedImage.referenceImage.name;
8.             ActivateTrackedObject(addedImage);
9.         }
10.  
11.         for (int i = 0; i < args.added.Count; i++)
12.         {
13.             if (args.added[i] != null)
14.                 LogUtil.Log.LogDebug("Tracked new image:" + i + " with name: " + args.added[i].referenceImage.name);
15.         }
16.  
17.         foreach (var updated in args.updated)
18.         {
19.             LogUtil.Log.LogInfo_1("Tracking State: " + updated.trackingState);
20.             //if tracked image tracking state is comparable to tracking
21.             if (updated.trackingState == TrackingState.Tracking)
22.             {
23.                 //set the image tracked ar object to active
24.                 currentTrackedImage = updated.referenceImage.name;
25.                 try
26.                 {
27.                     allOverlayObjects[updated.referenceImage.name].SetActive(true);
28.                     //allOverlayObjects[trackedImage.referenceImage.name].transform.SetParent(trackedImage.transform, false);
29.  
30.                     // Give the initial image a reasonable default scale
31.                     float tScale = 0.0001f * SCALE_;
32.                     allOverlayObjects[updated.referenceImage.name].transform.localScale = new Vector3(tScale, tScale, tScale);
33.                     allOverlayObjects[updated.referenceImage.name].transform.position = updated.transform.position;
34.                     allOverlayObjects[updated.referenceImage.name].transform.rotation = updated.transform.rotation;
35.                 }
36.                 catch (KeyNotFoundException)
37.                 {
38.                     LogUtil.Log.LogMessage("Error: Failed to track given ref image");
39.                     LogUtil.Log.LogError("Failed to track given reference image");
40.                 }
41.  
42.                 if (!isCRRunning)
43.                     StartCoroutine(SeqCoroutine());
44.             }
45.             else
46.             {
47.                 //deactivate the image tracked ar object
48.                 allOverlayObjects[updated.referenceImage.name].SetActive(false);
49.             }
50.         }
51.  
52.         // for each tracked image that has been removed
53.         foreach (var trackedImage in args.removed)
54.         {
55.             // destroy the AR object associated with the tracked image
56.             Destroy(trackedImage.gameObject);
57.         }
58.     }

and here is snippet for loading the reference Images. The code will help you get some idea and you may need some tweaks based on how you would want to use.

Code (CSharp):

1. IEnumerator CreateDynamicReferenceLibrary(string url)
2.  
3.     {
4.         arTrackedImageManager = gameObject.GetComponent<ARTrackedImageManager>();
5.         referenceImageLibrary = (MutableRuntimeReferenceImageLibrary)arTrackedImageManager.CreateRuntimeLibrary();
6.         arTrackedImageManager.enabled = false;
7.         //set allobject dictionary instane
8.         allOverlayObjects = new Dictionary<string, GameObject>(StringComparer.OrdinalIgnoreCase);
9.         allOverlayObjectTuple = new Dictionary<string, Tuple<float, float, float, float>>(StringComparer.OrdinalIgnoreCase);
10.  
11.         int count = 0;
12.         string line;
13.         UnityWebRequest www = UnityWebRequest.Get(url);
14.         www.certificateHandler = new AcceptAllCertificatesSignedWithASpecificKeyPublicKey();
15.         yield return www.SendWebRequest();
16.  
17.         if (www.isNetworkError || www.isHttpError)
18.         {
19.             LogUtil.Log.LogError(www.error);
20.             LogUtil.Log.LogMessage("Error: Failed to load ref images");
21.         }
22.         else
23.         {
24.             // Show results as text
25.             var sr = new StringReader(www.downloadHandler.text);
26.             line = sr.ReadLine(); //to skip the first line
27.             while ((line = sr.ReadLine()) != null)
28.             {
29.                 count++;
30.                 //split the line into prameters
31.                 List<string> list = new List<string>();
32.                 string curr = null;
33.  
34.                 //skip empty lines
35.                 if (string.IsNullOrEmpty(line))
36.                     continue;
37.  
38.                 foreach (Match match in csvSplit.Matches(line))
39.                 {
40.                     curr = match.Value;
41.                     list.Add(curr.TrimStart(',', ' '));
42.                 }
43.                 LogUtil.Log.LogWarning("Added Reference: " + count + " with ID:" + list[0] + " URL:" + list[1] + " with Width:" + list[2]);
44.                 //convert width to meter before sending it ref libraray
45.                 float w = float.Parse(list[2]) / 100;
46.                 //start and wait till it adds ref image
47.                 Coroutine arf = StartCoroutine(AddRefImage(list[1], list[0], w));
48.                 yield return arf;
49.             }
50.  
51.             if (referenceImageLibrary != null)
52.                 arTrackedImageManager.referenceLibrary = referenceImageLibrary;
53.             arTrackedImageManager.maxNumberOfMovingImages = 5;
54.             arTrackedImageManager.enabled = true;
55.             if (!LogUtil.Log.GetMessage().StartsWith("Error", StringComparison.CurrentCulture))
56.                 LogUtil.Log.LogMessage("Reference image loaded successfully");
57.         }
58.     }
59.  
60.     IEnumerator AddRefImage(string url, string imgName, float width)
61.     {
62.         if (arTrackedImageManager.descriptor.supportsMutableLibrary)
63.         {
64.             UnityWebRequest webRequest = UnityWebRequestTexture.GetTexture(url);
65.             webRequest.certificateHandler = new AcceptAllCertificatesSignedWithASpecificKeyPublicKey();
66.             yield return webRequest.SendWebRequest();
67.             if (webRequest.isNetworkError || webRequest.isHttpError)
68.             {
69.                 LogUtil.Log.LogError(webRequest.error);
70.             }
71.             else
72.             {
73.                 Texture2D imageToTexture2d = DownloadHandlerTexture.GetContent(webRequest);
74.                 JobHandle jobHandle = referenceImageLibrary.ScheduleAddImageJob(imageToTexture2d, imgName, width);
75.  
76.                 OFFSET_X = 0;
77.                 OFFSET_Y = 0;
78.  
79.                 //create a targetImage tuple
80.                 allOverlayObjectTuple.Add(imgName, ImageObject.GetTargetImageTuple(imageToTexture2d.width, imageToTexture2d.height, width * 100));
81.                 addObjectToDictionary(imgName, new Vector2(OFFSET_X, OFFSET_Y));
82.                 jobHandle.Complete();
83.             }
84.         }
85.     }
86.  
87.     private void addObjectToDictionary(string iName, Vector2 offsetXY)
88.     {
89.         GameObject newOverlay = overlayPrefab;
90.         //check if the object is prefab and need to be instantiated
91.         if (overlayPrefab.gameObject.scene.rootCount == 0)
92.         {
93.             newOverlay = (GameObject)Instantiate(overlayPrefab, transform.localPosition, Quaternion.identity);
94.         }
95.         //update dictionary to store the gameobject and its size
96.         allOverlayObjects.Add(iName, newOverlay);
97.         //allOverlayObjectOffsets.Add(iName, offsetXY);
98.         newOverlay.SetActive(false);
99.     }

I ran this on Moto Z(2) device. Let me know if you need any other help.

 

We've got a reply on the AR Core Github that there is in fact a difference between static and dynamic runtime image libraries.

https://github.com/google-ar/arcore-android-sdk/issues/1137

I hope they fix it soon!

 

Alright, so it was actually a bug on Unity's side, they fixed it in AR Foundation 4.0.1 preview 9 and up.

EDIT: My mistake they fixed it in ar foundation 4.1.0 preview 10.. However I can't find it in the changelog anymore.

 

Amy reference to this in a changelog somewhere?

 

Very interesting thread - and thank you all for chasing this down. I have gotten the dynamic library creation to work on IOS (hoping to move away from Vuforia . I do have a questions --> The "args.removed" is not triggered when the image is out of view? (args.add & args.updated works fine) Any thoughts?

 

I solved this using the TrackingState.Limited. The definitions were a bit confusing. See below. Thanks

foreach (var trackedImage in args.updated)
{

if (trackedImage.trackingState == TrackingState.Tracking)
{
trackedImage.gameObject.SetActive(true);
Debug.Log($"updated: {trackedImage.transform.position.x}, {trackedImage.transform.position.y}, {trackedImage.transform.position.z}");

}
else if (trackedImage.trackingState == TrackingState.Limited)
{
Debug.Log($"limited: {trackedImage.transform.position.x}, {trackedImage.transform.position.y}, {trackedImage.transform.position.z}");
trackedImage.gameObject.SetActive(false);
}
else
{
Debug.Log($"lost: {trackedImage.transform.position.x}, {trackedImage.transform.position.y}, {trackedImage.transform.position.z}");

}
}

 

What you have stored on the server and in which format can you please tell or show.
Thank you

 

have yall seen this? https://github.com/newyellow/arfoundation-helper

 

Does that helper assist with dynamic image libraries? A quick glance only looks like static image libraries setup in the editor, but maybe I missed it.

 

Can we switch multiple MutableRuntimeReferenceImageLibrary and while switching can we use previously added image for tracking?

I am trying this method but facing an issue, Not bale to get any data in Event.updated