Resolved AR Foundation using reference image library during runtime

Hi,

I am working on a project, and I am currently using the following code to recognize images and display the name of the recognized image. I am are planning on adding more functions later, but for that I need to be able to alter the reference image library during runtime. I have a reference image library with images, but I need to be able to add images to it during runtime, so in order to do that, I would have to first make the current reference image library work during runtime. I have read many answers and guide's on how to do this, but I have not been able to figure out how to implement this correctly.

In short, I need to:
1) Use the images in the ReferenceImageLibrary.
2) Have a RuntimeReferenceImageLibrary for adding pictures during runtime.
3) Have the app recognize images from the library, and display the name of the image.

Any help is greatly appreciated.

Here is the code:

using System.Collections;
using System.Collections.Generic;
using System.Collections.Specialized;
using UnityEngine;
using UnityEngine.XR;
using UnityEngine.XR.ARFoundation;
using UnityEngine.UI;
using System.Globalization;
using UnityEngine.XR.ARSubsystems;

[RequireComponent(typeof(ARTrackedImageManager))]
public class ImageTracking : MonoBehaviour
{

private ARTrackedImageManager trackedImageManager;
private XRReferenceImageLibrary runtimeImageLibrary;
private Text currentImageText;
private Text previousImageText;
string name = "";
int counter = 0;

private void Start()
{
currentImageText = GameObject.Find("CurrentImageName").GetComponent<Text>();
previousImageText = GameObject.Find("PrevImageName").GetComponent<Text>();
//the next part is the part I am having trouble with.
trackedImageManager = GameObject.AddComponent<ARTrackedImageManager>();
trackedImageManager.referenceLibrary = trackedImageMa-
nager.CreateRuntimeLibrary(runtimeImageLibrary);
trackedImageManager.enabled = true;
}

private void Awake()
{
trackedImageManager = FindObjectOfType<ARTrackedImageManager>();
}

private void OnEnable()
{
trackedImageManager.trackedImagesChanged += ImageChanged;
}

private void OnDisable()
{
trackedImageManager.trackedImagesChanged -= ImageChanged;
}

private void ImageChanged(ARTrackedImagesChangedEventArgs eventArgs)
{
foreach(ARTrackedImage trackedImage in eventArgs.added)
{
UpdateImage(trackedImage);
}
foreach (ARTrackedImage trackedImage in eventArgs.updated)
{
if (trackedImage.referenceImage.name != name)
{
UpdateImage(trackedImage);
counter++;
}
}
foreach (ARTrackedImage trackedImage in eventArgs.removed)
{
currentImageText.text = "Tracking: None";
}
}

private void UpdateImage(ARTrackedImage trackedImage)
{

name = trackedImage.referenceImage.name;
currentImageText.text = "Tracked:" + name;
previousImageText.text = "Counter: " + counter;

}
}

 

Here is an example of how to change the library at runtime. Please assign all fields from the inspector.

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.XR.ARFoundation;
3. using UnityEngine.XR.ARSubsystems;
4.  
5.  
6. public class ImageTrackingLibrarySwitchTest : MonoBehaviour {
7.     [SerializeField] XRReferenceImageLibrary firstLibrary = null;
8.     [SerializeField] XRReferenceImageLibrary secondLibrary = null;
9.     [SerializeField] ARTrackedImageManager manager = null;
10.  
11.     int curLibIndex = 0;
12.  
13.  
14.     void Update() {
15.         if (Input.GetMouseButtonUp(0)) {
16.             switchImageLibrary();
17.         }
18.     }
19.  
20.     void switchImageLibrary() {
21.         curLibIndex = curLibIndex == 0 ? 1 : 0;
22.         Debug.Log("setting lib with index " + curLibIndex);
23.         manager.enabled = false;
24.         var lib = curLibIndex == 0 ? firstLibrary : secondLibrary;
25.         manager.referenceLibrary = manager.CreateRuntimeLibrary(lib);
26.         manager.enabled = true;
27.     }
28. }
29.  

 

Thanks for the help! I figured out and understood the way to implement it thanks to your example.

 

Hello, can you share how were you able to add images to the library on runtime?

 

I followed the steps but I don't understand why it does not work.

Code (CSharp):

1.  
2. using UnityEngine;
3. using UnityEngine.XR.ARFoundation;
4. using System.Collections.Generic;
5. using UnityEditor.XR.ARKit;
6. using UnityEngine.XR.ARSubsystems;
7.  
8. public class ImageTracker : MonoBehaviour
9. {
10.     [SerializeField] Texture2D image;
11.     [SerializeField] string imageName;
12.     [SerializeField] GameObject[] arPrefabs;
13.  
14.     ARTrackedImageManager trackedImageManager;
15.  
16.     List<GameObject> ARObjects = new List<GameObject>();
17.  
18.     void Awake()
19.     {
20.         trackedImageManager = GetComponent<ARTrackedImageManager>();
21.  
22.         trackedImageManager.referenceLibrary = trackedImageManager.CreateRuntimeLibrary();
23.     }
24.  
25.     float timer;
26.     bool timerEnabled = true;
27.     void Update()
28.     {
29.         if (timerEnabled)
30.         {
31.             timer += Time.deltaTime;
32.  
33.             if (timer > 1f)
34.             {
35.                 timer = 0;
36.                 timerEnabled = false;
37.  
38.                 AddImage(IMGToSprite.DuplicateTexture(image));
39.             }
40.         }
41.     }
42.  
43.     void OnEnable()
44.     {
45.         trackedImageManager.trackedImagesChanged += OnChanged;
46.     }
47.  
48.     void OnDisable()
49.     {
50.         trackedImageManager.trackedImagesChanged -= OnChanged;
51.     }
52.  
53.     void AddImage(Texture2D texture2D)
54.     {
55.         Debug.Log("Add Image " + ARSession.state);
56.  
57.         if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
58.             return;
59.  
60.         Debug.Log("Add Image " + texture2D.name);
61.  
62.         if (trackedImageManager.referenceLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
63.         {
64.             Debug.Log("Add Image Mutable");
65.  
66.             mutableLibrary.ScheduleAddImageWithValidationJob(texture2D, imageName, 0.5f);
67.         }
68.     }
69.  
70.     void OnChanged(ARTrackedImagesChangedEventArgs eventArgs)
71.     {
72.         //Create object based on image tracked
73.         foreach (var trackedImage in eventArgs.added)
74.             foreach (var arPrefab in arPrefabs)
75.                 if (trackedImage.referenceImage.name == arPrefab.name)
76.                     ARObjects.Add(Instantiate(arPrefab, trackedImage.transform));
77.  
78.         //Update tracking position
79.         foreach (var trackedImage in eventArgs.updated)
80.             foreach (var gameObject in ARObjects)
81.                 if (gameObject.name == trackedImage.name)
82.                     gameObject.SetActive(trackedImage.trackingState == TrackingState.Tracking);
83.     }
84. }

I added a timer because if the AddImage is called inside the Awake, it does not meet the following condition:

Code (CSharp):

1. if (!(ARSession.state == ARSessionState.SessionInitializing || ARSession.state == ARSessionState.SessionTracking))
2.     return;

When I run the application, the mutableLibrary increased by one but the GameObject is not intanciate in the scene even if I am pointing at the image I gave.

And here is the setup in the inspector:

​

 

The duplicated method is here to set the Texture2D as readable cause there is an error if the texture is not readable.

Since I am doing everything at runtime so I wanted to make a try in the editor.

Here the duplicate method:

Code (CSharp):

1. public static Texture2D DuplicateTexture(Texture2D source)
2. {
3.     // Source: https://forum.unity.com/threads/easy-way-to-make-texture-isreadable-true-by-script.1141915/
4.     RenderTexture renderTex = RenderTexture.GetTemporary(
5.                 source.width,
6.                 source.height,
7.                 0,
8.                 RenderTextureFormat.Default,
9.                 RenderTextureReadWrite.Linear);
10.  
11.     Graphics.Blit(source, renderTex);
12.     RenderTexture previous = RenderTexture.active;
13.     RenderTexture.active = renderTex;
14.     Texture2D readableText = new Texture2D(source.width, source.height);
15.     readableText.ReadPixels(new Rect(0, 0, renderTex.width, renderTex.height), 0, 0);
16.     readableText.Apply();
17.     RenderTexture.active = previous;
18.     RenderTexture.ReleaseTemporary(renderTex);
19.     return readableText;
20. }