Dynamic XRReferenceImageLibrary in AR Foundation

Hello, I'd like to start a discussion about how the AR tracked images are referenced in the new AR Foundation system. I'm using Unity 2019.2.0b1 with AR preview packages installed (AR Extensions + Foundation + Subsystems and ARCore+ARKit plugins).
As far as I know there's no documentation yet on this feature but I managed to create a simple scene with a static library of images that, once detected by the phone camera, are used as anchors for 3D AR content. Perfect. The problem is that my aim was to have a dynamic library of images. With "dynamic" I mean a library that is not included in the app build but it is dynamically downloaded from a remote server (through an AssetBundle) and injected in the ARTrackedImageManager. After many tests I came to the conclusion that this is not possible with the current AR Foundation implementation. First of all, the manager dies if it does not find a library on initialization. My workaround was using a placeholder library with just one image (an empty library is not considered valid) and then replacing it at runtime:

Code (CSharp):

1. var manager = gameObject.GetComponent<ARTrackedImageManager>();
2. manager.referenceLibrary = myDynamicLibrary;

This approach does not generate any error and looked fair enough to me but it just doesn't work: the manager kept looking for the old library images.
After further investigation I came across the ARCoreImageTrackingProvider, which acts as a bridge between the Foundation and the low-level ARCore (i'm on Android) image tracking functionalities. From my understanding of this code, using a dynamic library for image tracking is just impossible: the provider expects to find a file .imgdb (which is the ARCore format for the images library) in the build itself. Unity creates this file just before building and ships it along the internal Android stuff (see ARCorePreprocessBuild).

Is there any solution for a dynamic approach? Maybe using Google ARCore implementation?

 

Today I tried to go low-level and directly call ARCore library method to update the AR images library. To make it compile I had to flag it as unsafe. I also had to recover the .imgdb file generated by the ARCore preprocessing routine and pass it to my function (second parameter):

Code (CSharp):

1. public static unsafe void SetARCoreLibrary(XRReferenceImageLibrary lib, byte[] arcoreLib)
2. {
3.     if (arcoreLib == null || arcoreLib.Length == 0)
4.     {
5.         throw new InvalidOperationException(string.Format(
6.             "Failed to load image library '{0}' - file was empty!", lib.name));
7.     }
8.  
9.     var guids = new NativeArray<Guid>(lib.count, Allocator.Temp);
10.     try
11.     {
12.         for (int i = 0; i < lib.count; ++i)
13.         {
14.             guids[i] = lib[i].guid;
15.         }
16.  
17.         fixed (byte* blob = arcoreLib)
18.         {
19.             UnityARCore_imageTracking_setDatabase(
20.                 blob,
21.                 arcoreLib.Length,
22.                 guids.GetUnsafePtr(),
23.                 guids.Length);
24.         }
25.     }
26.     catch (Exception e)
27.     {
28.         Debug.LogError(string.Format("Error while loading '{0}': {1}", lib.name, e));
29.     }
30.     finally
31.     {
32.         guids.Dispose();
33.     }
34. }
35.  
36.  
37. [DllImport("UnityARCore")]
38. static unsafe extern void UnityARCore_imageTracking_setDatabase(
39.     void* databaseBytes, int databaseSize, void* sourceGuidBytes, int sourceGuidLength);

Aaaand it doesn't work. I see no errors and the ARTrackerImageManager looks correctly configured (from the logs I can see that the number of images and their names is updated and correct)... but the system keeps detecting only the images of the old library.
Yeah, this is a nightmare.
My guess is that by replicating the [DllImport("UnityARCore")] attribute I'm actually calling another instance of that library... I'm not sure btw. I just know that a public keyword in the right place would have solved this problem...

 

Interesting, thanks for sharing your efforts! Although it looks discouraging at the moment, I'm hopeful there will be a way eventually. I am sure that Vuforia will be grateful for the delay, because the capability we're looking for here is what continues to make them relevant. But the worst case is that Unity intends to restrict this feature, seeing it as a potential profit center and locks down "Cloud Reco" in the same way Vuforia does. That would suck.

 

Update: we have a documentation for the AR Tracked Image Manager!

Quoting:

The reference image library can be set at runtime, but as long as the tracked image manager component is enabled, the reference image library must be non-null.
The reference image library is an instance of the ScriptableObject XRReferenceImageLibrary. This object contains mostly Editor data. The actual library data (containing the image data) is provider-specific. Refer to your provider's documentation for details.​

I'm quite confused... why are we allowed to set the library at runtime if the actual library data is created at build time?

 

Other details in the ARCore plugin documentation:

When building the Player for Android, each reference image library is used to generate a corresponding imgdb file, ARCore's representation of the reference image library. These files are placed in your project's StreamingAssets folder in a subdirectory called HiddenARCore to allow them to be accessible at runtime.​

The StreamingAssets folder is read-only and this path is hardcoded in the provider's code... any idea on how to override this behaviour?

 

I've been using EasyAR. It can pick up images directly from the StreamingAssets folder. I wanted to explore using ARFoundation but this is an issue for me also. Of course not running in the editor directly or using a working Remote also make me hold off on ARFoundation.

 

Guys I solved.
It works.
And I'm proud of my esoteric solution!
Basically I'm using reflection to call the private, static, unsafe method that updates the ARCore images database: ARCoreImageTrackingProvider.Provider.UnityARCore_imageTracking_setDatabase.
I know, this is so wrong but I guess it is the only way by now.

(WARNING, this C# code could make you cry).

Code (CSharp):

1. #if UNITY_ANDROID
2. private static unsafe void ChangeARCoreImagesDatabase(XRReferenceImageLibrary library, byte[] arcoreDB)
3. {
4.     if (arcoreDB == null || arcoreDB.Length == 0)
5.     {
6.         throw new InvalidOperationException(string.Format(
7.             "Failed to load image library '{0}' - file was empty!", library.name));
8.     }
9.  
10.     var guids = new NativeArray<Guid>(library.count, Allocator.Temp);
11.     try
12.     {
13.         for (int i = 0; i < library.count; ++i)
14.             guids[i] = library[i].guid;
15.  
16.         fixed (byte* blob = arcoreDB)
17.         {
18.             // Retrieve the ARCore image tracking provider by reflection
19.             var provider = typeof(ARCoreImageTrackingProvider).GetNestedType("Provider", BindingFlags.NonPublic | BindingFlags.Instance);
20.  
21.             // Destroy the current image tracking database
22.             var destroy = provider.GetMethod("UnityARCore_imageTracking_destroy", BindingFlags.NonPublic | BindingFlags.Static);
23.             destroy.Invoke(null, null);
24.  
25.             // Set the image tracking database
26.             var setDatabase = provider.GetMethod("UnityARCore_imageTracking_setDatabase", BindingFlags.NonPublic | BindingFlags.Static);
27.             setDatabase.Invoke(null, new object[]
28.             {
29.                 new IntPtr(blob),
30.                 arcoreDB.Length,
31.                 new IntPtr(guids.GetUnsafePtr()),
32.                 guids.Length
33.             });
34.         }
35.     }
36.     catch (Exception e)
37.     {
38.         Debug.LogError(string.Format("Error while loading '{0}': {1}", library.name, e));
39.     }
40.     finally
41.     {
42.         guids.Dispose();
43.     }
44. }
45. #endif

TLDR: I found a way to change the ARCore tracked images database in an AR Foundation based application at runtime. The AR Foundation on Android expects to find this file in the app assets (it is automatically created and included at build time by Unity) making impossible to use downloaded or imported libraries. With my method you can swap between libraries easly, you just need to read the .imgdb file and pass the bytes to the function

 

Hey guys, at the moment I'm trying to do the same on ARKit... and oh my god.
Here's the signature of the native call used to set the image library in iOS.

Code (CSharp):

1. [DllImport("__Internal")]
2. static extern SetReferenceLibraryResult UnityARKit_imageTracking_trySetReferenceLibrary([MarshalAs(UnmanagedType.LPWStr)] string name, int nameLength, Guid guid);

As you can see the parameter that matters is just the NAME of the library, string that it is internally used to find the file that Unity had created during the build processing... file that it won't find, since the library has been downloaded/generated dynamically.
From my understanding, [DllImport("__Internal")] means that the actual implementation of that function is not in an external plugin but it is compiled in the engine itself... am I right? Is there a way to find the actual code?

 

Hi Lorenzo,

Thanks for your efforts researching this and posting the code, I will need this soon and was planning on looking into it. Will post here if I have any new insights.

More on the internal statement here (but I'm sure you find this already):

https://docs.unity3d.com/Manual/NativePlugins.html

 

Hi nilsdr,
Thank you, I hope for new insights aswell, I've no idea how to solve this
Yes I saw that page and I found the statically linked plugin that ARKit uses for the internal calls: "Library/PackageCache/com.unity.xr.arkit@2.0.1/Runtime/iOS/UnityARKit.a". I tried to analyze its symbols (for the curious, you can do it using this unix command: nm UnityARKit.a) but I cannot see anything useful. I even tried to decompile it using Hopper

 

I admire your dedication, Lorenzo, but it's clear that this capability (setting image targets at runtime) is not a design priority by Unity ARF team, at least not yet. It would be nice to have an official comment from someone on the AR Foundation team as to whether this omission is intentional, or if changes are coming that will make it possible.

 

Thanks and yes, I totally agree with you, it is clear that this feature is not supported. I would love to hear news from the AR Foundation team!

 

Oh damn, what a ride, hope they introduce any sort of solution for all of this very soon.

 

Hey @LorenzoValente !

Thanks for sharing all this effort!
We are going to try it very soon.
It's quite sad that they leave this important functionality inaccessible.

 

Hey @LorenzoValente !
Great work with ARCore part.

I've managed to add reference image for ARKit, but only for now deprecated Unity ARKit Plugin. As far as I know, there is no way to add new reference image beside restarting native ARKit session. Check here for example.

So we have to change .a library to add new native call that restarting AR session with new reference images set.

 

Thank you Macoron!
Yes i saw that there was a way with the old ARKit plugin... but I was looking for a cleaner solution that didn't break the Foundation packages, they should not be changed

 

Hey Lorenzo, thanks for trying to get dynamic image targets working in ARFoundation. It's cool that you were able to get it working for the ARCore part of things (which makes sense, the official ARcore Unity plugin supports that feature, although the documentation is poor).

If you have any success in the future getting this working, please make another post here. I've started watching this thread.

PS - I did see that some people say they got at least a single dynamic image target working with ARKit 1.5 (not ARFoundation) via this thread. But since the Unity ARKit plugin is depreciated, we need a solution moving forward.

 

Hello! I was wondering if anyone has been able to find any solutions regarding dynamic image libraries on iOS? I have been working on a project for 6 months now that uses the dynamic loading of images, which I got working on the older versions of the ARKit and ARCore plugins. I'm stuck using those older, deprecated plugins until ARFoundation adds support for this feature, as dynamic image loading is an integral part of my project.

All the work of Lorenzo is very impressive, thanks for sharing it with us! It's a shame that Unity aren't adding dynamic image capability themselves, even though the feature is supported by both ARCore and ARKit.

 

Could you share the modified ARKit plugin code with us? That will help myself and others get this functionality working in at least the older versions of ARKit since it doesn't look like it's going to be brought over to ARFoundation.

Personally I'm trying to get the modified ARKit to support multiple dynamic image targets, added before the session starts.

 

Of course! There are 3 files that I modified, namely the UnityARCameraManager.cs, UnityARSessionNativeInterface.cs and ARSessionNative.mm. The first two are written in C#, the last on is written in Objective C.

This is the function that I added to ARSessionNative.mm. It allows you to start a session with multiple tracking images, which can be downloaded or retrieved from the device at runtime.

Code (CSharp):

1. extern "C" void StartWorldTrackingSessionWithOptionsAndImages(void* nativeSession, ARKitWorldTrackingSessionConfiguration unityConfig, UnityARSessionRunOptions runOptions, const void* a_imageBuffer, unsigned int* a_bufferOffsets, unsigned int a_bufferLength, float* a_physicalWidth, int* a_id, int a_count)
2. {
3.     UnityARSession* session = (__bridge UnityARSession*)nativeSession;
4.     ARWorldTrackingConfiguration* config = [ARWorldTrackingConfiguration new];
5.     ARSessionRunOptions runOpts = GetARSessionRunOptionsFromUnityARSessionRunOptions(runOptions);
6.     GetARSessionConfigurationFromARKitWorldTrackingSessionConfiguration(unityConfig, config);
7.     session->_getPointCloudData = (BOOL) unityConfig.getPointCloudData;
8.     session->_getLightEstimation = (BOOL) unityConfig.enableLightEstimation;
9.    
10.     if(UnityIsARKit_1_5_Supported() && unityConfig.referenceImagesResourceGroup != NULL && strlen(unityConfig.referenceImagesResourceGroup) > 0)
11.     {
12.         if (@available(iOS 11.3, *))
13.         {
14.             NSMutableSet *mSet = [[NSMutableSet alloc ] init];
15.             NSData* allData = [[NSData alloc] initWithBytes:a_imageBuffer length:a_bufferLength];
16.             for (int i = 0; i < a_count; i++) {
17.                 int j;
18.                 int y = 2;
19.                 j = i * y;
20.                 NSData* imageData = [allData subdataWithRange:NSMakeRange(a_bufferOffsets[j], a_bufferOffsets[j+1])];          
21.                 UIImage* uiimage = [[UIImage alloc] initWithData:imageData];
22.                 CGImageRef cgImage = [uiimage CGImage];
23.                 ARReferenceImage *image = [[ARReferenceImage alloc] initWithCGImage:cgImage orientation:kCGImagePropertyOrientationUp physicalWidth:a_physicalWidth[i]];
24.                 NSString *name = @(a_id[i]).stringValue;
25.                 image.name = name;
26.                 [mSet addObject:image];
27.             }
28.             config.detectionImages = mSet;
29.         }
30.     }
31.  
32.     if(UnityIsARKit_2_0_Supported())
33.     {
34.         if (@available(iOS 12.0, *))
35.         {
36.             NSMutableSet<ARReferenceObject *> *referenceObjects = nullptr;
37.             if (unityConfig.referenceObjectsResourceGroup != NULL && strlen(unityConfig.referenceObjectsResourceGroup) > 0)
38.             {
39.                 NSString *strResourceGroup = [[NSString alloc] initWithUTF8String:unityConfig.referenceObjectsResourceGroup];
40.                 [referenceObjects setByAddingObjectsFromSet:[ARReferenceObject referenceObjectsInGroupNamed:strResourceGroup bundle:nil]];
41.             }
42.            
43.             if (unityConfig.ptrDynamicReferenceObjects != nullptr)
44.             {
45.                 NSSet<ARReferenceObject *> *dynamicReferenceObjects = (__bridge NSSet<ARReferenceObject *> *)unityConfig.ptrDynamicReferenceObjects;
46.                 if (referenceObjects != nullptr)
47.                 {
48.                     [referenceObjects setByAddingObjectsFromSet:dynamicReferenceObjects];
49.                 }
50.                 else
51.                 {
52.                     referenceObjects = dynamicReferenceObjects;
53.                 }
54.             }          
55.             config.detectionObjects = referenceObjects;
56.         }
57.     }
58.    
59.     if (runOptions == UnityARSessionRunOptionsNone)
60.         [session->_session runWithConfiguration:config];
61.     else
62.         [session->_session runWithConfiguration:config options:runOpts];  
63.     [session setupMetal];
64. }

This is the code that I added to the UnityARSessionNativeInterface.cs. Don't forget to import it with the [DllImport("__Internal")], else you won't be able to call it.

Code (CSharp):

1. public void RunWithConfigAndOptionsAndImages(ARKitWorldTrackingSessionConfiguration config, UnityARSessionRunOption runOptions, byte[] a_ImageBuffer, uint[] a_BufferLengths, uint a_BufferLength, float[] a_physicalWidth, int[] a_id, int a_count)
2.         {
3. #if !UNITY_EDITOR && UNITY_IOS
4.             StartWorldTrackingSessionWithOptionsAndImages(m_NativeARSession, config, runOptions, a_ImageBuffer, a_BufferLengths, a_BufferLength, a_physicalWidth, a_id, a_count);
5. #elif UNITY_EDITOR
6.             CreateRemoteWorldTrackingConnection(config, runOptions);
7. #endif
8.         }

And finally, this is the code that I added to the UnityARCameraManager. This is the function that you should call to start an ARKit session with your own, dynamically loaded tracking images. Simply provide the images, reference image sizes and the IDs for the images. The IDs are integers here, but they could also be reworked to be strings. You'll be able to see which image has been detected by using the ARImageAnchor.referenceImageName function. This way you can have specific behavior when an image with a certain ID is tracked.

Code (CSharp):

1.  public void AddNewReferenceImages(Texture2D[] _images, float[] _sizes, int[] _ids) {
2.         UnityARSessionNativeInterface.ARSessionShouldAttemptRelocalization = true;
3.         m_session = UnityARSessionNativeInterface.GetARSessionNativeInterface();
4.         Application.targetFrameRate = 60;
5.         ARKitWorldTrackingSessionConfiguration config = sessionConfiguration;
6.         if (config.IsSupported) {
7.             List<byte> _totalBytes = new List<byte>();
8.             List<uint> _offsets = new List<uint>();
9.             for(int i = 0; i < _images.Length; i++) {
10.                 byte[] _bytes = _images[i].EncodeToJPG();
11.                 if(_offsets.Count > 0 ) {
12.                     _offsets.Add((uint)_totalBytes.Count);
13.                 } else {
14.                     _offsets.Add(0);
15.                 }
16.                 _totalBytes.AddRange(_bytes);
17.                 _offsets.Add((uint)_bytes.Length);            
18.             }
19.             uint _totalLength = (uint)_totalBytes.Count;
20.             m_session.RunWithConfigAndOptionsAndImages(config, UnityARSessionRunOption.ARSessionRunOptionRemoveExistingAnchors | UnityARSessionRunOption.ARSessionRunOptionResetTracking, _totalBytes.ToArray(), _offsets.ToArray(), _totalLength, _sizes, _ids, _images.Length);
21.             UnityARSessionNativeInterface.ARFrameUpdatedEvent += FirstFrameUpdate;
22.         } else {
23.             Debug.Log("Image config not supported!");
24.         }
25.         if (m_camera == null) {
26.             m_camera = Camera.main;
27.         }
28.     }

Hope it helps!

 

That's awesome, I'm going to post this in the ARKit feature request thread for others to see, this will help hold over a lot of people until ARFoundation catches up.

https://bitbucket.org/Unity-Technol...es/20/couldnt-dynamic-update-detection-images

 

Hi all,

I have also used a similar solution that @RobAEG describes for ARKit 1.5 and 2.0.

Now, I am waiting for a solution for this problem in ARFoundation + ARKit3, too.

 

I'm glad that I was able to provide some help! I actually got most of the information that I needed from that exact same thread. I posted there as Roverboef. The AR scene is a small world it seems.

 

Still waiting for a solution to the dynamic/runtime update of reference images or objects with ARKIT 3.0 and AR Foundation... Any idea, someone? I think we depend on a native iOS code or wait to Unity develop a way of doing this.

Is there a technical limitation for this or this is just a business decision?

 

I don't think it's a technical limitation, as it's an officially supported feature in ARKit native, but not on ARKit Unity Plugin. And clearly, there are ways to add in the missing functionality. I'm guessing that Unity's ARFoundation developers will only add features that both ARCore and ARKit officially support through their respective unmodified plugins. So that's why I'm doubtful we'll see that feature added anytime soon.

My speculation is that this would require Unity's ARFoundation team talking to the ARCore and ARKit plugin teams respectively and getting this feature added to those plugins officially, and then Unity's ARFoundation team would wrap it with their code. And since this is hard to market as a new feature, it will get less priority than other AR features.

Truthfully I haven't messed around with ARFoundation much, as I have my own wrapper for ARCore/ARKit that incorporates these features. But I'm guessing we're missing access to one of the modified files discussed in this thread that's preventing us from just adding in this feature ourselves to ARFoundation, similar to how it's been shown can be done for ARFoundation's implementation of ARCore.

 

So I've added this code to my ARKit Unity 2.0 plugin, however, the image anchor is never found after being added dynamically before the session is started.

I thought it might be because we never set up an ARReferenceImage data object in the session configuration, but adding a dynamic ARReferenceImageSet and a dynamically created ARReferenceImage to it with a test texture leads to the following error within xCode upon runtime...

Code (CSharp):

1. did fail with error: Error Domain=com.apple.arkit.error Code=300 "Invalid reference image." UserInfo={NSLocalizedFailureReason=One or more reference images have an invalid size: 1311512199, NSLocalizedRecoverySuggestion=Make sure that all reference images are greater than 100 pixels and have a positive physical size in meters.

From what I've read online, the iOS project needs to be set to iOS 12.0 and the image needs to be a .jpg with read/write enabled, and the texture size must be over 100x100 pixels. And finally, the supplied width in meters must be a positive value (1 meter for my test image). The test image I'm using fulfils all of these requirements.

@RobAEG did you encounter anything like this? If not would you be able to help out by providing a test project that you confirmed works on your end? Sorry for the request, but as you noted, the world of AR developers is a small one.

 

I'm not able to provide a test project at this very moment but I might be able to help you out! You don't have to add an ARReferenceImageSet the the UnityARCameraManager script for the code to work. We build our own in the ARSessionNative.mm.

Have you debugged your code to see what happens when you use the modified scripts to dynamically load tracking images? Note that the image sizes for ARKit can be a bit finicky, 1 meter is very big. I think it might work better if you get up close to say a 15 centimeter image. Image sizes should be entered with 0.1 being equal to 1 meter. So for an image of 15 centimeters you'd give a size of 0.015.

 

I found where I was having an issue, in ARSessionNative.mm code above we were checking...

Code (csharp):

1.  
2. if(UnityIsARKit_1_5_Supported() && unityConfig.referenceImagesResourceGroup != NULL && strlen(unityConfig.referenceImagesResourceGroup) > 0 )

but the
unityConfig.referenceImagesResourceGroup
variable is unused. We can safely remove it from that if statement to get the image targets working properly.

 

Can you share a test project that works with this code? I'm a little lost with the ARKit plugin.

 

Hi all, I want to add another way to dynamically load reference libraryies at runtime for Android - without the need for reflection.

First of all, thanks to @LorenzoValente for his great work. I explored his approach and that led me to my solution. So basically has the folloing steps:

1. Register a custom XRImageTrackingSubsystem which contains almost the same code as ARCoreImageTrackingProvider (except that it loads the database from a remote location).
   
2. Disable the ARTrackedImageManager in the Editor, so it doesn't start loading the image database directly at start.
3. In my script upon awake, deserialize a new XRReferenceImageLibrary with my new reference images (but that's just some meta info, not the imgdb file itself, yet).
4. Enable the ARTrackedImageManager in my script, which triggers loading of the database.
5. My custom XRImageTrackingSubsystem gets called and it in turn loads the imdb file from my server.

Now for the steps in more detail:

Register a custom XRImageTrackingSubsystem
The included XRImageTrackingSubsystem for ARCore is called ARCoreImageTrackingProvider and can be found in \Library\PackageCache\com.unity.xr.arcore@2.1.0-preview.5\Runtime\ARCoreImageTrackingProvider.cs. Copy this file to you own project and rename it. You only need some minor adjustments.First, change the point in time the provider gets registered. We need our custom provider to get registered before the built-in provider beacuse the only the first one gets used. So change this line...

Code (CSharp):

1. [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
2. static void RegisterDescriptor()
3. {
4.     // ...
5. }

... to this line...

Code (CSharp):

1. [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterAssembliesLoaded)]
2. static void RegisterDescriptor()
3. {
4.     // ...
5. }

Next, change the GetPathForLibrary method so it returns the URL of your imgdb file on the server. Here's the original code:

Code (CSharp):

1. internal static string GetPathForLibrary(XRReferenceImageLibrary library)
2. {
3.     if (library == null)
4.         throw new ArgumentNullException("library");
5.  
6.     return Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
7. }

And here's my code:

Code (CSharp):

1. internal static string GetPathForLibrary(XRReferenceImageLibrary library)
2. {
3.     if (library == null)
4.         throw new ArgumentNullException("library");
5.  
6.     return library is XRReferenceImageLibraryWithDynamicLoading libraryWithDynamicLoading && !string.IsNullOrWhiteSpace(libraryWithDynamicLoading.Url)
7.         ? libraryWithDynamicLoading.Url
8.         : Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
9. }

You may notice that I have introducted a type called XRReferenceImageLibraryWithDynamicLoading. This is just a subclass of XRReferenceImageLibrary that contains an additional property for supplying the URL. This is my aproach to manage URLs, your's might look differently.

Code (CSharp):

1. public class XRReferenceImageLibraryWithDynamicLoading : XRReferenceImageLibrary
2. {
3.     public string Url { get; set; }
4. }

Disable the ARTrackedImageManager in the Editor


Deserialize a new XRReferenceImageLibrary / Enable the ARTrackedImageManager
Your image library should match your image database.

Code (CSharp):

1.     public void Awake()
2.     {
3.         this.trackedImageManager = this.GetComponent<ARTrackedImageManager>();
4.  
5. #if !UNITY_EDITOR
6.         var json = @"{
7.            ""m_GuidLow"": 5140542808452585354,
8.            ""m_GuidHigh"": 2180237115512313227,
9.            ""Url"": ""http://172.20.10.5:6789/myimages.imgdb"",
10.            ""m_Images"": [
11.                {
12.                    ""m_SerializedGuid"": {
13.                        ""m_GuidLow"": 5679415376573207540,
14.                        ""m_GuidHigh"": 6089316183866679477
15.                    },
16.                    ""m_SpecifySize"": false,
17.                    ""m_Name"": ""poster-of-cool-movie""
18.                }
19.            ]
20.        }";
21.         var library = new XRReferenceImageLibraryWithDynamicLoading();
22.         JsonUtility.FromJsonOverwrite(json, library);
23.         this.trackedImageManager.referenceLibrary = library;
24. #endif
25.         this.trackedImageManager.enabled = true;
26.     }

Of course I'd normally get the JSON from a server as well, but for testing purposes I inlined it here. Notice the

#if !UNITY_EDITOR

preprocessor directive. If you omit that, this code will be called in the editor. Since I don't want any dynamic loading and server calls to be performed in the editor, I just disabled it that way.

The last line in the method above enables the ARTrackedImageManager which triggeres all the internal loading and with it our next step.

Load the imdb file from my server
Back in my custom XRImageTrackingSubsystem, since I provided the URL of a remote server instead of a path to a file on the device, the built-in code is already capable of handling it. I'll show the code here, but note that this wasn't code I added, but coded that was already included in the original ARCoreImageTrackingProvider. The only modification I made was inside the GetPathForLibrary method I described above.

Code (CSharp):

1. public unsafe override XRReferenceImageLibrary imageLibrary
2. {
3.     set
4.     {
5.         if (value == null)
6.         {
7.             UnityARCore_imageTracking_setDatabase(null, 0, null, 0);
8.         }
9.         else
10.         {
11.             using (var uwr = new UnityWebRequest(GetPathForLibrary(value)))
12.             {
13.                 uwr.downloadHandler = new DownloadHandlerBuffer();
14.                 uwr.disposeDownloadHandlerOnDispose = true;
15.                 uwr.SendWebRequest();
16.                 while (!uwr.isDone) {}
17.  
18.                 byte[] libraryBlob = uwr.downloadHandler.data;
19.                 if (libraryBlob == null || libraryBlob.Length == 0)
20.                 {
21.                     throw new InvalidOperationException(string.Format(
22.                         "Failed to load image library '{0}' - file was empty!", value.name));
23.                 }
24.  
25.                 var guids = new NativeArray<Guid>(value.count, Allocator.Temp);
26.                 try
27.                 {
28.                     for (int i = 0; i < value.count; ++i)
29.                     {
30.                         guids[i] = value[i].guid;
31.                     }
32.  
33.                     fixed (byte* blob = libraryBlob)
34.                     {
35.                         UnityARCore_imageTracking_setDatabase(
36.                             blob,
37.                             libraryBlob.Length,
38.                             guids.GetUnsafePtr(),
39.                             guids.Length);
40.                     }
41.                 }
42.                 finally
43.                 {
44.                     guids.Dispose();
45.                 }
46.             }
47.         }
48.     }
49. }

 

ARCore seems to have iOS support:

https://developers.google.com/ar/develop/unity/quickstart-ios

Would there be a way perhaps to run ARFoundation only on ARCore, even on iOS devices, and solve the dynamic target loading that way?

 

Looks like the ARCore support for iOS depends on UnityARKitPlugin, which is deprecated and already has a dynamic image detection solution already available by the Unity community.

In my case, I am interested more in Object Tracking with dynamic AR Objects. Today, like the image detection, it does not allow dynamic runtime object detection addition.

 

Good point there. Seems like I'll be ditching Unity and go with native implementations for iOS and Android in separate projects. As much as the AR community needs a good cross-platform abstraction, seems like ARFoundation is just not there yet. Vuforia is nice enough, but boy it's expensive.

 

Vuforia doesn't support dynamic image targets either. Although their algorithm for image tracking is far better than ARCore and slightly better than ARKit.

Also, the ARCore support for iOS did not include image tracking capabilities, it was mainly used for surface tracking.

 

Vuforia does support dynamic targetdatabases

 

Has anyone figured out how to programmatically add reference images to an XRReferenceImageLibrary?

 

So far there is just the aforementioned solution for ARCore, or going back to the deprecated plugins.

 

@LorenzoValente thank you for all the work you put into this!

Question: after I switch the imgdb for

XRReferenceImageLibrary

(ARCore) the

ARTrackedImage.referenceImage

objects that I get in my

ARTrackedImageManager.trackedImagesChanged

callback have empty name and guid properties. Without these properties, I'm not sure how to disambiguate between the different image anchors in my scene.

Have you noticed this? Is there a workaround?

Thanks so much!

 

I decided to build my app natively with ARKit. I wanted to use ARFundation, but I am disappointed with it.

 

I'm doing something like that (tested for iOS):
- save asset bundle with XRReferenceImageLibrary
- upload asset to server
- add ARTrackeImageManager to ARSessionOrigin
- disable it
- load asset from webserver
- assign to imgTrackedmanager.referenceLibrary
- enable imgTrackedmanager
Works fine.

Code (CSharp):

1.  
2.     private UnityWebRequest assetWebRequest;
3.     private ARTrackedImageManager imgTrackedmanager;
4.  
5.     IEnumerator GetAsset(string url, uint version)
6.     {
7.         Debug.Log("GetAsset");
8.  
9.         using (assetWebRequest = UnityWebRequestAssetBundle.GetAssetBundle(url, version, 0))
10.         {
11.  
12.             yield return assetWebRequest.SendWebRequest();
13.  
14.             if (assetWebRequest.isNetworkError || assetWebRequest.isHttpError)
15.             {
16.                 Debug.Log(assetWebRequest.error);
17.             }
18.             else
19.             {
20.                 // Get downloaded asset bundle
21.                 AssetBundle bundle = DownloadHandlerAssetBundle.GetContent(assetWebRequest);
22.                 Debug.Log("LoadImageLibrary:download finished:" + bundle);
23.  
24.                 var assetLoadRequest = bundle.LoadAssetAsync<XRReferenceImageLibrary>("ReferenceImageLibrary");
25.                 yield return assetLoadRequest;
26.  
27.                 XRReferenceImageLibrary lib = assetLoadRequest.asset as XRReferenceImageLibrary;
28.  
29.                 Debug.Log("LoadImageLibrary:lib:" + lib);
30.  
31.                 if (imgTrackedmanager == null)
32.                 {
33.                     imgTrackedmanager = GetComponent<ARTrackedImageManager>();
34.                 }
35.  
36.                 imgTrackedmanager.referenceLibrary = lib;
37.                 imgTrackedmanager.enabled = true;
38.  
39.                 Debug.Log("LoadImageLibrary:imgTrackedmanager.enabled:" + imgTrackedmanager.enabled);
40.  
41.                 bundle.Unload(false);
42.             }
43.         }
44.     }

 

@LorenzoValente how do you get the .imgdb file? Do you generate your own during runtime? I'm cannot find how to get it to add new images in runtime...

Also, i realised that all the approaches above described are to change between ARReferenceImageLibraries but not to add new images to an existing Library, right? I need the user to be able to upload their Image and 3DModels (in a desktop application) and then add these images to the Library when the user opens the mobile app... And i'm affraid i have to switch ARCore (i'm on Android) to make this easily happen...

PS: It would be great if someone from the AR Foundation team could inform us about whether this feature is going to be implemented or not in the future...

Best,
D

 

It seems that its actually possible now in the previews: https://docs.unity3d.com/Packages/c...elog/CHANGELOG.html#300-preview2---2019-09-05

https://forum.unity.com/threads/ar-foundation-multiple-tracked-image-managers.697625/

At least its possible to set it during runtime and then start the manager

 

Thank you @amichai2! No I'm sorry, this has never happened to me... are you using my esoteric code?

No @Dan_G, my need was to dynamically download the library from a remote server, so I packed the .imgdb file in an asset bundle statically created in the editor. Expanding the library at runtime sounds much more complicated

YES guys, it is happening! I am not working on that AR project anymore at the moment but I know that the AR Foundation is now supporting dynamic libraries I advise everyone to test the previews before trying my weird code

 

So, if i transform a ReferenceImageLibrary to a AssetBundle i can download it and load my Ar Project dynamically?

 

No, this is not enough, because both ARKit and ARCore produce a binary version of the AR Library that you need to put in the bundle and manually read.
Again, don't do this, with the newest AR Foundation should be much easier

 

I think we can get an idea from the documentation of AR Foundation package.

https://docs.unity3d.com/Packages/c...r.html#adding-new-reference-images-at-runtime

 

Can someone confirm this?

 

I cannot get the new ScheduleAddImageJob method to work at all for adding images to the reference library at runtime. I get no errors, but when I check the referenceLibrary.count after adding the new image it is still at 0. I'm pulling my hair out trying to figure out what I'm doing wrong, but according to everything I've found in the documentation it should be working.

 

@x70x I met the same problem in Android system too. But when I used the same codes build in iphoneX it just worked well .

 

Yes, I'm using this functionality successfully. It works using MutableRuntimeReferenceImageLibrary, and here's an example of how it's done:
https://github.com/Unity-Technologies/arfoundation-samples/issues/319

Don't pay attention to the issue reported, just look at the code. Incidentally, note the enabled=true at the bottom; one gotcha that tripped me up initially is that the ar manager is disabled automatically when it doesn't have an image library initially (which it won't if created using AddComponent, rather than setup in the editor).

I too am finding this works better on iPhone than on Android. However "better" does mean it is working on Android, just with room for improvement (that I am currently trying to figure out). Specifically, the added images aren't recognized on Android immediately, but only after I leave the scene and then reload it.

I'm about to ask about this on the forum. My best guess is that the imgdb takes longer to be ready, so I'm thinking I should monitor that using GetPathForLibrary as described here:
https://softwareproduction.eu/2019/...rreferenceimagelibrary-for-arcore-at-runtime/

Although x70x's post gives me another idea. Perhaps I can just make a coroutine that monitors referenceLibrary.count, and waits for the count to equal the number of images I added. hm I think I'll try that right now, before posting me question...

EDIT: rats the referenceLibrary.count idea didn't work, asking here:
https://forum.unity.com/threads/arf...works-properly-on-ios-but-not-android.785507/