Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Dynamic XRReferenceImageLibrary in AR Foundation

Discussion in 'AR/VR (XR) Discussion' started by LorenzoValente, May 21, 2019.

  1. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Hello, I'd like to start a discussion about how the AR tracked images are referenced in the new AR Foundation system. I'm using Unity 2019.2.0b1 with AR preview packages installed (AR Extensions + Foundation + Subsystems and ARCore+ARKit plugins).
    As far as I know there's no documentation yet on this feature but I managed to create a simple scene with a static library of images that, once detected by the phone camera, are used as anchors for 3D AR content. Perfect. The problem is that my aim was to have a dynamic library of images. With "dynamic" I mean a library that is not included in the app build but it is dynamically downloaded from a remote server (through an AssetBundle) and injected in the ARTrackedImageManager. After many tests I came to the conclusion that this is not possible with the current AR Foundation implementation. First of all, the manager dies if it does not find a library on initialization. My workaround was using a placeholder library with just one image (an empty library is not considered valid) and then replacing it at runtime:

    Code (CSharp):
    1. var manager = gameObject.GetComponent<ARTrackedImageManager>();
    2. manager.referenceLibrary = myDynamicLibrary;
    This approach does not generate any error and looked fair enough to me but it just doesn't work: the manager kept looking for the old library images.
    After further investigation I came across the ARCoreImageTrackingProvider, which acts as a bridge between the Foundation and the low-level ARCore (i'm on Android) image tracking functionalities. From my understanding of this code, using a dynamic library for image tracking is just impossible: the provider expects to find a file .imgdb (which is the ARCore format for the images library) in the build itself. Unity creates this file just before building and ships it along the internal Android stuff (see ARCorePreprocessBuild).

    Is there any solution for a dynamic approach? Maybe using Google ARCore implementation?
     
    Last edited: May 22, 2019
    iuriap, mende and Blarp like this.
  2. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Today I tried to go low-level and directly call ARCore library method to update the AR images library. To make it compile I had to flag it as unsafe. I also had to recover the .imgdb file generated by the ARCore preprocessing routine and pass it to my function (second parameter):

    Code (CSharp):
    1. public static unsafe void SetARCoreLibrary(XRReferenceImageLibrary lib, byte[] arcoreLib)
    2. {
    3.     if (arcoreLib == null || arcoreLib.Length == 0)
    4.     {
    5.         throw new InvalidOperationException(string.Format(
    6.             "Failed to load image library '{0}' - file was empty!", lib.name));
    7.     }
    8.  
    9.     var guids = new NativeArray<Guid>(lib.count, Allocator.Temp);
    10.     try
    11.     {
    12.         for (int i = 0; i < lib.count; ++i)
    13.         {
    14.             guids[i] = lib[i].guid;
    15.         }
    16.  
    17.         fixed (byte* blob = arcoreLib)
    18.         {
    19.             UnityARCore_imageTracking_setDatabase(
    20.                 blob,
    21.                 arcoreLib.Length,
    22.                 guids.GetUnsafePtr(),
    23.                 guids.Length);
    24.         }
    25.     }
    26.     catch (Exception e)
    27.     {
    28.         Debug.LogError(string.Format("Error while loading '{0}': {1}", lib.name, e));
    29.     }
    30.     finally
    31.     {
    32.         guids.Dispose();
    33.     }
    34. }
    35.  
    36.  
    37. [DllImport("UnityARCore")]
    38. static unsafe extern void UnityARCore_imageTracking_setDatabase(
    39.     void* databaseBytes, int databaseSize, void* sourceGuidBytes, int sourceGuidLength);
    Aaaand it doesn't work. I see no errors and the ARTrackerImageManager looks correctly configured (from the logs I can see that the number of images and their names is updated and correct)... but the system keeps detecting only the images of the old library.
    Yeah, this is a nightmare.
    My guess is that by replicating the [DllImport("UnityARCore")] attribute I'm actually calling another instance of that library... I'm not sure btw. I just know that a public keyword in the right place would have solved this problem...
     
    mende likes this.
  3. polytropoi

    polytropoi

    Joined:
    Aug 16, 2006
    Posts:
    677
    Interesting, thanks for sharing your efforts! Although it looks discouraging at the moment, I'm hopeful there will be a way eventually. I am sure that Vuforia will be grateful for the delay, because the capability we're looking for here is what continues to make them relevant. But the worst case is that Unity intends to restrict this feature, seeing it as a potential profit center and locks down "Cloud Reco" in the same way Vuforia does. That would suck.
     
  4. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Update: we have a documentation for the AR Tracked Image Manager!

    Quoting:
    The reference image library can be set at runtime, but as long as the tracked image manager component is enabled, the reference image library must be non-null.
    The reference image library is an instance of the ScriptableObject XRReferenceImageLibrary. This object contains mostly Editor data. The actual library data (containing the image data) is provider-specific. Refer to your provider's documentation for details.

    I'm quite confused... why are we allowed to set the library at runtime if the actual library data is created at build time?
     
  5. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Other details in the ARCore plugin documentation:
    When building the Player for Android, each reference image library is used to generate a corresponding imgdb file, ARCore's representation of the reference image library. These files are placed in your project's StreamingAssets folder in a subdirectory called HiddenARCore to allow them to be accessible at runtime.

    The StreamingAssets folder is read-only and this path is hardcoded in the provider's code... any idea on how to override this behaviour?
     
  6. TaiTex

    TaiTex

    Joined:
    May 14, 2014
    Posts:
    17
    I've been using EasyAR. It can pick up images directly from the StreamingAssets folder. I wanted to explore using ARFoundation but this is an issue for me also. Of course not running in the editor directly or using a working Remote also make me hold off on ARFoundation.
     
    LorenzoValente likes this.
  7. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Guys I solved.
    It works.
    And I'm proud of my esoteric solution!
    Basically I'm using reflection to call the private, static, unsafe method that updates the ARCore images database: ARCoreImageTrackingProvider.Provider.UnityARCore_imageTracking_setDatabase.
    I know, this is so wrong but I guess it is the only way by now.

    (WARNING, this C# code could make you cry).
    Code (CSharp):
    1. #if UNITY_ANDROID
    2. private static unsafe void ChangeARCoreImagesDatabase(XRReferenceImageLibrary library, byte[] arcoreDB)
    3. {
    4.     if (arcoreDB == null || arcoreDB.Length == 0)
    5.     {
    6.         throw new InvalidOperationException(string.Format(
    7.             "Failed to load image library '{0}' - file was empty!", library.name));
    8.     }
    9.  
    10.     var guids = new NativeArray<Guid>(library.count, Allocator.Temp);
    11.     try
    12.     {
    13.         for (int i = 0; i < library.count; ++i)
    14.             guids[i] = library[i].guid;
    15.  
    16.         fixed (byte* blob = arcoreDB)
    17.         {
    18.             // Retrieve the ARCore image tracking provider by reflection
    19.             var provider = typeof(ARCoreImageTrackingProvider).GetNestedType("Provider", BindingFlags.NonPublic | BindingFlags.Instance);
    20.  
    21.             // Destroy the current image tracking database
    22.             var destroy = provider.GetMethod("UnityARCore_imageTracking_destroy", BindingFlags.NonPublic | BindingFlags.Static);
    23.             destroy.Invoke(null, null);
    24.  
    25.             // Set the image tracking database
    26.             var setDatabase = provider.GetMethod("UnityARCore_imageTracking_setDatabase", BindingFlags.NonPublic | BindingFlags.Static);
    27.             setDatabase.Invoke(null, new object[]
    28.             {
    29.                 new IntPtr(blob),
    30.                 arcoreDB.Length,
    31.                 new IntPtr(guids.GetUnsafePtr()),
    32.                 guids.Length
    33.             });
    34.         }
    35.     }
    36.     catch (Exception e)
    37.     {
    38.         Debug.LogError(string.Format("Error while loading '{0}': {1}", library.name, e));
    39.     }
    40.     finally
    41.     {
    42.         guids.Dispose();
    43.     }
    44. }
    45. #endif
    TLDR: I found a way to change the ARCore tracked images database in an AR Foundation based application at runtime. The AR Foundation on Android expects to find this file in the app assets (it is automatically created and included at build time by Unity) making impossible to use downloaded or imported libraries. With my method you can swap between libraries easly, you just need to read the .imgdb file and pass the bytes to the function
     
    jhocking, IsDon, PTW_AR and 2 others like this.
  8. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Hey guys, at the moment I'm trying to do the same on ARKit... and oh my god.
    Here's the signature of the native call used to set the image library in iOS.
    Code (CSharp):
    1. [DllImport("__Internal")]
    2. static extern SetReferenceLibraryResult UnityARKit_imageTracking_trySetReferenceLibrary([MarshalAs(UnmanagedType.LPWStr)] string name, int nameLength, Guid guid);
    As you can see the parameter that matters is just the NAME of the library, string that it is internally used to find the file that Unity had created during the build processing... file that it won't find, since the library has been downloaded/generated dynamically.
    From my understanding, [DllImport("__Internal")] means that the actual implementation of that function is not in an external plugin but it is compiled in the engine itself... am I right? Is there a way to find the actual code?
     
  9. nilsdr

    nilsdr

    Joined:
    Oct 24, 2017
    Posts:
    230
    Hi Lorenzo,

    Thanks for your efforts researching this and posting the code, I will need this soon and was planning on looking into it. Will post here if I have any new insights.

    More on the internal statement here (but I'm sure you find this already):

    https://docs.unity3d.com/Manual/NativePlugins.html
     
    LorenzoValente likes this.
  10. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Hi nilsdr,
    Thank you, I hope for new insights aswell, I've no idea how to solve this :(
    Yes I saw that page and I found the statically linked plugin that ARKit uses for the internal calls: "Library/PackageCache/com.unity.xr.arkit@2.0.1/Runtime/iOS/UnityARKit.a". I tried to analyze its symbols (for the curious, you can do it using this unix command: nm UnityARKit.a) but I cannot see anything useful. I even tried to decompile it using Hopper
     
  11. polytropoi

    polytropoi

    Joined:
    Aug 16, 2006
    Posts:
    677
    I admire your dedication, Lorenzo, but it's clear that this capability (setting image targets at runtime) is not a design priority by Unity ARF team, at least not yet. It would be nice to have an official comment from someone on the AR Foundation team as to whether this omission is intentional, or if changes are coming that will make it possible.
     
    LorenzoValente likes this.
  12. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Thanks and yes, I totally agree with you, it is clear that this feature is not supported. I would love to hear news from the AR Foundation team!
     
  13. Alessio_Super

    Alessio_Super

    Joined:
    Aug 16, 2017
    Posts:
    2
    Oh damn, what a ride, hope they introduce any sort of solution for all of this very soon.
     
    LorenzoValente likes this.
  14. visualjoy

    visualjoy

    Joined:
    Aug 23, 2016
    Posts:
    19
    Hey @LorenzoValente !

    Thanks for sharing all this effort!
    We are going to try it very soon.
    It's quite sad that they leave this important functionality inaccessible.
     
    LorenzoValente likes this.
  15. Macoron

    Macoron

    Joined:
    Mar 11, 2017
    Posts:
    3
    Hey @LorenzoValente !
    Great work with ARCore part.

    I've managed to add reference image for ARKit, but only for now deprecated Unity ARKit Plugin. As far as I know, there is no way to add new reference image beside restarting native ARKit session. Check here for example.

    So we have to change .a library to add new native call that restarting AR session with new reference images set.
     
    LorenzoValente likes this.
  16. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Thank you Macoron!
    Yes i saw that there was a way with the old ARKit plugin... but I was looking for a cleaner solution that didn't break the Foundation packages, they should not be changed :(
     
    Heavenwinds likes this.
  17. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    Hey Lorenzo, thanks for trying to get dynamic image targets working in ARFoundation. It's cool that you were able to get it working for the ARCore part of things (which makes sense, the official ARcore Unity plugin supports that feature, although the documentation is poor).

    If you have any success in the future getting this working, please make another post here. I've started watching this thread.

    PS - I did see that some people say they got at least a single dynamic image target working with ARKit 1.5 (not ARFoundation) via this thread. But since the Unity ARKit plugin is depreciated, we need a solution moving forward.
     
    LorenzoValente likes this.
  18. RobAEG

    RobAEG

    Joined:
    Dec 20, 2018
    Posts:
    4
    Hello! I was wondering if anyone has been able to find any solutions regarding dynamic image libraries on iOS? I have been working on a project for 6 months now that uses the dynamic loading of images, which I got working on the older versions of the ARKit and ARCore plugins. I'm stuck using those older, deprecated plugins until ARFoundation adds support for this feature, as dynamic image loading is an integral part of my project.

    All the work of Lorenzo is very impressive, thanks for sharing it with us! It's a shame that Unity aren't adding dynamic image capability themselves, even though the feature is supported by both ARCore and ARKit.
     
    jhocking and LorenzoValente like this.
  19. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    Could you share the modified ARKit plugin code with us? That will help myself and others get this functionality working in at least the older versions of ARKit since it doesn't look like it's going to be brought over to ARFoundation.

    Personally I'm trying to get the modified ARKit to support multiple dynamic image targets, added before the session starts.
     
  20. RobAEG

    RobAEG

    Joined:
    Dec 20, 2018
    Posts:
    4
    Of course! There are 3 files that I modified, namely the UnityARCameraManager.cs, UnityARSessionNativeInterface.cs and ARSessionNative.mm. The first two are written in C#, the last on is written in Objective C.

    This is the function that I added to ARSessionNative.mm. It allows you to start a session with multiple tracking images, which can be downloaded or retrieved from the device at runtime.

    Code (CSharp):
    1. extern "C" void StartWorldTrackingSessionWithOptionsAndImages(void* nativeSession, ARKitWorldTrackingSessionConfiguration unityConfig, UnityARSessionRunOptions runOptions, const void* a_imageBuffer, unsigned int* a_bufferOffsets, unsigned int a_bufferLength, float* a_physicalWidth, int* a_id, int a_count)
    2. {
    3.     UnityARSession* session = (__bridge UnityARSession*)nativeSession;
    4.     ARWorldTrackingConfiguration* config = [ARWorldTrackingConfiguration new];
    5.     ARSessionRunOptions runOpts = GetARSessionRunOptionsFromUnityARSessionRunOptions(runOptions);
    6.     GetARSessionConfigurationFromARKitWorldTrackingSessionConfiguration(unityConfig, config);
    7.     session->_getPointCloudData = (BOOL) unityConfig.getPointCloudData;
    8.     session->_getLightEstimation = (BOOL) unityConfig.enableLightEstimation;
    9.    
    10.     if(UnityIsARKit_1_5_Supported() && unityConfig.referenceImagesResourceGroup != NULL && strlen(unityConfig.referenceImagesResourceGroup) > 0)
    11.     {
    12.         if (@available(iOS 11.3, *))
    13.         {
    14.             NSMutableSet *mSet = [[NSMutableSet alloc ] init];
    15.             NSData* allData = [[NSData alloc] initWithBytes:a_imageBuffer length:a_bufferLength];
    16.             for (int i = 0; i < a_count; i++) {
    17.                 int j;
    18.                 int y = 2;
    19.                 j = i * y;
    20.                 NSData* imageData = [allData subdataWithRange:NSMakeRange(a_bufferOffsets[j], a_bufferOffsets[j+1])];          
    21.                 UIImage* uiimage = [[UIImage alloc] initWithData:imageData];
    22.                 CGImageRef cgImage = [uiimage CGImage];
    23.                 ARReferenceImage *image = [[ARReferenceImage alloc] initWithCGImage:cgImage orientation:kCGImagePropertyOrientationUp physicalWidth:a_physicalWidth[i]];
    24.                 NSString *name = @(a_id[i]).stringValue;
    25.                 image.name = name;
    26.                 [mSet addObject:image];
    27.             }
    28.             config.detectionImages = mSet;
    29.         }
    30.     }
    31.  
    32.     if(UnityIsARKit_2_0_Supported())
    33.     {
    34.         if (@available(iOS 12.0, *))
    35.         {
    36.             NSMutableSet<ARReferenceObject *> *referenceObjects = nullptr;
    37.             if (unityConfig.referenceObjectsResourceGroup != NULL && strlen(unityConfig.referenceObjectsResourceGroup) > 0)
    38.             {
    39.                 NSString *strResourceGroup = [[NSString alloc] initWithUTF8String:unityConfig.referenceObjectsResourceGroup];
    40.                 [referenceObjects setByAddingObjectsFromSet:[ARReferenceObject referenceObjectsInGroupNamed:strResourceGroup bundle:nil]];
    41.             }
    42.            
    43.             if (unityConfig.ptrDynamicReferenceObjects != nullptr)
    44.             {
    45.                 NSSet<ARReferenceObject *> *dynamicReferenceObjects = (__bridge NSSet<ARReferenceObject *> *)unityConfig.ptrDynamicReferenceObjects;
    46.                 if (referenceObjects != nullptr)
    47.                 {
    48.                     [referenceObjects setByAddingObjectsFromSet:dynamicReferenceObjects];
    49.                 }
    50.                 else
    51.                 {
    52.                     referenceObjects = dynamicReferenceObjects;
    53.                 }
    54.             }          
    55.             config.detectionObjects = referenceObjects;
    56.         }
    57.     }
    58.    
    59.     if (runOptions == UnityARSessionRunOptionsNone)
    60.         [session->_session runWithConfiguration:config];
    61.     else
    62.         [session->_session runWithConfiguration:config options:runOpts];  
    63.     [session setupMetal];
    64. }
    This is the code that I added to the UnityARSessionNativeInterface.cs. Don't forget to import it with the [DllImport("__Internal")], else you won't be able to call it.

    Code (CSharp):
    1. public void RunWithConfigAndOptionsAndImages(ARKitWorldTrackingSessionConfiguration config, UnityARSessionRunOption runOptions, byte[] a_ImageBuffer, uint[] a_BufferLengths, uint a_BufferLength, float[] a_physicalWidth, int[] a_id, int a_count)
    2.         {
    3. #if !UNITY_EDITOR && UNITY_IOS
    4.             StartWorldTrackingSessionWithOptionsAndImages(m_NativeARSession, config, runOptions, a_ImageBuffer, a_BufferLengths, a_BufferLength, a_physicalWidth, a_id, a_count);
    5. #elif UNITY_EDITOR
    6.             CreateRemoteWorldTrackingConnection(config, runOptions);
    7. #endif
    8.         }
    And finally, this is the code that I added to the UnityARCameraManager. This is the function that you should call to start an ARKit session with your own, dynamically loaded tracking images. Simply provide the images, reference image sizes and the IDs for the images. The IDs are integers here, but they could also be reworked to be strings. You'll be able to see which image has been detected by using the ARImageAnchor.referenceImageName function. This way you can have specific behavior when an image with a certain ID is tracked.

    Code (CSharp):
    1.  public void AddNewReferenceImages(Texture2D[] _images, float[] _sizes, int[] _ids) {
    2.         UnityARSessionNativeInterface.ARSessionShouldAttemptRelocalization = true;
    3.         m_session = UnityARSessionNativeInterface.GetARSessionNativeInterface();
    4.         Application.targetFrameRate = 60;
    5.         ARKitWorldTrackingSessionConfiguration config = sessionConfiguration;
    6.         if (config.IsSupported) {
    7.             List<byte> _totalBytes = new List<byte>();
    8.             List<uint> _offsets = new List<uint>();
    9.             for(int i = 0; i < _images.Length; i++) {
    10.                 byte[] _bytes = _images[i].EncodeToJPG();
    11.                 if(_offsets.Count > 0 ) {
    12.                     _offsets.Add((uint)_totalBytes.Count);
    13.                 } else {
    14.                     _offsets.Add(0);
    15.                 }
    16.                 _totalBytes.AddRange(_bytes);
    17.                 _offsets.Add((uint)_bytes.Length);            
    18.             }
    19.             uint _totalLength = (uint)_totalBytes.Count;
    20.             m_session.RunWithConfigAndOptionsAndImages(config, UnityARSessionRunOption.ARSessionRunOptionRemoveExistingAnchors | UnityARSessionRunOption.ARSessionRunOptionResetTracking, _totalBytes.ToArray(), _offsets.ToArray(), _totalLength, _sizes, _ids, _images.Length);
    21.             UnityARSessionNativeInterface.ARFrameUpdatedEvent += FirstFrameUpdate;
    22.         } else {
    23.             Debug.Log("Image config not supported!");
    24.         }
    25.         if (m_camera == null) {
    26.             m_camera = Camera.main;
    27.         }
    28.     }
    Hope it helps!
     
  21. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    That's awesome, I'm going to post this in the ARKit feature request thread for others to see, this will help hold over a lot of people until ARFoundation catches up.

    https://bitbucket.org/Unity-Technol...es/20/couldnt-dynamic-update-detection-images
     
    RobAEG likes this.
  22. samuelmorais

    samuelmorais

    Joined:
    Aug 3, 2012
    Posts:
    39
    Hi all,

    I have also used a similar solution that @RobAEG describes for ARKit 1.5 and 2.0.

    Now, I am waiting for a solution for this problem in ARFoundation + ARKit3, too.
     
  23. RobAEG

    RobAEG

    Joined:
    Dec 20, 2018
    Posts:
    4
    I'm glad that I was able to provide some help! I actually got most of the information that I needed from that exact same thread. I posted there as Roverboef. The AR scene is a small world it seems.
     
  24. samuelmorais

    samuelmorais

    Joined:
    Aug 3, 2012
    Posts:
    39
    Still waiting for a solution to the dynamic/runtime update of reference images or objects with ARKIT 3.0 and AR Foundation... Any idea, someone? I think we depend on a native iOS code or wait to Unity develop a way of doing this.

    Is there a technical limitation for this or this is just a business decision?
     
  25. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    I don't think it's a technical limitation, as it's an officially supported feature in ARKit native, but not on ARKit Unity Plugin. And clearly, there are ways to add in the missing functionality. I'm guessing that Unity's ARFoundation developers will only add features that both ARCore and ARKit officially support through their respective unmodified plugins. So that's why I'm doubtful we'll see that feature added anytime soon.

    My speculation is that this would require Unity's ARFoundation team talking to the ARCore and ARKit plugin teams respectively and getting this feature added to those plugins officially, and then Unity's ARFoundation team would wrap it with their code. And since this is hard to market as a new feature, it will get less priority than other AR features.

    Truthfully I haven't messed around with ARFoundation much, as I have my own wrapper for ARCore/ARKit that incorporates these features. But I'm guessing we're missing access to one of the modified files discussed in this thread that's preventing us from just adding in this feature ourselves to ARFoundation, similar to how it's been shown can be done for ARFoundation's implementation of ARCore.
     
    Last edited: Jul 24, 2019
  26. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    So I've added this code to my ARKit Unity 2.0 plugin, however, the image anchor is never found after being added dynamically before the session is started.

    I thought it might be because we never set up an ARReferenceImage data object in the session configuration, but adding a dynamic ARReferenceImageSet and a dynamically created ARReferenceImage to it with a test texture leads to the following error within xCode upon runtime...

    Code (CSharp):
    1. did fail with error: Error Domain=com.apple.arkit.error Code=300 "Invalid reference image." UserInfo={NSLocalizedFailureReason=One or more reference images have an invalid size: 1311512199, NSLocalizedRecoverySuggestion=Make sure that all reference images are greater than 100 pixels and have a positive physical size in meters.
    From what I've read online, the iOS project needs to be set to iOS 12.0 and the image needs to be a .jpg with read/write enabled, and the texture size must be over 100x100 pixels. And finally, the supplied width in meters must be a positive value (1 meter for my test image). The test image I'm using fulfils all of these requirements.

    @RobAEG did you encounter anything like this? If not would you be able to help out by providing a test project that you confirmed works on your end? Sorry for the request, but as you noted, the world of AR developers is a small one.
     
    rsodre likes this.
  27. RobAEG

    RobAEG

    Joined:
    Dec 20, 2018
    Posts:
    4
    I'm not able to provide a test project at this very moment but I might be able to help you out! You don't have to add an ARReferenceImageSet the the UnityARCameraManager script for the code to work. We build our own in the ARSessionNative.mm.

    Have you debugged your code to see what happens when you use the modified scripts to dynamically load tracking images? Note that the image sizes for ARKit can be a bit finicky, 1 meter is very big. I think it might work better if you get up close to say a 15 centimeter image. Image sizes should be entered with 0.1 being equal to 1 meter. So for an image of 15 centimeters you'd give a size of 0.015.
     
  28. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    I found where I was having an issue, in ARSessionNative.mm code above we were checking...
    Code (csharp):
    1.  
    2. if(UnityIsARKit_1_5_Supported() && unityConfig.referenceImagesResourceGroup != NULL && strlen(unityConfig.referenceImagesResourceGroup) > 0 )
    but the
    unityConfig.referenceImagesResourceGroup
    variable is unused. We can safely remove it from that if statement to get the image targets working properly.
     
  29. gphilip

    gphilip

    Joined:
    Jan 31, 2015
    Posts:
    4
    Can you share a test project that works with this code? I'm a little lost with the ARKit plugin.
     
  30. mhusseini

    mhusseini

    Joined:
    Jul 12, 2019
    Posts:
    1
    Hi all, I want to add another way to dynamically load reference libraryies at runtime for Android - without the need for reflection.

    First of all, thanks to @LorenzoValente for his great work. I explored his approach and that led me to my solution. So basically has the folloing steps:
    1. Register a custom XRImageTrackingSubsystem which contains almost the same code as ARCoreImageTrackingProvider (except that it loads the database from a remote location).
    2. Disable the ARTrackedImageManager in the Editor, so it doesn't start loading the image database directly at start.
    3. In my script upon awake, deserialize a new XRReferenceImageLibrary with my new reference images (but that's just some meta info, not the imgdb file itself, yet).
    4. Enable the ARTrackedImageManager in my script, which triggers loading of the database.
    5. My custom XRImageTrackingSubsystem gets called and it in turn loads the imdb file from my server.
    Now for the steps in more detail:

    Register a custom XRImageTrackingSubsystem
    The included XRImageTrackingSubsystem for ARCore is called ARCoreImageTrackingProvider and can be found in \Library\PackageCache\com.unity.xr.arcore@2.1.0-preview.5\Runtime\ARCoreImageTrackingProvider.cs. Copy this file to you own project and rename it. You only need some minor adjustments.First, change the point in time the provider gets registered. We need our custom provider to get registered before the built-in provider beacuse the only the first one gets used. So change this line...

    Code (CSharp):
    1. [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
    2. static void RegisterDescriptor()
    3. {
    4.     // ...
    5. }
    ... to this line...

    Code (CSharp):
    1. [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterAssembliesLoaded)]
    2. static void RegisterDescriptor()
    3. {
    4.     // ...
    5. }
    Next, change the GetPathForLibrary method so it returns the URL of your imgdb file on the server. Here's the original code:

    Code (CSharp):
    1. internal static string GetPathForLibrary(XRReferenceImageLibrary library)
    2. {
    3.     if (library == null)
    4.         throw new ArgumentNullException("library");
    5.  
    6.     return Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
    7. }
    And here's my code:

    Code (CSharp):
    1. internal static string GetPathForLibrary(XRReferenceImageLibrary library)
    2. {
    3.     if (library == null)
    4.         throw new ArgumentNullException("library");
    5.  
    6.     return library is XRReferenceImageLibraryWithDynamicLoading libraryWithDynamicLoading && !string.IsNullOrWhiteSpace(libraryWithDynamicLoading.Url)
    7.         ? libraryWithDynamicLoading.Url
    8.         : Path.Combine(k_StreamingAssetsPath, library.guid.ToString() + ".imgdb");
    9. }
    You may notice that I have introducted a type called XRReferenceImageLibraryWithDynamicLoading. This is just a subclass of XRReferenceImageLibrary that contains an additional property for supplying the URL. This is my aproach to manage URLs, your's might look differently.

    Code (CSharp):
    1. public class XRReferenceImageLibraryWithDynamicLoading : XRReferenceImageLibrary
    2. {
    3.     public string Url { get; set; }
    4. }
    Disable the ARTrackedImageManager in the Editor
    upload_2019-8-2_12-1-39.png

    Deserialize a new XRReferenceImageLibrary / Enable the ARTrackedImageManager
    Your image library should match your image database.
    Code (CSharp):
    1.     public void Awake()
    2.     {
    3.         this.trackedImageManager = this.GetComponent<ARTrackedImageManager>();
    4.  
    5. #if !UNITY_EDITOR
    6.         var json = @"{
    7.            ""m_GuidLow"": 5140542808452585354,
    8.            ""m_GuidHigh"": 2180237115512313227,
    9.            ""Url"": ""http://172.20.10.5:6789/myimages.imgdb"",
    10.            ""m_Images"": [
    11.                {
    12.                    ""m_SerializedGuid"": {
    13.                        ""m_GuidLow"": 5679415376573207540,
    14.                        ""m_GuidHigh"": 6089316183866679477
    15.                    },
    16.                    ""m_SpecifySize"": false,
    17.                    ""m_Name"": ""poster-of-cool-movie""
    18.                }
    19.            ]
    20.        }";
    21.         var library = new XRReferenceImageLibraryWithDynamicLoading();
    22.         JsonUtility.FromJsonOverwrite(json, library);
    23.         this.trackedImageManager.referenceLibrary = library;
    24. #endif
    25.         this.trackedImageManager.enabled = true;
    26.     }

    Of course I'd normally get the JSON from a server as well, but for testing purposes I inlined it here. Notice the
    #if !UNITY_EDITOR
    preprocessor directive. If you omit that, this code will be called in the editor. Since I don't want any dynamic loading and server calls to be performed in the editor, I just disabled it that way.

    The last line in the method above enables the ARTrackedImageManager which triggeres all the internal loading and with it our next step.

    Load the imdb file from my server
    Back in my custom XRImageTrackingSubsystem, since I provided the URL of a remote server instead of a path to a file on the device, the built-in code is already capable of handling it. I'll show the code here, but note that this wasn't code I added, but coded that was already included in the original ARCoreImageTrackingProvider. The only modification I made was inside the GetPathForLibrary method I described above.

    Code (CSharp):
    1. public unsafe override XRReferenceImageLibrary imageLibrary
    2. {
    3.     set
    4.     {
    5.         if (value == null)
    6.         {
    7.             UnityARCore_imageTracking_setDatabase(null, 0, null, 0);
    8.         }
    9.         else
    10.         {
    11.             using (var uwr = new UnityWebRequest(GetPathForLibrary(value)))
    12.             {
    13.                 uwr.downloadHandler = new DownloadHandlerBuffer();
    14.                 uwr.disposeDownloadHandlerOnDispose = true;
    15.                 uwr.SendWebRequest();
    16.                 while (!uwr.isDone) {}
    17.  
    18.                 byte[] libraryBlob = uwr.downloadHandler.data;
    19.                 if (libraryBlob == null || libraryBlob.Length == 0)
    20.                 {
    21.                     throw new InvalidOperationException(string.Format(
    22.                         "Failed to load image library '{0}' - file was empty!", value.name));
    23.                 }
    24.  
    25.                 var guids = new NativeArray<Guid>(value.count, Allocator.Temp);
    26.                 try
    27.                 {
    28.                     for (int i = 0; i < value.count; ++i)
    29.                     {
    30.                         guids[i] = value[i].guid;
    31.                     }
    32.  
    33.                     fixed (byte* blob = libraryBlob)
    34.                     {
    35.                         UnityARCore_imageTracking_setDatabase(
    36.                             blob,
    37.                             libraryBlob.Length,
    38.                             guids.GetUnsafePtr(),
    39.                             guids.Length);
    40.                     }
    41.                 }
    42.                 finally
    43.                 {
    44.                     guids.Dispose();
    45.                 }
    46.             }
    47.         }
    48.     }
    49. }
     
  31. gphilip

    gphilip

    Joined:
    Jan 31, 2015
    Posts:
    4
  32. samuelmorais

    samuelmorais

    Joined:
    Aug 3, 2012
    Posts:
    39
    Looks like the ARCore support for iOS depends on UnityARKitPlugin, which is deprecated and already has a dynamic image detection solution already available by the Unity community.

    In my case, I am interested more in Object Tracking with dynamic AR Objects. Today, like the image detection, it does not allow dynamic runtime object detection addition.
     
  33. gphilip

    gphilip

    Joined:
    Jan 31, 2015
    Posts:
    4
    Good point there. Seems like I'll be ditching Unity and go with native implementations for iOS and Android in separate projects. As much as the AR community needs a good cross-platform abstraction, seems like ARFoundation is just not there yet. Vuforia is nice enough, but boy it's expensive.
     
  34. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    Vuforia doesn't support dynamic image targets either. Although their algorithm for image tracking is far better than ARCore and slightly better than ARKit.

    Also, the ARCore support for iOS did not include image tracking capabilities, it was mainly used for surface tracking.
     
  35. nilsdr

    nilsdr

    Joined:
    Oct 24, 2017
    Posts:
    230
    Vuforia does support dynamic targetdatabases
     
    DrSharky likes this.
  36. petitbas

    petitbas

    Joined:
    Sep 9, 2014
    Posts:
    5
    Has anyone figured out how to programmatically add reference images to an XRReferenceImageLibrary?
     
  37. sphira_co

    sphira_co

    Joined:
    Jun 13, 2019
    Posts:
    4
    So far there is just the aforementioned solution for ARCore, or going back to the deprecated plugins.
     
  38. amichai2

    amichai2

    Joined:
    Jul 30, 2018
    Posts:
    3
    @LorenzoValente thank you for all the work you put into this!

    Question: after I switch the imgdb for
    XRReferenceImageLibrary
    (ARCore) the
    ARTrackedImage.referenceImage
    objects that I get in my
    ARTrackedImageManager.trackedImagesChanged
    callback have empty name and guid properties. Without these properties, I'm not sure how to disambiguate between the different image anchors in my scene.

    Have you noticed this? Is there a workaround?

    Thanks so much!
     
  39. tomicz

    tomicz

    Joined:
    Mar 16, 2015
    Posts:
    125
    I decided to build my app natively with ARKit. I wanted to use ARFundation, but I am disappointed with it.
     
  40. grzegorn

    grzegorn

    Joined:
    Mar 3, 2015
    Posts:
    3
    I'm doing something like that (tested for iOS):
    - save asset bundle with XRReferenceImageLibrary
    - upload asset to server
    - add ARTrackeImageManager to ARSessionOrigin
    - disable it
    - load asset from webserver
    - assign to imgTrackedmanager.referenceLibrary
    - enable imgTrackedmanager
    Works fine.

    disabled_artrackedimagemanager.png

    Code (CSharp):
    1.  
    2.     private UnityWebRequest assetWebRequest;
    3.     private ARTrackedImageManager imgTrackedmanager;
    4.  
    5.     IEnumerator GetAsset(string url, uint version)
    6.     {
    7.         Debug.Log("GetAsset");
    8.  
    9.         using (assetWebRequest = UnityWebRequestAssetBundle.GetAssetBundle(url, version, 0))
    10.         {
    11.  
    12.             yield return assetWebRequest.SendWebRequest();
    13.  
    14.             if (assetWebRequest.isNetworkError || assetWebRequest.isHttpError)
    15.             {
    16.                 Debug.Log(assetWebRequest.error);
    17.             }
    18.             else
    19.             {
    20.                 // Get downloaded asset bundle
    21.                 AssetBundle bundle = DownloadHandlerAssetBundle.GetContent(assetWebRequest);
    22.                 Debug.Log("LoadImageLibrary:download finished:" + bundle);
    23.  
    24.                 var assetLoadRequest = bundle.LoadAssetAsync<XRReferenceImageLibrary>("ReferenceImageLibrary");
    25.                 yield return assetLoadRequest;
    26.  
    27.                 XRReferenceImageLibrary lib = assetLoadRequest.asset as XRReferenceImageLibrary;
    28.  
    29.                 Debug.Log("LoadImageLibrary:lib:" + lib);
    30.  
    31.                 if (imgTrackedmanager == null)
    32.                 {
    33.                     imgTrackedmanager = GetComponent<ARTrackedImageManager>();
    34.                 }
    35.  
    36.                 imgTrackedmanager.referenceLibrary = lib;
    37.                 imgTrackedmanager.enabled = true;
    38.  
    39.                 Debug.Log("LoadImageLibrary:imgTrackedmanager.enabled:" + imgTrackedmanager.enabled);
    40.  
    41.                 bundle.Unload(false);
    42.             }
    43.         }
    44.     }
     
    polytropoi likes this.
  41. Dan_G

    Dan_G

    Joined:
    Dec 4, 2017
    Posts:
    1
    @LorenzoValente how do you get the .imgdb file? Do you generate your own during runtime? I'm cannot find how to get it to add new images in runtime...

    Also, i realised that all the approaches above described are to change between ARReferenceImageLibraries but not to add new images to an existing Library, right? I need the user to be able to upload their Image and 3DModels (in a desktop application) and then add these images to the Library when the user opens the mobile app... And i'm affraid i have to switch ARCore (i'm on Android) to make this easily happen...

    PS: It would be great if someone from the AR Foundation team could inform us about whether this feature is going to be implemented or not in the future...

    Best,
    D
     
    Last edited: Sep 20, 2019
  42. polytropoi

    polytropoi

    Joined:
    Aug 16, 2006
    Posts:
    677
    Yes, I think the official silence on this striking. But perhaps the reason can be found at the bottom of this page, under the "Partners" heading, where you'll see the vuforia logo. Vuforia has been a very important partnership for Unity for years, that has benefitted all of us. Hesitating to destroy the way your partner makes money may be the charitable explanation.
     
  43. BlackclawsK

    BlackclawsK

    Joined:
    Jan 9, 2019
    Posts:
    3
  44. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    Thank you @amichai2! No I'm sorry, this has never happened to me... are you using my esoteric code?

    No @Dan_G, my need was to dynamically download the library from a remote server, so I packed the .imgdb file in an asset bundle statically created in the editor. Expanding the library at runtime sounds much more complicated

    YES guys, it is happening! I am not working on that AR project anymore at the moment but I know that the AR Foundation is now supporting dynamic libraries :) I advise everyone to test the previews before trying my weird code
     
    Last edited: Oct 28, 2019
    DrSharky likes this.
  45. AlbyDj90

    AlbyDj90

    Joined:
    Feb 7, 2015
    Posts:
    29
    So, if i transform a ReferenceImageLibrary to a AssetBundle i can download it and load my Ar Project dynamically?
     
  46. LorenzoValente

    LorenzoValente

    Joined:
    May 2, 2019
    Posts:
    23
    No, this is not enough, because both ARKit and ARCore produce a binary version of the AR Library that you need to put in the bundle and manually read.
    Again, don't do this, with the newest AR Foundation should be much easier
     
  47. unity-top-man

    unity-top-man

    Joined:
    Monday
    Posts:
    1
  48. tomicz

    tomicz

    Joined:
    Mar 16, 2015
    Posts:
    125