Search Unity

  1. Calling all beginners! Join the FPS Beginners Mods Challenge until December 13.
    Dismiss Notice
  2. It's Cyber Week at the Asset Store!
    Dismiss Notice

Dynamic XRReferenceImageLibrary in AR Foundation

Discussion in 'AR/VR (XR) Discussion' started by LorenzoValente, May 21, 2019.

  1. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    Yes, I'm using this functionality successfully. It works using MutableRuntimeReferenceImageLibrary, and here's an example of how it's done:
    https://github.com/Unity-Technologies/arfoundation-samples/issues/319

    Don't pay attention to the issue reported, just look at the code. Incidentally, note the enabled=true at the bottom; one gotcha that tripped me up initially is that the ar manager is disabled automatically when it doesn't have an image library initially (which it won't if created using AddComponent, rather than setup in the editor).

    I too am finding this works better on iPhone than on Android. However "better" does mean it is working on Android, just with room for improvement (that I am currently trying to figure out). Specifically, the added images aren't recognized on Android immediately, but only after I leave the scene and then reload it.

    I'm about to ask about this on the forum. My best guess is that the imgdb takes longer to be ready, so I'm thinking I should monitor that using GetPathForLibrary as described here:
    https://softwareproduction.eu/2019/...rreferenceimagelibrary-for-arcore-at-runtime/

    Although x70x's post gives me another idea. Perhaps I can just make a coroutine that monitors referenceLibrary.count, and waits for the count to equal the number of images I added. hm I think I'll try that right now, before posting me question...

    EDIT: rats the referenceLibrary.count idea didn't work, asking here:
    https://forum.unity.com/threads/arf...works-properly-on-ios-but-not-android.785507/
     
    Last edited: Dec 3, 2019 at 10:14 PM
  2. x70x

    x70x

    Joined:
    Aug 20, 2011
    Posts:
    49
    Interesting. I'm using AR Foundation to hopefully keep my app cross-platform, but I have not started iOS testing yet. I'm only at the very start of this project and trying out different APIs and solutions. Is there a place to submit bugs for this?
     
  3. x70x

    x70x

    Joined:
    Aug 20, 2011
    Posts:
    49
    Wow. This actually worked for me. I was able to load an image from a web url, add it to my reference library, and then start tracking that image immediately after it finished loading. I didn't need to reload the scene or anything.

    The problem is that this code is extremely similar to my own, so I have no idea why this worked and why my code did not. I will need to do some more testing.

    Thank you for the link!
     
  4. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    damn really, this code worked without issue? I hadn't thought to actually run this code for testing, only referred to it while writing my own code. I guess I better try this now...

    EDIT: hm well I'm seeing "myRuntimeReferenceImageLibrary: 0" in the console, which is the debug message for library count. I'm guessing there is something subtle you accidentally got right, like maybe it matters which object this code is attached to.

    EDIT2: I just noticed something that may be related in the logcat for my application. I see this OpenGL error the first time I load the scene, but not the second:

    OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_ENUM: enum argument out of range
    (Filename: ./Runtime/GfxDevice/opengles/GfxDeviceGLES.cpp Line: 348)
     
    Last edited: Dec 2, 2019 at 10:25 PM
  5. x70x

    x70x

    Joined:
    Aug 20, 2011
    Posts:
    49
    This is working for me. Check if your setup is similar to mine and then try the code below.

    Game Objects in the Scene:
    • AR Session Origin (with AR Session Origin component)
      • AR Camera (with AR Pose Driver, AR Camera Manager, and AR Camera Background components)
    • AR Session (with AR Session and AR Input Manager components)
    • EventSystem (with Event System and Standalone Input Module components)
    • Directional Light
    Then let's create a new script. I called mine DynamicImageLibrary and I attached it to the AR Session Origin object.

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using System.IO;
    4. using UnityEngine;
    5. using UnityEngine.Networking;
    6. using UnityEngine.XR.ARFoundation;
    7. using UnityEngine.XR.ARSubsystems;
    8.  
    9. public class DynamicImageLibrary : MonoBehaviour
    10. {
    11.     public ARTrackedImageManager mARTrackedImageManager;
    12.     public GameObject mTrackedImagePrefab;
    13.     public Texture2D imageToTexture2d;
    14.     public MutableRuntimeReferenceImageLibrary myRuntimeReferenceImageLibrary;
    15.  
    16.     private void Awake()
    17.     {
    18.         Screen.sleepTimeout = SleepTimeout.NeverSleep;
    19.         mARTrackedImageManager = gameObject.AddComponent<ARTrackedImageManager>();
    20.         mARTrackedImageManager.enabled = false;
    21.         myRuntimeReferenceImageLibrary = mARTrackedImageManager.CreateRuntimeLibrary() as MutableRuntimeReferenceImageLibrary;
    22.         mARTrackedImageManager.maxNumberOfMovingImages = 1;
    23.         mARTrackedImageManager.trackedImagePrefab = mTrackedImagePrefab;
    24.         mARTrackedImageManager.trackedImagesChanged += OnImageChanged;
    25.     }
    26.  
    27.     void Start()
    28.     {      
    29.         StartCoroutine(AddImageTrackerByUrl("https://upload.wikimedia.org/wikipedia/commons/9/97/The_Earth_seen_from_Apollo_17.jpg[/URL]"));
    30.     }
    31.  
    32.     public void OnImageChanged(ARTrackedImagesChangedEventArgs args)
    33.     {
    34.         foreach (var trackedImage in args.added)
    35.         {
    36.             Debug.Log(trackedImage.referenceImage.name);
    37.         }
    38.     }
    39.  
    40.     IEnumerator AddImageTrackerByUrl(string url)
    41.     {
    42.         mARTrackedImageManager.enabled = false;
    43.         if (mARTrackedImageManager.descriptor.supportsMutableLibrary)
    44.         {
    45.             UnityWebRequest webRequest = UnityWebRequestTexture.GetTexture(url);
    46.             yield return webRequest.SendWebRequest();
    47.             imageToTexture2d = DownloadHandlerTexture.GetContent(webRequest);
    48.  
    49.             Unity.Jobs.JobHandle jobHandle = myRuntimeReferenceImageLibrary.ScheduleAddImageJob(imageToTexture2d, Path.GetFileName(url), 0.2f);
    50.             jobHandle.Complete();
    51.  
    52.             if (myRuntimeReferenceImageLibrary != null)
    53.             {
    54.                 Debug.Log("Image Library Count: " + myRuntimeReferenceImageLibrary.count);
    55.  
    56.                 mARTrackedImageManager.referenceLibrary = myRuntimeReferenceImageLibrary;
    57.             }
    58.         }
    59.         mARTrackedImageManager.enabled = true;
    60.     }
    61. }

    I have only tested this on Android so far. If you get it to work on either iOS or Android please let me know!
     
    Last edited: Dec 3, 2019 at 3:03 PM
    jhocking-bundlar likes this.
  6. Usoka

    Usoka

    Joined:
    Tuesday
    Posts:
    4
    I've been trying to get this working for a while now, and currently have something very similar to yours (except I'm loading images locally, not from web). Still doesn't seem to be working for me though.
    Do you mind specifying what the versions of everything you're using is? ARFoundation, ARCore, and Unity itself. Might be something which only works in specific version.
     
    jhocking-bundlar likes this.
  7. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    Great idea, and thanks for putting together this minimal test. When I build that for Android it doesn't work, with these messages in logcat:

    12-02 21:22:42.124 5597 5662 E Unity : OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_ENUM: enum argument out of range
    12-02 21:22:42.124 5597 5662 E Unity : (Filename: ./Runtime/GfxDevice/opengles/GfxDeviceGLES.cpp Line: 348)

    12-02 21:22:43.868 5597 5618 I Unity : Image Library Count: 0
    12-02 21:22:43.868 5597 5618 I Unity : <AddImageTrackerByUrl>d__7:MoveNext()
    12-02 21:22:43.868 5597 5618 I Unity : UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
    12-02 21:22:43.868 5597 5618 I Unity : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

    That's the same OpenGL error I was seeing before, so I'm gonna try googling that error message to see if I come up with anything. And yeah I second specifying which version of stuff you are using. I've got Unity 2019.2.13 with preview.3 3.0.0 for both AR Foundation and ARCore; I know there's a newer preview package available, but I've tried upgrading and that didn't change anything.

    In the meantime, I'm going to build this for iOS to let you know if it works there.
     
    Last edited: Dec 3, 2019 at 3:43 AM
  8. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    Confirmed, it is working on iOS. Incidentally, I added more logging so that I can tell in the console when it's tracking:

    Code (CSharp):
    1.     public void OnImageChanged(ARTrackedImagesChangedEventArgs args)
    2.     {
    3.         foreach (var trackedImage in args.added)
    4.         {
    5.             Debug.Log($"added: {trackedImage.referenceImage.name}");
    6.         }
    7.         foreach (var trackedImage in args.updated)
    8.         {
    9.             Debug.Log($"updated: {trackedImage.transform.position.x}, {trackedImage.transform.position.y}, {trackedImage.transform.position.z}");
    10.         }
    11.     }
    EDIT: whoah now it seems to be working on Android! I built it again cause I didn't have this additional logging before and was curious. Incidentally, I still see that OpenGL error, but the library count was 1 this time, so I guess that OpenGL error is a red herring.
     
    Last edited: Dec 3, 2019 at 4:09 AM
  9. x70x

    x70x

    Joined:
    Aug 20, 2011
    Posts:
    49
    Interesting. I am using AR Foundation 3.1.0 which I believe is the latest preview release. Installed via the Unity Package Manager. Unity is 2019.2.8f1.

    This seems good enough for testing and early development prototyping. Hopefully as the updates for AR Foundation move out of preview they will become more reliable and have better documentation. This would make it so that I don't have to rely on Vuforia.
     
  10. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    yeah replacing Vuforia in our application is why I started messing with the AR Foundation preview package. The prototyping has been going great, except that it is pretty annoying that I can't test the AR tracking within the editor, and have to keep doing builds every time.
     
  11. x70x

    x70x

    Joined:
    Aug 20, 2011
    Posts:
    49
    Yes, that is a major downside. At one point they had a remote option where you could use your phone's video feed (connected via usb) in the editor play mode, but they removed that for some reason. I hope they bring it back soon.
     
  12. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    oh hey I just got it working immediately (ie. I don't need to reload the scene for tracking) in my big app! While looking at the minimal sample you put together, it occurred to me that there is a pause for downloading the image between when the tracking manager is created and when the image is added. On a hunch I figured maybe on Android the AR system needs more initialization time, so I did things in a coroutine and paused for a bit before starting to add images to the created image library.
     
  13. Usoka

    Usoka

    Joined:
    Tuesday
    Posts:
    4
    Does seem to be something to do with timing. I changed my code back to being done in a co routine (which hadn't worked previously) and added
    yield return null
    just after where it loads a list of all the images as byte[] and it works now!
    For testing purposes I just tried moving it to the end of the method (where I'd had it before just so it'd work as a co routine) and it didn't work. So it definitely depends on where it is and needs to be after you load/get your images, and before you start adding them to the library.
     
  14. Usoka

    Usoka

    Joined:
    Tuesday
    Posts:
    4
    Just did a fresh install of my app and it didn't work when it first started up. However closing and reopening it did work. So the different between the files already being in persistent storage and having to load them in there from the jar first also slows things down enough that it doesn't work.
    So for anyone that's developing an app for Android using ARFoundation, be aware of timing and that having an extra `yield return` can be the difference between having count = 0 and count = n.
     
    jhocking-bundlar likes this.
  15. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    ah, it did seem like my code needed two 'yield return null;' lines (ie. pause two frames) not just one, I guess that's why.
     
  16. Usoka

    Usoka

    Joined:
    Tuesday
    Posts:
    4
    New challenge. Creating the library dynamically seem to not work as well for actual image tracking. Looking at the statuses on trackedImagesChanged, for those in args.updated the only TrackingStatus they ever have is TrackingStatus.None, it no longer shows up with Tracking or Limited whilst it did previously. This is an issue for me as previously this is how I was getting around Android's limitation that it never removes images from tracking in when they're no longer visible.

    Anyone else had this? Or any thoughts about why this might be happening?
     
    Last edited: Dec 4, 2019 at 12:04 AM
  17. jhocking-bundlar

    jhocking-bundlar

    Joined:
    Nov 12, 2019
    Posts:
    12
    I don't have any thoughts why that's happening to you, I'll think about that since the tracking status is working for me with the library created dynamically.

    I also have a new issue that just started happening after this change to the AR initialization. The camera view has become really choppy. Several odd details:

    This only affects one of the two Android phones I've tested on. It happens on a Pixel 2, but this Galaxy S9 is fine.

    The camera view is choppy but tracking is still smooth, implying that the issue is solely with the camera texture in the scene, not the camera input itself.

    The camera view is actually fine at first, but then turns choppy when I reload the scene. It's almost as if the previous camera feed persists between loads, and thus interferes with a newly created camera background.
     
    Last edited: Dec 5, 2019 at 7:00 PM