Search Unity

  1. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Getting access to the ARKit ARFrame pointer with ARFoundation

Discussion in 'AR' started by renaudbedard, Nov 27, 2018.

  1. renaudbedard

    renaudbedard

    Joined:
    Jul 2, 2012
    Posts:
    20
    As I understand, the official way to do computer vision with ARFoundation is to use the mechanisms described in "Accessing the Camera Image on the CPU", but when using CoreML it's much more convenient to pass the ARFrame captured image to a Vision request. I used to do this with the ARKit plug-in by passing the
    UnityARSessionNativeInterface.GetNativeFramePtr() to my Objective-C plug-in, which worked great.

    With ARFoundation, there is no obvious way to get access to this ARFrame pointer, but it's still defined in the plug-in, so I tried accessing it via reflection like this :

    Code (CSharp):
    1. var arKitApiClass = typeof(UnityEngine.XR.ARKit.ARWorldMap).Assembly.GetType("UnityEngine.XR.ARKit.Api");
    2. var getNativeFramePtrMethod = arKitApiClass.GetMethod("UnityARKit_getNativeFramePtr", BindingFlags.NonPublic | BindingFlags.Static);
    This works, and I can invoke it statically to get access to the pointer, but when I try to use it from Objective-C (by making a bridged-cast back to an
    ARFrame*
    ), it dies with a BAD_ACCESS exception.
    I assume that this is because the ARFrame was created from protected memory and I don't have access to it...

    Is there a way around this? It would be hugely convenient to have access to the ARKit session objects from Objective-C or Swift plug-ins.

    Thanks!
     
  2. tdmowrer

    tdmowrer

    Unity Technologies

    Joined:
    Apr 21, 2017
    Posts:
    604
    You want XRCameraExtensions.GetNativePtr. The reason it didn't work is that this isn't the raw ARFrame, but rather a pointer to a struct of pointers. We'll be releasing a C header soon which will let you interpret them properly, but for now you just need to know that each type of native ptr is a pointer to a struct whose first member is an int and the second is the native pointer you want. For example, the native pointer you get on iOS points to a struct similar to

    Code (CSharp):
    1. struct NativeCameraPtr {
    2.     int version;
    3.     ARFrame* frame;
    4. };
    However, I wouldn't recommend using the ARFrame directly on iOS, since there is nothing retaining this pointer and ARKit has its own frame rate independent of Unity, which means the pointer may become invalidated at any point (there are stronger guarantees on ARCore, so this isn't as much of an issue there). It might be better to get the ARSession pointer, and use the currentFrame from that. Again, ARFoundation gives you a pointer to a struct of version and native pointer pair.
     
    renaudbedard likes this.
  3. renaudbedard

    renaudbedard

    Joined:
    Jul 2, 2012
    Posts:
    20
    Awesome, that worked!
    For now I'm just defining the struct like you posted, I'll update with the official header once it's out.

    Thanks for the help!
     
  4. renaudbedard

    renaudbedard

    Joined:
    Jul 2, 2012
    Posts:
    20
    Hi again,

    I'm trying this method in another project just now, and XRSessionSubsystem.GetNativePtr() is returning null on iOS within a working ARKit session...
    Is there anything that would cause the register hook not to hook itself? I see that the ARKit module uses [RuntimeInitializeOnLoadMethod] to register its GetNativePtr() method, what could cause this method not to run?

    Thanks for any info! It's pretty hard to debug this on-device since XCode is stepping through delegates and method pointers, and it's nearly impossible to follow.

    Edit : I just found out that calling GetNativePtr() ends up in the DefaultGetNativePtr() implementation on device, which returns 0. So that means the register failed or never ran?
     
    Last edited: Jan 18, 2019
  5. renaudbedard

    renaudbedard

    Joined:
    Jul 2, 2012
    Posts:
    20
    Update on my issue, it looks like just touching the UnityEngine.XR.ARKit assembly by doing a typeof() on any type inside it will load the assembly and register the method, which makes GetNativePtr() work as expected. It's surprising that it wasn't loaded otherwise... I did have tracking working with points and planes on iOS.
     
  6. tdmowrer

    tdmowrer

    Unity Technologies

    Joined:
    Apr 21, 2017
    Posts:
    604
    That sounds like an issue with IL2CPP stripping an extension method. That can be addressed by adding a link.xml, but the latest ARKit package should prompt to do this for you. What versions of the packages do you have?
     
  7. mjfang

    mjfang

    Joined:
    Feb 17, 2018
    Posts:
    1
    I've used the method of grabbing the ARSession successfully with ARKit and am trying to setup a similar pipeline with Android. With regards to ARCore, it seems like we can call GetNativePtr for the XRSessionSubsystem as well, does it point to something equivalent to the ARSession? Alternatively is there a proper way to access the current ARCore Frame through ARFoundation?

    Thanks!
     
  8. tdmowrer

    tdmowrer

    Unity Technologies

    Joined:
    Apr 21, 2017
    Posts:
    604
    Yes you can do the same trick on Android. It's a similar struct which stores a pointer to the ArSession.

    The frame is also available. I believe it's the one on the camera subsystem.
     
    mjfang likes this.
  9. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    80
    Hi! Is this broken in the latest ARFoundation ? I can't seem to find XRSessionSubsystem.GetNativePtr() since ARExtensions are deprecated and XRSessionSubsystem.nativePtr returns BAD_EXCEPTION error when I try to use it in Objective C. I made sure to disable stripping for XR.ARKit.

    Any help is appreciated.
     
  10. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    80
    EDIT: Seems that the pointer is always returning nil in objective C. even though the AR session is running.
     
  11. tdmowrer

    tdmowrer

    Unity Technologies

    Joined:
    Apr 21, 2017
    Posts:
    604
  12. URoutine

    URoutine

    Joined:
    Nov 11, 2017
    Posts:
    3
    I'm trying to access the cvPixelBuffer from ar foundation to pass it over for some coreML stuff. Where do i get the UnityEngine.XR.ARKit.ARKitSessionSubsystem object? Is it even the correct class to use?
     
  13. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    171
    To get access to the ARKitSessionSubsystem, downcast the XRSessionSubsystem:
    Code (CSharp):
    1. ARKitSessionSubsystem arkitSessionSubsystem = ((m_ARSession != null) && (m_ARSession.subsystem is ARKitSessionSubsystem)) ? (ARKitSessionSubsystem)m_ARSession.subsystem : null;
    However, if you are trying to access the camera images, just follow the Accessing the Camera Image on the CPU documentaion.
     
  14. jforder

    jforder

    Joined:
    Jun 30, 2012
    Posts:
    22
    Hey, trying to get access to the Front TrueDepth data with ARFoundations, just wondering whats the best approach to getting that? I see the ARFrame has captureDepthData, but not sure which path to take to access that infomation. Any help would be great! Cheers
     
unityunity