Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Resolved How to access ARFrame image in Unity ARKit

Discussion in 'AR' started by natwales, Sep 20, 2017.

  1. natwales

    natwales

    Joined:
    Dec 5, 2012
    Posts:
    10
    I'm trying to integrate OpenCV with ARKit and I can't figure out how to access the captured image from ARKit.

    I was hoping that in the ARFrameUpdatedEvent delegate I'd be able to access ARFrame and presumably the captured image - which I could then pipe into OpenCV, but this only returns a UnityARCamera object that doesn't seem to expose ARFrame or the captured frame image in any way.

    Can anyone point me to a way I might accomplish this? Is it possible?
     
  2. jimmya

    jimmya

    Joined:
    Nov 15, 2016
    Posts:
    793
    In UnityARCamera, look at videoParams.cvPixelBufferPtr - might be what you need (its void* pointer to the capturedImage pixelbuffer).
     
    CatalinMustata and natwales like this.
  3. quitebuttery

    quitebuttery

    Joined:
    Mar 12, 2011
    Posts:
    327
    @jimmya Do you have an example of this working? I'm trying to use Texture2D.CreateExternalTexture with cvPixelBufferPtr and the resultant texture is always white. RGBA: .804, .804, .804 .804. Is there perhaps a timing issue with accessing this pointer? Are the contents only valid at certain times or something? Or maybe I'm using the wrong format to create this texture? ( TextureFormat.ARGB32)
     
    Last edited: Oct 2, 2017
    natwales likes this.
  4. quitebuttery

    quitebuttery

    Joined:
    Mar 12, 2011
    Posts:
    327
    In fact....from checking out the YUVMaterial shader, I don't think there is a valid camera frame texture anywhere? It seems like the fragment shader combines the Y and CbCr textures and spits them out to the screen. But this video frame isn't actually stored anywhere...?
     
    natwales likes this.
  5. quitebuttery

    quitebuttery

    Joined:
    Mar 12, 2011
    Posts:
    327
    Ok--so the answer is...make a new camera, put the UnityARVideo behavior on it, have the culling flags set to it doesn't render anything, have this camera output to a RenderTexture. That RenderTexture will include only the video frame! Works great!
     
  6. byrd61bar

    byrd61bar

    Joined:
    Apr 11, 2016
    Posts:
    7
    Hi,

    Can you provide sample code or the steps on how to set the culling flags not to render anything and how to create a RenderTexture on the camera? Thanks
     
  7. CatalinMustata

    CatalinMustata

    Joined:
    Oct 12, 2017
    Posts:
    6
    I think the problem you're having is related to the fact that cvPixelBufferPtr is a pointer to a native CVPixelBuffer object, which wraps a pixel buffer and not a native texture. Therefore, you're getting garbage.

    If you want to extract the actual buffer, you would need to write some native iOS code I think. Have a look at CVPixelBufferLockBaseAddress and then CVPixelBufferGetBaseAddress (possibly per plane if it's YUV420).
     
  8. DHein

    DHein

    Joined:
    Jan 26, 2016
    Posts:
    38
    Any new insights concerning this issue? There are actually a few developers that would like to use the camptured ARKit frame in OpenCV. By using the Rendertexture approach it is actually hard to get a decent frame rate due to the necessary conversion from Rendertexture to Texture2D (which is needed in order to be converted to a Mat object).
     
    CarinaMclane likes this.
  9. jimmya

    jimmya

    Joined:
    Nov 15, 2016
    Posts:
    793
    You should be able to use a Rendertexture directly in a material. Notice a material uses a Texture, and a RenderTexture is a Texture.
     
  10. DHein

    DHein

    Joined:
    Jan 26, 2016
    Posts:
    38
    Oh I think we have a misunderstanding here, I am talking about the Mat class of Open CV which is basically a matrix that contains image data and not the texture for a Unity Material.
     
  11. CarinaMclane

    CarinaMclane

    Joined:
    Mar 12, 2014
    Posts:
    23
    You can extract pixel values from the RenderTexture and pass it to the OpenCV mat, but it is an expensive operation. Would be great if there was a cheap (as cheap as possible) way of doing this as AR has loads of places where you want to do custom computer vision operations, on top of those provided. The suggestion I have seen is to extract the pixel values natively, but it feels like rather than each of us doing this separately it should be part of some core functionality (pretty please! :) )
     
    Blarp and DHein like this.
  12. jimmya

    jimmya

    Joined:
    Nov 15, 2016
    Posts:
    793
    If you look in code how the arkit remote sends video frames across to editor, there is a way to do it there.
     
  13. DHein

    DHein

    Joined:
    Jan 26, 2016
    Posts:
    38
    Thanks for the hint, I've tried to use the ARKit Remote connection code, but it nonetheless relies on the Texture2D and the Apply function and therefore results in immense FPS drops. I guess the only viable option that remains is writing native code or waiting for an update where this feature is implemented.
     
    Blarp likes this.
  14. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    81
    If I am fine with R8 VideoTextureY from UnityARVideo, how to convert it (it's a CreateExternalTexture) into Unity Texture2D so I can access pixels? I tried accessing with IntPtr but I am only getting garbage. :(

    I am trying to avoid RenderTexture solution mentioned above.

    +1 to have it exposed. In ArCore they use System.Runtime.InteropServices.Marshal.Copy and it works great!
     
  15. jimmya

    jimmya

    Joined:
    Nov 15, 2016
    Posts:
    793
    DHein and BinaryBanana like this.
  16. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    81
    Thanks! It works great! You are awesome! :)
     
  17. pavank1

    pavank1

    Joined:
    Jun 26, 2017
    Posts:
    5
    Apologies if this is a noob question - How do extract the image from the separate Y and UV texture byte arrays? Ideally I would like a png encoded image bytes. Any pointers are appreciated on this.

    The current workflow seems so - > Y, UV arrays passed to a shader
    Shader converts pixel by pixel

    const float4x4 ycbcrToRGBTransform = float4x4(
    float4(1.0, +0.0000, +1.4020, -0.7010),
    float4(1.0, -0.3441, -0.7141, +0.5291),
    float4(1.0, +1.7720, +0.0000, -0.8860),
    float4(0.0, +0.0000, +0.0000, +1.0000)
    );

    float2 texcoord = i.texcoord;
    float y = tex2D(_textureY, texcoord).r;
    float4 ycbcr = float4(y, tex2D(_textureCbCr, texcoord).rg, 1.0);

    rgba = multiply(ycbcrToRGBTransform,ycbcr);

    Any better method to abstract this and write it more clean and easy?
     
    Last edited: Mar 6, 2018
  18. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    81
    The only other way I know is to use render texture from second camera and read from there. I don't think there is a way today to get RGB image from AR Core or AR Kit.
     
  19. pavank1

    pavank1

    Joined:
    Jun 26, 2017
    Posts:
    5
    If I use the render texture, I lose the video background when using ARKitRemote :( Also there seems to be a delay in the render if I apply it to quad and re render it again on the plane.

    Is there any way to capture the image on click of button while the video keeps rendering on the screen? Any pointers on this?

    I am not exactly sure how to add the second camera. I tried that and I get errors that I have two listeners etc.,
     
  20. rpasquini

    rpasquini

    Joined:
    Feb 1, 2018
    Posts:
    5
    Did you ever figure out a good way to get RGB from this?
     
  21. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    81
    I followed this comment from above:

    with the culling flag, it doesn't render anything, just capturing background. which is exactly what you need. The issue with two listeners comes most likely form audio listener component, just remove it on the second camera.
     
  22. rpasquini

    rpasquini

    Joined:
    Feb 1, 2018
    Posts:
    5
    To get RGB bytes, I arrived at a solution that took the example in UnityRemoteVideo.cs. Take the bytes from SetCapturePixelData() and set those on my own version of the YUVMaterial, which I used Graphics.Blit() to render onto a RenderTexture. Then use that RenderTexture to get a Texture2D to then access the RGB bytes.
     
  23. pavank1

    pavank1

    Joined:
    Jun 26, 2017
    Posts:
    5
    Do you have an example ?

    Are you using the remote connection and the built version separately using this method ?

    I did look at the UnityRemoteVideo and it sends the YByteArray to a player connection for rendering in the editor.

    How can I access this while using ARKit remote and eventually also after I package the app and not using the editor?
     
  24. pavank1

    pavank1

    Joined:
    Jun 26, 2017
    Posts:
    5
    I just get a green screen when I follow this -

    I created a camera, made the culling mask to nothing, created a render texture and set the camera output to this texture. I can see the video on unity editor but the texture is green in color (the same color which the editor shows before the phone connects)
     
  25. MikkoHaapoja

    MikkoHaapoja

    Joined:
    Dec 2, 2016
    Posts:
    6
    If you look at the current source code the `ARKitRemoteConnection` will call `arVideo.SetYTexure` and `arVideo.SetUVTexure` on the Main Camera. This means if you're using a second camera which renders to a render texture you'll never see it update which means it'll stay "green". If you run it on device however since it'll use the shared ptrs then you'll see it work.
     
  26. Dadi897

    Dadi897

    Joined:
    Mar 5, 2018
    Posts:
    8
    After I managed to get the frame from ARKit into an OpenCV Mat and do some operations, like making the image in grayscale, can I then output this image to the screen?

    EDIT: I just put a RawImage covering the entire screen and then put the texture on that. But would it be possible then to combine the OpenCV image and the ARKit image with all the planes etc?
     
    Last edited: Apr 13, 2018
  27. Vander-Does

    Vander-Does

    Joined:
    Dec 22, 2015
    Posts:
    19
  28. Pilltech101

    Pilltech101

    Joined:
    Jul 25, 2014
    Posts:
    35
    Would you please be kind and share on how you
    got this working.
    You would be saving my life.
     
    Last edited: Nov 17, 2018
  29. jimmya

    jimmya

    Joined:
    Nov 15, 2016
    Posts:
    793
    This is how the YUV texture format has been defined from its origins - the Cb and Cr channels are at half resolution to save bandwidth. The idea is that in general color does not have to have as much resolution as intensity.
     
    Vander-Does likes this.
  30. smtabatabaie

    smtabatabaie

    Joined:
    May 6, 2013
    Posts:
    45
    Hi, Are there any updates or sample codes to this? (except from using rendertexture)
    I'm also trying to get ARKit camera feed to do some other processing on it but haven't been able to do it.
     
    Last edited: Feb 26, 2019
    Blarp likes this.
  31. RaymingW

    RaymingW

    Joined:
    Nov 11, 2017
    Posts:
    36
    Hi,

    I tried to create a similar script that attaches to the camera. The goal is to load the image on to a material's texture.
    I basically duplicated your script and changed the OnPreRender() function to:


    Code (CSharp):
    1. private void OnPreRender()
    2.         {
    3.             if (!bTexturesInitialized)
    4.                 return;
    5.            
    6.             currentFrameIndex = (currentFrameIndex + 1) % 2;
    7.             m_Session.SetCapturePixelData (true, PinByteArray(ref m_pinnedYArray,YByteArrayForFrame(currentFrameIndex)), PinByteArray(ref m_pinnedUVArray,UVByteArrayForFrame(currentFrameIndex)));
    8.             Texture2D tex = new Texture2D(1, 1);
    9.             tex.LoadImage(UVByteArrayForFrame(1-currentFrameIndex));
    10.             m_PreviewMaterial.mainTexture = tex;
    11.         }
    However, The texture seems corrupted because it shows a red marker with a white background.

    Also. what is the different between m_textureYBytes and m_textureUVBytes?

    Many thanks!
     
  32. smtabatabaie

    smtabatabaie

    Joined:
    May 6, 2013
    Posts:
    45
    I'm really struggling to solve this, Are there any ways to get the texture of camera (Beside rendertexture)
    @jimmya a sample code will be really appreciated.
     
  33. Dev_Sebas

    Dev_Sebas

    Joined:
    Jul 10, 2014
    Posts:
    19
    Hey,
    I'm currently using the render texture way, its not perfect but it works :)
    Suppose with ARFundation all this stuff will be more direct and easy accessed, no?
    Thanks
    Seb
     
    Blarp likes this.
  34. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    81
    If only AR Foundation team would reply to any of theirs threads on Unity forum, live would be easier :)

    From my experience with AR Foundation, it is easier to get a frame but getting a frame, camera matrices and feature points at the same time is almost impossible. AR Foundation doesn't expose useful info when camera was captured and when feature points are captured. You get for one frame count and timestamp (in not know format) for the other. Matching them is useless.

    I hope they will fix is with the next release.

    --
    Smart AR Home -
    Manage your smart home devices with augmented reality!
    http://smartarhome.com
     
    Blarp and Dev_Sebas like this.
  35. Dev_Sebas

    Dev_Sebas

    Joined:
    Jul 10, 2014
    Posts:
    19
    Thanks for the information provided! :)
    Thanks
    Seb
     
  36. smtabatabaie

    smtabatabaie

    Joined:
    May 6, 2013
    Posts:
    45
    Is it possible to help us a little bit with it kindly with a sample code?
    I did it with RenderTexture but it is not very optimized.
     
  37. adrian-taylor09

    adrian-taylor09

    Joined:
    Dec 22, 2016
    Posts:
    63
    Hi All,

    I'm trying to use the render texture method described above and I'm curious...

    What is the highest resolution image I can capture from the camera during an AR session? I am trying to capture single frames which will be used in an offline image recognition algorithm (not done on the device in real time) where high resolution images are desirable.

    Also, how do I figure this out at runtime?
     
  38. arcade_jon

    arcade_jon

    Joined:
    Feb 9, 2018
    Posts:
    11
    You can get the AR camera feed using the handles that return pointers to the camera texture:
    ARTextureHandles handles = UnityARSessionNativeInterface.GetARSessionNativeInterface().GetARVideoTextureHandles();
    Then, create a Texture2D out of them:
    videoTextureY = Texture2D.CreateExternalTexture(currentResolution.width, currentResolution.height, TextureFormat.R8, false, false, (System.IntPtr)handles.TextureY);

    This is the most optimised way I've found of converting the camera feed into a Texture2D. however, I'm stuck converting the Texture2D into OpenCV Mat for processing. I've tried using Utils.copyToMat (using the IntPtr directly) as well as Utils.fastTexture2DToMat, but have only managed to get a blank output (sometimes with distorted, white noise). Does anyone know how to successfully convert a Texture2D as created with the code above to a Mat? Thanks
     
    Blarp likes this.
  39. Blarp

    Blarp

    Joined:
    May 13, 2014
    Posts:
    269
  40. smtabatabaie

    smtabatabaie

    Joined:
    May 6, 2013
    Posts:
    45
    Really There are no ways to just get a texture2d from ARKit feed???
     
  41. smtabatabaie

    smtabatabaie

    Joined:
    May 6, 2013
    Posts:
    45
    I can't find GetARVideoTextureHandles() in my code and it gives me error, what version of ARKit you use?
     
  42. x2stone

    x2stone

    Joined:
    Dec 17, 2014
    Posts:
    22
  43. mtellezleon46

    mtellezleon46

    Joined:
    Dec 9, 2020
    Posts:
    4
    Hi!! I finally get to work with the data from ARFoundation with OpenCV.
    Well.... I´ve created an script where I get the distance of a raycasted point cloud to the camera. The main problem is that I couldn´t be able to use the managers like RayCastManager and Point Cloud using the information from the ARCamera. If I just enable those managers, the rendered texture for OpenCV just don´t show and if I unenable the managers, everything works fine. Could you please give a tip or what Am I doing wrong? Thank you. I´ve to say that I´m using the ARCamera to bring infomation for an algorithm of OpenCV.
     
  44. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    AR Foundation is a wrapper for native AR solutions (ARKit, ARCore, etc.), it doesn't do any image processing itself. That's why you can't feed a video to AR Foundation and expect it to track planes, point clouds, etc.