Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Question Access Camera Image and/or Environment Probes

Discussion in 'Unity MARS' started by tteneder, Jul 21, 2020.

  1. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    162
    Hi,

    I'm porting a legacy ARKit Plugin based project to the latest MARS. What I did in the old system was access the camera background textures and manually build a reflection map of it (cheap but efficient).

    How can I access the Camera Image in MARS?

    The new XR camera subsystem promises to provide a camera image, but I don't even know how to get this instance ^^. I'd prefer the "external" GPU texture btw.

    An alternative solution would be to use ARKit's environment probes instead. How is this done with MARS?

    Thanks in advance,

    Andi
     
  2. jmunozarUTech

    jmunozarUTech

    Unity Technologies

    Joined:
    Jun 1, 2020
    Posts:
    296
    hello @atti ,

    Have you checked this? https://forum.unity.com/threads/grab-the-camera-image-texture.922934/ From @amydigiov own words:

    If you have a MonoBehaviour that implements IUsesCameraTexture you can call
    this.GetCameraTexture()
    ("this" is necessary because GetCameraTexture is an extension method for IUsesCameraTexture).

    At runtime on device this will return the camera texture provided by AR Foundation. In simulation in the editor it will return the texture of the video in "Recorded" mode or the webcam in "Live" mode. In other simulation modes it returns null.
     
    tteneder likes this.
  3. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    162
    Hi!

    sorry I missed this thread.

    This got me further/was helpful, but the texture I receive has just information in the red-channel (with ARKit on an iPad Pro 11").

    I presume this is just the Y channel of a Yuv video signal. Can you confirm and ideally give a way to get the uv-texture as well?

    Or an RGB texture right away (if it's performant)....or the reflection probes from ARKit/Core :)

    thanks!
     
  4. amydigiov_Unity

    amydigiov_Unity

    Unity Technologies

    Joined:
    May 24, 2016
    Posts:
    16
    Hi @atti, this is a bug with our camera texture provider implementation for AR Foundation. We will make sure to fix this as soon as possible.

    In the meantime you can grab the camera image directly from the ARCameraManager, which you can get from the active MARS camera game object:
    MarsRuntimeUtils.GetActiveCamera(true).GetComponent<ARCameraManager>()


    Check out Unity's AR Foundation Samples for an example of how to grab the CPU image: https://github.com/Unity-Technologi...test-preview/Assets/Scripts/CpuImageSample.cs
     
    tteneder and jmunozarUTech like this.
  5. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    162
    Hi Amy,

    perfect, works like a charm!

    I presume this has a bit of an performance overhead, since it's aquireing a CPU image texture, right?

    Thanks so much!
     
  6. CiaranWills

    CiaranWills

    Unity Technologies

    Joined:
    Apr 24, 2020
    Posts:
    195
    If you get the ARCameraBackground component from the camera you can use the method suggested by the AR Foundation documentation to make a GPU copy:
    Code (CSharp):
    1. Graphics.Blit(null, m_MyRenderTexture, m_ARBackgroundCamera.material);
    2.  
     
    tteneder likes this.
  7. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    162
    Sweet!
     
  8. shibi2017

    shibi2017

    Joined:
    Jan 18, 2018
    Posts:
    153
    hi tteneder, how's your plugin? or where I can find it? I really needs this function to make realtime reflection in ar.
     
  9. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    162
    Hi,
    It was not a plugin, but a commercial project I moved on from ever since (hence I cannot share it).

    The principle was simple:
    - Create a sphere with normals pointing inwards
    - UV unwrap it so that the camera texture looks alright and repeats in a way, so that it fills the whole spere.
    - Render a cubemap with a camera placed in the middle of the sphere (did this ~once a second only, since it's expensive)
    - Use this cubemap as reflection map

    The replies above should help you getting the camera texture.

    hth