Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice
  2. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Help Wanted Access Camera Image and/or Environment Probes

Discussion in 'Unity MARS' started by tteneder, Jul 21, 2020.

  1. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    133
    Hi,

    I'm porting a legacy ARKit Plugin based project to the latest MARS. What I did in the old system was access the camera background textures and manually build a reflection map of it (cheap but efficient).

    How can I access the Camera Image in MARS?

    The new XR camera subsystem promises to provide a camera image, but I don't even know how to get this instance ^^. I'd prefer the "external" GPU texture btw.

    An alternative solution would be to use ARKit's environment probes instead. How is this done with MARS?

    Thanks in advance,

    Andi
     
  2. jmunozarUTech

    jmunozarUTech

    Unity Technologies

    Joined:
    Jun 1, 2020
    Posts:
    258
    hello @atti ,

    Have you checked this? https://forum.unity.com/threads/grab-the-camera-image-texture.922934/ From @amydigiov own words:

    If you have a MonoBehaviour that implements IUsesCameraTexture you can call
    this.GetCameraTexture()
    ("this" is necessary because GetCameraTexture is an extension method for IUsesCameraTexture).

    At runtime on device this will return the camera texture provided by AR Foundation. In simulation in the editor it will return the texture of the video in "Recorded" mode or the webcam in "Live" mode. In other simulation modes it returns null.
     
    tteneder likes this.
  3. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    133
    Hi!

    sorry I missed this thread.

    This got me further/was helpful, but the texture I receive has just information in the red-channel (with ARKit on an iPad Pro 11").

    I presume this is just the Y channel of a Yuv video signal. Can you confirm and ideally give a way to get the uv-texture as well?

    Or an RGB texture right away (if it's performant)....or the reflection probes from ARKit/Core :)

    thanks!
     
  4. amydigiov

    amydigiov

    Unity Technologies

    Joined:
    May 24, 2016
    Posts:
    13
    Hi @atti, this is a bug with our camera texture provider implementation for AR Foundation. We will make sure to fix this as soon as possible.

    In the meantime you can grab the camera image directly from the ARCameraManager, which you can get from the active MARS camera game object:
    MarsRuntimeUtils.GetActiveCamera(true).GetComponent<ARCameraManager>()


    Check out Unity's AR Foundation Samples for an example of how to grab the CPU image: https://github.com/Unity-Technologi...test-preview/Assets/Scripts/CpuImageSample.cs
     
    tteneder and jmunozarUTech like this.
  5. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    133
    Hi Amy,

    perfect, works like a charm!

    I presume this has a bit of an performance overhead, since it's aquireing a CPU image texture, right?

    Thanks so much!
     
  6. CiaranWills

    CiaranWills

    Unity Technologies

    Joined:
    Apr 24, 2020
    Posts:
    73
    If you get the ARCameraBackground component from the camera you can use the method suggested by the AR Foundation documentation to make a GPU copy:
    Code (CSharp):
    1. Graphics.Blit(null, m_MyRenderTexture, m_ARBackgroundCamera.material);
    2.  
     
    tteneder likes this.
  7. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    133
    Sweet!
     
  8. sizheng_2017

    sizheng_2017

    Joined:
    Jan 18, 2018
    Posts:
    24
    hi tteneder, how's your plugin? or where I can find it? I really needs this function to make realtime reflection in ar.
     
  9. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    133
    Hi,
    It was not a plugin, but a commercial project I moved on from ever since (hence I cannot share it).

    The principle was simple:
    - Create a sphere with normals pointing inwards
    - UV unwrap it so that the camera texture looks alright and repeats in a way, so that it fills the whole spere.
    - Render a cubemap with a camera placed in the middle of the sphere (did this ~once a second only, since it's expensive)
    - Use this cubemap as reflection map

    The replies above should help you getting the camera texture.

    hth
     
unityunity