Help Wanted Access Camera Image and/or Environment Probes

Hi,

I'm porting a legacy ARKit Plugin based project to the latest MARS. What I did in the old system was access the camera background textures and manually build a reflection map of it (cheap but efficient).

How can I access the Camera Image in MARS?

The new XR camera subsystem promises to provide a camera image, but I don't even know how to get this instance ^^. I'd prefer the "external" GPU texture btw.

An alternative solution would be to use ARKit's environment probes instead. How is this done with MARS?

Thanks in advance,

Andi

 

hello @atti ,

Have you checked this? https://forum.unity.com/threads/grab-the-camera-image-texture.922934/ From @amydigiov own words:

If you have a MonoBehaviour that implements IUsesCameraTexture you can call

this.GetCameraTexture()

("this" is necessary because GetCameraTexture is an extension method for IUsesCameraTexture).



At runtime on device this will return the camera texture provided by AR Foundation. In simulation in the editor it will return the texture of the video in "Recorded" mode or the webcam in "Live" mode. In other simulation modes it returns null.

 

Hi!

sorry I missed this thread.

This got me further/was helpful, but the texture I receive has just information in the red-channel (with ARKit on an iPad Pro 11").

I presume this is just the Y channel of a Yuv video signal. Can you confirm and ideally give a way to get the uv-texture as well?

Or an RGB texture right away (if it's performant)....or the reflection probes from ARKit/Core

thanks!

 

Hi @atti, this is a bug with our camera texture provider implementation for AR Foundation. We will make sure to fix this as soon as possible.

In the meantime you can grab the camera image directly from the ARCameraManager, which you can get from the active MARS camera game object:

MarsRuntimeUtils.GetActiveCamera(true).GetComponent<ARCameraManager>()

Check out Unity's AR Foundation Samples for an example of how to grab the CPU image: https://github.com/Unity-Technologi...test-preview/Assets/Scripts/CpuImageSample.cs

 

Hi Amy,

perfect, works like a charm!

I presume this has a bit of an performance overhead, since it's aquireing a CPU image texture, right?

Thanks so much!

 

If you get the ARCameraBackground component from the camera you can use the method suggested by the AR Foundation documentation to make a GPU copy:

Code (CSharp):

1. Graphics.Blit(null, m_MyRenderTexture, m_ARBackgroundCamera.material);
2.  

 

Sweet!

 

Hi,
It was not a plugin, but a commercial project I moved on from ever since (hence I cannot share it).

The principle was simple:
- Create a sphere with normals pointing inwards
- UV unwrap it so that the camera texture looks alright and repeats in a way, so that it fills the whole spere.
- Render a cubemap with a camera placed in the middle of the sphere (did this ~once a second only, since it's expensive)
- Use this cubemap as reflection map

The replies above should help you getting the camera texture.

hth