Resolved AR Foundation Camera output to Render Texture?

Hi all! I'm using AR foundation on an Android device, and I want to take the output of the camera and put it onto a render texture live. Whenever I try to do this, I just get a black image. What I have is as follows:
On a canvas I have a little Raw Image with a render texture applied to it. Then I'm using this line that was in the documentation in Update(). Nothing works the way it should. Am I doing something wrong?

Code (CSharp):

1. Graphics.Blit(null, myRenderTexture, myARCameraBackground.material);

I want this render texture to update live with the motion of the camera. The plan is to use FMStream (on asset store) to live stream it to a PC for further processing.

I appreciate any and all help!

 

It's possible, but there are some caveats. While Android uses one camera texture, iOS uses two textures. So for iOS, you'll have to Blit two times and output these two textures with a custom shader.

1. Subscribe to ARCameraManager.frameReceived event.
2. For each ARCameraFrameEventArgs.textures do the Graphics.Blit():
   - For Android: Graphics.Blit(texture, targetRenderTexture, arCameraBackground.material). Passing arCameraBackground.material is required for Android.
   - For iOS: Graphics.Blit(texture, targetRenderTexture). No need to pass arCameraBackground.material, this will produce green texture.

The other possible solution is to access the camera image on CPU.

 

Works perfectly, thank you! I'm on Android though, will just that code work for iOS (without passing in the CameraBackground), or is there anything else I need to do?

Thanks so much!

 

Yes, this will work on iOS too, but you'll have to write a custom shader that will display two render textures (Y and CbCr) correctly.

 

Got it, thank you. I will look into that in the future.

 

@KirillKuzyk: Thank you so much for the insight, really helps me points to the right direction with a project, but I wonder though, do you have a link to documentation regarding the 2 textures output on IOS? Not sure how it is outputted, it is by index? (ie [0] and [1] for the two textures needed)

 

If you subscribe to ARCameraManager.frameReceived event, then you can access these textures via ARCameraFrameEventArgs.textures.
To be sure which texture is TextureY and which one is TextureCbCr, I would not recommend accessing them by index, rather than find the index of property id and match it with the ARCameraFrameEventArgs.textures index. I think a code sample is better than a thousand of words
Not optimized for performance, just to show an idea:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.XR.ARFoundation;
3.  
4.  
5. public class ARKitCameraTextures : MonoBehaviour {
6.     [SerializeField] ARCameraManager cameraManager = null;
7.  
8.  
9.     void Awake() {
10.         cameraManager.frameReceived += args => {
11.             var textureYPropId = Shader.PropertyToID("_textureY");
12.             var textureYPropIdIndex = args.propertyNameIds.IndexOf(textureYPropId);
13.             var textureY = args.textures[textureYPropIdIndex];
14.  
15.             var textureCbCrPropId = Shader.PropertyToID("_textureCbCr");
16.             var textureCbCrPropIdIndex = args.propertyNameIds.IndexOf(textureCbCrPropId);
17.             var textureCbCr = args.textures[textureCbCrPropIdIndex];
18.         };
19.     }
20. }
21.  

 

Thank you so much for the code samples, it is very helpful indeed @KirillKuzyk!

 

Hello there, I would like to know if someone know how to get the projected face texture in ARFoundation

 

Hello again @kirill!
Mhmm, I don't think that works anymore. If I run

Code (CSharp):

1.  
2. var textureYPropId = Shader.PropertyToID("TextureY");
3. var textureCbCrPropId = Shader.PropertyToID("TextureCbCr");
4. Debug.Log($"textureYPropId: {textureYPropId}, textureCbCrPropId: {textureCbCrPropId}");
5. // textureYPropId: 1788, textureCbCrPropId: 1789
6. // but
7. foreach (var item in args.propertyNameIds)
8.    Debug.Log($"propertyNameId: {item}");
9. // prints 703, 707
10.  

Could this be related to me using URP and not built-in?

PS: Is there a way of getting which keys the 703 and 707 stand for?

 

Turns out there is no need for "blit" or inspecting the raw textures, all I had to do, to get the AR-Camera Images on a custom object was:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.UI;
3. using UnityEngine.XR.ARFoundation;
4.  
5. [RequireComponent(typeof(RawImage))]
6. [RequireComponent(typeof(ARCameraBackground))]
7. public class ARTextureResponder : MonoBehaviour
8. {
9.     void Start()
10.     {
11.         var rawImage = GetComponent<RawImage>();
12.         var camBack = GetComponent<ARCameraBackground>().material;
13.         rawImage.material = camBack;
14.     }
15. }
16.  

In this case I used a `RawImage` , I guess any other object that has a material would also work.

 

Hmm, this is strange. I can confirm that using texture display names (TextureY, TextureCbCr) doesn't work. But using the actual property names (_textureY, _textureCbCr) does work. I updated the original response, huge thanks for pointing that out!