Making ARFoundation Fit Within a Camera's Rect

For the app I am developing, I am wanting to always render ARFoundation's camera into a widescreen view (16:9, or 9:16 if in portait); the reason being I am using camera-based recording to record the camera's output, and want all output videos to have the same aspect ratio on any device. To do so, the most obvious solution to me was to do some math to adjust the ARCamera's rect into a 16:9 aspect ratio, essentially letterboxing the view on devices that aren't naturally 16:9 (iPad, iPhoneX, etc.).

However, ARFoundation scales the camera feed based on the device's entire screen size, rather than only using the ARCamera's rect, so if the camera's rect does not maintain the same aspect ratio as the device's screen, the view gets distorted. Is there a method to essentially crop the output and make ARFoundation use the size of the camera's viewport on the screen instead of the entire screen?

EDIT: Clarified that I intend to use 16:9 or 9:16, depending on if the device is landscape or portrait.

 

I have two recommendations for you:

1) Trying to crop a 16:9 images from a phone in portrait mode (which would be 9:16 or taller) will be an extremely narrow field of view, be very low resolution, and waste a lot of screen space.

Instead have the user hold the device in landscape mode when recording. You can permanently set the orientation to landscape in settings. Or use scripting to set the orientation to landscape while recording.

2) If you are not using the scriptable render pipeline, you can use OnRenderImage to intercept a camera render and output it to a new RenderTexture of your desired resolution.

Code (CSharp):

1. using UnityEngine;
2. [RequireComponent(typeof(Camera))]
3. public class VideoCrop : MonoBehaviour
4. {
5.     public Material cropMaterial;
6.     public RenderTexture videoOutput;
7.  
8.     void OnRenderImage(RenderTexture src, RenderTexture dest)
9.     {
10.         // Pass through to the screen
11.         Graphics.Blit(src, dest);
12.  
13.         // Crop render to the video texture
14.         Graphics.Blit(src, videoOutput, cropMaterial);
15.     }
16. }

You can also use an unlit material to crop the texture to your desired offset and ratio. This would be the most efficient and straightforward technique if you are not ready to do a deep dive into render pipelines.

You could alternatively use Texture2D.ReadPixels to transfer and crop a Texture2D or RenderTexture to a Texture2D, however this is usually slower than Graphics.Blit
https://docs.unity3d.com/ScriptReference/Texture2D.ReadPixels.html

 

My bad for not being more specific, but when the user records in portrait I am letterboxing to 9:16, not 16:9.

Unfortunately the recording plugin I'm using (NatCorder) would likely not work with this, given it renders the camera into a separate RenderTexture for recording. Buuuuuuut that does give me an idea to leave the camera alone and simply modify the recorder instead to grab a 16:9 (or 9:16) area of the camera's texture instead of the whole thing. I'll have to see if that works.

EDIT: This did work for me with Natcorder. I was able to just change what area of the screen was recorded and essentially fake the letterboxing (by just putting an overlay UI). There's a little bit more overhead for doing so but it should work for now.