Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Request for help: RenderTexture Camera Issues in Unity for Pico4 with WebRTC on Android

Discussion in 'General Graphics' started by JMeer, Dec 1, 2023.

  1. JMeer

    JMeer

    Joined:
    Mar 3, 2017
    Posts:
    29
    Hello Unity Community,

    I'm facing a challenge in our Unity project (initially on 2020.3, now on 2021.3 LTS) involving the Pico4 headset and WebRTC integration. Built-in Render Pipeline. Our aim is to display real-time coaching sessions within our app, but we're struggling to correctly render the camera view on Android OpenGLES3. The WebRTC part is fully functional. Rendertextures seem to not work that great and seem buggy.

    Here's a summary of our attempts and issues:

    1. ScreenCapture.CaptureScreenshotIntoRenderTexture: Works in the editor, but results in a black texture on Android. I've tried a lot of settings.

    2. OnRenderImage with New Render Texture: Functional on Android works, but causes a 55% performance drop and had a culling bug (fixed in 2021.3 LTS). However even creating a rendertexture once without using it, drops the performance in half. volumeDepth and vrUsage settings helped in making the rendertexture atleast visible on android.

    3. CommandBuffer Approach: Leads to a gray texture on Android. (works in editor)
      Code (CSharp):
      1. commandBuffer = new CommandBuffer { name = "Capture Camera" };
      2. commandBuffer.Blit(BuiltinRenderTextureType.CurrentActive, renderTexture);
      3. cam.AddCommandBuffer(CameraEvent.AfterEverything, commandBuffer);
    Our goal is to capture the camera view in a Texture2D or RenderTexture on the Pico4 without compromising performance. We don't need a big resolution. We're using Multiview for efficiency. Any insights or solutions for sharing the camera view with minimal performance impact would be incredibly valuable.

    I've been tackling this for a week and would greatly appreciate your expertise and suggestions. My skills in graphics and command buffer is not that great. We have customers waiting for the feature and I feel a lot of pressure. I really hope you guys can help me out!

    Thanks for your time and help.
     
    Last edited: Dec 1, 2023
  2. JMeer

    JMeer

    Joined:
    Mar 3, 2017
    Posts:
    29
    Quick update and request for assistance: I've managed to get CommandBuffer working for camera rendering in Multi Pass mode, but not in Single Pass Instanced yet (Multi View). I'm using Unity's Default render pipeline.

    Does anyone know how to make a CommandBuffer that blits camera to a RenderTexture compatible with Single Pass Instanced? Tried a custom shader for blitting, but no luck so far.

    Here is my current full code:

    Code (CSharp):
    1. using Unity.WebRTC;
    2. using UnityEngine;
    3. using UnityEngine.Rendering;
    4.  
    5. public class VRCommandBuffer : MonoBehaviour
    6. {
    7.     public Material targetMaterial;
    8.     private CommandBuffer commandBuffer;
    9.     private Camera vrCamera;
    10.     private RenderTexture vrRenderTexture;
    11.  
    12.     void Start()
    13.     {
    14.         Setup();
    15.     }
    16.  
    17.     private void Setup()
    18.     {
    19.         vrCamera = GetComponent<Camera>();
    20.  
    21.         vrRenderTexture = new RenderTexture(vrCamera.pixelWidth, vrCamera.pixelHeight, 0, RenderTextureFormat.ARGB32)
    22.         {
    23.             antiAliasing = vrCamera.allowMSAA ? QualitySettings.antiAliasing : 1,
    24.             format = WebRTC.GetSupportedRenderTextureFormat(SystemInfo.graphicsDeviceType),
    25.         };
    26.  
    27.         commandBuffer = new CommandBuffer();
    28.         commandBuffer.name = "VRCommandBuffer";
    29.  
    30.         commandBuffer.Blit(BuiltinRenderTextureType.CurrentActive, vrRenderTexture);
    31.  
    32.         vrCamera.AddCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);
    33.  
    34.         if (targetMaterial != null)
    35.             targetMaterial.mainTexture = vrRenderTexture;
    36.     }
    37.  
    38.     void OnDestroy()
    39.     {
    40.         vrCamera.RemoveCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);
    41.         CommandBufferPool.Release(commandBuffer);
    42.    
    43.         if (vrRenderTexture != null)
    44.             vrRenderTexture.Release();
    45.     }
    46. }
     
    Last edited: Dec 5, 2023