Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct here to familiarize yourself with the rules and how to post constructively.

  2. Unity 2022.1 is now available as the latest Tech release.
    Dismiss Notice
  3. Improve your project's performance with our new guide on profiling in Unity.
    Dismiss Notice

Improving render streaming latency from PC > Quest 2

Discussion in 'Unity Render Streaming' started by CloudyVR, Nov 19, 2021.

  1. CloudyVR

    CloudyVR

    Joined:
    Mar 26, 2017
    Posts:
    630
    I am very new to WebRTC and am finding it very cool. I have a project idea in mind that I need super low latency video stream from one Unity project on PC into another Unity project running on a Quest 2. I've managed to build the examples and I can stream a render texture from the PC game into the Quest and see the video in VR !

    The only issue I am having is the latency is ~ 1 second:

    It appears to be working okay but I really need less than 0.25 second latency for my project idea to work.

    Is the render streaming codec lossless? Is there a lossy option? Could the latency be minimized if the Quest somehow dropped old frames and only rendered the most recent frame?

    If I use PC for both projects then the video appears perfectly synced in real time, and even over a local network in web browsers the video appears instant.

    It's only in the Quest 2 I notice the latency becomes very high and sometimes video playback is a little choppy.

    Is there anything i can do to improve render streaming performance or use faster decoding? I tried reducing the stream resolution to 800x600 but the frame rates in the Quest weren't affected and remained kinda slow.

    I have the Quest connected to gigabit WIFI and the PC directly connected to the router, I don't think the network is causing the latency.

    I know the Quest 2 can stream realtime HD full screen VR over WIFI, I sometime play VR games from my PC over WIFI directly on the Quest and it works phenomenally well.

    But Unity's render streaming is really struggling to keep up.

    Is there a faster decoder or more efficient way to stream video on a Quest/Android?
     
    Last edited: Nov 19, 2021
  2. CloudyVR

    CloudyVR

    Joined:
    Mar 26, 2017
    Posts:
    630
    I just tested by explicitly enabling hardware encoding by setting:

    WebRTC.WebRTC.Initialize(EncoderType.Hardware);
    .

    Then when I tried to connect the Receiver the Broadcast log showed a error:
    [impolite-Unity.RenderStreaming.PeerConnection] Failed to set remote offer sdp: Failed to set remote video description send parameters for m-section with mid='0'.

    Does this mean the hardware codec is not available? Could it be the reason why the Quest video stream looks so delayed? Can I be sure if h264 is being used instead of v8?

    When using the WebBrowserInput demo it works with and without Hardware Encoding enabled, And I do believe I can see a slight difference in frame rate/quality when switching between them. My hunch is that the codecs are changing and h264 is working for WebBrowser demo, but I am unsure how to enable h264 codec for the Broadcast and Receiver demo - it always produces the above error.

    I am using a 2080TI GPU, I believe it supports HW h264.
     
    Last edited: Nov 19, 2021
  3. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    418
    Hi, I am not sure Quest 2 has a hardware encoder.
    Is this document by official announcement?
    https://creator.oculus.com/blog/encoding-high-resolution-360-and-180-video-for-oculus-go/
     
unityunity