Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. The 2023.1 beta is now available for testing. To find out what's new, have a look at our 2023.1 feature highlights.
    Dismiss Notice

WebRTC support for float format textures/point cloud

Discussion in 'Unity Render Streaming' started by Momofil31, Jun 23, 2022.

  1. Momofil31

    Momofil31

    Joined:
    Apr 27, 2022
    Posts:
    4
    As suggested by @kazuki_unity729 I'm posting here our private conversation about this thread's topic.

     
  2. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    568
    Thank you for your question and sharing us!
     
  3. Momofil31

    Momofil31

    Joined:
    Apr 27, 2022
    Posts:
    4
    @kazuki_unity729
    I had an idea for a workaround, which is not usable in production, but as a proof of concept could work. Basically the idea for streaming the ARGBFloat texture is to copy raw byte data using Texture2D.SetRawTextureData into a BGRA32 texture with width multiplied by 4 (i.e. if float texture is 1280x720, bgra would be 5120x720), stream this texture and on the receiver convert it back to float analogously. In this way one float channel is encoded by 4 int channels.

    I tested this conversion part and it works.
    When it comes to stream the texture however there is a loss of information due to the downsampling of YUV420 on the chrominance values. As a consequence information is lost and received texture is meaningless and therefore unusable.

    Is it possible to switch to YUV444 color space instead of 420 in order to avoid loss of information? I think it is necessary to modify the c++ code you linked and re-build the entire unity.webrtc package.
    Would this approach solve the information loss problem? Or the underlying codec might be a problem as well?

    As I said I don't need to really use this app but it is a proof of concept so compatibilty and related stuff is not an issue.

    Thank you very much!
     
  4. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    568
    It looks a good idea for me.
    YUV format support is a significant feature to make flexibility of video streaming.

    I cannot promise when we start the task, but it is valuable to discuss about the topic just now.
     
  5. Momofil31

    Momofil31

    Joined:
    Apr 27, 2022
    Posts:
    4
    I gave a look at native webRTC and a recent contribution you can find here added native support for YUV444 H.264 streams.
    https://bugs.chromium.org/p/chromium/issues/detail?id=1251096

    I did not understand if it is already merged into production but it is a good starting point.

    Unfortunately I don't think I am able to start working on it since this is beyond my skills. Unity WebRTC plugin code is really complex and not documented that much...

    Thank you for you support Kazuki
     
  6. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    568
    We supports H.264 format using Hardware codec like NvCodec(Windows/Linux) and VideoToolbox(macOS/iOS).
    We can do if these codecs supports YUV444 format.