WebRTC support for float format textures/point cloud

As suggested by @kazuki_unity729 I'm posting here our private conversation about this thread's topic.

 

@kazuki_unity729
I had an idea for a workaround, which is not usable in production, but as a proof of concept could work. Basically the idea for streaming the ARGBFloat texture is to copy raw byte data using Texture2D.SetRawTextureData into a BGRA32 texture with width multiplied by 4 (i.e. if float texture is 1280x720, bgra would be 5120x720), stream this texture and on the receiver convert it back to float analogously. In this way one float channel is encoded by 4 int channels.

I tested this conversion part and it works.
When it comes to stream the texture however there is a loss of information due to the downsampling of YUV420 on the chrominance values. As a consequence information is lost and received texture is meaningless and therefore unusable.

Is it possible to switch to YUV444 color space instead of 420 in order to avoid loss of information? I think it is necessary to modify the c++ code you linked and re-build the entire unity.webrtc package.
Would this approach solve the information loss problem? Or the underlying codec might be a problem as well?

As I said I don't need to really use this app but it is a proof of concept so compatibilty and related stuff is not an issue.

Thank you very much!

 

I gave a look at native webRTC and a recent contribution you can find here added native support for YUV444 H.264 streams.
https://bugs.chromium.org/p/chromium/issues/detail?id=1251096

I did not understand if it is already merged into production but it is a good starting point.

Unfortunately I don't think I am able to start working on it since this is beyond my skills. Unity WebRTC plugin code is really complex and not documented that much...

Thank you for you support Kazuki