Official Unity Render Streaming Introduction & FAQ

Thanks. I read the links many times before but I still have many questions. For instance, let's say that my users have average network connections (decent bandwidth but a few packet loss from time to time) and I want to force the encoder (in the Unity side) to have more frequent keyframes to compensate for this packet drops. Is there any way to do this today?

 

I want to connect Mixed Reality WebRtc Library with normal webrtc library. Can anyone with experience on this subject help?

Note: I could not run the Mixed Reality WebRtc library on IOS, I would appreciate it if anyone could help me about this issue.

 

Hey,
been following this thread for quite some time,
currently im trying to implement webRTC client for windows 10 (64),
I have python httpsignalling server, and im trying to connect it to magic leap webrtc example,

webRTC sdk:
3.0.0-pre.6

everything works well, only issue is, when I receiver offer on windows client from ML, it gives below error:
Failed to set remote video description send parameters for m-section with mid='0'.

incoming offer sdp looks like:
v=0
o=- 3224918172084906319 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS localCamOnly
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Cmle
a=ice-pwd:1C7j4WYrQHqwITbsZkD6c5g5
a=ice-options:trickle
a=fingerprint:sha-256 5B:35:FA:5D:80E:A9:C8:CAA:13:AF:4E:7FB:9F:2B:50:BF:80:4A:5C:62:16E:90:71:0A:B1:4A:9F:C3
a=setup:actpass
a=mid:0
a=extmap:14 urn:ietfarams:rtp-hdrext:toffset
a=extmap:13 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:12 urn:3gpp:video-orientation
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:3 urn:ietfarams:rtp-hdrext:sdes:mid
a=extmap:4 urn:ietfarams:rtp-hdrext:sdes:rtp-stream-id
a=extmap:5 urn:ietfarams:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:localCamOnly localCamOnly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 red/90000
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 ulpfec/90000
a=ssrc-group:FID 2122193244 487483518
a=ssrc:2122193244 cnameRQlTGcgeLB3/4mp
a=ssrc:2122193244 msid:localCamOnly localCamOnly
a=ssrc:2122193244 mslabel:localCamOnly
a=ssrc:2122193244 label:localCamOnly
a=ssrc:487483518 cnameRQlTGcgeLB3/4mp
a=ssrc:487483518 msid:localCamOnly localCamOnly
a=ssrc:487483518 mslabel:localCamOnly
a=ssrc:487483518 label:localCamOnly
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Cmle
a=ice-pwd:1C7j4WYrQHqwITbsZkD6c5g5
a=ice-options:trickle
a=fingerprint:sha-256 5B:35:FA:5D:80E:A9:C8:CAA:13:AF:4E:7FB:9F:2B:50:BF:80:4A:5C:62:16E:90:71:0A:B1:4A:9F:C3
a=setup:actpass
a=mid:1
a=extmap:1 urn:ietfarams:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:3 urn:ietfarams:rtp-hdrext:sdes:mid
a=extmap:4 urn:ietfarams:rtp-hdrext:sdes:rtp-stream-id
a=extmap:5 urn:ietfarams:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:localCamOnly audio_track
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:450921227 cnameRQlTGcgeLB3/4mp
a=ssrc:450921227 msid:localCamOnly audio_track
a=ssrc:450921227 mslabel:localCamOnly
a=ssrc:450921227 label:audio_track
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:Cmle
a=ice-pwd:1C7j4WYrQHqwITbsZkD6c5g5
a=ice-options:trickle
a=fingerprint:sha-256 5B:35:FA:5D:80E:A9:C8:CAA:13:AF:4E:7FB:9F:2B:50:BF:80:4A:5C:62:16E:90:71:0A:B1:4A:9F:C3
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

I have read few related posts,
most of them identify this issue as H264 decoding issue on windows platform,

my question is how can I bypass this issue?
- some tell to disable hardware encoding, but I think this option is deprecated asd I cant see it any where in sdk?
- there seems to support for nvdia drivers? does this support is automatic or do I need to add third party plugin for it?
- cant I override preferred codec somehow to use any codec other than h264? like v8 or v9?

what are my options? please guide me!!

 

Update:
Fixed the issue by updating nvdia driver

now facing yet another 2 issues:
1) Remote video incoming on windows app is not rendering (can only see first frame then its stuck), although I have added coroutine for webrtc.update()
2) whenever I try to disconnect, my windows app hangs on pc.close() (this happends only when im connected to ML unity webrtc exmaple, if im connected to another windows app instance it does not have any issue)

By guess is im somehow getting issue of decoder even after updating nvdia driver as my remote stream is stuck and cant closer peer connection.

P.S : im testing on windows 11, can this be issue? as I have read somewhere it has support for windows 10 only??

 

hey,
thanks for getting back,
ML already has an example on unity
https://ml1-developer.magicleap.com/en-us/learn/guides/unity-webrtc-guide

their webrtc example includes a browser client as well, so both can easily connect with each other,

what I'm trying to do is replicate browser client code to make windows app connect with ML unity example,
my windows app connect with browser client as well,
previously I was not able to establish the connection at all (windows app + ML unity example),
after upgrading nvdia driver on windows, I could establish connection, audio is fine, I can send messages via channel, ML example can see my video from windows app clearly as well,
After studying sdp payload coming from ML and coming from browser, I saw the ML payload only have H264, ulpfec, rtx, red support while browser payload include v8, v9, h264 etc,
so ML is forcing my windows unity webrtc app to use h264 decoders, which causing 2 issues I mentioned above,
I think there some issue with compatibility with decoder itself

 

@kazuki_unity729 thanks for getting back,
actually I have read the requirements, my testing PC has Nvdia GeForce RTX 3050 with latest driver version 536.40 on windows 10,
also surprizingly enough, if I switch platform to mac and make mac build from my mac book, it works 100%

so at this point, im really not sure whats happening, even tried another system with nvida graphics card on windows 10, had same issue as described in above messages.
currently seem lost to make my current implementation to work on windows platform

 

Yes. we tried this fix (we rebuild plugin with this PR) same error. we tried other graphics devices as well.same freeze of first frame.

i attached dump from webrtc stats panel

 

hey @kazuki_unity729 thanks for the suggestion,
already tried with DX11 and DX12 (both), havent tried vulkan though


Got Nvdia GeForce RTX 3050 with latest driver version 536.40

 

Hello @kazuki_unity729 , I am working with version 3.1.0-exp.3, and I am considering the option to upgrade to the latest, or at least to the next exp.4, but before, I wanted to ask for your suggestion. The issue I want to fix with this upgrade is that in my current version, the quality of the streaming at some points is not so good when the camera is in movement (some weird noise or artifacts are added to the stream), however, when it stops moving it usually tends to improve. For this, I have seen that in the changelog: (https://docs.unity3d.com/Packages/c...hangelog/CHANGELOG.html#310-exp4---2022-10-06) that for version 3.1.0-exp.4, it already has a bitrate option.

Do you think that the changes that has been made since version 3.1.0-exp.3 could help improve the quality of the Web RTC stream?

At this point, I am only streaming the video, because it is what matters the most to my application. I want to be able to stream 4k with the best possible quality. So, in a few words: Do you believe that upgrading will improve the quality of the stream?

Or maybe another way to achieve a higher quality on the stream? Even at the cost of a higher latency? Can this be somehow achieved? Basically, somehow influencing the latency directly?

 

sorry I don't understand question, the result is always same remote stream on windows is stuck when connected to ML device and when try to disconnect peer connection windows app gets stuck.

 

Hello guys
hello master kazuki_unity729

Could you clarify one moment with Unity Render Streaming

Is it possible to stream video feed from VR build ( Meta Quest 2 ) to WebGL build with help of URS?
And if possible then what costs would be: picture quality drop in headset? fps drop?

Or may be it's overkill to use Unity Render Streaming for such task and you could suggest better way to stream from oculus to web page?

Thanks

 

I cannot find the option for encoding h264 in my VideoStreamSender inspector. My computer is a win10, NVIDIA 3080 graphics card, and I am sure to install the graphics card driver correctly. How to handle it?

I want to know how to install the H264 encoder so that it can be used. I saw some clues in the code, but due to being encapsulated in C++code, I cannot continue.

Code (CSharp):

1. WebRTC.Context.GetSenderCapabilities(kind, out IntPtr ptr);

 

I posted this issue, which several others have also encountered, to the Renderstreaming forum topic, but have not received any response yet. Using the latest exp.7 point release, this TURN SERVER issue is apparently not yet fixed.

See the post "
Unable to connect over TURN server from WebBrowserInput sample on WAN. Bug?"
https://forum.unity.com/forums/unity-render-streaming.413/

 

Hey @kazuki_unity729
My team is building spatial audio chat feature to run on Unity WebGL. Currently, it seems impossible because of AudioClip's limitations on WebGL. My question is, Would it be possible to send audio streams from n users to n users via RenderStreaming?
Thanks!

 

1. Can I use this package to get the camera stream from a Unity app running in an android and send it to another Unity app running on another android ?
2. Can I use this package without the input ?
3. My goal is actually to create a screen sharing app. Would that work ? Shoud I use another better unity package for my use case ?

 

Hello , soory for my bad English.
Multiplayer is work correctly on local network with WebApp but when I build for Linux Server and connect with guest on chrome screen is black and nothing show. what can mak problem?
please help...

 

Hi, when I use this package and try to provide any kind of command line argument to the unity application (it doesn't matter if it's command line argument for the render streaming, or unity standalone command line args), the streaming is not working. I'm pretty sure that my argument is valid because it's changing the desired value. In the player logs I see this message:

ArgumentException: Could not instantiate RTCPeerConnection
at Unity.WebRTC.RTCPeerConnection..ctor (Unity.WebRTC.RTCConfiguration& configuration) [0x00066] in <052e0185aa62428b89b00bfd35ff5e1a>:0
at Unity.RenderStreaming.PeerConnection..ctor (System.Boolean polite, Unity.WebRTC.RTCConfiguration config, System.Single resendInterval, System.Func`2[T,TResult] startCoroutine, System.Action`1[T] stopCoroutine) [0x0002f] in <a90540334eaa47d8ad1b8b7ff105b539>:0
at Unity.RenderStreaming.SignalingManagerInternal.CreatePeerConnection (System.String connectionId, System.Boolean polite) [0x0002f] in <a90540334eaa47d8ad1b8b7ff105b539>:0
at Unity.RenderStreaming.SignalingManagerInternal.OnOffer (Unity.RenderStreaming.Signaling.ISignaling signaling, Unity.RenderStreaming.DescData e) [0x0003a] in <a90540334eaa47d8ad1b8b7ff105b539>:0
at Unity.RenderStreaming.Signaling.WebSocketSignaling+<>c__DisplayClass39_1.<WSProcessMessage>b__2 (System.Object d) [0x00014] in <a90540334eaa47d8ad1b8b7ff105b539>:0
at UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () [0x00002] in <420f675e2d6b4f17877563ada0d329b3>:0
at UnityEngine.UnitySynchronizationContext.Exec () [0x0005d] in <420f675e2d6b4f17877563ada0d329b3>:0
at UnityEngine.UnitySynchronizationContext.ExecuteTasks () [0x00014] in <420f675e2d6b4f17877563ada0d329b3>:0

 

Thank you for the answers.

Because it is for VR headset Lynx-R1 =)

 

Hello
I am trying to stream an app render over to the client using WebRTC, a TURN server, and a rendering Google Cloud server with T4 GPU.
but no matter what I do or what codec I use, I always have issues on the client side. the two main issues are:
1- Controller delay
2- Render being pixlated by movement, the faster the movement is the more pixelation happens

I have tried using different turn servers, I have streamed a 4k colorful video with the same setup on the same server with no issue, I have tried a windows build of the project and no quality loss happens there either.
can anyone help me understand what the issue might be?

 

Hello!!
I've been having trouble getting the WebBrowserInput Sample Project from Unity's Render Streaming to work. I am getting an error saying that .zsh cannot find my downloaded web app. May I ask how I can fix this error? Currently, I am working on a Macbook Pro (Apple Silicon M1) and have installed Unity Render Streaming (3.1.0-exp.3) and WebRTC (2.3.0-exp.6).

Thank you in advance!