Official Unity Render Streaming Introduction & FAQ

I am trying to send video from the browser to Unity and I get an error when setting the local description in Unity OnOffer from the browser
"Network Error: Failed to set local answer sdp: Failed to set local video description recv parameters for m-section with mid='0'."

 

Or just a guide for how to use the app2.js example on the Unity side would help. I must be missing something.
I thought something with the transceiver configuration could be wrong

 

Hello I have the same error sending from Unity to Unity. Did you find a solution for this?

 

Hi,

I’m getting a scrambled video stream and wanted to ask if this image indicates the likely problem. A codec mismatch? A mismatch is frame sizes? (I have tried to regularize to 1280 x 720.)

I have had success with hardware encoding in various setups. This problem is related to software encoding on my local PC (Windows 10, NVIDIA video card).

I’ve updated to WebRTC 2.3.1 in both of the Unity projects.

Components in my setup:

• Unity Server, based on the Unity Render Streaming package version 2.2.2, which I manually updated for compatibility with WebRTC 2.3.1.
• Unity Client, based on the WebRTC 2.3.1 sample called “VideoReceive.”
• Signaling Server, based on the NodeJS server included with Unity Render Streaming 2.2.2.

Hardware encoding, whether on my local PC or in the cloud, works great for any configuration involving:

• Unity Server
• Signaling Server
• Web Browser (again using the sample web app from Unity Render Streaming 2.2.2.)

The image is from testing with software encoding on my local PC with:

• Unity Server
• Signaling Server
• Unity Client

Please let me know if you have any ideas. I’d be happy to provide more info if necessary.

 

I'm using 2.3.1 already. This version solves the wrong string ICE error but not this one.

Network Error: Failed to set local answer sdp: Failed to set local video description recv parameters for m-section with mid='0'

 

I still have this issue, was about to try WebRTC 2.3.1 but if you say its not working as well we need some help from Unity then

 

Is there a way to stream video from the browser using python? I'm trying to use opencv on a video stream from Unity and WebRTC seems like it might be an easy way to do that.

 

any news about looking in to this?

 

Confirmed, it works with software encoding.
Any road map on H.264 decoder and mobile support as well

 

Hello,
I created a renderTexture1 to use as the targetTexture for the camera, and used the constructor videoStreamTrack (String Label, renderTexture Source) to create a track from Texutre1,
but the screen is black on the browser side.
do you have any suggestions to fix this or can you provide the code for captureStreamTrack ()

 

@kazuki_unity729 - Thank you for the reply. Turns out the issue was the resolution. I just stumbled on one more place in the configuration where I had a different video resolution. Once I changed that to 1280x720 the video came through clearly.

So: confirmed that software-based streaming works from a Unity server to a Unity client.

 

I wonder how to stream the specific track to the specific channel (by channel id or other method), thanks!

 

[WebRTC] SetAnswerDesc : error : Failed to set remote answer sdp: Failed to set remote video description send parameters for m-section with mid='1'.
I have developed with unity webrtc library 2.3.1, and have used it well with software encoding.
But when I tested nvidia [NVIDIA Tesla T4], I got the error.
The answer sdp was the following. What's the problem?

v=0
o=- 323691294067975285 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS
m=audio 54722 UDP/TLS/RTP/SAVPF 111 110
c=IN IP4 1.227.95.66
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:2698998123 1 udp 2113937151 f66dc19f-2b5b-4ba2-ab03-196f453ec558.local 54722 typ host generation 0 network-cost 999
a=candidate:842163049 1 udp 1677729535 1.227.95.66 54722 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999
a=ice-ufrag:1Xot
a=ice-pwd:+NEt3AVx+4p6SjXfameS/0ah
a=ice-options:trickle
a=fingerprint:sha-256 18:63:3F:F4:BA:87:7B:52:4C:B7:3B:97:698:30:3A:77E:1B:96:A6:43:2D:FF:1C:637:F8:3C:6C:40:48
a=setup:active
a=mid:0
a=extmap:1 urn:ietfarams:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietfarams:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietfarams:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietfarams:rtp-hdrext:sdes:repaired-rtp-stream-id
a=inactive
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:110 telephone-event/48000
m=video 9 UDP/TLS/RTP/SAVPF 127 126 125 102 98 99 100
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:1Xot
a=ice-pwd:+NEt3AVx+4p6SjXfameS/0ah
a=ice-options:trickle
a=fingerprint:sha-256 18:63:3F:F4:BA:87:7B:52:4C:B7:3B:97:698:30:3A:77E:1B:96:A6:43:2D:FF:1C:637:F8:3C:6C:40:48
a=setup:active
a=mid:1
a=extmap:14 urn:ietfarams:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietfarams:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietfarams:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietfarams:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:127 VP8/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=rtpmap:126 VP9/90000
a=rtcp-fb:126 goog-remb
a=rtcp-fb:126 transport-cc
a=rtcp-fb:126 ccm fir
a=rtcp-fb:126 nack
a=rtcp-fb:126 nack pli
a=fmtp:126 profile-id=0
a=rtpmap:125 VP9/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 profile-id=2
a=rtpmap:102 VP9/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 profile-id=1
a=rtpmap:98 red/90000
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 ulpfec/90000
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:1Xot
a=ice-pwd:+NEt3AVx+4p6SjXfameS/0ah
a=ice-options:trickle
a=fingerprint:sha-256 18:63:3F:F4:BA:87:7B:52:4C:B7:3B:97:698:30:3A:77E:1B:96:A6:43:2D:FF:1C:637:F8:3C:6C:40:48
a=setup:active
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

 

Is it possible to pause the webrtc transmission? I have case where I need to stream a render texture only when user input something, basically I only render and transmit as needed. Maybe I should use socket in this case? Thanks !

 

I already managed to stream on my local network, but when I run webstreming.exe I only have the option to use the local IP (127.0.0.0 or the local lan 192.168.14.17) I have a fixed public IP and I want to be able to enter from outside of my LAN for streaming.








When I open the window of my public IP, if I have the menus but not the streaming of the 3D space:







Additionally I have this error:


And I do not know what happens.

Any ideas?

 

Hi!
I tried to test the operation of the SimpleRenderStreamingURP scene. In the browser, the 'Light on', 'Light off' and 'Play audio' buttons work as expected, but you can't control the camera from the browser window. In the Unity console, an error appears:

I use Unity 2020.2.3.f1, Unity Render Streaming Version 2.2.2-preview, WebRTC Version 2.2.1-preview. Unity, webserver.exe and Firefox browser are running localy on the same PC (Windows 10 x64).

Please tell me how can I get rid of the error?

UPD. Sometimes a miracle happens and I manage to control the camera in Unity from the browser window and vice versa (control the browser camera from the Unity window). However, the error described above still appears in the console. There were also cases where I could either control the camera in Unity from the browser window, or control the camera in the browser from the Unity window, but not both at the same time... In general, the behavior changes whenever I refresh the page in the browser and restart the webserver.exe

 

Yes, same behaviour also reproduces in Chrome. May be the problem is in not stable connection with stun:stun.l.google.com:19302 Ice server... I will try to set up local Ice server and test this issue.

 

HttpSignaling, url: http://localhost

 

No, this is not an error I receive. Mine is about: "Network Error: Failed to set remote answer sdp: Called in wrong state: kStable"

Local ice server did not fix the issue.

 

Any chance you have your source for this connection flow posted anywhere ? (Unity server to client instead of browser). I am working on a similar implementation and would appreciate it as a resource.

 

Hi!
I'm trying to deal with buttons. How to handle button click events in renderstreaming? Thank you all!

 

Button in Unity.
I have two ideas to respond to the button event in unity in renderstreaming. The web side detects the button in unity canvas and sends the button click event to the unity side after clicking the mouse (is it difficult for the web side to implement it?). The other is that the unity side detects whether the button has been clicked according to the position of the mouse click and performs related operations (this may have applicability problems).

I hope you can provide some ideas to realize it, thank you!

 

I have trouble visualizing the second stream and I was wondering if any of you had the same issues.
If I stream locally using the default sample scene and a modified version of the webApp I manage to see the videoThumbnail almost all the time, while if I stream with a public IP only the first stream get shown on the main page.
I have no error in the browser console log and nothing in the unity one as well.
How can I discover what is happening?

Unity version 2019.4.11f1
PKG version 2.2.2

 

It's not difficult, the webapp already reads click and mouse position continuously and send it to the Unity Instance.
Then this one creates a fake input as it was really happening in unity, this is done in RemoteInput.cs while the input created is then read in SimpleCameraController.cs.

I know it may sound complicated but reading this two scripts you will manage to do what you want

 

Right on, thank you!

 

Sorry, I can't understand it.

In my opinion, webapp can only judge whether the mouse clicks or not, and where the mouse clicks. When the mouse clicks, it will only send the sendmouse event information, not the buttonclick event. Therefore, webapp can't judge whether the clicking position is a button element in unity. The ButtonClick event in remoteinput.cs only occurs when the mouse clicks on the button element in browser.

So how can I simply and universally respond to the button click event in unity?

 

You are right, button click event has nothing to do with the button click event you would like to achieve.
What would probably help you is understanding well what happens after the send mouse event in the webApp and how those information are used by the unity Instance.

This function

Code (CSharp):

1.  InputSystem.QueueStateEvent(RemoteMouse, new MouseState {
2.                 position = pos,
3.                 delta = delta,
4.                 buttons = button
5.             });

creates a fake input based on the information received from the internet. If I were you I would try to add a canvas and see if it already works without any more efforts, in the negative case I would try to create a fake ray-cast as It was done with this function for the mouse movement.

 

I see what you mean. But in my case, the unity instance can't capture the button click event. I can't find the reason. Is it because of the coordinate problem that I didn't actually click the button?

 

Thank you for your contribution to this awesome project.

But I have trouble when I use the latest version of render streaming (v2.2.2)

How can I stream only one camera and allow all other browser can control the camera? The Sample is multi camera but it seem to write badly cannot stream one camera.

Also, client of unity editor cannot control the camera as well

 

First, nice work with the render streaming, we have found it very useful!

However, at the moment we are having trouble getting high enough framerates even though hardware is not fully utilized. It seems that video encoding is currently fully synchronous with rendering meaning that the render thread waits for the encoder to finish before it starts rendering cameras for the next frame. It does not affect latency but has a major effect on throughput. Are there any plans to implement asynchronous encoding or multithreading with the render streaming plugin? We are using Nvidia hw encoding (NVENC).

 

Please can anyone walk me through the process of getting user inputs from the browser? I have followed the guide in the documentation but it didn’t work.

I tried this from the URS docs but it didn’t.
// Create a RemoteInput instance
RemoteInput input = RemoteInputReceiver.Create();
channel.OnMessage = bytes => input.ProcessInput(bytes);


// Get the keyboard device, process on w key press
Keyboard keyboard = input.RemoteKeyboard;
if(keyboard.wKey.isPressed)
{
// ...
}

Can anyone give me a simple example of how to get user input from the browser to control a game object in my scene?

 

I want to attach a new script to game object and control it with keyboard input from the browser. Please can you walk me through the process of doing that? I followed the documentation but is still not working.

 

I want to attach a new script to game object and control it with keyboard input from the browser. Please can you walk me through the process of doing that? I followed the documentation but is still not working

 

I want to attach a new script to game object and control it with keyboard input from the browser. Please can you walk me through the process of doing that? I followed the documentation but is still not working