Official Unity Render Streaming Introduction & FAQ

Hi,Check out this Unity Forma from the release: https://portal.furioos.com/share/134.761?autoRun=true

The codec and CPU usage in this example are fairly balanced

 

Anyone gotten the tutorial in the documentation to work?

I can get a stream to work but I want to send input. I have followed
https://docs.unity3d.com/Packages/com.unity.renderstreaming@3.1/manual/control-camera.html

However I feel like there is stuff missing from that documentation.
The WebBrowserInput example uses ways to register devices like
receiver.onDeviceChange += OnDeviceChange;

Which is not present in the above documentation. However if I add the Simple Camera controller, and copy the Canvas from that example then my input from the "control-camera.html" instruction works. So something is missing

I am running 3.1.exp.3 .

Also, is there a way to get the unity UI to get the mouse events? Like if I have a slider?

 

Your project needs to be setup with the new input system and add the InputSystemUIInputModule component to your EventSystem object. That should get the UI working. There is a section in the new input system's documentation talking about these changes if you want to hunt down more details.

 

Is there a way to find out which streaming resolutions are supported when enabling Hardware Encoder Support? I can only stream 640x480 and 1280x720. No luck with 1600x1200, FullHD or anything above that.

Without hardware encoding support, I can reach larger resolutions, but I would love to use higher resolutions with HW encoding support on.

 

yes

 

yes, that was me. Thanks for the info, I'll switch to Software Encoding for the time being

 

I'm also facing another issue:

I am currently implementing an SFU using Janus. So far, I can send and receive audio and video streams. What does not work, however, are the data channels. Negotiation is successful and I can send data to Janus. However, on the receiving side,

pc.OnDataChannel

never gets invoked although the offer sent by Janus includes a data channel. Here is an example jsep offer that Unity receives by Janus:

Code (CSharp):

1.    "jsep": {
2.       "type": "offer",
3.       "sdp": "v=0\r\no=- 1652698575225406 1 IN IP4 164.90.176.95\r\ns=VideoRoom exhibition-h264\r\nt=0 0\r\na=group:BUNDLE 0\r\na=ice-options:trickle\r\na=fingerprint:sha-256 E6:CD:63:25:3A:5D:FD:9E:A8:C5:ED:E3:E8:58:9D:18:D0:5A:B4:74:13:8C:42:F3:E0:BF:C8:32:AE:2B:57:C0\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS janus\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 164.90.176.95\r\na=sendonly\r\na=mid:0\r\na=sctp-port:5000\r\na=ice-ufrag:A+S/\r\na=ice-pwd:d9X1i3QqwysJk3WbeTs/Xr\r\na=ice-options:trickle\r\na=setup:actpass\r\na=mid:0\r\n"
4.    }

and the answer Unity RenderStreaming creates:

Code (CSharp):

1. "jsep":{
2. "type":"answer",
3. "sdp":"v=0\r\no=- 7811705463152725371 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=application 55406 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 10.1.8.127\r\na=candidate:3880229 1 udp 2122260223 10.1.8.127 55406 typ host generation 0 network-id 1 network-cost 50\r\na=ice-ufrag:Shk9\r\na=ice-pwd:gDVVYjI1S4v7NNRXEjnBx0Sp\r\na=ice-options:trickle\r\na=fingerprint:sha-256 DD:E6:DD:91:FD:5C:9E:42:22:2E:C2:AC:C9:53:04:C3:C2:9E:3D:8C:CF:F0:34:53:0A:42:99:B4:A7:AD:17:A5\r\na=setup:active\r\na=mid:0\r\na=sctp-port:5000\r\n"
4. }

My question is now: When exactly does RenderStreaming invoke

pc.OnDataChannel

and how can I trace where the error could stem from?

 

Hey, thanks for your reply, but I think my question was a bit confusing. I know that onDataChannel is in the RenderStreamingInternal class and was already able to trace its invoking to this method in RTCPeerConnection:

Code (CSharp):

1.         [AOT.MonoPInvokeCallback(typeof(DelegateNativeOnDataChannel))]
2.         static void PCOnDataChannel(IntPtr ptr, IntPtr ptrChannel)
3.         {
4.             WebRTC.Sync(ptr, () =>
5.             {
6.                 if (WebRTC.Table[ptr] is RTCPeerConnection connection)
7.                 {
8.                     connection.OnDataChannel?.Invoke(new RTCDataChannel(ptrChannel, connection));
9.                 }
10.             });
11.         }

However, because of the .dll call, I don't really know when this method is called.

 

How can I send and receive data between different Unity projects?
I want to send integer data from project 1 to project 2.

 

Is GPU necessary? My machine doesn't be equipped GPU, but still run the sample of this project successfully.

 

Hi Have you already run the project on container? I have a question here.
https://forum.unity.com/threads/how-to-run-unity-standalone-player-on-container-eg-docker.1283063/

 

I am using Multiplay Sample it working fine on the same devices browser where the unity app is running but it not working on othe devices browsers.

 

Thanks! So this should basically fire using that SDP?

Code (CSharp):

1. "sdp":"v=0\r\no=- 7811705463152725371 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=application 55406 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 10.1.8.127\r\na=candidate:3880229 1 udp 2122260223 10.1.8.127 55406 typ host generation 0 network-id 1 network-cost 50\r\na=ice-ufrag:Shk9\r\na=ice-pwd:gDVVYjI1S4v7NNRXEjnBx0Sp\r\na=ice-options:trickle\r\na=fingerprint:sha-256 DD:E6:DD:91:FD:5C:9E:42:22:2E:C2:AC:C9:53:04:C3:C2:9E:3D:8C:CF:F0:34:53:0A:42:99:B4:A7:AD:17:A5\r\na=setup:active\r\na=mid:0\r\na=sctp-port:5000\r\n"

Could it be that the SDP is formatted wrongly in any way and that the native code does produce an exception or anything similar that I'm not being notified of?

 

Hi, I use Render streaming and AR foundation kit to build my project, the purpose is to achieve the function of AR remote rendering.
The client side collects the modelview matrix, projection matrix, and timestamp, and sends it to the server for scene rendering, and then streaming it to the client side. The scene frame and camera frame will be matched through Timestamp, and the matched frame will be used Shader to merge.
Now I encounter a problem that the model of the scene frame is not synchronized on the camera frame. The model seems to drift. Similar to the video in this discussion thread, I have tried all the suggested methods in it, but I still have the same problem. Is there any other solution?

The link is my code. (Client)
https://gist.github.com/a55987482/49b9ede2ef1905e9a9f6e78f4b54fb56
PlaceOnPlane.cs is collects the modelview matrix, projection matrix, Timestamp, and sending them to the server.
CameraFrame.cs is responsible for storing Camera frame in Dictionary.
AnchorSample.cs is responsible for receiving server-side streaming images (each frame includes Timestamp), and uses Timestamp to extract the matching Camera frame, and finally merges through Shader.

Render streaming version is 3.1.0-exp.2
AR foundation version is 4.2.2

 

Thank you so much for your help! Simply pasting the SDP into MungeSDP does not work and gives me either an "Invalid SDP line" exception or an "invalid fingerprint" exception. But I'm guessing this is because the copied SDP does not stem from the same application? In my application, the incoming SDP does not lead to an exception in any way. And if the SPD contains more than one m-line (e.g., a video track) than these are processed correctly and work.

If I compare an answer SDP generated by Unity to an answer SDP created by the Janus server, they seem identical, expect from the dynamically generated values such as the fingerprint.

Here is a link to the diff between the two SDPs. The left one is the one generated by Unity, the right one by Janus:

https://www.diffchecker.com/aCpVyAI1

 

I was wondering the same thing, because I want players to be able to stream from other cameras than the main camera that they look at, so screen streaming won't work in that case. I think you can use a Browser source in OBS then stream to twitch, youtube or whatever you like. I am yet to test that for my project.

 

Update: I can, however, exchange datachannel-part in the Munge SDP web sample with my SDP and this works! So, I can replace anything that starts from

m=application 9 UDP/DTLS/SCTP webrtc-datachannel

with this:

Code (csharp):

1.  
2. m=application 55406 UDP/DTLS/SCTP webrtc-datachannel
3. c=IN IP4 10.1.8.127
4. a=candidate:3880229 1 udp 2122260223 10.1.8.127 55406 typ host generation 0 network-id 1 network-cost 50
5. a=ice-ufrag:Shk9
6. a=ice-pwd:gDVVYjI1S4v7NNRXEjnBx0Sp
7. a=ice-options:trickle
8. a=fingerprint:sha-256 DD:E6:DD:91:FD:5C:9E:42:22:2E:C2:AC:C9:53:04:C3:C2:9E:3D:8C:CF:F0:34:53:0A:42:99:B4:A7:AD:17:A5
9. a=setup:active
10. a=mid:0
11. a=sctp-port:5000
12.  

and the web application is still able to send and receive via data channel. If I do this in Unity, the application throws an Invalid SDP line error.

 

Hi. Are there any solutions to solve the problem that render view shows slowly?

Will it be better if I use ice server or turn server?

 

Is it possible to negotiate with Vanilla ICE?

 

Services such as Wowza and Ant Media that support WebRTC delivery can also be connected with Tricle ICE, but they are basically negotiated with Vanilla ICE.

 

Hello,

I am currently developing an app that use Unity render streaming to stream it content on a local web page. The Unity executable and web page client are on the same computer. We use it only to render 3D object and interact with it, the web page are used for the main user interface.

I am currently facing a bug where the input sends twice the drag event (once coming from the web page and the other one from the unity executable) which leads to a jump of my object at two different position, as the mouse position sends two different values. I have a way to correct locally the problem, but I want to know if it is a bug ? It does not happen when we are on two different computer.

 

For now as I only use the mouse position to calculate a delta between two mouse position, I set a threshold so I don't use the delta that are too high. I still get the two values and in case the executable and the web page are on the same screen (or at least not enough separate), I keep both, what I shouldn't, and a scintillation appears.

Is it possible to get two different device Id in case we use the input from the render streaming ? Or I missed something that enables to difference the inputs from streaming and directly from the app ?

 

I would appreciate it if you could add an option to switch.

 

I would like a separate FAQ thread for com.unity.webrtc.

 

Instead of having all users connect to the same stream, want each person to end up in their own interactive experiences. How To do this, can provide some toturs , thank

 

How to run multiple instances of the unity using render streaming on the same computer

 

Hi I would like to know how to use Unity Render Streaming to control my project via browser. Thanks

 

Good Day!! I installed and run the Unity Render Streaming package as is and completely followed the documentation on how it works. However, the imported sample scenes doesn't work on my Unity project. I wonder what could be the problem?? Thanks and have a good day

 

Hello All,
I want to use Unity Render Streaming for my VR Oculus quest app to stream to web browser. It's awesome.
Currently I'm using my own websocket solution for streaming but it has issue FPS drop in Oculus Quest app, because due to web socket is TCP so packet drop result VR app down FPS.

Is this Unity Render Streaming has anyone integrated in Oculus app, Anyone have faced FPS drop issue when streaming started?

 

Hello dhruv
i have the same setup and i use the websoket instead of tcp