Official Unity Render Streaming Introduction & FAQ

I saw there's a new version of WebRTC and RenderStreaming uploaded yesterday to Package Manager.

Is the "input team" even interested in fixing the Gui issue?
@markj_pw first reported this in Oct '19 (!) https://forum.unity.com/threads/unity-render-streaming-introduction-faq.742481/page-2#post-5096984

It's really a major negative point to expect people to design buttons and sliders in node.js and compile it into the webserver.

 

I have been successful in getting our application streaming on an AWS EC2 instance. Thanks for everyone's help. Pulling in the new input system to our Packages folder and making the focus based changes fixed it so we can attach without having the application to have focus. The audio streaming is not working for us. I have enabled audio on the Windows VM. I believe I may need to install a virtual soundcard (recommended by this post https://www.cloudar.be/awsblog/how-to-use-aws-ec2-gpu-instances/#150).

The current hurdle is trying to manage remote commands to stop / query and start the application. Using 'aws-cli' can use send powershell commands but, windows does not like to allow launching the application in the foreground. If it runs in the background and therefore headless, I cannot connect to it.

Should we be able to stream from a "headless" application? I assume the application need to think it is rendering, if we go in batchmode that assumes nographics correct?

 

Hi, I'm trying to use WebRTC to establish a p2p connection between my iPhone and Oculus Quest. However, I am developing on a Mac. I imported the WebRTC package by importing via git URL. When I try to build I get a build error: "The type or namespace 'WebRTC' does not exist in the namespace 'Unity'". I've tried reimporting all assets to no avail.

The issue seems to be related to https://github.com/Unity-Technologies/com.unity.webrtc/issues/91, as Oculus uses an Android build. Just wanted to confirm and reach out for any suggested workarounds. Thanks!

 

Hi I tired this unity render stream and it works good on my chrome browser when i try to connect from other computer it doesn't work it simply shows grey screen. Can I know why this is happening is some things i am missing i checked out the documentation but still no clue. When I try to connect from other computer chrome browser I see play icon and when i click the play icon its grey. I am attaching the error for your reference. Please help we have requirement based on this feature.

 

I have faced a similar issue when my internet connection is not stable. As soon as I connected to the stable internet everything works fine.

 

Hi, I am using a second UI camera to render some object pointers in my scene. Is there any way I can merge the two camera renderings before sending to the stream?

 

Good morning from Spain. First of all, thank you very much Kazuki (and team) for the great work and effort that is being done in this project.

I am using the latest version (2.0.2) and I am trying to modify the example scene to send a single video signal as in version 1.2.3, which I have done correctly.

However, I am modifying the operation so that each incoming user has their own camera and input. I have successfully achieved this behavior, but streaming video is only shown in the web browser for the first two users that connect; after the third user, the web browser screen appears gray; although observing the browser console, the video streaming reaches the browser correctly but is not displayed, although the inputs and other button behaviors do work.

Does anyone have this behavior or do you know how it could be solved? I don't know if the problem is from Unity or the browser and I'm a bit stuck.

Thank you!!

 

I want to interact with browser scripting similar to how you can in WebGL Unity builds, for example, I would like to be able to call a function to show a button on the url page to open from an ingame event. is this possible with the current version(s)? If so, where should I look/how should it be implemented?

 

Ya you are right but sometimes it connects sometimes after trying out for 5 to 10 attempts only it connects are you sure this is not a bug because when I tried with Wifi network where I connected to my Unity thorugh my mobile browser and my internet speed is so high

 

I have couple of questions please help.

1.Hello everyone when I am in Play mode in unity editor I was able to connect from browser but when I make it as build i.e(.exe) and when i run the executable build I am not able to connect from my browser but in documentation it says it supports Unity(Editor/Application).

2.How to send input from browser to unity say for example I have a Toggle UI button in the scene and like to toggle a gameobject visibility when i click the toggle button from browser how can i achieve this?They say in FAQ uGUI can't be operated with browser does that mean all user inputsystem interaction needs to be handled from the application developer?

Looking forward for your reply

 

I got this working but sometimes i got this error from chrome and the window would grey out.
It happens like once a while but not always. Any idea how to fix it?


video-player.js:153 Uncaught (in promise) DOMException: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Error processing ICE candidate
async function (async)
(anonymous) @ video-player.js:152
loopGetCandidate @ signaling.js:67
async function (async)
loopGetCandidate @ signaling.js:59
start @ signaling.js:32
async function (async)
start @ signaling.js:23
setupConnection @ video-player.js:156
async function (async)
setupConnection @ video-player.js:74
setupVideoPlayer @ app.js:107
onClickPlayButton @ app.js:48

 

Hi Kazuki,

Thanks for the response. I just wanted to confirm you're referring to WebRTC package as a whole and not Unity Render Streaming.

 

Excited to find out that this works now, Is there any documentation on how to set this up? We are running our application on AWS g4 on a Nvidia Tesla T4 GPU which has unrestricted Max # of concurrent sessions. Would that suffice to have multiple sessions from the same application?

How many sessions can one control from a single application?

 

Even I am facing this issue if someone helps it would be nice

 

Hi kazuki,

When we we stream from chrome/firefox while unity editor is in play mode it Grey's out and It happens like once a while but not always. Any idea how to fix it?
Also when I make as .exe file and try to stream it from browser it always show grey screen in browser as of now only streaming work with unity editor and that too sometime it shows grey color in browser. I already asked this question please help me on this topic.

 

I have successfully set up WebRTC streaming for my project, and it works, however I have noticed a serious memory leak in it. My application can run just fine for hours, but when I turn on the WebRTC streaming in it, which is set up just like in the demo scene (but streaming only 1 camera) the memory consumption starts climbing up, by approximately up to 6 mb per second, until it fills up all available RAM.

Unity Memory Profiler (the separate tool by Unity, not the one that comes within the standard Profiler) shows no objects that might have been the cause of this. This makes me think that the leak happens on the native side.

Does anybody else experience the same issue?

 

@kazuki_unity729 can one use this to stream to Twitch or YouTube? (having a server address + stream key)

Thank you

 

I believe he is referring to this issue, i took a video of it

After clicking "Play" button on chrome, everything is stuck and this error pops in Chrome
video-player.js:153 Uncaught (in promise) DOMException: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Error processing ICE candidate

It happens once a while, after i refresh it 1 - 2 times it will work again.
My iPhone safari and Window 10 Firefox doesn't have this problem, only happens on Android Chrome and Desktop Chrome

OS: Window 10
Browser: Chrome Version 81.0.4044.138 (Official Build) (64-bit)

 

Hi kazuki,
whatever youtube link herohki attached in the reply post that's what I meant its not gray color its black one I am wrong.
Below are my config
My OS Version:windows10
Unity Editor Version:2020.1.0a16.1913 personal edition
Graphics APIirectx12
Chrome browser version:Version 83.0.4103.61 (Official Build) (64-bit)
Graphics Driver Version: Attached as a png with this post.

Also when I make as .exe file and try to stream it from browser it always show black screen in browser like the one herohki attached in youtube link. Also as of now only streaming work with unity editor and that too sometime it shows black color in browser(Firefox/chrome). Also streaming doesn't work on all devices for example I have laptop and mobile both connected to Same wifi network but when i try to stream from mobile browser streaming happens but with laptop it shows a black screen. Even we had done turn server setup. Please help looking forward for your reply.

 

Hi kazuki,
Attaching graphic driver version image for your reference. Also my graphics API is directx12.

 

Hi kazuki,

Attaching the video for your reference where I try to stream from the .exe file in the chrome browser for your reference. Below is the drive link https://drive.google.com/file/d/1ZkMFR1AvnqNITg1Y4lQNqUFZDC5KT7ns/view?usp=sharing.

 

My platform is Windows 10, I am using Unity 2019.3.14f1, packages: WebRTC preview-2.0.2, Unity Render Streaming preview-2.0.2

Some additional info:

• the increase of 6Mb/s happens only in a built application, when I run it as a standalone.
• when doing the same thing in the editor, the memory increase is instead around 1Mb/s. It is not very constant, one second it can be 100kb/s, but the next second it can be 1.2Mb/s. I have also noticed that there are periods when it reaches about 3Mb/s, but after several seconds, like 10-20 seconds maybe, it returns back to ~1Mb/s

In both cases the increase is very persistent and does never reduce (nothing gets garbage collected).

Also, an important fact: if I test this inside the editor, and I stop the application, then a part of the consumed RAM is returned, like normal, but the part that resulted from using WebRTC does not return. So, for example, if I wait several minutes and let it consume 5Gb of RAM, then after stopping the application, the Unity Editor process is going to always have these additional 5Gb of RAM, it does not release them. And this is going to continue up to when it fills up 99% of RAM, at which point the Editor simply becomes unresponsive.

 

Thanks is there going to be any documentation on how to setup multiple streams from same application?

 

Hi kazuki,

I have one question related to building web-server.exe. If I want to build web-server.exe from this repo https://github.com/Unity-Technologies/UnityRenderStreaming is it possible? Currently I don't see any documentation for building web-server.exe. Am I missing something? Please help me on these. Looking forward for your reply.

 

You have to run pack_webapp.cmd (or pack_webapp.sh on UNIX).

 

I would like to report some issue when i test the webrtc under different service provider

Under different provider the webrtc would behave differently even with the same computer.
The issue happened as shown in the video above, I am not sure if it is because the service provider blocked some port or communication problem, anyone experiencing similar issue?

 

Mr. Kazuki,

TLDR: The referenced sample projects do not work at all. My GPU and its drivers are very much suitable for WebRTC and indeed, as I have mentioned, it works, but the only problem is the constantly increasing memory consumption and eventual crash (one of the reports is attached ). The problem still remains after I've tested it through and through on all the versions of Unity and sample projects.

I am very much interested in fixing this problem, so if you see something that I did wrong or something else I could try, or if you need some more information from me, I am all open for discussion.

The detailed story:

Part I: Trying to replicate the working case you are referring to

There are no project available by the link, there are only packages. But when I try to open the ProjectData~ folder in them as a project (the only thing that is reminiscent of a said project there at all) that does not work well. Firstly when I try opening it, I'm getting this message:



I am not sure how is it possible that it was created in Unity version before 5.0, but probably the project files are corrupted. But ok, I click Continue and let Unity upgrade it. Right after upgrading and loading the project I am getting these errors:



There are also errors in the rendering pipeline so after running the Rendering Pipeline Wizard and fixing all these errors I get:



The Rendering Pipeline is clearly broken and nothing is visible on the scene, both in the Editor and in the Game windows. And when I press play, I get this:



I've tried this multiple times, with the same result.

Trying to open the non-RTX project, simply the HDRP one, it starts with this, where the project does not even know what version it is supposed to be opened with, again indicating some data corruption:


And the rest of the errors is the same as for the RTX project.

I have also tried to open the `com.unity.template.renderstreaming-hd` and `com.unity.template.renderstreaming-rtx` projects from the `Source code.zip` file, but the result and the errors are the same. Except that these at least know their desired Unity version which is for some reason 2019.3.6f1 and not the latest one or at least pre-latest (why?).

I've tried all this in both Unity 2019.3.6f1 and Unity 2019.3.14f1 (latest atm), and it's always the same.

Conclusion/TLDR: I am totally not sure how you manage to run these projects and get any wokring results out of them, so I was unable to test their behavior. Please tell me if I am doing something wrong so I could try to do it more correctly.

Part II: Explanation on how was I able to make something work in my project in spite of Part I

My project runs in Unity 2019.3.14f (current atm) and my GPU is RTX 2080 Ti, I have tested the behavior of WebRTC on 3 latest versions of the driver and it was all the same.

The way I was able to obtain a working example of a WebRTC streaming demo was through installing the `Unity Render Streaming preview-2.0.2` from the Package Manager into my own project. And then installing the HDRP sample by clicking the button available in the description of the package.

Can you confirm that the way I did it is also a correct way to get WebRTC Video Streamnig into my project?

Part III: The bug still remains

Throughout any experiments I have ever conducted with WebRTC, across all Unity versions, driver versions and sample projects, at times when I was able to make it run, I always observed that constant memory consumption that eventully started to taking all the available RAM. In the end, if you don't stop it, the application always crashes.

I am very much interested in fixing this problem, so if you see something that I did wrong or something else I could try, or if you need some more information from me, I am all open for discussion.

Thank you for helping in investigating this issue!

 

Hi @subramaniyanvgatmoback,

1. Whenever you run the executable file you need to make sure that server application launch before the unity application.

2. To get the input from the browser you need to integrate the new input system into your project which means developer need to handle the interaction from both sides.

 

Thanks for the reply ashwani. I did run the server application before unity but it works sometimes on Browser and many times it shows grey screen but anyway they said it will take some time to fix. Also at present I am creating the interaction for unity GUI by using new input system if in case of any issues i will post. Thanks again

 

Thank you so much it's working. Now I am able to create .exe .

 

Hello!

So, I have followed this tutorial and was able to create a project template from UnityRenderStreaming 2.0.2-preview, and I get absolutely the same behavior, here you can see it yourself:

the video is quite long, but you can quickly skip parts of it and see the general memory consumption increase and Unity crash at the end.

I followed the instructions on importing the project template this time! But it did not change anything. I'm also running this on Unity 2019.3.6f1, but the result is the same on 2019.3.14f1 and 2019.3.15f1.

There is also another problem in the video -- sometimes the video feed glithches and starts to loop over the last few frames over and over again. This problem is not related to memory consumption and does not happen in 100% of the cases. That's another problem but I decided to not re-film the video because of it, to demonstrate it too.

What do you tink is happening?

 

@kazuki_unity729 Would you know if this issue of creating GUIs has been addressed and fixed by the Input team?

@ashwani_9 A few posts earlier it seems you might be already doing this? (creating canvas/gui buttons that work)
If yes, would you be able to take some time explain a bit more. Very grateful if you could.
Kind Regards

 

When I deploy the service on the server，Enter the server's IP and port in another computer's browser，The play button appears。but when I hit Play，i got an error
This is normal both locally and in the editor
I don't know why. I hope to get your reply

 

I am currently fighting this too. We get a pause of 5 to 30 seconds before the stream comes up. With potentially a refresh or two before it catches. Unreal's demo with Web-RTC takes 1 to 3 seconds to connect and start streaming. Currently walking Unreal's and Unity's WebApp / Node.js code to find the difference. My current thought is some of the objects are not fully initialized.

https://stackoverflow.com/questions/38198751/domexception-error-processing-ice-candidate

If anyone else has progress on this or is seeing the same pause, I am sure we all would love to hear about.