Official Unity Render Streaming Introduction & FAQ

When i use WebSocket as singnaling The error gone away。but it is Still not working

 

Can anyone tell me the current status of this? Can we now use different cameras for different streams etc? So could we use this to stream an app to say 4 browsers and have them all be a different camera with different input?

 

Another month and no progress.
@kazuki_unity729 Could you invite the person in-charge of the Input manager product to this forum.
I'd like to fast forward the lag this team is facing.
Kind Regards

 

Hi @kazuki_unity729

I am using the latest preview package 2.0.2 and tried to run in 3D unity project instead of the HDRP project type. I got the below error in CameraStreamer class where it fails to initialize the videostreamtrack.


NullReferenceException: Object reference not set to an instance of an object
Unity.WebRTC.VideoStreamTrack..ctor (System.String label, System.IntPtr ptr, System.Int32 width, System.Int32 height) (at Library/PackageCache/com.unity.webrtc@2.0.2-preview/Runtime/Scripts/MediaStreamTrack.cs:173)
Unity.WebRTC.VideoStreamTrack..ctor (System.String label, UnityEngine.Texture source, UnityEngine.RenderTexture dest, System.Int32 width, System.Int32 height) (at Library/PackageCache/com.unity.webrtc@2.0.2-preview/Runtime/Scripts/MediaStreamTrack.cs:104)
Unity.WebRTC.VideoStreamTrack..ctor (System.String label, UnityEngine.RenderTexture source) (at Library/PackageCache/com.unity.webrtc@2.0.2-preview/Runtime/Scripts/MediaStreamTrack.cs:147)
Unity.WebRTC.CameraExtension.CaptureStreamTrack (UnityEngine.Camera cam, System.Int32 width, System.Int32 height, System.Int32 bitrate, Unity.WebRTC.RenderTextureDepth depth) (at Library/PackageCache/com.unity.webrtc@2.0.2-preview/Runtime/Scripts/MediaStream.cs:169)
Unity.RenderStreaming.CameraStreamer.OnEnable () (at Assets/Scripts/CameraStreamer.cs:23)

Does Unity render streaming works only in the HDRP project type? Am I missing something to configure in order to work with the default 3D project type?

 

I included that component to a GameObject already.

 

Thanks a lot. It worked after explicitly adding the renderstreaming in script execution order.

 

As a temporary fix you can pull the input system into your unity project's packages folder and modify the two places necessary to get it to accept input without focus. This post on this forum gave me the info I
needed.
https://forum.unity.com/threads/unity-render-streaming-introduction-faq.742481/page-2#post-5096528

 

When I don't use hdrp and URP, how to get the final screen picture after multiple cameras are mixed? The picture I get through id3d11rendertargetview contains all the contents of game window (as shown in the figure below). But I just want to get the real result picture of the operation, and I don't need unity's own window

 

Hi I am trying to build to Linux so I can run the project on a linux container.
However, whenever I try to run the linux build i get the following error:

Unable to preload the following plugins:

libwebrtc.so

Unable to load player prefs

Plugins: Couldn't open /root/project/buildtest/linusbuild_Data/Plugins/libwebrtc.so, error: libcuda.so.1: cannot open shared object file: No such file or directory 

(Filename:  Line: 330)

I am using render streaming V2.0.0 and building from a windows pc.
Is there support for linux builds? If so, i'd really appreciate some help on what steps im missing (note: i've never build to linux before)

thanks in advance

 

Thanks for your response. Yes, NVIDIA is equiped for both. The project runs and streams as expected within the editor (with target platform set to Linux) as well as a Win64 build but unfortunatley continues to abort when attemping to run a Linux build.

 

What is webserver.exe? And what is it for? Thanks for the answer.

 

I'm having a namespace issue that happens when trying to use a fresh init of the Unity Render Streaming template. I've done quite some searching on this already but the error is so ambiguous that its hard to understand what is conflicting.

Basically creating a new project and then installing Photon PUN2 doesn't get recognized when calling Photon from within the scripts folder.

Code (CSharp):

1. Assets\Scripts\Test.cs(4,14): error CS0234: The type or namespace name 'Pun' does not exist in the namespace 'Photon' (are you missing an assembly reference?)
2.  

It works fine when I create scripts in the root Asset folder and I can use Photon, but any call from the Scripts folder yields the above error.

Does anyone have any hints to what might be going on? I'm pretty sure its a namespace conflict related to code from Unity Render Streaming, or the location of an assembly file or something.

When initially importing PUN you also get an error related to Websocket, but thats easy to fix by removing one of the two copies of Websocket (I removed the one from the Plugins folder).

EDIT: I think I've solved my own issue.. there's an assembly file in the scripts folder which needs to have those files added in order to compile.

 

Anybody found a fix to this? We are experiencing the same problem

 

Im trying to run my project headless on a Linux VM with Nvidia M60 GPU. Whenever I try to run it headless I get the following error:

Plugins: Couldn't open /mnt/batch/tasks/applications/echoremote0.0.4.2038-linux2020-06-23-09-51/EchoRemote/EchoRemote_Data/Plugins/libwebrtc.so, error: /mnt/batch/tasks/applications/echoremote0.0.4.2038-linux2020-06-23-09-51/EchoRemote/EchoRemote_Data/Plugins/libwebrtc.so: undefined symbol: glCopyImageSubData

Is headless rendering without display supported as of today?

 

Hi, I tried to run the renderstreaming example and in LAN all works fine.
But how can I allow user from remote network to access the application?
I tried to setup conturn on a linux virtual machine following this guide https://docs.unity3d.com/Packages/com.unity.renderstreaming@2.0/manual/en/turnserver.html but at some point it ask me to write "turn:xx.xx.xx.xx:3478?transport=tcp" in the Render Streaming object but what the "xx.xx.xx.xx" stands for? Did I need a internet domain?
I've also tried to use a public turn server "numb.viagenie.ca" but nothing seems to work.
Can someone guide me with a more detailed step by step guide ?

 

Hi, I'm working on our project with RTC on mobile ... so we have a login screen with 2 input field for ID and password -- I've already deal with the text input issue so it can input text into the input field from a remote PC. However, when access via mobile, it doesn't show the touch keyboard so there is no method to input login info. Is there anyway to send signals to the remote browser to open the touch keyboard? Or need to insert a virtual keyboard in game for visiting via mobile browser?

 

Unity webgl is no supported on mobile browser, and keyboard input is not supported too. If you want to show keyboard when your user click on the input field you have to call a javascript script that show a javascript input field to the user and have the user type into that input field and then the javascript input filed send the string to unity

Code (CSharp):

1. Javascript code will be something like
2.             var _inputTextData = prompt("Insert string", Pointer_stringify(_str));
3.             if (_inputTextData == null )
4.             {
5.                 //canceled text
6.             }
7.             else
8.             {
9.                 //send data to unity
10.                 SendMessage(Pointer_stringify(_name), 'ReceiveInputData', _inputTextData);
11.             }

 

The issue is not about how to send text -- the viewer of browser has a listener of input(so I guess if I use a physic keyboard via mobile phone, it should get the input text directly -- I need an otg cable to connect phone and keyboard to test this, just ordered one via Net-shopping so will need another 2 days). I've also caught up the idea of using JavaScript, make a HTML input field so the browser will call the system IME automatically -- I've added a request at github issue page, put this as an alternative method. But, before dealing with the questions above, you must do one thing first -- let the browser know you want input something.
If you checked the elements of the viewer page, you'll find that the streaming is only a video element, all our operations through the brower are turned into "signals" then transfer to the server by a javascript, and the server read those signals and restore them to local operations -- i.e., I clicked my mouse at position (X,Y), the viewer of browser will send a byte array containing operartion type(Mouse event), mouse position, and button action, then the Unity application, or server side receive the signal, and recurring what you did -- click the mouse at position (X,Y) in the app's screen space. Our project is still using Unity 2018.4, and is not possible to update to 2019 at once, that means we don't have the new input system, so we deconstruct the remote input and linked it to our own input controller -- however, the basic theory doesn't change, we just remapping the key code manually from the signal, while the new input system deals them by itself. So when dealing with the signals, I found the solution of interaction with UGUI input field -- the viewer of browser send back almost all the key code when you typing(there are exceptions, like TAB key, the browser will take it first, so it will not be captured by the viewer of browser), so what I do is just get those input singals and insert the chararcter into InputField.text(and move carret, define backspace/delete normal actions and selection actions, etc.) when a InputField is focused. And here is the question: as I said above, the actions on server side(Unity application) are just reflections from your operations on browser, so when you focus an Input Field, the unity application will know that, and by using my method, input the received character into the input field -- BUT, the browser cannot know that if we don't do something -- all it can do is just receive video streaming from server side, and send input signals to the server side.
So... after these long explainations, I hope you get what I mean. Currently, the input works well on a PC/Mac, because they have direct input method. For mobile phone, I guess it may also work if I connect a physic keyboard to it, but you cannot ask every end-user to prepare a physic keyboard right? So we always need a touch keyboard(called by system IME) to do the text input. It doesn't matter that input directly via the touch keyboard or using JavaScirpt to make an intermediate transmission for current step, instead, frist thing of first, is let the browser know that an input field is focused on server side, and this is what I'm asking for -- send signals from server to viewer of browser that "I want to input text". The viewer's reaction could be considered after this.

 

I made sure libcuda.so.1 was installed. Now I'm receiving the following error:

Code (CSharp):

1. Plugins: Couldn't open /root/project/buildtest/linuxbuild_Data/Plugins/libwebrtc.so, error: /root/project/buildtest/linuxbuild_Data/Plugins/libwebrtc.so: undefined symbol: glCopyImageSubData

Also should I be using OpenGL or Vulkan ?

 

OpenGL version is 3.3

 

How is audio streaming suppose to work, because i can't hear any audio on my stream.

Ok i managed to get it working, for some reason i have to put thread to sleep

 

Ok, tested mobile phone with a physic keyboard(a Mac keyboard, same one I used for PC platform test), and ... works. Or partly works, I'm not sure if every key code returns the same one as PC because some keys don't work, like backspace and numpad keys have no reaction when press, but arrow keys and delete key work well.
So like I guessed before, the javascript in RTC viewer of the browser does catch input, even on Android ones, and as physic keyboard works, so should the virtual one, and what I need is let the browser/system know I need to input something and call out the touch keyboard.

 

I've managed to enable microphone on server side but how do i play that in unity build?

 

ok, I have another idea.
I can add a floating button to the browser viewer page with JS and try force call the touch keyboard out on mobile phone platform.
So ... the question is now become: where should I put the script if I want to customize the viewer page?

 

I've enabled microphone through javascript but I'm not getting anything in my local unity build that's streaming to my browser. Our streaming experience has multiplayer so i need to send microphone data from browser to unity build and then send it elsewhere.

 

Is streaming audio from the browser on the roadmap? Or is there a fundamental limitation blocking this?
Thanks

 

Hi，I ran into the same problem.
When I use software encoder instead of hardware encoder, I can successfully receive the video stream.
But when I reinstall the graphics card driver and use hardware encoder, I still cannot receive the video stream
The graphics card I use is GeForce RTX 2080 ti/GeForce Game Ready Driver 451.67
Anything else I should be considering? thanks.

 

Hi there!

Is it possible to run Render Streaming on a cloud server like GCP, AWS or Azure? I know it is mentioned in the FAQ that it is not supported, but I am wondering why? Is it a because of license issues or technical considerations?

Running Render Streaming in the cloud would be great. For example, the cloud server could join a multiplayer game and stream it to web browsers for spectator mode.

Is it a use-case that is supported?

Thanks!

 

Yes it can be run on a cloud server and is being done by a number of us on this board as far as I can tell. The example you mentioned could be done. It is a pretty flexible solution and designing a title specifically with "local multiplayer" in mind is pretty interesting too.

 

@Maktech Thanks for the reply. That's encouraging! Would you say it's "production" ready at this stage?

Our use case would require us to spawn several instances of the game. So I assume that would mean several GPU instances, and therefore, several AWS server instances? I am not very familiar with AWS. Does anyone have experience with this?

Thanks again

 

Hi

Is it possible to run the webserver/application multiple times on the same machine?

We would like to run multiple instances of the application and stream it to different users.

Thanks!

 

Yes. You have to use different ports for each web server and your app instance needs to know which one to use. Same applies to your clients. As this is somewhat inconvenient I suggest extending the server component to support multiple instances.

Bear in mind, that the number of hardware encoded sessions is limited by your NVidia card. Depending on the GPU you might be forced to use software encoding (most models allow only 2 sessions in parallel s. Video Encode and Decode GPU Support Matrix.

 

Thanks for the great answer @kayy ! I will look into it.