Search Unity

Unity Render Streaming Introduction & FAQ

Discussion in 'Unity Render Streaming' started by kazuki_unity729, Sep 10, 2019.

  1. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    You will have to modify
    RemoteInput::ProcessMouseMoveEvent
    , so that it puts the position in the queue. Something like this, (not tested).

    Code (CSharp):
    1.  
    2. static void ProcessMouseMoveEvent(short x, short y, byte button)
    3. {
    4.     var position = new UnityEngine.Vector2Int(x, y);
    5.     var delta = position - m_prevMousePos;
    6.     InputSystem.QueueStateEvent(RemoteMouse, new MouseState { position = pos, delta = delta, buttons = button });
    7.     m_prevMousePos = position;
    8. }
    9.  
     
    bartburkhardt-sim-ci likes this.
  2. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    The certificate is "invalid", because you created the certificate yourself and it wasn't assigned to you by a trusted Certificate Authority.
    For local testing, this is okay and you should be able to ignore the warning and proceed to see the page.
     
    bartburkhardt-sim-ci likes this.
  3. polygonfuture

    polygonfuture

    Joined:
    Feb 4, 2016
    Posts:
    5
    Yeah, except this does not work when the server webpages are loaded on iOS. The iPad Pro is not able to get past these invalid certificates, and no visuals are displayed. Either in Chrome or Safari.
     
  4. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    It works, but you have to press some buttons to get past it.
    Try googling "safari ios bypass certificate error".
     
  5. bartburkhardt-sim-ci

    bartburkhardt-sim-ci

    Joined:
    Aug 21, 2017
    Posts:
    6
    Cool, that works!. It would be nice to have that position = pos in there in the example code so others can use it as well
    So now I am able to precisely click from the browser in the Unity world using this

    Code (CSharp):
    1.  
    2. var ray = MainCamera.ScreenPointToRay(new Vector2( (x/1280) * Screen.width, (y/720) * Screen.height ));
    3. if (Physics.Raycast(ray, out var hit, 1000))
    4. {
    5.    //do something with  hit.point or hit.collider
    6. }
    7.  
    Thanks!

     
  6. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    Thanks for the suggestion. We are not using position in the demo, but as you said, it would be a good idea to just put it in and let others customize what they want to use with it.
     
  7. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    75
    I like to integrate the UnityRenderStremaing and WebRTC packages into an existing project to provide a prototye for the Windows version. Now the nightly build crashes for all non-Windows platforms like WebGL, Android, ...

    How can I deal with this kind of problem?
    In another project I ended up with manipulating the package manifest in an editor callback method. This worked basically but was a pretty unstable approach.
     
  8. Tobs-

    Tobs-

    Joined:
    Feb 12, 2016
    Posts:
    12
    I can see Unity.webrtc release 2.0 is already in git, but is it actually official released yet? I am asking because I need the multi-camera support :)
     
  9. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    version 2.0 is in development now. I am sorry to have kept you waiting.
     
  10. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    75
    I post the workaround I found for my own problem. I made a copy of the WebRTC package content from /Library/PackageCache/com.unity.webrtc@1.0.1-preview to /Packages/com.unity.webrtc (without version! ) and make the following 2 changes:
    • Unity.WebRTC.Runtime.asmdef:
      Exlude non-Widnows-Platforms
    • Add some #if UNITY_STANDALONE_WIN || UNITY_EDITOR_WIN to Assets/Scripts/RenderStreaming.cs so that it is considered only for Windows (or to whatever classes you have on your own that uses namespace Unity.WebRTC)
    It would be great if the asmdef changes will be included in the next version of WebRTC or an alternative approach like providing dummy DLLs, ... Thanks
     
  11. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    Thank you for the feedback. As you said, the current released version only works on Windows, but we are also working to support other platforms as well. We'll try to see how we can prevent the problems that occurred on your side in the next version.
     
    kayy likes this.
  12. globalikslivov

    globalikslivov

    Joined:
    Dec 6, 2019
    Posts:
    1
    Why web server does not using web socket as signaling mechanism??
     
  13. cchang326

    cchang326

    Joined:
    Dec 6, 2019
    Posts:
    1
    I'm trying the plugin and got the following error for RenderStreaming in the Inspector window:
    The associated script can not be loaded. Please fix any compile errors and assign a valid script.

    I'm following the tutorial. Any idea how to resolve the error?
     
  14. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    I agree to use Websocket as signaling technology but Websocket is not officially supported on Unity.
    If you want to use it, you can try "websocket-sharp" http://sta.github.io/websocket-sharp/
     
  15. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
  16. RanLehr

    RanLehr

    Joined:
    Nov 1, 2018
    Posts:
    1
    How can I use it in Android WebView ?
     
  17. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    I am not sure about that but it might be working or not depends on the implementation of the WebView.

    This article is showing working WebRTC normally on Android WebView. (sorry this is written in Japanese)
    https://qiita.com/v-cube/items/09cf8d59b7a8e7821b89
     
  18. Staale

    Staale

    Joined:
    Apr 3, 2013
    Posts:
    3
    Yes, I've been running this on AWS EC2 for a couple of months now, as a HMI/SCADA interface with HDRP, and I'm really impressed by the low latency.. I chose a g4dn.xlarge NVIDIA Gaming Server with Windows Server. It will cost you around $0.7 per hour, so make sure you shut it down when not using it...

    Edit: I should add that the AWS SDK for .NET works well from the EC2, giving direct Lambda and DynamoDb (and a few other services) access. This, in combination with a Amazon Kinesis Video Stream to a texture in the WebRTC stream is kind of kick-ass, especially if your other AWS stuff is in the same region...
     
    Last edited: Dec 20, 2019
    ninestar09 likes this.
  19. ThomasSheppard99

    ThomasSheppard99

    Joined:
    Nov 18, 2019
    Posts:
    1
    When is version 2.1 likely to be released? I am currently trying to use the render streaming features in a project however we use dependencies which require the OpenGLCore Graphics API for Windows. Is there an estimated time when OpenGL will be supported?
     
  20. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    Unfortunately, NVIDIA codec sdk doesn't supported OpenGL API graphics on Windows. That's why Unity Render Streaming doesn't support OpenGL API graphics on Windows.
    On the other hand, we are working on supporting OpenGL API graphics on Linux. This feature will be released next version.
     
    ThomasSheppard99 likes this.
  21. fuzymarshmello

    fuzymarshmello

    Joined:
    Feb 22, 2015
    Posts:
    12
    Hi, sorry if this has been asked or is easy to solve. everything is working locally, but I want to deploy this to a website but I am confused as to what should be deployed to the client side.

    I've been looking here and have everything done for the webserver with the certificates and keys. Do I have to add a url to the hosts file? How do I point the client side to the server?
     
    Last edited: Dec 19, 2019
  22. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    Have you checked the parameter "URL Signaling" in the inspector?
    This reference page may help you.
     
  23. fuzymarshmello

    fuzymarshmello

    Joined:
    Feb 22, 2015
    Posts:
    12
  24. Jelmer123

    Jelmer123

    Joined:
    Feb 11, 2019
    Posts:
    94
    Hi, what other GPUs can we expect to be supported and when? I have to decide now whether to replace my 2019 macbook pro or not...

    Also, on guthub it says "This solution is optimised for NVIDIA graphics cards."


    This implies that the package also works on other GPUs, albeit with lower performance. Is that correct?

    Although on the github FAQ page it says:
    "Make sure you're using a graphics card that supports Nvcodec"

    this is very conflicting information!
     
  25. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    In the current version, you have to use it with NVIDIA graphics card.
    In the next quarter, we will release a new version that supports the software encoder.
     
    Jelmer123 likes this.
  26. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    Does this mean that want to know how to set a domain to node.js server?
    It is difficult to answer your question because depend on many things there is not a generic solution for that.
     
  27. arielfel

    arielfel

    Joined:
    Mar 19, 2018
    Posts:
    34
    Hi guys, as we excited for the future update regarding the Linux support :)... can tell us if all of this will work in Headless mode?
     
  28. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    I agree with your request, but `Camera.targetTexture` returns an empty image when we use the headless mode. This behavior of runtime is by design.
    Using headless mode excludes graphics processing.
     
    Last edited: Jan 6, 2020
  29. arielfel

    arielfel

    Joined:
    Mar 19, 2018
    Posts:
    34
    You are right, just tried some sample project on Docker (ubuntu 18 with X) it didn't get any pixel data on headless.
    but it did in -batchmode! (and that good enough!! :)
     
  30. sbganesh

    sbganesh

    Joined:
    Dec 1, 2014
    Posts:
    1
    Is it possible to stream the Windows desktop or Unity Editor? If I wanted to make some runtime customization when working on a 3D model concept? If so can you provide me with some guidance?
     
  31. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
  32. Staale

    Staale

    Joined:
    Apr 3, 2013
    Posts:
    3
    Kazuki, could you please post an example for output from Unity to JavaScript in browser?

    I'm guessing it's as easy as e.g. channel.Send("hello"); in RenderStreaming.cs -- and then
    create a ondatachannel with channel.OnMessage in videoplayer.js?
     
    Last edited: Jan 12, 2020
  33. Phoenix_J

    Phoenix_J

    Joined:
    Apr 16, 2016
    Posts:
    1
    Good Afternoon @kazuki_unity729
    I have write a demo for webrtc on unity, and it works well on PC chrome browser.
    Now I want to make a unity application which contains chrome base browser, and use the browser to see the streaming output. After I click the play button, the render streaming application paused and there is a error log "Network Error: Unity.WebRTC.RTCError"
     
  34. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    This page is beneficial for you.
    https://developer.mozilla.org/en-US/docs/Web/API/RTCDataChannel/onmessage

    JavaScript sample
    Code (JavaScript):
    1.  
    2. this.channel.onmessage = function(e) {
    3.   console.log(e.data);
    4. }
    5.  
     
  35. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    Are you trying to embed WebView to Unity app (the client side) to receive the streaming ? And do you also have another Unity app that acts as the server ?
     
  36. varun_naval

    varun_naval

    Joined:
    Oct 20, 2019
    Posts:
    4
    Hi @kazuki_unity729

    The render streaming seems to be working for PC and Mac platforms over a local network as well as via the URL signalling for WiFi networks. However, when I try to test it on a smartphone (Android and iOS) then no rendering occurs. The basic framework shows up on the screen, however nothing after that, whereas the Mac and Windows versions have everything from rendering to input.

    Any idea what the problem might be?

    Also, any timeframe for when microphone input will be accepted from the browser side?

    Thanks
     
  37. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    - What browser do you use?
    - What about disable the firewall?
    - What about launches the signaling server as https?
     
  38. varun_naval

    varun_naval

    Joined:
    Oct 20, 2019
    Posts:
    4
    I got the mobile version working on Android (Chrome) but not on iOS. Tried using Chrome, Firefox and Safari in iOS but it does not seems to work. Are iOS devices supported or is only Android currently supported?

    Also, any update on when microphone input will be added to this component?

    Thanks again for your help and support
     
  39. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    It seems the specification of Safari.

    In Safari and iOS Safari, WebRTC features cannot be used with http. Instead, https must be used.

    https://github.com/Unity-Technologi...Documentation~/en/webserver.md#the-web-client
     
  40. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    75
    I found a similar problem on iOS: It works (of course under https only) but only if the ocnnection goes over Wifi. Whenever I turn off Wifi and switch to a pretty good LTE connection, the OnDataChannel callback is never called. The last Unity console output shows signaling.PostCandidate().
    The same behaviour occurs when I pause Wifi during an established connection. The data channel gets lost reproducably afterwards.
     
  41. varun_naval

    varun_naval

    Joined:
    Oct 20, 2019
    Posts:
    4
    Lastly, when is the microphone input being added or planned for this component?

    Thanks for your help and support
     
  42. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    Sorry I was overlooked.
    We have never tested it with the microphone input.

    You have asked the way to integrate with this API. Right?
    https://docs.unity3d.com/ScriptReference/Microphone.html
     
  43. varun_naval

    varun_naval

    Joined:
    Oct 20, 2019
    Posts:
    4
    Yes specifically accepting microphone input from the browser side to send back to the running project. That would be a very useful feature to include in this already amazing API

    Will be patiently waiting for further development on this component! Thanks again for the help! :)
     
  44. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    I got it.
    This is the same issue you asked.
    We would test the issue sometime soon.
    https://github.com/Unity-Technologies/UnityRenderStreaming/issues/188
     
  45. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    60
    How about Android with an LTE connection? Did you get the same issue?

    > The same behavior occurs when I pause Wifi during an established connection.
    It should be implemented the feature to reconnect when the connection is lost.
     
  46. andrew-newmandilfer

    andrew-newmandilfer

    Joined:
    Dec 16, 2015
    Posts:
    4
    When is Unity going to add in all of the mouse input functions so that we don't have to code around this? This WebRTC capability is amazing, but it's really lacking in the input department - the other glaring issue being that I can't click on UI buttons and wasPressedThisFrame does not get picked up (I realize the browser click may be delayed getting to Unity, but this should be handled). This package has absolutely phenomenal potential, but these issues are HUGE hindrances. I'd just love to know what the roadmap is for addressing them. Thanks for all the hard work though!
     
  47. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    The sample code to handle MouseMove (by displaying a red dot in the correct position) has just been submitted to Github (version 1.1.3).
    We are doing a bit more testing before releasing this version to the public, so please be patient.
     
  48. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    75
    Yes. I just tested it: Wifi works, LTE not. There are 3 calls to Signaling.PostCandidate() and the WebRequests get back without any errors after 1 second. But I observed a different bahavior of RTCIceConnectionState changes in the RTCPeerConnection.OnIceConnectionChange delegate:
    • Wifi to ethernet in the same company network directly jumps to Connected after about 1 second.
    • LTE shows Checking after 6 secs and Failed after another 15 secs.
    The Safari web developer tools on my Mac show the same output from video-player.js.

    I see. The control center suspends the browser and thus the connection is dropped.
     
  49. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    75
    It looks like some firewall problem. The LTE based IP seems to be blocked according to the firewall logs. A similar behavior is found when a Windows host is connected via VPN. On the other hand a Mac with Tunnelblick VPN has no problem. I guess it depends on either the VPN implementation or the VPN connection was preferred.

    I am no networking expert but as far as I understand it, a STUN server might not be sufficient in some cases and a TURN server is needed. Is this correct?
     
  50. markj_pw

    markj_pw

    Joined:
    Oct 27, 2016
    Posts:
    8
    Is there any way I can get RenderStreaming to work whilst in -batchmode on Windows 10? I tried manually grabbing the camera and calling camera.Render() on an update/coroutine. I still get a black screen in my browser (audio and input works though).

    This would be necessary for my setup of running Unity on a server machine.

    Thanks Kazuki, Sindharta
     
unityunity