Search Unity

  1. Curious about what's going to be in 2020.1? Have a look at the 2020.1 beta blog post.
    Dismiss Notice
  2. Want to see 2020.1b in action? Sign up for our Beta 2020.1 Overview Webinar on April 20th for a live presentation from our evangelists and a Q&A session with guests from R&D.
    Dismiss Notice
  3. Interested in giving us feedback? Join our online research interviews on a broad range of topics and share your insights with us.
    Dismiss Notice
  4. New Unity Live Help updates. Check them out here!

    Dismiss Notice

Unity Render Streaming Introduction & FAQ

Discussion in 'Unity Render Streaming' started by kazuki_unity729, Sep 10, 2019.

  1. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    Yes.
    It may need the TURN server to relay around NAT.
    This document shows that integrate "coturn" as a OSS implemented TURN server.
    I am sorry we haven't translated the document into English.

    https://github.com/Unity-Technologi...nderstreaming/Documentation~/jp/turnserver.md
     
  2. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    Specifying -batchmode would still perform rendering normally, unless you are also specifying -nographics ?

    After looking around a bit, it seems that the above answer is not correct. That said though, there should be no problem when camera.Render() is called. Perhaps you can share your project ?
     
    Last edited: Jan 24, 2020
  3. markj_pw

    markj_pw

    Joined:
    Oct 27, 2016
    Posts:
    8
    Thanks for your reply. Knowing that it should work filled me with determination to spend the time to further investigate.

    It turns out the issue was happening only in an older state of RenderStreaming codebase back in October. It looks like the newer codebase uses Unity 2020.3 along with HD Render Pipeline version 7.1.7. Using 2020.2 and HDRP 6.9.0 wasn't working.

    All is well, thanks!
    Mark
     
    Last edited: Jan 25, 2020
  4. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    Fortunately the screenshots are in English and I was able to get it running without speaking Japanese.
    Especially Chrome (Android, iOS and Windows) has still some hickups but this seems to be related to the javascript part. I will investigate this further.

    One more question came up when discussing an n:m scenario i.e. using one big NVidia server for running n instances of the Windows exe on one machine to their content to thin clients per render streaming:
    Do we need more than one server (i.e. node.js) and or TURN server instance to make this happen?

    As far as I understand, the node.js components need to be extended to register separate streaming providers and the logic to connect a client to its favoured exe by request parameter or whatever. But everything should be possible with one node.js service listening on one port. Am I right? Thanks!
     
  5. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    I appreciate your effort!

    Since we have never tested the "N:M scenario", probably the signaling server works incorrectly.
    As you said, the signaling server will need to extend the feature in which the receiving signaling message from multiple Unity instances.
     
  6. markj_pw

    markj_pw

    Joined:
    Oct 27, 2016
    Posts:
    8
    Hey guys,

    I successfully have RenderStreaming working from a Windows Service. This new tech you guys have developed is really cool!

    One issue I have is controlling the resolution/aspect ratio. Since there is no window, Screen.SetResolution() will make the game error and quit. The game seems to have a resolution of 752x516 when I check Screen.width and Screen.height.

    My resulting stream is essentially square, whereas my desired stream would be 16:9, and with a desired resolution for bandwidth reasons. Since my windows service is running on Local System, I don't believe there is a registry entry for setting resolution either.

    Is there a way I can set the rendered resolution without going through Screen.SetResolution?

    Thanks,
    Mark
     
    kayy likes this.
  7. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    Sorry, we have never tried it from Windows Service.
    The resolution might be affected from these command-line arguments.

    -screen-height, -screen-width
    https://docs.unity3d.com/Manual/CommandLineArguments.html
     
  8. markj_pw

    markj_pw

    Joined:
    Oct 27, 2016
    Posts:
    8
    Looks like those arguments don't work when launched from a Windows Service. Interestingly, removing all supported aspect ratios besides 16:9 and setting default screen width and screen height in Player Settings also does not work. It appears that any screen-related settings have no effect.

    Two possible solutions I can think of:
    1) If there is a way to change the resolution of the screen buffer independently of the Screen Resolution.
    2) Point the main Camera to an offscreen RenderTexture of higher resolution, and then point the RenderStreaming to use that as the Capture Camera.
     
  9. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    Mm, To solve this issue we should dig more into the detail.
    I made a ticket on the GitHub issues, we are going to work on the issue in the future.

    https://github.com/Unity-Technologies/UnityRenderStreaming/issues/223
     
    markj_pw likes this.
  10. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    27
    We are preparing to release the next version of RenderStreaming soon, and we noticed this question you posted a while back. If you follow the template, the UI should already be visible in both Unity Editor and browser. Just press any key on the browser side, and you should see it.

    upload_2020-2-5_15-56-11.png
     
  11. markj_pw

    markj_pw

    Joined:
    Oct 27, 2016
    Posts:
    8
    One small note on the ticket: the issue is with the Unity Screen.width & Screen.height not being able to be changed. Unity will stop with the exception of not being able to change the window resolution (there is no window).

    The way it relates to streaming resolution, at least in my experience, is that setting the streaming resolution to any value other than the window resolution will result in distorted and/or cropped video in the browser.

    Thanks,
    Mark
     
  12. eproca_SimRTR

    eproca_SimRTR

    Joined:
    Nov 8, 2018
    Posts:
    3
    I managed to implement renderstreaming in a little test project and everything went more or less smooth. It is really cool. Some notes:

    As others as pointed out, it was not possible to make it to work in MacOSX or IOS Safari. I followed the full process for “Configure Self-Signed Credentials for https” and bypass the security warning….but not. The web server is connecting, but not streaming.

    The current solution seems to be thought to integrate the render streaming as part of a web page, with the (clear and sharp) UI provided by the web page itself, and not by the Unity instance running. Any UI text element provided in the streaming (with canvas render mode in “Screen Space – Camera”) is looking not really great, and a bit awful with any camera movement. Is there any plans to provide a clear a sharp UI directly from the Unity instance running?

    As others as pointed out, one big potential of this is having several Unity instances running independent streaming in some sort of GPU server. Any chance for having a template with a web server for this case scenario?
     
  13. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    Make sure that your iOS device is connected over WiFi and not via mobile network. Mobile data connections work fine but require a more sophisticated STUN/TURN server than the provided default test server stun.l.google.com.
     
  14. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    Some news about this: I got it working so far. After extending the signaling server to show a list of connected server instances, I can click on the one I am interested in and then get it streamed to the browser. That way other browsers can join in to the same session or connect to another serving instance. I tested successfully with 3 different machines serving 2 instances on each machine. This is really exciting, thanks a lot for your great work :)

    Now the next scenario includes running a larger number of instances on each machine but starting the 3rd instance makes WebRTC.Initialize() return the status Initialization Failed. A colleague of mine found out that there are some restrictions regarding the number of concurrent sessions according to the NVidia Support Matrix Document.

    I saw that most recent release M72 of WebRTC provides the ability to use a software encoder and seems to run on Linux/macOS. I installed the M72 source code package from the Releases page but cannot get it to work. WebRTC.Initialize() runs but Context.GetCodecInitializationResult() returns NotInitialized. I updated to 2019.3.2 and tried it on my Mac too (there with 2019.2.18 and having the tests deleted) but with no success.
    1. Any chance to get M72 working?
    2. Are we right that the limit of 2 instances is caused by the NVidia Encoder's Max # of concurrent sessions == 2?
    3. I don't see any library log output althoughWetRTC.Init has NativeMethods.RegisterDebugLog as first statement but it seems to be never called. What can I do to get more information about what's going on inside?
    Thanks in advance.
    Kay
     
  15. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    In your situation, Already kayy pointed out, it seems to need to use a TURN server.

    This is expected due to the nature of encoding to save bandwidth, and therefore clear and sharp UI directly from UI may unfortunately be hard, if not impossible, to achieve.
    A couple of options to reduce the artifacts:
    1. Stream with a higher resolution
    2. We are planning to add bitrate support. We can’t say when we are going to release this feature, but it is expected that you might be able to tune the quality.
    We have not yet provided the web server sample for that. If there are many requests about that, we may need to make a sample or publish a document.
     
    ninestar09 likes this.
  16. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    Awesome!

    Right now we are preparing to release new version that support Linux/macOS. Please wait a time!

    Yes, usually NVIDIA gaphics card makes restriction the count of concurrent streams.

    This method is called from only main thread at the moment, because UnityEngine.Debug.Log may cause a crash if called in other threads (Rendering, etc). It is possible to omit this check in debugLog in the plugin code to output debug logs.Another better alternative would be to output the logs in multiple files separated by threads, but we don’t have this feature yet.
     
  17. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    I am looking forward to the new release.
    Ah I see, WebRTCPlugin.cpp. I think omitting the check should be fine. From my understanding Debug.Log itself is thread safe and I used it in the past when doing asynchronous MQTT messaging from HoloLens without any problems. But of course it will cause exceptions if the callback method tries to access other stuff from the Unity API like GameObject.
     
  18. tsfk

    tsfk

    Joined:
    Feb 10, 2020
    Posts:
    2
    Is it possible to have this solution stream uncompressed video?

    I have the solution running locally.

    I understand that WebRTC and nvenc only supports compressed formats.

    I wonder what it would take to stream in a specific uncompressed format.
     
    Last edited: Feb 24, 2020
  19. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    Could you let me know what kind of uncompressed format do you want?
    Usually, most browsers don't support the uncompressed format in my thought.
    Also, we would like to know what you want to do with that.
     
  20. tsfk

    tsfk

    Joined:
    Feb 10, 2020
    Posts:
    2
    A raw YUV 4:2:2. I am exploring options for streaming video over a local connection through udp; not to be received by a browser. I realize I'm not after all the other cool WebRTC functionality, and that this might be the wrong place to ask.

    I would be happy if you could point me to or explain how frames are captured and prepared for transfer.
     
  21. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    I am sorry but it seems difficult to promise support for your request as it looks like a special situation.

    everyone:
    Please let me know if you have the same issue.
     
  22. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    Hi developers,

    We have released the new version for Unity Render Streaming.
    For the detail, please check it out on the other thread. And have fun!
     
    fuzzy3d and kayy like this.
  23. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
  24. gtk2k

    gtk2k

    Joined:
    Aug 13, 2014
    Posts:
    12
    I want to be able to send my own RenderTexutre stream.
     
    markj_pw likes this.
  25. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
  26. tk_sg

    tk_sg

    Joined:
    Mar 2, 2020
    Posts:
    4
    Hi, the same problem as the following URL occurred:
    https://forum.unity.com/threads/unity-render-streaming-introduction-faq.742481/page-2#post-5109068
    I am experiencing video freezing and grey screen when my (Chrome) browser is on a screen that is 2880 x 1620 and larger.

    My environment was using only localhost, the Unity server was connected from a single client web browser.
    I could stream 2650x1600 in my environment, but I expected Unity render streaming to be able to stream 4K (3840x2160) 25fps frames.
    Please tell me where to fix or check.

    I look forward to hearing from you.
     
  27. gtk2k

    gtk2k

    Joined:
    Aug 13, 2014
    Posts:
    12
    @tk_sg
    Predict if the size of the monitor is relevant.
    My monitor is 5120x1440, but it succeeded at 3840x1440 but not at 3840x1920.
     
    Last edited: Mar 4, 2020
  28. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    We are very interested for your challenge to stread 4k resolution, but there are some challenges.

    One of the most important issue is the network bandwidth.
    As you can see on this site, distributing a 4k video requires the internet speed in the range of 15-20 Mbps.
    https://insights.enablex.io/4k-video-webrtc/
     
  29. r3dok1

    r3dok1

    Joined:
    Feb 16, 2020
    Posts:
    1
    Can you stream to unity 4 cameras using WebRTC for composing a single image?
     
  30. tk_sg

    tk_sg

    Joined:
    Mar 2, 2020
    Posts:
    4
    Your information is very interesting, but confused me...

    I was able to stream 2880x1620 using a FHD (1920x1080) monitor.
    However, neither 4K (3840x2160) nor FHD monitors could stream 3840x2160.
     
    gtk2k likes this.
  31. tk_sg

    tk_sg

    Joined:
    Mar 2, 2020
    Posts:
    4
    Since all applications (Unity, signaling server, and browser) were running on the same PC, so I thought there would be no network bandwidth issue.

    I accessed all local URLs ( local address, loopback address, and "localhost") presented from the node.js web app, but the same problem occurred.
     
  32. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    I appreciate you sharing the issue. We'll investigate it.
    This is the ticket for the issue, and we are going to work on it soon.

    https://github.com/Unity-Technologies/UnityRenderStreaming/issues/236
     
  33. tk_sg

    tk_sg

    Joined:
    Mar 2, 2020
    Posts:
    4
  34. MW_MWJ

    MW_MWJ

    Joined:
    Mar 13, 2019
    Posts:
    1
    Hi,

    Touch Phase always enters Stationary when I added the volume component.

    Do you know how to solve this problem?

    It's okay when I play on the editor, but only on the build.
     
    Last edited: Mar 11, 2020
  35. James_Bentley

    James_Bentley

    Joined:
    Mar 11, 2020
    Posts:
    1
    Hello,

    Sorry if this is quite a beginner question, but I'm fairly new to web stuff! I am trying to host the Unity Render Streaming template on a computer, however I want to be able to access the stream from a device on a different network.

    I have read the documentation for the webapp and the streaming etc, but I am struggling to make sense of it.

    Does anyone have a direction they could point me in for accessing the locally hosted application and stream from different networks?

    Thanks :)
     
  36. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
  37. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    I am glad for you interested in our product.
    Have you read the tutorial document? Please let me know what you feel difficult.
    https://docs.unity3d.com/Packages/com.unity.renderstreaming@1.2/manual/en/tutorial.html
     
  38. gtk2k

    gtk2k

    Joined:
    Aug 13, 2014
    Posts:
    12
    kazuki_unity729 likes this.
  39. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    I tested the new WebRTC release 1.1.1-preview. It works fine out of the box on Windows with both hardware and the software encoder.

    I then tried to make it working on macOS with the same code base. The browser client connects to my app but the videoPlayer stays black. I can see that the connection is established and data is transferred because RemoteInput is working. about:webrtc shows a significant amount of bytes sent although it's less than a working video stream takes.

    The log is very similar to the one from Windows but there are a few things I don't know.
    At the top threetimes:
    (ice/WARNING) z:/task_1583751780/build/src/media/mtransport/third_party/nICEr/src/net/nr_socket_multi_tcp.c:617 function nr_socket_multi_tcp_listen failed with error 3

    Then at the end again threetimes:
    (ice/ERR) ICE(PC:1584532964607000 (id=2147483651 url=https://myhost/myurl)): peer (PC:1584532964607000 (id=2147483651 url=https://myhost/myurl):default), stream(PC:1584532964607000 (id=2147483651 url=https://myhost/myurl) transport-id=transport_0 - 281605e7:619a730ca0742dc799bef4041272ff0d) tried to trickle ICE in inappropriate state 4

    Any hints what I am doing wrong here? Thanks
    ___________
    Update: I found out that the Codec initialization always fails. Even if I am waiting for 30 seconds WebRTC.CodecInitializationResult Always returns NotInitialized
    But I don't have yet any idea what is the cause.
     
    Last edited: Mar 18, 2020
  40. gdbbv

    gdbbv

    Joined:
    Apr 20, 2018
    Posts:
    6
    It is working as expected with the exception of iOS 13.3.1 Safari with iPhone 11 Pro. It works on Safari with iPad Pro (which has a full version of Safari). https was needed for both (as the documentation suggested) but adding TURN server (according to the instructions which I translated from Japanese) didn't make a difference even though the messages on this forum suggested that it might. The results are the same over LTE or WiFi. On iPhone only the first frame of the video stream is shown. The touch events are still transmitted and affect the server application and the other connections but no video update on the iPhone 11 connection. Any ideas? I tried most of the iOS Safari experimental flags but still no luck.
     
  41. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62

    Thank you for sharing.
    Unfortunately, we have not reproduced your issue yet.

    Please let me know the below
    • Which browser and version you are using
    • The result that trying the latest version of the template project

    WebRTC.CodecInitializationResult has been deprecated since this version.
     
  42. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    62
    We already tested using iOS Safari's latest version but we can't reproduce the issue you reported.
    Can you give us more info, for example, by attaching the console log of your browser?
     
  43. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    Thanks for your reply. My original post was related to upgrading to com.unity.webrtc 1.1.1-preview (1.1.2 was not out at the time of writing). I already worked for a while with 1.0.1 which runs fine in our project but without the template.

    I just tested the template project out of the box and got it running with a packed webserver. I did this similar to the Tutorial but on macOS instead of Windows. I will upgrade to 1.1.2 and then check for diffs between our own and the tutorial's code.

    Hmm, I don't understand. It's still there in WebRTC.cs' version from Fri 20/03/2020 and there is no obsolete marker.
    Anyway this was a mistake on my part. I have learned that WebRTC.InitializeResult() itself does to not change state as long as the call to CaptureStream is performed successfully.

    Update: 1.1.2 did not change anything. Both Firefox and Chrome show the same behaviour: browser console and Unity log appears fine and remote input is working but no video. Again I see these " tried to trickle ICE in inappropriate state 4" messages at the end in protocol section of about:webrtc.
    I made a couple of extensions to the code and there might be some race condition or other timing issues. However as macOS is not that important, I will check this at some point in the future - maybe it's gone then.
    Thanks anyway
     
    Last edited: Mar 23, 2020
  44. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    I am working on some way to perform resizing of the app (and the render texture output dimensions) triggered by the browser. The use case is to provide the users with the possibility to resize the browser and finally adjust the sending app's dimensions so that the stream perfectly fits into the browser window i.e. listen to javascript's resize event and fire a message after a time out.

    Based on the UnityRenderStreaming sample I managed to get a prototype working. As render textures cannot be resized, the only way I found is to call WebRTC.Finalize() and then .Initialize() again with the new dimensions. This can be achieved by destroying and instantiating the RenderStreaming game object. This works so far, but an exception is thrown when I teminate the application. It occurs when the CleanerCallback tries to release the RenderTextures a second time. To avoid this I patched the MediaStream.cs code temporarily and made Cleaner public so that I can destroy the component before calling WebRTC.Finalize(). Not the best way but works as expected.

    My Questions:
    1. Can you provide a way to cleanly start and stop WebRTC more than once?
    2. Or even better: Do you have any idea for a better approach?
    Thanks
     
    gtk2k likes this.
  45. eproca_SimRTR

    eproca_SimRTR

    Joined:
    Nov 8, 2018
    Posts:
    3
    Just tested 1.2.3. Everything went smooth, but this time I didn't check IOS or MacOS. (In windows and Android is fine).
    In local host or in the local network, it looks the same, but users accessing truly remotely to my web server notice a better performance than the previous version (1.1).

    I am streaming 1080p with an optional internal oversampling of 1.5X or 2X. With minimum bit rate of 50000, there is not obvious compression artifacts when using Chrome. The quality is pretty good and the latency is very reasonable. Of course you notice a difference compared with running directly the app in local. But it is quite good for what it is.

    As I have already commented, the issue that is not possible to have a clear and sharp UI directly from Unity is very frustrating. I understand that because of the nature of the encoding, if the UI is included in the video feed to be compressed, it is probably not possible (i have tried very high resolutions and bit rates, and if the camera moves it is always very bad). But maybe it is possible to send the UI in another way different that included in the compressed video feed.

    I have the feeling that the current version of the Unity Render streaming is too much thought for very specific case scenarios. Like for example car web configurators, when the UI will be always in the web side. And it is really a pity, because the potential is huge.

    It is interesting to compare with a similar Unity app running via web in Furioos (recently acquired by Unity). The performance when using the Unity Render streaming is a lot better, provided that you use Chrome and a high bit rate. But in Furioos, the "normal" Unity UI looks perfect. Maybe I am comparing apples with oranges, I don't know.....
     
    Cascho01 and gtk2k like this.
  46. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    1,161
    Thanks for sharing your experiences.
    I also tried Furioos and I´m happy with its easyness and the streaming quality.
    Yes, Unity bought it, but there´s no information about how/if we could benefit from it.

    I heard about camerastacking in Unity2020, maybe it is possible to render the GUI into a second texture, blend it over the cameras rendertotexture and finally stream this blended texture....
     
  47. gtk2k

    gtk2k

    Joined:
    Aug 13, 2014
    Posts:
    12
    kazuki_unity729 and kayy like this.
  48. gdbbv

    gdbbv

    Joined:
    Apr 20, 2018
    Posts:
    6
    This is what I get consistently and I can't figure out how to "allow" for such permissions. Was able to get it to update temporarily (and not giving the error) but still not sure why ... and then it went back to giving the error and not updating the video.

    Screen Shot 2020-03-24 at 8.10.45 AM.png
     
    Last edited: Mar 25, 2020
    jimstewartson likes this.
  49. kayy

    kayy

    Joined:
    Jul 26, 2011
    Posts:
    76
    I got it working on macOS! :)
    The player settings were to blame for the black frame:
    Other settings / Rendering / Color Space has be set to Linear, Gamma does not work with software encoder on macOS. But I have no explanation why the same settings work fine on Windows with the software encoder.
     
  50. jimstewartson

    jimstewartson

    Joined:
    Jun 8, 2018
    Posts:
    2
    Hi! Is there a way to have the webserver listening on a port other than 80?
     
unityunity