Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

[RELEASED] [OSS] .NET bindings for WebRTC, with Unity package

Discussion in 'Assets and Asset Store' started by akihikodaki, Oct 2, 2019.

  1. akihikodaki

    akihikodaki

    Joined:
    Aug 4, 2018
    Posts:
    1
    pixiv Inc. released .NET binding library for WebRTC, and is available on GitHub now!

    Releases
    Release 77.0 · pixiv/webrtc
    https://github.com/pixiv/webrtc/releases/tag/pixiv-m77.0

    What you can achieve with this library
    You can implement video/audio communication on a Unity application via WebRTC. WebRTC is:
    • A standardized protocol implemented by major browsers and mobile applications.
    • The protocol is capable of P2P communication behind firewalls.
    WebRTC provides efficient communication between clients for wide use.

    This library is based on the native API, which is somewhat low-level. This means:
    • You can do whatever with inputs/outputs. For example, a server-side application may render a video on UnityEngine.RenderTexture offscreen, and broadcast it with this library, using the texture as an input. Another example is the case that UnityEngine.Texture2D is set as an output, and the texture is mapped to a 3D object. The capability provides infinite creativity when combining it with Unity's powerful features such as 3D rendering, post processing, AR, etc.
    • You can extend it with C++. The reason to extending the library may vary. A platform-specific optimization can be added using a hardware encoder. Another reason is to give an access to hardware functioning as inputs/outputs of audio/video with C++ APIs of the target platform.
    • More code is required. I'm pretty sure that a skilled Unity developer can deal with that, but it costs time anyway...
    • The details of WebRTC communication must be negotiated with a independent means. The process is called signaling, and you have to implement it by your own. A simple configuration is not that difficult, but the more knowledge of WebRTC is required for the finer configuration.

    This library is carefully crafted to avoid even minor pitfalls of native binding to provide maximum compatibility and maximum stability (include thread safety) at minimum costs:
    • Advanced features of the runtime like System.GCHandle are employed if necessary.
    • C++11 (which WebRTC uses) and C99 (required by P/Invoke a.k.a native plugin) are respected even in details like language linkage of function pointers.
    • Lifetime of C++ objects represented in the WebRTC library by extensive use of smart pointers is respected and mapped carefully to C# utilizing basic infrastructure like System.IDisposable.
    • Resources provided by authorities such as ISO C++, Microsoft, Mono, and Unity were examined during the development. They describe common best practices as well as specifics such as Mono's unique P/Invoke threading model and IL2CPP requirements.
    This library utilizes WebRTC's own build system. It brings portability and further improvements from the upstream can be expected.

    Comparison with Unity Render Streaming
    Unity Render Streaming is a similar feature provided by Unity. However, there are number of differences due to different approaches, different goals, and different development stage.
    • No NVIDIA encoder support on Windows
    • This library is compatible with Android, Linux, Windows, iOS, and macOS. Unity Render Streaming currently (1.0) only supports Windows with NVIDIA encoder. Version 2.1 is expected to have Linux support.
    • This library supports both of receiving and sending, while Unity Render Streaming only supports sending.
    • This library is more up-to-date. Based on WebRTC implemented in Chromium 77.
    • Compatible with IL2CPP.
    • This library allows you to extend WebRTC with C++, while Unity Render Streaming is likely to need some code change.

    Learning how to use
    Basic installation
    Download com.pixiv.webrtc.tgz of the release you would like to install, and specify it as com.pixiv.webrtc package. Unity's documentation has detailed descriptions:
    https://docs.unity3d.com/Manual/upm-localpath.html

    Example
    Code tells you many things you are likely to want to know. See https://github.com/pixiv/webrtc/tree/branch-heads/pixiv-m77/examples/unity for an example. Note that the example requires Sora, an SFU implementation sold in Japan. It is still a good point to start even if you have to communicate with another signaling server implementation.

    Native API
    The API is based on the native API. See C++ header files included in https://github.com/pixiv/webrtc/tree/branch-heads/pixiv-m77. Header files include detailed descriptions of the API. The use of smart pointers implies expected lifetime of objects.
    webrtc.org has a page about the native API, but it basically says "read the headers" (https://webrtc.org/native-code/native-apis/)

    Web API
    The native API is not that different from the Web API. MDN provides a nice documentation at https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API.
    W3C has the specification of the Web API. Header files also often refers to the specification. https://w3c.github.io/webrtc-pc/

    Signaling and other server-side facilities
    There are several server implementations, and some are available as SaaS. They may provide reference client signaling implementations for Web browsers. In such a case, you may simply copy JavaScript, transform the syntax, and replace each API calls with ones provided by this library, effectively translating it to C#.

    Questions, suggestions, ideas, discussions, etc.
    WebRTC protocol and underlaying C++ codebase are complex and not easy to understand. Please feel free to ask questions if you have a difficulty.
    Questions about the internal implementation will also be welcomed. Hopefully it gives you an insight if you are building a library like Unity Render Streaming by your own, or writing a binding specific to your application.
    Suggestions for further improvements will be appreciated. If you have an idea of application, please share here. We may improve the binding to allow you to make it real if necessary. (e.g. adding hardware encoder support for Windows, data channel support, ...)

    Contributions, bug reports
    Contributions and bug reports will be greatly appreciated. An issue can be opened at https://github.com/pixiv/webrtc/issues. Please open a pull request for https://github.com/pixiv/webrtc on GitHub to contribute changes.
     
    Last edited: Oct 8, 2019