[RELEASED] [OSS] .NET bindings for WebRTC, with Unity package

pixiv Inc. released .NET binding library for WebRTC, and is available on GitHub now!

Releases
Release 77.0 · pixiv/webrtc
https://github.com/pixiv/webrtc/releases/tag/pixiv-m77.0

What you can achieve with this library
You can implement video/audio communication on a Unity application via WebRTC. WebRTC is:

• A standardized protocol implemented by major browsers and mobile applications.
• The protocol is capable of P2P communication behind firewalls.

WebRTC provides efficient communication between clients for wide use.

This library is based on the native API, which is somewhat low-level. This means:

• You can do whatever with inputs/outputs. For example, a server-side application may render a video on UnityEngine.RenderTexture offscreen, and broadcast it with this library, using the texture as an input. Another example is the case that UnityEngine.Texture2D is set as an output, and the texture is mapped to a 3D object. The capability provides infinite creativity when combining it with Unity's powerful features such as 3D rendering, post processing, AR, etc.
• You can extend it with C++. The reason to extending the library may vary. A platform-specific optimization can be added using a hardware encoder. Another reason is to give an access to hardware functioning as inputs/outputs of audio/video with C++ APIs of the target platform.
• More code is required. I'm pretty sure that a skilled Unity developer can deal with that, but it costs time anyway...
• The details of WebRTC communication must be negotiated with a independent means. The process is called signaling, and you have to implement it by your own. A simple configuration is not that difficult, but the more knowledge of WebRTC is required for the finer configuration.

This library is carefully crafted to avoid even minor pitfalls of native binding to provide maximum compatibility and maximum stability (include thread safety) at minimum costs:

• Advanced features of the runtime like System.GCHandle are employed if necessary.
• C++11 (which WebRTC uses) and C99 (required by P/Invoke a.k.a native plugin) are respected even in details like language linkage of function pointers.
• Lifetime of C++ objects represented in the WebRTC library by extensive use of smart pointers is respected and mapped carefully to C# utilizing basic infrastructure like System.IDisposable.
• Resources provided by authorities such as ISO C++, Microsoft, Mono, and Unity were examined during the development. They describe common best practices as well as specifics such as Mono's unique P/Invoke threading model and IL2CPP requirements.

This library utilizes WebRTC's own build system. It brings portability and further improvements from the upstream can be expected.

Comparison with Unity Render Streaming
Unity Render Streaming is a similar feature provided by Unity. However, there are number of differences due to different approaches, different goals, and different development stage.

• No NVIDIA encoder support on Windows
• This library is compatible with Android, Linux, Windows, iOS, and macOS. Unity Render Streaming currently (1.0) only supports Windows with NVIDIA encoder. Version 2.1 is expected to have Linux support.
• This library supports both of receiving and sending, while Unity Render Streaming only supports sending.
• This library is more up-to-date. Based on WebRTC implemented in Chromium 77.
• Compatible with IL2CPP.
• This library allows you to extend WebRTC with C++, while Unity Render Streaming is likely to need some code change.

Learning how to use
Basic installation
Download com.pixiv.webrtc.tgz of the release you would like to install, and specify it as com.pixiv.webrtc package. Unity's documentation has detailed descriptions:
https://docs.unity3d.com/Manual/upm-localpath.html

Example
Code tells you many things you are likely to want to know. See https://github.com/pixiv/webrtc/tree/branch-heads/pixiv-m77/examples/unity for an example. Note that the example requires Sora, an SFU implementation sold in Japan. It is still a good point to start even if you have to communicate with another signaling server implementation.

Native API
The API is based on the native API. See C++ header files included in https://github.com/pixiv/webrtc/tree/branch-heads/pixiv-m77. Header files include detailed descriptions of the API. The use of smart pointers implies expected lifetime of objects.
webrtc.org has a page about the native API, but it basically says "read the headers" (https://webrtc.org/native-code/native-apis/)

Web API
The native API is not that different from the Web API. MDN provides a nice documentation at https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API.
W3C has the specification of the Web API. Header files also often refers to the specification. https://w3c.github.io/webrtc-pc/

Signaling and other server-side facilities
There are several server implementations, and some are available as SaaS. They may provide reference client signaling implementations for Web browsers. In such a case, you may simply copy JavaScript, transform the syntax, and replace each API calls with ones provided by this library, effectively translating it to C#.

Questions, suggestions, ideas, discussions, etc.
WebRTC protocol and underlaying C++ codebase are complex and not easy to understand. Please feel free to ask questions if you have a difficulty.
Questions about the internal implementation will also be welcomed. Hopefully it gives you an insight if you are building a library like Unity Render Streaming by your own, or writing a binding specific to your application.
Suggestions for further improvements will be appreciated. If you have an idea of application, please share here. We may improve the binding to allow you to make it real if necessary. (e.g. adding hardware encoder support for Windows, data channel support, ...)

Contributions, bug reports
Contributions and bug reports will be greatly appreciated. An issue can be opened at https://github.com/pixiv/webrtc/issues. Please open a pull request for https://github.com/pixiv/webrtc on GitHub to contribute changes.

 

Hi,

I am in the need of building a unity app where I can use webrtc on ios and android and webapps for screensharing. So far I read through some explanations of webrtc and did the codelab example from Firebase Wbrtc to have a simple signaling server using firebase: https://github.com/webrtc/FirebaseRTC/tree/solution

Is it possible to implement this basic signaling process with these bindings in Unity?
And if so could you maybe give me some guidiance on how to "translate" this simple javascript
signaling into the .Net bindings?

I was looking into the unity exampe of the bindings: https://github.com/pixiv/webrtc/tree/branch-heads/pixiv-m78/examples/unity but I dont have access to Sora so I couln't run it.
It is quite hard to figure out how everything works together and I am not even sure wether I am on the right track with trying to do the firebase-webrtc signaling with these bindings.

I also started to look into the header files but for me there is no obvious point of entry in terms of how to start.

Maybe there are already solutions for a signaling server which works right away with these bindings (which is not japanese only)?

Best regards
Lukas

 

Hi, Sorry for late reply.

Recently I pushed an implementation compatible with the example server implementation located at "examples/peerconnection/server" in the repository. You may run the example to get WebRTC work first, and then start to implement your signaling implementation. Note that the server implementation and its official client implementation, located at "examples/peerconnection/client" is written in C++ and somewhat messy.

You may start implementing your signaling code by reading header files located at "api" directory. Although they are in C++, they have detailed documentations and resemble Web APIs quite a well. If you find an appropriate C++ API, see how their .NET bindings are used in "examples/unity".

I'm actually not actively watching the Unity forum. I may be able to respond quickly on GitHub. Feel free to open an issue there; it is OK to open an issue for a question.