Multithreaded rendering and GL.IssuePluginEvent

Hi,

Background:

A few days ago I started noticing problems with some plugins I wrote that issue OpenGL commands. One of those plugins is mainly for uploading textures with custom formats to the GPU, since Texture2D.SetPixels is rather limited (i.e., it only understands ARGB32, RGB24 and Alpha8). The other plugin I wrote is for creating textures with embedded text using the device's system font.

About the problem:

The new multithreaded rendering introduced by Unity 3.5 is really cool but it introduces some limitations to how we should develop plugins that issue OpenGL commands. That is, we should issue OpenGL commands from the same one thread: the rendering thread. OK, now into the problem I am having. My plugins issue "glBindTexture" using the native id of the texture created by Unity. On mobile platforms, there is (currently) no problem but on the desktop where Unity does multithreading the rendering thread may be issuing a batch of OpenGL commands, for instance a glBegin ... glEnd and in-between those batch commands, the plugin can call "glBindTexture" from another thread (not the rendering thread) putting the OpenGL context in an invalid state. Normally, glBindTexture will just fail with a GL_INVALID_OPERATION (and that's fine), but it may even crash for lengthier operations such us uploading textures with glTexImage2D, and so on.
When I first saw the manifestation of this problem I didn't immediately realize what was the core issue behind the curtain, as I have just moved recently my projects from 3.4 to Unity3.5, but as soon as I understood the real issue here I checked the official documentation again on how to write native plugins for Unity and found out the following:

http://unity3d.com/support/documentation/Manual/NativePluginInterface.html

The documentation above, mainly refers the existence of a new still undocumented API called GL.IssuePluginEvent for calling your plugin on the rendering thread. On the plugin's side, you will have to provide a void UnityRenderEvent(int eventID) callback.

Discussion and proposal an improvement:

Let's be fair. Even if we say that "currently, Unity does not use multithread rendering on mobile platforms", that won't be true forever. Right now we have iPad 2 and iPhone4S that only have 2 cores, and that's the main reason why "there's no much point in using multithreading", but in the very near future iPad3 and most probably iPhone5 too will have 4 or cores at least. Of course, when that time comes Unity will probably enable multithreading on mobile platforms too, as it does already on the desktop platforms. My problem here is how to easily develop and, more importantly, maintain a plugin not only for mobile platforms (namely Android and iOS) but also the desktop in order to be able to see equivalent results on my development environment. I would like to discuss and ask for advices on that subject.
First, I find GL.IssuePluginEvent API a pita because 1) you cannot specify "which" plugin's UnityRenderEvent you want to call and 2) because you can only specify 1 single parameter: an integer that will be passed into your plugin's UnityRenderEvent (you can do whatever handling you want based on this parameter).
Regarding point number 1, on iOS I can't see a way where you could just use several independent rendering plugins using this new UnityRenderEvent API. For instance, you would put them all under Assets/Plugins/iOS folder but during compilation Unity woud find several UnityRenderEvent symbols being exported and, therefore, you have a problem. Probably this is different for the desktop where you can build .bundles and .dlls and export a symbol for each dll. The only way I see would be creating a middleware plugin that would be responsible for handling UnityRenderEvent, and depending on the ID integer it gets it would dispatch the request to the plugin that knows how to deal with the given ID.
Regarding point number 2 (which is also kind of related to point number 1), since you cannot pass parameters to the plugin but just "magical" a ID integer, I think the "easy" way to pass custom arguments into the plugin would be to call a function on the plugin (as you would normally do) with the parameters you want to call including an ID that (on the Unity side) you would issue for identifying those parameters. The plugin, would have to manage a list of registered parameters and the corresponding IDs for pending "requests". Then, you could call GL.IssuePluginEvent using that ID, so the plugin would know what parameters are associated to the ID. While this is possible, it is a real pain in the ass which becomes even more painful if you use several rendering plugins, because each plugin would have to maintain a list of its own "issued IDs" so that each plugin would know what IDs to process and what IDs to ignore (since all plugins exporting UnityRenderEvent will get called back with an ID even if its not its own).
I thought of the following ways to alleviate the "pain". From Unity, rather than calling a GL.IssuePluginEvent, it would be nice to have at least one the following 2 APIs:

Code (csharp):

1.  
2. // Synchronize with Unity rendering thread (It somehow tells the rendering thread
3. // to stop temporarily in a "sane" state, while we execute our callBack called
4. // mySyncRender. Since I decided to prefix the API name with "Sync",
5. // "more code here" would not be executed until mySyncRender is called and
6. // finishes executing. As an extra it would also be nice to get the return value
7. // from UnityEngine.SyncRender as the value returned by mySyncRender itself:
8.  
9. UnityEngine.SyncRender(mySyncRender);
10. // more code here...
11.  

Code (csharp):

1.  
2. // The previously proposed API would be useful for non-frequent/non-incessant
3. // rendering, where you would not have a considerable performance hit, but
4. // following the same line of GL.IssuePluginEvent, it would be also nice to
5. // have an API like the one below. This API would be a mix of the previous
6. // UnityEngine.SyncRender(mySyncRender) and GL.IssuePluginEvent: you
7. // would register myAsyncRender to be called back from the rendering thread
8. // without waiting for the call to complete going and without going through all the
9. // hassle of asking the render thread to stop executing temporarily:
10.  
11. UnityEngine.AsyncRender(myAsyncRender);

My purpose is to develop useful standalone plugins (i.e., that can exist by their own without having to using some middleware glue for getting parameters from event IDs and stuff) and, if possible, to be able to reuse most of the code for several platforms despite the inevitable differences.

I know it's easier said than done, but please let me know of your thoughts on this.
I am also curious on how you are handling these new changes introduced by Unity3.5.

Best regards

Final Note:

Regarding my second plugin, I think I will probably do the device font rendering part on the native plugin side, and rather than overwriting the OpenGL texture directly on the plugin side too, I think it will be easier to pass the generated bitmap buffer to the Unity side and then deal with Texture2D.SetPIxels and so on. I don't like the fact that I will have to allocate/pin/copy buffers but in the meantime (while there are no better APIs for that purpose) I find it the easiest way to do it.

 

Hi Kaspar,

Thanks, that would be highly appreciated!
Looking forward to new developments!

 

I've just come across this problem myself. Had some lovely working c# code using ​InteropServices, e.g

Code (csharp):

1. [System.Runtime.InteropServices.DllImport("​opengl32.dll", EntryPoint = "glBindTexture", ExactSpelling = true)]
2. internal extern static void BindTexture(int target, uint textureID);
3.      
4. [System.Runtime.InteropServices.DllImport("​opengl32.dll", EntryPoint = "glTexSubImage2D", ExactSpelling = true)]
5. internal extern static void TexSubImage2D(int target, Int32 level, Int32 xoffset, Int32 yoffset, Int32 width, Int32 height, int format, int type, IntPtr pixels);

That no longer works in 3.5, I assume due to multi-threaded rendering? What was nice about this was it didn't need a c/c++ plugin by virtue of how opengl works and could be made to work cross-platform (Mac PC).

I've been trying to find a replacement without resorting to a c/c++ plugin, but so far without success. As far as I can tell you can't simply make a c# plugin as it can't export global functions in order to tie into the new multi-threaded commands/callbacks. Neither is there a method to get the rendering thread Unity is using?


So as far as I can tell there is no longer any way to do the above without resorting to writing per platform plugins in c/c++ in 3.5?

If so is there anything that could be added to a future Unity release that might give us back that sort of access without resorting to a plugin?