Render image into shared memory

Hey, first forum post so go easy on me! I am building a simulation in Unity which interfaces with a collection of C++ programs. I need to render several cameras and then copy those images to my C++ program.

So far it seems like the .NET

MemoryMappedFile

class is the way to do this, but I want to make sure the latency is as low as possible.

From what I've read,

Texture2D.ReadPixels

can cause "pipeline stall" and may wreck performance. The images I want to stream are relatively small (256x256) but there may be 2-4 cameras. I'm not using the render texture for any visual effects, I just need to stream it to another application.

Is there any way to copy a RenderTexture directly into shared memory, rather than copying to a Texture2D, then copying from that?

 

Did you ever find a performant solution? Thanks.

 

I was planning on using the AsyncGPUReadbackRequest API to copy directly into a

NativeArray

object but for some asinine reason that API isn't available in OpenGL. So I decided to go with

ReadPixels()

which copies into a managed byte array. I then copy from this into the native shared memory buffer.

In my setup code I have:

Code (CSharp):

1.             _renderTextureConfig = new RenderTextureDescriptor(
2.                 256,
3.                 256,
4.                 RenderTextureFormat.ARGBFloat,
5.                 32
6.             );
7.  
8.             _renderTexture = new RenderTexture(_renderTextureConfig);
9.             _renderTexture.Create();
10.             _camera.targetTexture = _renderTexture;
11.  
12.             _cpuTextureBuffer = new Texture2D(
13.                 256,
14.                 256,
15.                 TextureFormat.RGBAFloat,
16.                 false
17.             );

Then when I want to render a frame:

Code (CSharp):

1.         public byte[] CopyRenderTextureToCpu()
2.         {
3.             _camera.Render();
4.             // Set active RenderTexture to the camera and copy all the pixels into CPU memory.
5.             RenderTexture.active = _renderTexture;
6.             _cpuTextureBuffer.ReadPixels(_textureRect, 0, 0, false);
7.             _cpuTextureBuffer.Apply();
8.             RenderTexture.active = null;
9.  
10.             // A small optimization that tells the GPU it doesn't need to "restore" the contents of
11.             // the render texture the next time it is active.
12.             _renderTexture.DiscardContents();
13.  
14.             return _cpuTextureBuffer.GetRawTextureData();
15.         }

Finally to copy this into shared memory, I have a native plugin which allocates shared memory and returns an

IntPtr

which I can then copy the frame into with this:

Code (CSharp):

1. byte[] imageBuffer = CopyRenderTextureToCpu();
2.  
3. IntPtr sharedMemoryAddress = getSharedMemoryAddress();
4.  
5. Marshal.Copy(imageBuffer, 0, sharedMemoryAddress, imageBufferLengthInBytes);

It's very dissapointing that there isn't a way to copy directly to a NativeArray with

ReadPixels()

. I hope the Unity team is planning on adding that (or making the async one work with OpenGL). But happily it turns out that this has not become a performance bottleneck so far. I'm able to copy 256x256 frames at at least 60fps (I haven't tried going higher because it's not important for my use case, so who knows what the limit is).

 

They made asynchronous readback for opengl support in the new 2021.2 alpha build but I don't think they will port it back to the older LTS.