Search Unity

Scraping the back buffer

Discussion in 'General Graphics' started by guy123, Jan 8, 2019.

  1. guy123

    guy123

    Joined:
    May 18, 2011
    Posts:
    16
    Hello folks,

    In my company we use Unity on both Windows desktop machines, and Android tablets/phones. We have a requirement to be able to stream raw imagery from these devices over the network during their use (so that we can run algorithms on the image data).

    On a Windows machine, running DirectX9 this was really easy, I just wrote a C++ native plugin that created an offscreen surface and copied the contents of the backbuffer into it. However I'm having difficulty in working out how we can do this for OpenglES (phones and tablets).

    I have two main questions:

    1. Does the Native C++ plugin API work on android devices? (does Unity build the .so files into the .apk?)
    2. How do I scrape the backbuffer in OpenGL ES?
    And also, I am curious as to whether there is a quick and efficient way to do this via script in Unity itself? I managed to get the buffer pixels, but had trouble running a separate network thread and getting the two working in sync without issues surrounding having to do everything on the rendering thread.

    Any help and advice would be much appreciated.

    Thanks!
     
    guy123, Jan 8, 2019
    #1

  2. arnaud-carre

    arnaud-carre

    Unity Technologies

    Joined:
    Jun 23, 2016
    Posts:
    97
    You mention DX9, I guess you're not using recent Unity version? With 2018.2, the best way of doing that is using
    "async GPU readback". So you can get GPU data without blocking CPU. Unfortunatly this is not supported in OpenGL ( but at least you can use it on PC directX ).

    you can read the main camera activeTexture using async readback and a callback like that:

    void OnRequestDone(AsyncGPUReabackRequest request)
    {
    if (!request.hasError)
    {
    var data = request.GetData<uint>();
    // ...
    }
    }

    AsyncGPUReadback.Request(camera.activeTexture, OnRequestDone);

    ( you check if async readback is supported using "SystemInfo.supportsAsyncGPUReadback" )

    Regarding openGL, you still have to use classic blocking texture read function. ( I guess you're using ReadPixels? )
     
    arnaud-carre, Jan 10, 2019
    #2

  3. guy123

    guy123

    Joined:
    May 18, 2011
    Posts:
    16
    Hey Arnaud-carre, thanks for your response.

    We have an existing plugin for another application that uses DirectX9 so I was able to re-use my previous code for scraping the back buffer - hence the use of DX9. We are however using a recent version of Unity so in the long run will want to switch over to using DX11.

    The intention of this plugin is actually to support all of our typical rendering pipelines (DX9 and DX11 for PC and OpenGLES for Android devices). Therefore being able to do it entirely in Unity C# script would be ideal, however I was concerned about holding up the rendering thread as mentioned above.

    Currently I have a C++ native plugin running in the background that is hosting a TCP server and also handling the compression and distribution of the images that are taken from the back buffer.

    The solution you have suggested is very interesting. Presumably I could just read the pixels and then push them out to the native plugin for compression and distribution?

    When you say
    Do you mean that the AsyncGPUReadback call is not available when using OpenGL/ES?

    Thanks again
     
    guy123, Jan 10, 2019
    #3

  4. guy123

    guy123

    Joined:
    May 18, 2011
    Posts:
    16
    I have been trying to look into your suggested solution, however I can't seem to find the
    AsyncGPUReadback call anywhere.
    I'm using Unity Pro... is there something disabled somewhere in my project settings that might preclude this reference?
     
    guy123, Jan 10, 2019
    #4

  5. arnaud-carre

    arnaud-carre

    Unity Technologies

    Joined:
    Jun 23, 2016
    Posts:
    97
    arnaud-carre, Jan 14, 2019
    #5

  6. citron8000

    citron8000

    Joined:
    Nov 25, 2014
    Posts:
    22
    @guy123 Hi Gu123, we have the same problem here with Android and OpenGL. Did you end up solving your issue?

    Thanks!
     
    citron8000, Jan 20, 2021
    #6