Search Unity

Question Advice on how to get and process ARCamera frame/texture

Discussion in 'AR' started by mpriem, Nov 7, 2022.

  1. mpriem

    mpriem

    Joined:
    Dec 31, 2016
    Posts:
    9
    I'm working on an ARFoundation + ARCore Android project and would like to introduce a computer vision model into my project. I have a working PoC using a webcamtexture but would like to use the ARCamera texture/frame instead.

    As I understand - correct me if I'm wrong - the AR Camera Background component is what ultimately passes the frame through to an URP render feature.

    Given I would like to hook into the pipeline what is the best place to intercept/grab the frame and do my CV magic on it?
     
    mpriem, Nov 7, 2022
    #1

  2. davidmo_unity

    davidmo_unity

    Unity Technologies

    Joined:
    Jun 18, 2019
    Posts:
    99
    davidmo_unity, Nov 7, 2022
    #2
    KyryloKuzyk likes this.

  3. mpriem

    mpriem

    Joined:
    Dec 31, 2016
    Posts:
    9
    Thanks. I will need to read up on a few things as this doesn't ring any bells yet. Is there a good resource to start with to understand the different stages of rendering and where the options to interface are?
     
    mpriem, Nov 7, 2022
    #3

  4. davidmo_unity

    davidmo_unity

    Unity Technologies

    Joined:
    Jun 18, 2019
    Posts:
    99
    You'll need to look into URP and how it handles render features. I'd recommend starting with the URP documentation and working from there.

    One thing to note, depending on the XRCameraBackgroundRenderingMode you choose, you may need to handle the rendering features slightly differently. You may want to look into actually copying and customizing the code found in `ARBackgroundRendererFeature.cs` found in AR Foundation to append a read back at the end of the command buffer configuration and then use that custom renderer feature instead of the AR Renderer Feature. You could add this in the Execute method.
     
    davidmo_unity, Nov 8, 2022
    #4

  5. mpriem

    mpriem

    Joined:
    Dec 31, 2016
    Posts:
    9
    Thanks. I was playing around with the trygetlatestcpuimage but due to it being raw without rotation it was quite difficult to handle. I will look at your tip to just work inside the pipeline and maybe even recreating the background rendering feature.
     
    mpriem, Nov 8, 2022
    #5