Search Unity

Resolved How do I transform a XrCpuImage point into a Screen position

Discussion in 'AR' started by RedGirafeGames, Jan 13, 2023.

  1. RedGirafeGames

    RedGirafeGames

    Joined:
    Sep 30, 2016
    Posts:
    32
    Hi,

    I am trying to convert a point from a XrCpuImage (a point that i get from an OpenCv detection) to a Screen position, in order to use this position in a Camera raycast.

    The problem is that the image from the AR Camera has smaller resolution and a different aspect than the screen. I understand from other posts that the solution is to use the displayMatrix from the
    ARCameraFrameEventArgs.

    My question is : how do I do that ???

    Should I manage to apply the matrix4x4 to a Vector2? but I have no idea how to do it, I tried to adapt https://forum.unity.com/threads/how-to-assign-matrix4x4-to-transform.121966/ solution... but nothing good came out of it.

    Anyone would have clues on that?

    Thomas
     
    RedGirafeGames, Jan 13, 2023
    #1

  2. davidmo_unity

    davidmo_unity

    Unity Technologies

    Joined:
    Jun 18, 2019
    Posts:
    99
    The shaders that use the display matrix show how to use the display matrix to acquire the UV coordinates that should be sampled by a given fragment during rendering:

    Code (CSharp):
    1.  
    2. // ARCore Shader Code
    3. textureCoord = (_UnityDisplayTransform * vec4(gl_MultiTexCoord0.x, 1.0f - gl_MultiTexCoord0.y, 1.0f, 0.0f)).xy;
    4.  
    5. // ARKit Shader Code
    6. float2 texcoord = mul(float3(v.texcoord, 1.0f), _UnityDisplayTransform).xy;
    7.  
    So what you need to do is calculate the UV of the texel returned from your OpenCV. Additionally, you're gonna need to account for the fact that the UV coordinates used in rendering are based on a texture with a different aspect ratio so you'll have to calculate what portion of the CPU image has been sliced on each side to figure out where the UV coordinates origin should be. Once you have all that you can apply the UVs to the current screen resolution to figure out what Vec2 to use for the raycast.
     
    davidmo_unity, Jan 13, 2023
    #2

  3. RedGirafeGames

    RedGirafeGames

    Joined:
    Sep 30, 2016
    Posts:
    32
    First, thanks for your reply.

    But this is exactly the moment I am completly lost (as other posts pointed to this shader), is it really possible to do all that out of a shader, without all the apis from the snippet you quoted?
    I don't even understand how are UV pertinent as there is no 3D model here.
     
    RedGirafeGames, Jan 13, 2023
    #3

  4. davidmo_unity

    davidmo_unity

    Unity Technologies

    Joined:
    Jun 18, 2019
    Posts:
    99
    It's definitely possible. To break it down a bit, the ARCameraBackground renders the camera background texture as a fullscreen quad so the UV coordinates are a direct mapping from screen space to texel space. This means that the UV coordinates of a texel can be treated as a viewport point. See: https://docs.unity3d.com/ScriptReference/Camera.ViewportPointToRay.html

    There are some potential caveats here.

    First, I am not totally sure if the above method accepts negative or greater than 1 values off the top of my head but that is a possibility if OpenCV returns a texel that is out of the bounds of the rendered image.

    Second, the CPU image may be mirrored or flipped depending on your platform so you'll need to apply an additional transform to account for that or use the conversion api. See https://docs.unity3d.com/Packages/c...#accessing-the-device-camera-image-on-the-cpu for a brief on using the conversion api (you can also possibly slice the image to match the rendered texture so the first issue becomes a non-issue).
     
    Last edited: Jan 13, 2023
    davidmo_unity, Jan 13, 2023
    #4
    RedGirafeGames likes this.

  5. RedGirafeGames

    RedGirafeGames

    Joined:
    Sep 30, 2016
    Posts:
    32
    Thank you for the details.
    Yes the image is mirrored, I already have a transformation for that.
    I'll look on your link for the UV and try to use it, thanks a lot !
     
    RedGirafeGames, Jan 13, 2023
    #5

  6. RedGirafeGames

    RedGirafeGames

    Joined:
    Sep 30, 2016
    Posts:
    32
    It must be full of holes and could break easily compared to a version based on the displayMatrix and UVs, but for the moment I'm going with this custom transformation algorithm :

    Code (CSharp):
    1.     /**
    2.      * Transforms a point from an Xr Camera image to a screen point.
    3.      * Assuming Xr Image is rotated 90° (so width is height), keeps the height fixed and crops width with equal length on left and right.
    4.      */
    5.     private bool XrImagePointToScreenPoint(Vector2 xrPoint, out Vector2 screenPoint, Vector2 xrImageSize, Vector2 screenSize)
    6.     {
    7.         var xrToScreenRatio = screenSize.y / xrImageSize.x;
    8.         var theoricalXrWidth = xrImageSize.y * xrToScreenRatio;
    9.  
    10.         var exceededWidth = theoricalXrWidth - screenSize.x;
    11.         var margin = exceededWidth / 2;
    12.  
    13.         var x = (xrPoint.y * xrToScreenRatio) - margin;
    14.         var y = xrPoint.x * xrToScreenRatio;
    15.  
    16.         if (x < 0 || x > screenSize.x)
    17.         {
    18.             screenPoint = Vector2.zero;
    19.             return false;
    20.         }
    21.      
    22.         screenPoint = new Vector2((int) x, (int) y);
    23.      
    24.         return true;
    25.     }
    Best I could do for the moment, thanks to your explanations. I put it here in case it's useful to someone.
     
    Last edited: Jan 15, 2023
    RedGirafeGames, Jan 15, 2023
    #6
    davidmo_unity likes this.