Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Resolved ARFoundation; mirror/flip horizontally ARCamera image with virtual objects?

Discussion in 'AR' started by neutrum_unity, Jun 14, 2020.

  1. neutrum_unity

    neutrum_unity

    Joined:
    Sep 4, 2019
    Posts:
    1
    Hi,
    I am struggling with how can I flip/mirror screen to create "real mirror" while iPhone is in landscape mode (or portrait doesn't matter at this stage) and connected to TV.

    I am successfully using this code:
    Code (CSharp):
    1. // Attempt to get the latest camera image. If this method succeeds,
    2.         // it acquires a native resource that must be disposed (see below).
    3.         XRCameraImage image;
    4.         if (!cameraManager.TryGetLatestImage(out image))
    5.         {
    6.             return;
    7.         }
    8.  
    9.         // Once we have a valid XRCameraImage, we can access the individual image "planes"
    10.         // (the separate channels in the image). XRCameraImage.GetPlane provides
    11.         // low-overhead access to this data. This could then be passed to a
    12.         // computer vision algorithm. Here, we will convert the camera image
    13.         // to an RGBA texture and draw it on the screen.
    14.  
    15.         // Choose an RGBA format.
    16.         // See XRCameraImage.FormatSupported for a complete list of supported formats.
    17.         var format = TextureFormat.RGBA32;
    18.  
    19.         if (m_Texture == null || m_Texture.width != image.width || m_Texture.height != image.height)
    20.         {
    21.             m_Texture = new Texture2D(image.width, image.height, format, false);
    22.         }
    23.  
    24.         // Convert the image to format, flipping the image across the Y axis.
    25.         // We can also get a sub rectangle, but we'll get the full image here.
    26.         var conversionParams = new XRCameraImageConversionParams(image, format, CameraImageTransformation.MirrorY | CameraImageTransformation.MirrorX);
    27.  
    28.         // Texture2D allows us write directly to the raw texture data
    29.         // This allows us to do the conversion in-place without making any copies.
    30.         var rawTextureData = m_Texture.GetRawTextureData<byte>();
    31.         try
    32.         {
    33.             image.Convert(conversionParams, new IntPtr(rawTextureData.GetUnsafePtr()), rawTextureData.Length);
    34.         }
    35.         finally
    36.         {
    37.             // We must dispose of the XRCameraImage after we're finished
    38.             // with it to avoid leaking native resources.
    39.             image.Dispose();
    40.         }
    41.  
    42.         // Apply the updated texture data to our texture
    43.         m_Texture.Apply();
    44.  
    45.         // Set the RawImage's texture so we can visualize it.
    46.         //m_RawImage.texture = m_Texture;
    47.         material.SetTexture(MainTexId, m_Texture);
    I practically converted the latest image from ARCameraManager and set it to the custom material texture on ARCameraBackground. It does work but there are no virtual objects. It makes sense because as I understand it I am taking the frame from live camera feed.

    So how can I make it flipped/mirrored and show the virtual objects? Any pointers, suggestions?

    ad: I tried to use .blit() with ARCameraBackground.material and RenderTexture and then set it to RawImage stretched on Canvas, but that was really expensive.

    iPhone X iOS 13.5.1
    ARFoundation Preview.4 - 3.0.0
    Unity 2019.3.14f1
     
  2. fxgear

    fxgear

    Joined:
    Jan 22, 2015
    Posts:
    1
    Hi, it's not the solution you are looking for, but I'm struggling with using .blit() command to get camera frame.
    I use Graphics.Blit(null,RT,ARCameraBackground.material); code and RenderTexture.active=RT; before the Blit Code.
    The code is used when OnFrameReceived Callback function, but all I got is black RT.
    Is there any difference between my code and yours?
     
  3. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    The simplest and the most performant solution is to write a custom shader and apply a custom material to ARCameraBackground:
    upload_2020-12-10_11-15-51.png

    Make a copy of ARCoreBackground.shader, and replace this line:
    Code (CSharp):
    1. // Remap the texture coordinates based on the device rotation.
    2. textureCoord = (_UnityDisplayTransform * vec4(gl_MultiTexCoord0.x, 1.0f - gl_MultiTexCoord0.y, 1.0f, 0.0f)).xy;
    3.  
    With this one:
    Code (CSharp):
    1. // Remap the texture coordinates based on the device rotation.
    2. textureCoord = (_UnityDisplayTransform * vec4(1.0f - gl_MultiTexCoord0.x, 1.0f - gl_MultiTexCoord0.y, 1.0f, 0.0f)).xy;
     

    Attached Files:

  4. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Oops, just realized that your question was about ARKit. The approach for ARKit is very similar.
    Replace this line:
    Code (CSharp):
    1. // Remap the texture coordinates based on the device rotation.
    2. float2 texcoord = mul(float3(v.texcoord, 1.0f), _UnityDisplayTransform).xy;
    3.  
    With this one:
    Code (CSharp):
    1. // Remap the texture coordinates based on the device rotation.
    2. float2 texcoord = mul(float3(1.0 - v.texcoord.x, v.texcoord.y, 1.0), _UnityDisplayTransform).xy;
     
  5. PabloSE94

    PabloSE94

    Joined:
    Feb 4, 2021
    Posts:
    2
    Hi, is there a solution where you can flip the texture 180º degrees (horizontal axis) from this code? I dont have a lot of background in the shader stuff and i was wondering this.
    Thanks.
     
  6. bzalexDev

    bzalexDev

    Joined:
    Jul 2, 2019
    Posts:
    4
    Hi, I have successfully flipped the camera feed but how do I flip the body tracking as well?
     
    Bassem102 likes this.
  7. Bassem102

    Bassem102

    Joined:
    Oct 11, 2017
    Posts:
    6
    Hi, i want to know if you managed to find a way to flip body tracking?

    Thanks.
     
    mpower91 likes this.
  8. Bassem102

    Bassem102

    Joined:
    Oct 11, 2017
    Posts:
    6
    Hello, if someone is searching for a solution for flipping body tracker, i did by flipping horizontally the camera with this script attached to any GameObject:
    Code (CSharp):
    1. using System.Collections;
    2. using UnityEngine;
    3.  
    4. public class CameraMirror : MonoBehaviour
    5. {
    6.     [SerializeField]
    7.     private Camera cameraToMirror;
    8.  
    9.     void OnPreCull()
    10.     {
    11.         Matrix4x4 scale = Matrix4x4.Scale(new Vector3(-1, 1, 1));
    12.         cameraToMirror.ResetWorldToCameraMatrix();
    13.         cameraToMirror.ResetProjectionMatrix();
    14.         cameraToMirror.projectionMatrix = cameraToMirror.projectionMatrix * scale;
    15.     }
    16.     void OnPreRender()
    17.     {
    18.         GL.invertCulling = true;
    19.     }
    20.     void OnPostRender()
    21.     {
    22.         GL.invertCulling = false;
    23.     }
    24. }