Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question ARCore: Accurate Environment Depth with ArFoundation?

Discussion in 'AR' started by J_Kost, Nov 20, 2021.

  1. J_Kost

    J_Kost

    Joined:
    Feb 16, 2021
    Posts:
    32
    Hi,

    I am looking to maximize the accuracy of depth values in the environment depth image in order to make real-world measurements. Here are things I would like to do along with the pertaining questions:
    1. Get the highest possbile resolution environment depth image.
      A am using a Samsung S20+ 5G with a 640x480 TOF sensor. The depth images I get from the AR Occlusion Manager are 640x360 and appear to be a cropped version of the full res TOF image.
      Is there any why to change the resolution – like with the camera configurations in the AR Camera Manager – or is there just this one resolution?

    2. Get the depth image with as little prior processing possible. Ideally with no smoothing at all.
      In my environment depth images, there is a "cloud" of smoothing over and around objects. To try to get rid of this, I set the EnvironmentDepthMode to "Fastest" on the AROcclusionManager in the Editor:

      forum_post_1.PNG

      According to Tim Mowrer in [2], this disables all processing of the depth image on Unity's side. Unfortunately, the depth images still have the smoothing clouds when using the "Fastest" mode. These clouds are evident when comparing the downscaled camera image (l.) with the depth image (r.) side by side:

      aura_visualized.png

      I overlaid the object in the lower image to illustrate roughly what the depth image should look like without any smoothing.
      Can I conclude from this that ARFoundation uses ARCore's pre-processed "Full Depth API" [1] Image by default?

      2.1 According to the ARCore docs for ArFoundation [1] there is an option to get the raw Depth image from ARCore in ARFoundation. What confuses me here is that the docs say to use AROcclusionManager.environmentDepthTemporalSmoothingRequested() in order to expose the raw depth image. This seems counter-intutitive given the nomenclature. Is this true? Has raw depth image passthrough somehow been hacked into this functionality intended for temporal smoothing? Unfortunately, I am unable to test this for myself, because environmentDepthTemporalSmoothingRequested requires ARFoundation >= 4.2.0 and updating from 4.1.7 breaks my app on Android (I no longer get any Camera configurations from the ARCameraManager & logcat is full of lower level android errors).
      Is there any way to access the raw depth API image in ARFoundation at all? If so, is this also possible in 4.1.7?
    3. Use the Depths to estimate world 3D Positions.
      Todd Stinson says in [3] that the values in the environment depth textures are in meters. I am not sure though, whether these are meters of depth or distance. The ARCore docs for ArFoundation [4] clearly state that the values are depth along the camera view direction, but the same docs also claim the values are in millimeters, so I am not sure whether this page is trustworthy. After all, the values I am seeing in my textures (*) only make sense if interpreted as meters, so am am inclined to believe Todd, and disbelieve the ARCore docs.
      {(*)N.b.: I convert the depth XRCpuImages into Texture2Ds with Textureformat
      XRDepthImg.format.AsTextureFormat() (i.e. RFloat). So the values should be correct.}

      As things stand, I can't find any documentation that clearly states whether the environment depth textures contain distance to the real world point or depth of the real world point (i.e. the component of the distance along the camera z axis/principal axis/whatever you would like to call it). I'm pretty confident the values are depth, but "depth" and "distance" are used interchangeably in the DepthGradient shader in the ARF samples [5]. Also, the ARFoundation manual speaks of environment depth as the "distance from the device to any part of the environment in the camera field of view"[6], so I figured I'd better make sure:
      Are the values in a environment depth XRCpuImage from ARCore – after conversion to an RFloat Texture2D (native resolution of the XRCpuImage) – given in meters? Should they be interpreted as depth or distance in the sense of the image below?
      Finally, to be on the safe side:
      Are these values to be understood as relative to the camera position( = center = eye) or are they, for example, relative to some offset near plane?

      forum_post_2.PNG

    [1] https://developers.google.com/ar/develop/unity-arf/depth/raw-depth
    [2] https://github.com/Unity-Technologies/arfoundation-samples/issues/616#issuecomment-704600241
    [3] https://forum.unity.com/threads/env...nd-depth-in-arfoundation.919076/#post-6076761
    [4] https://developers.google.com/ar/develop/unity-arf/depth/developer-guide
    [5] https://github.com/Unity-Technologi...blob/main/Assets/Shaders/DepthGradient.shader
    [6] https://docs.unity3d.com/Packages/com.unity.xr.arsubsystems@4.1/manual/occlusion-subsystem.html
     
    Last edited: Nov 22, 2021
    hellowill89 likes this.
  2. hellowill89

    hellowill89

    Joined:
    Jan 8, 2013
    Posts:
    45
    Excellent questions. I too am looking for better depth maps from AR foundation.
     
  3. J_Kost

    J_Kost

    Joined:
    Feb 16, 2021
    Posts:
    32
    Unfortunately, there does not seem to be much interest in this topic.
     
  4. todds_unity

    todds_unity

    Joined:
    Aug 1, 2018
    Posts:
    324
    Changing the camera resolution on some devices may also change the corresponding depth image resolution. However, no API exists to independently modify the depth image resolution.


    In AR Foundation 4.2, you can define this behavior using AROcclusionManager.environmentDepthTemporalSmoothingRequested.
    https://docs.unity3d.com/Packages/c...er_environmentDepthTemporalSmoothingRequested

    On ARCore, a value for AROcclusionManager.environmentDepthTemporalSmoothingEnabled of true is using the ARCore full depth API and of false is using the ARCore raw depth API.

    Yes. Each RFloat pixel value in a environment depth XRCpuImage represents a distance in meters from the physical device's camera to the detected surface. Or, if the pixel value is 0, no surface is detected.

    Note that in devices which do not have LiDAR or ToF sensors, this depth information is estimated based on parallax from normal camera motion.
     
    KyryloKuzyk likes this.
  5. J_Kost

    J_Kost

    Joined:
    Feb 16, 2021
    Posts:
    32
    Thanks for the clarifications!