Unity Environmental Occlusion and Depth in ARFoundation

Do you need XCode 12 to compile for ARKit4? Still waiting for iOS 14 beta to download for my iPad and haven't come across any development requirements (besides updated packages and device iOS beta)

 

ARCore's depth API is now out of beta. Will the Occlusion Manager be updating to support that on Android at some point?

 

I was just about to post the exact same question. Thanks for beating me to it! It would be great to get the support for ARCore as soon as possible for android.

 

Wonderful thanks!

 

For iPad Pro with LiDAR running iOS13.x (not iOS14 Beta) this is not supported?

 

My Galaxy Tab S6(2019) says "ARCore unsupported device".
Please fix this. Thanks~

 

I have the same issue with my Samsung Galaxy S8+

 

I'm not understanding this list
Are all Android devices on that list supported, or only the ones with notes that says "supports depth API"?

 

Yes, only with notes "Support depth API" support depth API Rest support just AR Core without Depth API.

 

Does Environmental Occlusion and Depth work out of the box (just with the AROcclusionManager) in URP?

 

what a mess android is. as good as the depth api is, there is about 20 devices that support this out of hundreds of android phones ...

 

Is it possible to use the AROcclusionManager to get access to the environment depth texture without having automatic occlusion? My use cases are -
- I want to use the depth texture for collision tests only
- I want to generate my own occlusion from the depth texture so that I can try excluding around planar regions that my game content is sitting on
- Also occlusion doesn't seem correct when a scale is applied to the AR Session Origin so I want to see if I can work around that

 

Perfect, thanks! Do you have any tips for converting the values in the depth texture into depth in metres?

 

Handy - thanks again.

 

What is the best way to check for Depth mode support at runtime? I know ARCore provides the following
isDepthModeSupported (src: https://developers.google.com/ar/develop/java/depth/developer-guide#depth-api-supported). Is there an equivalent through AR Foundation?

Thanks!

 

I wish I had a checkbox in `AROcclusionManager` to toggle occlusion per platform (iOS, Android). I know I can do that with an extra script thought.

For the moment I am not quite happy with the Occlusion that I get on Android, part of objects that are placed close to the floor are getting clipped, although on iOS the Human Occlusion work quite good.

 

Is it somehow possible to get the human stencil texture in combination with the environment depth texture on iPad + LiDAR + iOS14?

 

The environment depth feature does not function after testing with the Simple Occlusion sample in ARFoundation Samples. Setup:

- ARFoundation 4.1.0-preview.5 (also tested with preview.2)
- ARKit XR Plugin 4.1.0-preview.5 (also tested with preview.2)
- ARSubsystems 4.1.0-preview.5 (also tested with preview.2)
- Unity 2019.4.3
- iPadOS 14 developer beta 3 (18A5332f) on an iPad Pro 4th Gen
- Xcode 12 beta 3 (12A8169g)

ARKit-specific demos from developer.apple.com appear to still function correctly.

Any other suggested troubleshooting (i.e. perhaps a public beta iPadOS version)?

 

Yes, me too.
Did you find any solutions?

 

I just verified the same problem with my iPad Pro 4th generation, using Unity 2020.1 and ARFoundation 4.1.0-preview.5, iPad OS 14.0 beta 3. I tried the arfoundation-samples latest preview branch, and other demos work, such as Classification Meshes, but neither SimpleOcclusion nor DepthImages demos work. I verified that m_OcclusionManager.descriptor?.supportsEnvironmentDepthImage is true.

This is the output I'm seeing in XCode's console in SimpleOcclusion, starting with the loading of the UnityARKit plugin:

Code (CSharp):

1. Subsystems] Loading plugin UnityARKit for subsystem ARKit-Input...
2. [Subsystems] UnityARKit successfully registered Provider for ARKit-Input
3. [Subsystems] UnityARKit successfully registered Provider for ARKit-Input
4. [Subsystems] UnityARKit successfully registered Provider for ARKit-Meshing
5. [Subsystems] Loading plugin UnityARKit for subsystem ARKit-Meshing...
6. 2020-07-26 17:15:42.958376-0700 samples[413:11445] Unbalanced calls to begin/end appearance transitions for <UnityViewControllerStoryboard: 0x10d705240>.
7. UnloadTime: 0.476458 ms
8. 2020-07-26 17:15:44.565873-0700 samples[413:11445] UnityARKit: Updating ARSession configuration with <ARWorldTrackingConfiguration: 0x283d80790 worldAlignment=Gravity lightEstimation=Disabled frameSemantics=Unknown videoFormat=<ARVideoFormat: 0x2829de490 imageResolution=(1920, 1440) framesPerSecond=(60)> autoFocus=Enabled environmentTexturing=None wantsHDREnvironmentTextures=Enabled planeDetection=None collaboration=Disabled userFaceTracking=Disabled sceneReconstruction=None>
9. 2020-07-26 17:15:44.974870-0700 samples[413:11445] UnityARKit: Updating ARSession configuration with <ARWorldTrackingConfiguration: 0x283d880b0 worldAlignment=Gravity lightEstimation=Disabled frameSemantics=Unknown videoFormat=<ARVideoFormat: 0x2829de490 imageResolution=(1920, 1440) framesPerSecond=(60)> autoFocus=Enabled environmentTexturing=None wantsHDREnvironmentTextures=Enabled planeDetection=None collaboration=Disabled userFaceTracking=Disabled sceneReconstruction=None>
10. 2020-07-26 17:15:44.975027-0700 samples[413:11620] [General] ARWorldTrackingConfiguration <0x283d88370>: Expected 3 jasper framerate values: 0
11. 2020-07-26 17:15:46.732889-0700 samples[413:11622] [Sensor] ARDepthSensor <0x283189ba0>: (AVCaptureDeviceTypeBuiltInTimeOfFlightCamera - Back): capture session dropped jasper frame: 1115.505557, LateData

In the DepthImages demo, I tried touching the button that changes the resolution, didn't make it start working.

 

In the environment of iPadOS 14 beta 3, when ARKit Sample code “Visualizing a Point Cloud Using Scene Depth(*1)" is built with Xcode 12.0 beta 3 (12A8169g), the point cloud is displayed, but when built with Xcode 12.0 beta (12A6159), the point cloud is not displayed. According to this result, it seems that Xcode needs to use "beta 3".

According to Changelog of ARKit XR Plugin 4.1.0-preview.5 (*2),
it says "Static library was built with Xcode 12.0 beta 2 (12A6163b)".
By the above result, I guess the static library should be built with Xcode 12.0 beta 3.

*1 https://developer.apple.com/documen...a_point_cloud_using_scene_depth?language=objc
*2 https://docs.unity3d.com/Packages/com.unity.xr.arkit@4.1/changelog/CHANGELOG.html

 

I tried running Normals Meshing example with Ar Foundation 4.0.2 on Unity 2019.4.4f1 and after 8-10 minutes application freezes. The problem seems to be with the Unity garbage collector and reproduces even if I don't move iPad (very few meshes are being generated). Anyone else faced this issue?

 

Is it possible to have environment occlusion and human occlusion working simultaneously?

So far, setting AROcclusionManager OcclusionPreferenceMode to "Human Occlusion" seems to make environment occlusion not work - setting it to "Environment Occlusion" seems to make human occlusion not work.

Is this intended?

 

I tried the same example with latest preview and unfortunately the problem persists. Maybe you can give some general direction / advice on how I can fix it.

 

One more issue I've run into. When an AR app is locked in a landscape orientation the perspective is off and the virtual objects do not appear in their correct locations while in landscape mode. Turn the phone to portrait orientation and everything looks great. Back to landscape and its off again. Its as if the app is not detecting the initial orientation setting and re-projecting.

Would setting the FOV Axis on the AR camera to Horizontal fix this issue if the app is locked in landscape?

 

Environmental Occlusion worked for me (in my own project) with 4.1.0-preview.6 (though it wasn’t as detailed/stable as I hoped it would be). Unfortunately, I don’t have the project available right now to give any specifics/tips, aside from putting the occlusion manager on the camera.

 

Is it possible to use AR Core's depth capabilities on non-LIDAR iOS devices through AR Foundation? Or would I need to use AR Core without AR Foundation?

 

In a portrait app, the environmentDepthTexture is rotated 90 degrees. Is there a way to automatically adjust this to portrait?

We also are noticing that when using getPixels on the environmentDepthTexture at a given location the values returned for the r value are only returning between 0 and 1. Is this the correct behavior? With the distance in meters being stored in the r value we anticipated values over 1 meter, but maybe that was a mis judgment on our end. Want to verify its working correctly. Thank you!

 

In a portrait app, the environmentDepthTexture is rotated 90 degrees. Is there a way to automatically adjust this to portrait?

It's my understanding that the values returned by the environmentDepthTexture are in meters, the distance from the camera to the pixel in the environmentDepthTexture. According to this comment:

We are noticing that when using getPixel on the environmentDepthTexture at a given location the values returned for the r value are only returning between 0 and 1. Is this the correct behavior? With the distance supposedly in meters we anticipated values over 1 meter. Or are the Color.r values returned by the environmentDepthTexture actually not the distance in meters and we then have to calculate the distance based on the clipping planes etc?

For example, doing the following never gives us a value over 1:

Code (CSharp):

1. Color depthValue = m_OcclusionManager.environmentDepthTexture.GetPixel((int)touchPos.x,(int)touchPos.y);
2. Debug.Log(depthValue.r)

 

From what I can tell, the depth values are in meters but the format of the texture means that if you use GetPixel you get a value clamped to 1 meter max. To get full-range values what I ended up doing is blitting the depth texture into a render texture (format RenderTextureFormat.ARGBFloat) then render that into a new Texture2D, which I could then use GetPixel on. I'm sure there's a more efficient way but it worked. I also use a shader when blitting the initial texture that rotates the texture based on Screen.orientation so I don't have to handle that later on - the initial texture is always in 'Landscape Right' orientation.

 

UPDATE:
This worked perfectly, thank you again for your help! I've been stuck on this for a week and this finally got me over the hump. Thank you!

ORIGINAL POST:
Thank you so much for this information! I tried blitting into a rendertexture, but was using a different texture format. I will give that a try and see what results we get.

Any chance you'd be willing to share the shader you used for rotating? I've been using the shader provided with the DisplayDepthImage example, with some modifications. This may work with the change you recommended, but if not it would be awesome if you were able to share. If not, I definitely understand.

Thanks for your help!

 

No problem. This is the shader I've been using - pass in Screen.orientation in "_Orientation" -

Code (CSharp):

1. Shader "Unlit/CopyShader"
2. {
3.     SubShader
4.     {
5.         Tags { "RenderType"="Opaque" }
6.         LOD 100
7.         ZTest Always
8.         Cull Off
9.         ZWrite Off
10.        
11.         Pass
12.         {
13.             Name "Unlit"
14.                
15.             HLSLPROGRAM
16.             #pragma vertex vert
17.             #pragma fragment frag
18.            
19.             #include "UnityCG.cginc"
20.  
21.             struct appdata
22.             {
23.                 float4 vertex : POSITION;
24.                 float2 uv : TEXCOORD0;
25.             };
26.  
27.             struct v2f
28.             {
29.                 float2 uv : TEXCOORD0;
30.                 float4 vertex : SV_POSITION;
31.             };
32.            
33.             Texture2D _MainTex;
34.             SamplerState sampler_MainTex;
35.             float4 _MainTex_ST;
36.             int _Orientation;
37.  
38.             v2f vert (appdata v)
39.             {
40.                 v2f o;
41.                 o.vertex = UnityObjectToClipPos(v.vertex);
42.                
43.                 // Flip X
44.                 o.uv = float2(1.0 - v.uv.x, v.uv.y);
45.                
46.                 if (_Orientation == 1) {
47.                     // Portrait
48.                     o.uv = float2(1.0 - o.uv.y, o.uv.x);
49.                 }
50.                 else if (_Orientation == 3) {
51.                     // Landscape left
52.                     o.uv = float2(1.0 - o.uv.x, 1.0 - o.uv.y);
53.                 }
54.                
55.                 o.uv = TRANSFORM_TEX(o.uv, _MainTex);
56.                 return o;
57.             }
58.  
59.             fixed4 frag (v2f i) : SV_Target
60.             {
61.                 return  _MainTex.Sample(sampler_MainTex, i.uv);
62.             }
63.             ENDHLSL
64.         }
65.     }
66. }
67.  

 

Would this also let you know if an iOS device has lidar? Trying to figure out how to detect the lidar hardware on iOS as well.

Thank you!

 

Right now it would work, because only the LIDAR supports creating depth images in ARKit (or mesh reconstruction for that matter). But that's not guaranteed for the future, it's possible that the next ARKit could add depth sensing and mesh reconstruction to the normal camera for older devices.

Another option is to check for the device model, and check against devices you know have a LIDAR. For that you would parse the string returned from SystemInfo.deviceModel.

 

Thanks!

 

Does the environment depth features of AR Foundation 4.1.0-preview.7 work with the latest iPadOS 14 beta 7 and Xcode 12.0 beta 6?
I'm worried that similar above problems will occur.

 

If I use ARKore Deptx API in my project. I cannot see the entire 3D model. I see only part of the 3D model and the rest is hidden. How to make me see the entire 3D model completely?

 

Hello ! I've started testing the Depth API, but I have the same problem as j0schm03, could'nt retrieve depth values.
I tried to understand the tagged post, but I'm not sure how you did it.

I tried to Convert the Texture2D using ARGBFloat and GetPixels on the converted texture, but I'm still getting values between 0 and 1.

What I am doing wrong ?

Thanks in advance !