Official Human occlusion has been added to ARFoundation / ARKit

Looks awesome. Any idea if the Android guys are working on a similar thing so we can down the line (hopefully soonish) get same across AR Foundation for both ios and Android?

 

I am using stenciling for "portals" (materials have a stencil ref and compare func, and a mask is in front of the camera), will this collide with that functionality? Also, does it play well with LW/URP?

 

I am using URP with an iPhone 11 Pro and I get a black screen with the latest preview version of ARFoundation and ARKit

 

Does this work along side human body tracking?

 

Aside from fixing the black screen issue, I have a question. I am using SSAO as a render feature with URP, is it possible to render the depth stencil on top of post processing effects?

 

Just upgraded my project to 3.1.0-preview.1 and is still functioning properly (showing camera data as background). I am using the ARPipelineAsset from a branch of the ARFoundation-samples repo, not the default SRPAsset.

 

Hmm, not sure if I’m doing something wrong because I basically just upgraded an existing project using URP which worked on version 3.0.0. I then created a new project and set everything up to work with URP and still got a black screen. Note I’m using Unity 2020.

 

Ah, I am on 2019.2.11. I cannot speak to 2020.

 

For me version 3.1.0 preview 1 works with LWRP but not URP. I have tested both 2019.3 and 2020 with URP. 2019.2 with LWRP works.

 

Thank you for the feedback. I’m unable to get the latest preview version of ARFoundation working with URP and I’ve tried both 2019.3 as well as 2020. I made a barebones project just setting up the pipeline and forward renderer with the ARbackground feature and link the pipeline in the build settings. The build gives me a black screen. I am using an iPhone 11 Pro Max

 

Update, it works on 2020.1.0a14

 

The black screen when using ARFoundation/ARkit 3.1.0 preview 1 with URP is reproducible by enabling post processing on the camera.

 

Thank you for the update.

 

There is no update for android yet

 

The ARFoundation works for human occlusion! can i expect it to work for building and other physical objects as well??

 

Any update on support for new iPad Pro (LIDAR) and ARKit 3.5 ?
Also , there was a mention about making a better example that handles depth and stencil for Human Segmentation. Is that still in the works?
Thanks.

 

What about support for this https://developers.googleblog.com/2019/12/blending-realities-with-arcore-depth-api.html?
Object blending seems like it's a bit different (not specifically supporting humans or moving objects) but it works really well for everything static. Can we expect to see this supported sometime soon?

 

@todds_unity are you aware of a sample that translates a pixel in the humanDepthTexture to an x,y,z value in world coordinates, if one, or example, wanted to place a GameObject or a particle at one or more of the spots where a human was detected?

 

Is there a built-in way in AROcclusionManager to only display the human without the background? (Basically “green screening”)? Maybe using the depth texture? How should one approach this effectively performance wise?

 

I think everyone here is still waiting for similar functionality. The sample that's currently there is from a year ago (ARKit 3.0). It would be great to have a functional sample that is ready to use.

 

I am using "ARkit XR Plugin 3.1.3" & "AR foundation 3.1.3"
But I can't find "AR Occlusion Manager" in "Add Component"

I can find "AR Occlusion Manager" in "Add Component" with "ARkit XR Plugin 4.0.1" & "AR Foundation 4.0.2"

but camera become black screen in 4.0.1 with all setting remain the same : (

UPDATE: I can find "AR Occlusion Manager" in 3.1.0 , but not 3.1.3....

 

Is it possible to use the AROcclusionManager in the user-facing camera direction? It seems that AROcclusionManager is somehow forcing the camera to be world-facing.

I understand if segmentation is not supported in the front facing camera, but can someone point me to where ARFoundation is forcing my camera the other way? I don't see any logs indicating 'user facing camera not supported, switching to world facing' etc. and I haven't tracked down where it's happening yet.

 

Thanks that's good to know about the extra logs!

It seems that for some reason outside of the sample scene I am able to get the raw occlusion mask for the front facing camera which is great. I'm doing a bit of a homebrew solution to blur the background behind the user (similar to portrait camera on iOS). With the depth mask and the camera texture I think I should be able to accomplish this with a shader and re-apply my new 'processed' camera texture. Still trying it out.

Just for my reference. I'm able to query the 'requested camera' and the 'actual camera', but do you know off the top of your head what subsystem is saying "You can't use the requested camera so here's the other one"? Still trying to track that down.

 

Sorry to dredge up this thread, but is there a recommended way to switch between the two features?

We would like to use people occlusion for the majority of our game, but switch to body tracking for a few missions. Then we'll switch back to occlusion (ideally we would use both simultaneously, but we'd live with the switch). Is this possible in Unity at the moment? What's the recommended way to do it?

 

Ok, answered my own question... at runtime you can set the "requestedDepthMode" and "requestedStencilMode" to disabled on the AROcclusionManager component, and "pose3Drequested" flag on your ARHumanBodyManager component to true (or vice versa), and ARFoundation will redetermine which configuration to use on the fly. Nice! I guess this is the recommended way to do things, if you don't want to or can't change scenes.

 

Can occlusion settings be set at runtime? Such as prefer human vs environment occlusion?

 

That sounds great!
How did you manage to get the raw occlusion mask from the front facing camera? Are you getting it from AROcclusionManager? I'm trying to mask 3D objects with someone's head occlusion mask while in front-facing camera mode.

 

HI @all

Is Human occlusion supports in latest android devices
If yes can someone post me some working unity examples

thanks in advance

 

Human Occlusion is not supported by ARCore.

 

I can not see AROcculussionManager.cs in ARFoundation3.1.6, Is this version of ARFoundation not availiable for Human Occulussion?

 

it worked on 3.1.3

 

Referring back to the question @HidingGlass posted a while back: "...do you know off the top of your head what subsystem is saying "You can't use the requested camera so here's the other one"? Still trying to track that down."

I would like to use the front facing camera with the AROcclusionManager, but something is forcing it to always use the back camera even when I specifically try to switch to the front camera in code.

 

Current implementations of ARKit and ARCore don't support occlusion on the front-facing camera. Here is an extension method I made to check if all features can be satisfied simultaneously using the XRSessionSubsystem.DetermineConfiguration():

Code (CSharp):

1. using JetBrains.Annotations;
2. using UnityEngine.Assertions;
3. using UnityEngine.XR.ARFoundation;
4. using UnityEngine.XR.ARSubsystems;
5.  
6.  
7. public static class ARSessionExtensions {
8.     public static bool AreAllFeaturesSupportedSimultaneously([NotNull] this ARSession session, Feature features) {
9.         var subsystem = session.subsystem;
10.         Assert.IsNotNull(subsystem, $"Please ensure that {nameof(ARSession)} was enabled at least once before calling the {nameof(AreAllFeaturesSupportedSimultaneously)}() method.");
11.         var config = subsystem.DetermineConfiguration(features);
12.         return config.HasValue && features.SetDifference(config.Value.features) == 0;
13.     }
14. }
15.  

 

Thanks for the reply! I'm actually using your ARFoundationsEssentials repo (and your AR Remote plugin) for my testing, thanks so much for all your efforts!

Without trying the above extension, it sounds like it just confirms that the front camera won't work with my config - is that about right? Simply adding AROcclusionManager to my my AR setup is what forces the system to use the back camera only, without the component, I can switch cameras at will.

@todds_unity or @mfaud, do you guys know why the front camera isn't supported? Is it a hardware limitation of the front camera or is it a limitation of ARKit itself? And if neither of those, do you know if you're planning to add support for occlusion on the front camera in the future?

 

Thank you

Yes, this is correct.

This is a limitation in the native implementation of ARKit/ARCore. AR Foundation is a wrapper around these native implementations and can't provide any extra functionality.
For example, here is the ARKit documentation that describes all available session configurations. As you can see, the user-facing camera (they call it 'selfie' in the article) is only available in ARFaceTrackingConfiguration.

 

Hi, sorry my question might be strange; but, Is there any solution in AR occlusion manager to switch off the depth mode while keeping the stencile mode on ?
In my specific case, I need to use AR occlusion manager to occlude my virtual object constently with hand/body whether the virtual object is behind the hand/body or infront of it. Indeed, I need to disable the depth detection. In the current setting both modes (Human Segmentation stencile mode and Human segmentation depth mode) needs to be enabled to make the occlusion manager work and disabling the depth mode will result in build error. So, Is there any way to reach the explained objective? Any help is appreciated!

 

Hi. Is there a way to detect if my body is colliding with some GameObject? I mean, when the camera detects my hand, does it turn my hand into a game object too?