Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Unity Human occlusion has been added to ARFoundation / ARKit

Discussion in 'Handheld AR' started by todds_unity, Nov 22, 2019.

  1. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
    The latest AR Foundation 3.1.0-preview.1 release introduces a new component AROcclusionManager. Adding this new component to the AR camera in your scene provides depth data to the ARCameraBackground component so that any available depth information may be passed to the camera background shader.

    In this initial implementation, the people occlusion functionality of ARKit 3 is used to generate depth information about people detected in the real world. The camera background rendering then uses this depth information, thus allowing the people to occlude any virtual content that appears behind them in the scene.

    At this time, only iOS devices that support the ARKit 3 people occlusion functionality will produce occlusion effects with the new AROcclusionManager. These are devices with the A12, A12X, or A13 chips running iOS 13 (or later).

    Future devices will be supported when depth map functionality is added to the respective SDKs.


    Simply adding the new AROcclusionManager component to the AR camera (along with both the ARCameraManager and ARCameraBackground components) will enable automatic human occlusion to occur on supported devices.

    occlusion-manager.png

    The new AROcclusionManager has 2 parameters: HumanSegmentationStencilMode and HumanSegmentationDepthMode. These two settings allow you to balance the quality of the depth information from ARKit and the performance cost for rendering the occlusion pass.

    HumanSegmentationStencilMode has 4 possible values as follows:
    • Disabled - No human stencil image is produced, and automatic human occlusion is disabled.
    • Fastest - A human stencil image with dimensions 256x192 is produced.
    • Medium - A human stencil image with dimensions 960x720 is produced.
    • Best - A human stencil image with dimensions 1920x1440 is produced.
    HumanSegmentationDepthMode has 3 possible values as follows:
    • Disabled - No human depth image is produced, and automatic human occlusion is disabled.
    • Fastest - A human depth image with dimensions 256x192 is produced.
    • Best - A filtering pass is applied to enhance the 256x192 human depth image.
    Note the previous dimensions/behaviors are produced by the ARKit 3 implementation and are subject to change in future devices and/or ARKit SDK versions.

    Modify the HumanSegmentationStencilMode value to alter the boundaries of the human segmentation. Modify the HumanSegmentationDepthMode to alter how the real world depth is measured. Disabling either setting will disable automatic human occlusion.

    stencil-combined.png
     
  2. newguy123

    newguy123

    Joined:
    Aug 22, 2018
    Posts:
    362
    Looks awesome. Any idea if the Android guys are working on a similar thing so we can down the line (hopefully soonish) get same across AR Foundation for both ios and Android?
     
  3. justdizzy

    justdizzy

    Joined:
    Mar 8, 2016
    Posts:
    63
    I am using stenciling for "portals" (materials have a stencil ref and compare func, and a mask is in front of the camera), will this collide with that functionality? Also, does it play well with LW/URP?
     
  4. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    I am using URP with an iPhone 11 Pro and I get a black screen with the latest preview version of ARFoundation and ARKit
     
  5. xzodia

    xzodia

    Joined:
    Jan 24, 2013
    Posts:
    35
    Does this work along side human body tracking?
     
  6. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    Aside from fixing the black screen issue, I have a question. I am using SSAO as a render feature with URP, is it possible to render the depth stencil on top of post processing effects?
     
  7. justdizzy

    justdizzy

    Joined:
    Mar 8, 2016
    Posts:
    63
    Just upgraded my project to 3.1.0-preview.1 and is still functioning properly (showing camera data as background). I am using the ARPipelineAsset from a branch of the ARFoundation-samples repo, not the default SRPAsset.
     
  8. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    Hmm, not sure if I’m doing something wrong because I basically just upgraded an existing project using URP which worked on version 3.0.0. I then created a new project and set everything up to work with URP and still got a black screen. Note I’m using Unity 2020.
     
  9. justdizzy

    justdizzy

    Joined:
    Mar 8, 2016
    Posts:
    63
    Ah, I am on 2019.2.11. I cannot speak to 2020.
     
  10. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    For me version 3.1.0 preview 1 works with LWRP but not URP. I have tested both 2019.3 and 2020 with URP. 2019.2 with LWRP works.
     
  11. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
    ARKit people occlusion and ARKit motion tracking cannot be used simultaneously. This is an ARKit restriction.
     
  12. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
    I would not expect any functionality collision.

    The automated occlusion has been tested with LWRP 6.9.2 in Unity 2019.2.12f1 and with URP 7.1.5 in Unity 2019.3.0b12.
     
  13. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
    The AROcclusionManager provides two texture properties humanStencilTexture and humanDepthTexture that can be used in your custom render passes.
     
    VadimPyrozhok and DiveshNaidoo like this.
  14. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    Thank you for the feedback. I’m unable to get the latest preview version of ARFoundation working with URP and I’ve tried both 2019.3 as well as 2020. I made a barebones project just setting up the pipeline and forward renderer with the ARbackground feature and link the pipeline in the build settings. The build gives me a black screen. I am using an iPhone 11 Pro Max
     
  15. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    Update, it works on 2020.1.0a14
     
    newguy123 likes this.
  16. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    The black screen when using ARFoundation/ARkit 3.1.0 preview 1 with URP is reproducible by enabling post processing on the camera.
     
    newguy123 likes this.
  17. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
    DiveshNaidoo and VadimPyrozhok like this.
  18. DiveshNaidoo

    DiveshNaidoo

    Joined:
    Apr 1, 2018
    Posts:
    22
    todds_unity likes this.
  19. Leonardo_Carrasco

    Leonardo_Carrasco

    Joined:
    Aug 2, 2019
    Posts:
    1
    There is no update for android yet
     
    Doraemon231 likes this.
  20. enigmatic

    enigmatic

    Joined:
    Feb 6, 2015
    Posts:
    2
    The ARFoundation works for human occlusion! can i expect it to work for building and other physical objects as well??
     
  21. eyalfx

    eyalfx

    Joined:
    Oct 8, 2010
    Posts:
    101
    Any update on support for new iPad Pro (LIDAR) and ARKit 3.5 ?
    Also , there was a mention about making a better example that handles depth and stencil for Human Segmentation. Is that still in the works?
    Thanks.
     
    Last edited: Mar 27, 2020
  22. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
    The SimpleAR scene in https://github.com/Unity-Technologies/arfoundation-samples contains a working sample. The image from the original post in this thread was made using this scene.
     
  23. sbethge

    sbethge

    Joined:
    Mar 20, 2019
    Posts:
    13
  24. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
  25. poshaughnessey

    poshaughnessey

    Joined:
    Jul 12, 2012
    Posts:
    42
    @todds_unity are you aware of a sample that translates a pixel in the humanDepthTexture to an x,y,z value in world coordinates, if one, or example, wanted to place a GameObject or a particle at one or more of the spots where a human was detected?
     
  26. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    150
    The floating point value of each pixel in the human depth texture is either (1) 0.0 if there is no human in the pixel or (2) the estimated depth in meters from the device camera to the human.

    Using information from the camera matrix, it should be possible to determine the world coordinate position of that human pixel.

    There is no sample currently that demonstrates this functionality.
     
    poshaughnessey likes this.
  27. herra_lehtiniemi

    herra_lehtiniemi

    Joined:
    Feb 12, 2017
    Posts:
    4
    Is there a built-in way in AROcclusionManager to only display the human without the background? (Basically “green screening”)? Maybe using the depth texture? How should one approach this effectively performance wise?
     
  28. zulaman

    zulaman

    Joined:
    May 12, 2017
    Posts:
    17
    I think everyone here is still waiting for similar functionality. The sample that's currently there is from a year ago (ARKit 3.0). It would be great to have a functional sample that is ready to use.
     
  29. enzokwan

    enzokwan

    Joined:
    Mar 21, 2020
    Posts:
    1
    I am using "ARkit XR Plugin 3.1.3" & "AR foundation 3.1.3"
    But I can't find "AR Occlusion Manager" in "Add Component"

    I can find "AR Occlusion Manager" in "Add Component" with "ARkit XR Plugin 4.0.1" & "AR Foundation 4.0.2"

    but camera become black screen in 4.0.1 with all setting remain the same : (

    UPDATE: I can find "AR Occlusion Manager" in 3.1.0 , but not 3.1.3....
     
    Last edited: Jun 17, 2020
unityunity