Search Unity

How to Setup People Occlusion

Discussion in 'Handheld AR' started by Cruxx, Jun 8, 2019.

  1. Cruxx

    Cruxx

    Joined:
    Nov 3, 2018
    Posts:
    6
    I have just have my unity AR foundation and IOS updated to the latest preview version. I tried to get the new fancy feature people occlusion works but have no idea where to start with.
    I read the doc and look through the sample repo and did not find anything helpful (maybe it's just me being stupid). Is there a option I should turn on in the unity editor, or is that function something being similar with the light estimation which will require external scripts?
    Thank you
     
  2. tdmowrer

    tdmowrer

    Unity Technologies

    Joined:
    Apr 21, 2017
    Posts:
    480
    Check out the "HumanSegmentationImages" sample scene. Right now, it just surfaces the stencil and depth buffers as raw images. We're working on a better sample, but that should get you started.
     
  3. virtualHCIT

    virtualHCIT

    Joined:
    Aug 20, 2018
    Posts:
    10
    I was wondering if you could suggest how to go about using the depth and stencil raw buffers to perform occlusion. Would these have to be used in a custom postprocess effect? Can they be applied directly or do they have to be cropped/rotated to match the screen resolution/orientation?

    Thanks!
     
    Last edited: Jun 10, 2019
  4. virtualHCIT

    virtualHCIT

    Joined:
    Aug 20, 2018
    Posts:
    10
    Some other questions: does UnityARKit_HumanBodyProvider_TryGetHumanDepth and UnityARKit_HumanBodyProvider_TryGetHumanStencil use Apple's generateDilatedDepthFromFrame and generateMatteFromFrame to acquire the buffers? (i.e. are the buffers already scaled and refined using ARKit's API?)
     
  5. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    141
    Is there any chance that there will be a way to use the human segmentation-generated Depth Buffer as the base depth buffer or will it be necessary to sample it separately inside a shader and then perform some kind of check with the frag depth or something like that?
     
  6. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    38
    You will need to write your own shader that uses both the stencil and depth image.

    The values in the depth buffer are in meters with the range [0, infinity) and need to be converted into the view space with the depth value [0, 1] mapped between the near & far clip plane. Additionally, because non-human pixels in the human depth image are 0, you will need to use the stencil buffer to know whether that 0 value means a human occluding pixel at the near clip plane or a non-human pixel which should get the value of the far clip plane.

    Todd
     
  7. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    141
    Thanks for the extra info! Sorry if this is a dumb question, but would there potentially be some way of blitting the depth texture onto the frame buffer's depth buffer before the whole scene draws (like right after the Camera clears it)? For scenarios where we don't need the stencil buffer functionality and only want to use the depth buffer, this seems like it would really simplify the workflow. I have an AR project that I'm working on adding people occlusion to, but it uses some rather complex shaders from the asset store that are really tough to modify because they use all kinds of #include .cginc files and compiler preprocessors...

    Also, any estimate on when we might see a sample demonstrating the technique you outlined in your previous post?

    Thanks!
     
  8. Keion92

    Keion92

    Joined:
    Sep 21, 2015
    Posts:
    2
    The examples provides us with the stencil and depth data for People Occlusion but how can we get the original depth map? Is it possible to control how far it will detect the people occlusion or is it a fixed value?
     
  9. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    141
    I feel like i'm a little out of my depth (no pun intended) here, but it seems like drawing a full screen quad with a shader that only outputs to SV_Depth before the rest of the scene draws would be the simplest implementation of basic occlusion.

    I'm a bit of a neophyte when it comes to View space... I don't really understand the range of values derived from something like
    Code (CSharp):
    1. UnityObjectToViewPos( float4( v.vertex.xyz, 1.0 ) )
    Clip space I understand a bit more because the x and y components are (-1 -> 1), but I don't really understand the z and w components.

    Anyway, I hope Unity will include an example that demonstrates just writing the depth stuff to SV_Depth because that seems like a much more user friendly way of adding this to any project.
     
    ina likes this.
  10. eco_bach

    eco_bach

    Joined:
    Jul 8, 2013
    Posts:
    1,275
    @todds_unity So if I understand correctly, this custom shader will only be concerned with returning numerical depth data, not necessarily displaying anything ?
     
  11. Keion92

    Keion92

    Joined:
    Sep 21, 2015
    Posts:
    2
    @todds_unity Is there a reason why the image in the HumanSegmentationImages is reversed? We are attempting to unrevert it so we can apply the shader. If you know how to uninvert the output from the humansegmentationimage scene, please let us know.

    Thank you.
     
  12. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    141
    In whatever shader you're using to sample from the depth/stencil textures, just invert the y coordinate:
    Code (CSharp):
    1. uv.y = 1.0 - uv.y;
     
  13. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    774
    waiting also for a working example