Search Unity

How to Setup People Occlusion

Discussion in 'Handheld AR' started by Cruxx, Jun 8, 2019.

  1. Cruxx

    Cruxx

    Joined:
    Nov 3, 2018
    Posts:
    6
    I have just have my unity AR foundation and IOS updated to the latest preview version. I tried to get the new fancy feature people occlusion works but have no idea where to start with.
    I read the doc and look through the sample repo and did not find anything helpful (maybe it's just me being stupid). Is there a option I should turn on in the unity editor, or is that function something being similar with the light estimation which will require external scripts?
    Thank you
     
  2. tdmowrer

    tdmowrer

    Unity Technologies

    Joined:
    Apr 21, 2017
    Posts:
    490
    Check out the "HumanSegmentationImages" sample scene. Right now, it just surfaces the stencil and depth buffers as raw images. We're working on a better sample, but that should get you started.
     
  3. virtualHCIT

    virtualHCIT

    Joined:
    Aug 20, 2018
    Posts:
    13
    I was wondering if you could suggest how to go about using the depth and stencil raw buffers to perform occlusion. Would these have to be used in a custom postprocess effect? Can they be applied directly or do they have to be cropped/rotated to match the screen resolution/orientation?

    Thanks!
     
    Last edited: Jun 10, 2019
  4. virtualHCIT

    virtualHCIT

    Joined:
    Aug 20, 2018
    Posts:
    13
    Some other questions: does UnityARKit_HumanBodyProvider_TryGetHumanDepth and UnityARKit_HumanBodyProvider_TryGetHumanStencil use Apple's generateDilatedDepthFromFrame and generateMatteFromFrame to acquire the buffers? (i.e. are the buffers already scaled and refined using ARKit's API?)
     
  5. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    164
    Is there any chance that there will be a way to use the human segmentation-generated Depth Buffer as the base depth buffer or will it be necessary to sample it separately inside a shader and then perform some kind of check with the frag depth or something like that?
     
  6. todds_unity

    todds_unity

    Unity Technologies

    Joined:
    Aug 1, 2018
    Posts:
    38
    You will need to write your own shader that uses both the stencil and depth image.

    The values in the depth buffer are in meters with the range [0, infinity) and need to be converted into the view space with the depth value [0, 1] mapped between the near & far clip plane. Additionally, because non-human pixels in the human depth image are 0, you will need to use the stencil buffer to know whether that 0 value means a human occluding pixel at the near clip plane or a non-human pixel which should get the value of the far clip plane.

    Todd
     
    Colin_MacLeod likes this.
  7. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    164
    Thanks for the extra info! Sorry if this is a dumb question, but would there potentially be some way of blitting the depth texture onto the frame buffer's depth buffer before the whole scene draws (like right after the Camera clears it)? For scenarios where we don't need the stencil buffer functionality and only want to use the depth buffer, this seems like it would really simplify the workflow. I have an AR project that I'm working on adding people occlusion to, but it uses some rather complex shaders from the asset store that are really tough to modify because they use all kinds of #include .cginc files and compiler preprocessors...

    Also, any estimate on when we might see a sample demonstrating the technique you outlined in your previous post?

    Thanks!
     
  8. Keion92

    Keion92

    Joined:
    Sep 21, 2015
    Posts:
    5
    The examples provides us with the stencil and depth data for People Occlusion but how can we get the original depth map? Is it possible to control how far it will detect the people occlusion or is it a fixed value?
     
    cjensen1 and ina like this.
  9. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    164
    I feel like i'm a little out of my depth (no pun intended) here, but it seems like drawing a full screen quad with a shader that only outputs to SV_Depth before the rest of the scene draws would be the simplest implementation of basic occlusion.

    I'm a bit of a neophyte when it comes to View space... I don't really understand the range of values derived from something like
    Code (CSharp):
    1. UnityObjectToViewPos( float4( v.vertex.xyz, 1.0 ) )
    Clip space I understand a bit more because the x and y components are (-1 -> 1), but I don't really understand the z and w components.

    Anyway, I hope Unity will include an example that demonstrates just writing the depth stuff to SV_Depth because that seems like a much more user friendly way of adding this to any project.
     
    ina likes this.
  10. eco_bach

    eco_bach

    Joined:
    Jul 8, 2013
    Posts:
    1,322
    @todds_unity So if I understand correctly, this custom shader will only be concerned with returning numerical depth data, not necessarily displaying anything ?
     
  11. Keion92

    Keion92

    Joined:
    Sep 21, 2015
    Posts:
    5
    @todds_unity Is there a reason why the image in the HumanSegmentationImages is reversed? We are attempting to unrevert it so we can apply the shader. If you know how to uninvert the output from the humansegmentationimage scene, please let us know.

    Thank you.
     
  12. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    164
    In whatever shader you're using to sample from the depth/stencil textures, just invert the y coordinate:
    Code (CSharp):
    1. uv.y = 1.0 - uv.y;
     
  13. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    799
    waiting also for a working example
     
  14. Keion92

    Keion92

    Joined:
    Sep 21, 2015
    Posts:
    5
    What variable tells us that the depth or stencil value is 0 or 1? I can only find the texture humanStencil and humanDepth. How can I get the depth data from what we already have?
     
  15. Keion92

    Keion92

    Joined:
    Sep 21, 2015
    Posts:
    5
    Is there a reason why the provided sample has the camera output very zoomed in compared to the depth/stencil data output? How can we change the output camera so that it sees what the original camera sees / stencil depth output sees. Because they are not the same outputs.

    @todds_unity @tdmowrer
     
  16. eco_bach

    eco_bach

    Joined:
    Jul 8, 2013
    Posts:
    1,322
    Yes agree. To get anything working took a lot of trial and error hackery using Blitting and even then the results are barely acceptable
     
    Cruxx, FlyingRio and ina like this.
  17. Keion92

    Keion92

    Joined:
    Sep 21, 2015
    Posts:
    5
    For people depth value, is there a limitation for the distance between the camera and the people detected? After 1 meter, the people depth value is a pure color meaning that there's no more change in value.
     
  18. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    164
    It goes beyond 1 meter, but in a fragment shader, 1.0 in the red channel is fully red. If you try dividing the depth by 10, you wouldn't reach full red until 10 meters away. Make sense? It's a floating point texture so it can store values beyond 1.0.
     
    FlyingRio likes this.
  19. HVRAT

    HVRAT

    Joined:
    Aug 28, 2018
    Posts:
    6
    IMG_0065.jpg IMG_0066.jpg

    Why under landscape mode, depth/stencil is flipped horizontally (which can be fixed in shader by uv.x = 1 - uv.x;
    ), and when holding my phone in portrait mode, depth/stencil is not changing accordingly, anyone how to fix this?
     
  20. edo_m18

    edo_m18

    Joined:
    Nov 17, 2014
    Posts:
    3
    I have same issue.

    Can anyone tell me about what the problem is?
    I'm working on masking the AR scene with stencil texture.

    I tried it as just mask value texture first, but it didn't show me correctly. Next, I tried to flip and scale the texture. But it didn't work. Please see the attached image. It was a little bit wrong. I also attached my shader code. Please tell me what how to fix the problem?

    This shader code is used in posteffect.

    Code (CSharp):
    1. v2f vert (appdata v)
    2. {
    3.     v2f o;
    4.     o.vertex = UnityObjectToClipPos(v.vertex);
    5.     o.uv = v.uv;
    6.     return o;
    7. }
    8.  
    9. sampler2D _MainTex;
    10. sampler2D _DepthTex;
    11. sampler2D _StencilTex;
    12.  
    13. fixed4 frag (v2f i) : SV_Target
    14. {
    15.     fixed4 col = tex2D(_MainTex, i.uv);
    16.  
    17.     float2 uv = i.uv;
    18.     uv.x = 1.0 - uv.x;
    19.  
    20.     // The reason of "1.62" is correcting ratio.
    21.     // Passed stencil texture is not same ratio to the display.
    22.     uv.y = (uv.y + 0.5) / 1.62;
    23.  
    24.     float stencil = tex2D(_StencilTex, uv).r;
    25.  
    26.     return lerp(col, float4(1, 0, 0, 1), stencil);
    27. }
     

    Attached Files:

    Last edited: Aug 8, 2019
  21. edo_m18

    edo_m18

    Joined:
    Nov 17, 2014
    Posts:
    3
    I found out correct code. (But it works only portrait left)

    Code (CSharp):
    1. fixed4 frag (v2f i) : SV_Target
    2. {
    3.     fixed4 col = tex2D(_MainTex, i.uv);
    4.  
    5.     // Correcting ratio from 2688x1242 to 1920x1440
    6.     float ratio = 1.62;
    7.  
    8.     float2 uv = i.uv;
    9.     uv.x = 1.0 - uv.x;
    10.     uv.y /= ratio;
    11.     uv.y += 1.0 - (ratio * 0.5);
    12.  
    13.     float stencil = tex2D(_StencilTex, uv).r;
    14.     if (stencil >= 0.9)
    15.     {
    16.         return col * float4(stencil, 0.0, 0.0, 1.0);
    17.     }
    18.     else
    19.     {
    20.         return col;
    21.     }
    22. }
     

    Attached Files:

  22. dilmer

    dilmer

    Joined:
    Jun 15, 2013
    Posts:
    223
    That looks great, I have the same issue where is a bit off, can you post the entire shader and possibly setup?
     
  23. edo_m18

    edo_m18

    Joined:
    Nov 17, 2014
    Posts:
    3
    Thank you for replying.

    Here is my code. The code have to a little bit optimize I think.
    And I attached an example of result this code. IMG_0114.PNG

    First, This is the shader code.

    Code (CSharp):
    1. Shader "Hidden/PeopleOcclusion"
    2. {
    3.     Properties
    4.     {
    5.         _MainTex ("Texture", 2D) = "white" {}
    6.     }
    7.     SubShader
    8.     {
    9.         Cull Off ZWrite Off ZTest Always
    10.  
    11.         Pass
    12.         {
    13.             CGPROGRAM
    14.             #pragma vertex vert
    15.             #pragma fragment frag
    16.  
    17.             #include "UnityCG.cginc"
    18.  
    19.             struct appdata
    20.             {
    21.                 float4 vertex : POSITION;
    22.                 float2 uv : TEXCOORD0;
    23.             };
    24.  
    25.             struct v2f
    26.             {
    27.                 float2 uv : TEXCOORD0;
    28.                 float4 vertex : SV_POSITION;
    29.             };
    30.  
    31.             v2f vert (appdata v)
    32.             {
    33.                 v2f o;
    34.                 o.vertex = UnityObjectToClipPos(v.vertex);
    35.                 o.uv = v.uv;
    36.                 return o;
    37.             }
    38.  
    39.             sampler2D _MainTex;
    40.             sampler2D _BackgroundTex;
    41.             sampler2D _DepthTex;
    42.             sampler2D _StencilTex;
    43.  
    44.             UNITY_DECLARE_DEPTH_TEXTURE(_CameraDepthTexture);
    45.  
    46.             fixed4 frag (v2f i) : SV_Target
    47.             {
    48.                 fixed4 col = tex2D(_MainTex, i.uv);
    49.  
    50.                 float2 uv = i.uv;
    51.  
    52.                 // Flip x axis.
    53.                 uv.x = 1.0 - uv.x;
    54.  
    55.                 // Correcting textures ratio that can be got by ARHumanBodyManager to the screen ratio.
    56.                 float ratio = 1.62;
    57.                 uv.y /= ratio;
    58.                 uv.y += 1.0 - (ratio * 0.5);
    59.  
    60.                 float stencil = tex2D(_StencilTex, uv).r;
    61.                 if (stencil < 0.9)
    62.                 {
    63.                     return col;
    64.                 }
    65.  
    66.                 // Check depth delta. If delta is over zero, it means pixels that estimated like human is in front of AR objects.
    67.                 float depth = tex2D(_DepthTex, uv).r;
    68.                 float sceneZ = LinearEyeDepth(tex2D(_CameraDepthTexture, i.uv));
    69.                 float delta = saturate(sceneZ - depth);
    70.                 if (delta > 0.0)
    71.                 {
    72.                     return tex2D(_BackgroundTex, i.uv);
    73.                 }
    74.                 else
    75.                 {
    76.                     return col;
    77.                 }
    78.             }
    79.             ENDCG
    80.         }
    81.     }
    82. }
    83.  

    Next, here is the C# code.
    I cloned it from AR foundation example code and customize a little bit it.

    My approach is to render as post-effect.

    Code (CSharp):
    1. using System.Text;
    2. using UnityEngine;
    3. using UnityEngine.UI;
    4. using UnityEngine.XR.ARFoundation;
    5.  
    6. public class PeopleOcclusion : MonoBehaviour
    7. {
    8.     [SerializeField, Tooltip("The ARHumanBodyManager which will produce frame events.")]
    9.     private ARHumanBodyManager _humanBodyManager;
    10.  
    11.     [SerializeField]
    12.     private Material _material = null;
    13.  
    14.     [SerializeField]
    15.     private ARCameraBackground _arCameraBackground = null;
    16.  
    17.     [SerializeField]
    18.     private RawImage _captureImage = null;
    19.  
    20.     private RenderTexture _captureTexture = null;
    21.  
    22.     public ARHumanBodyManager HumanBodyManager
    23.     {
    24.         get { return _humanBodyManager; }
    25.         set { _humanBodyManager = value; }
    26.     }
    27.  
    28.     [SerializeField]
    29.     private RawImage _rawImage;
    30.  
    31.     /// <summary>
    32.     /// The UI RawImage used to display the image on screen.
    33.     /// </summary>
    34.     public RawImage RawImage
    35.     {
    36.         get { return _rawImage; }
    37.         set { _rawImage = value; }
    38.     }
    39.  
    40.     [SerializeField]
    41.     private Text _imageInfo;
    42.  
    43.     /// <summary>
    44.     /// The UI Text used to display information about the image on screen.
    45.     /// </summary>
    46.     public Text ImageInfo
    47.     {
    48.         get { return _imageInfo; }
    49.         set { _imageInfo = value; }
    50.     }
    51.  
    52.     #region ### MonoBehaviour ###
    53.     private void Awake()
    54.     {
    55.         Camera camera = GetComponent<Camera>();
    56.         camera.depthTextureMode |= DepthTextureMode.Depth;
    57.  
    58.         _rawImage.texture = _humanBodyManager.humanDepthTexture;
    59.  
    60.         _captureTexture = new RenderTexture(Screen.width, Screen.height, 0);
    61.         _captureImage.texture = _captureTexture;
    62.     }
    63.     #endregion ### MonoBehaviour ###
    64.  
    65.     private void LogTextureInfo(StringBuilder stringBuilder, string textureName, Texture2D texture)
    66.     {
    67.         stringBuilder.AppendFormat("texture : {0}\n", textureName);
    68.         if (texture == null)
    69.         {
    70.             stringBuilder.AppendFormat("   <null>\n");
    71.         }
    72.         else
    73.         {
    74.             stringBuilder.AppendFormat("   format : {0}\n", texture.format.ToString());
    75.             stringBuilder.AppendFormat("   width  : {0}\n", texture.width);
    76.             stringBuilder.AppendFormat("   height : {0}\n", texture.height);
    77.             stringBuilder.AppendFormat("   mipmap : {0}\n", texture.mipmapCount);
    78.         }
    79.     }
    80.  
    81.     private void Update()
    82.     {
    83.         var subsystem = _humanBodyManager.subsystem;
    84.  
    85.         if (subsystem == null)
    86.         {
    87.             if (_imageInfo != null)
    88.             {
    89.                 _imageInfo.text = "Human Segmentation not supported.";
    90.             }
    91.             return;
    92.         }
    93.  
    94.         StringBuilder sb = new StringBuilder();
    95.         Texture2D humanStencil = _humanBodyManager.humanStencilTexture;
    96.         Texture2D humanDepth = _humanBodyManager.humanDepthTexture;
    97.         LogTextureInfo(sb, "stencil", humanStencil);
    98.         LogTextureInfo(sb, "depth", humanDepth);
    99.  
    100.         if (_imageInfo != null)
    101.         {
    102.             _imageInfo.text = sb.ToString();
    103.         }
    104.  
    105.         _material.SetTexture("_StencilTex", humanStencil);
    106.         _material.SetTexture("_DepthTex", humanDepth);
    107.         _material.SetTexture("_BackgroundTex", _captureTexture);
    108.     }
    109.  
    110.     private void LateUpdate()
    111.     {
    112.         if (_arCameraBackground.material != null)
    113.         {
    114.             Graphics.Blit(null, _captureTexture, _arCameraBackground.material);
    115.         }
    116.     }
    117.  
    118.     private void OnRenderImage(RenderTexture src, RenderTexture dest)
    119.     {
    120.         Graphics.Blit(src, dest, _material);
    121.     }
    122. }
    123.  
     
    thorikawa and JurreFonk like this.
  24. knotttrodt

    knotttrodt

    Joined:
    Nov 27, 2017
    Posts:
    1
    Have you fixed the rotation problem?