Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

How to Setup People Occlusion

Discussion in 'AR' started by Cruxx, Jun 8, 2019.

  1. vovo801

    vovo801

    Joined:
    Feb 3, 2015
    Posts:
    18
    @GSO_GT If you are using the PeopleOcclusionPostEffect.cs from the post by @Tanktop_in_Feb , I have removed the testTexture field from that script and just used m_cameraFeedTexture in this line:

    m_material.SetTexture("_CameraFeed", m_cameraFeedTexture);
     
  2. GSO_GT

    GSO_GT

    Joined:
    Aug 30, 2019
    Posts:
    17
    @vovo801 thank you so much! I think it should work fine now.
     
  3. balloonanimals

    balloonanimals

    Joined:
    Jul 16, 2015
    Posts:
    4
    I cleaned up the people occlusion cs and shader to make it a bit easier to use.

    Now it world in landscape and portrait.
    It also now with auto adjust the material's scale and offset so the screen render and camera image will align.
     

    Attached Files:

  4. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,080
    Tried this in the Simple AR example just placing the scripts in the AR camera... doesn't seem to occlude fine-ly?

    On standard resolution it produces:

    https://gyazo.com/78ce24d53378c68244c1d7605e462342

    On full screen it produces:

    https://gyazo.com/7eda3210247c36f86116f510ec5780b6
     
    Last edited: Oct 12, 2019
    vovo801 likes this.
  5. balloonanimals

    balloonanimals

    Joined:
    Jul 16, 2015
    Posts:
    4
    That is the limitation of iOS arKit3. It doesn't work perfectly. But the alignment of the mask in your photos looks good.
     
  6. mayocoo

    mayocoo

    Joined:
    Apr 1, 2017
    Posts:
    7
    I tried HumanSegmentationImages, it only shows a white image. Does anyone know the reason? I test in iphone11,build by unity 2019.2.8f1,Xcode 11.1. 20191015104535.jpg
     
  7. balloonanimals

    balloonanimals

    Joined:
    Jul 16, 2015
    Posts:
    4
    the white image is ui canvas render of the stencil texture that is to be use in the material for people occlusion. make sure your human management script instance has it enabled. If it is disabled, you don't get the stencil image
     
  8. mayocoo

    mayocoo

    Joined:
    Apr 1, 2017
    Posts:
    7
    Thank you for your reply. They all set to Standard Resolution in ARHumanBodyManager. And when I enable the ImageInfo gameobject(default exist in the sample scene), it shows m_HumanBodyManager.humanStencilTexture is null(read in TestDepthImage.cs).
     

    Attached Files:

    Last edited: Oct 16, 2019
  9. thibautvdumont

    thibautvdumont

    Joined:
    Oct 6, 2015
    Posts:
    25
    If it's ticked, try disabling 'Human Body Pose 2D Estimation' and 'Human Body Pose 3D Estimation' on your ArHumanBodyManager. It seems you can't have both the segmentation and the pose estimation.
     
  10. mayocoo

    mayocoo

    Joined:
    Apr 1, 2017
    Posts:
    7
    It already set to disable.
     
  11. thibautvdumont

    thibautvdumont

    Joined:
    Oct 6, 2015
    Posts:
    25
    Did anyone manage to get a solution that works with transparent objects as well ?
    All the shaders here are using _CameraDepthTexture and post-process on the final image, but the depth texture does not include transparent objects unfortunately. I was wondering if it wouldn't make more sense to write the segmentation depth texture directly into the ZBuffer without outputting colors. Would that be possible ? Something like :

    Code (CSharp):
    1.  
    2.    Shader "Custom/HumanSegmentationZWrite" {
    3.  Properties {
    4.     _MainTex ("Texture", 2D) = "white" {}
    5.     _StencilTex ("StencilTex", 2D) = "white" {}
    6.  }
    7.  SubShader {
    8.     Tags { "Queue"="Geometry-10" }
    9.     ColorMask 0
    10.      Pass{
    11.          ZWrite On
    12.          CGPROGRAM
    13.          
    14.              #pragma vertex vert
    15.              #pragma fragment frag
    16.  
    17.              #include "UnityCG.cginc"
    18.  
    19.              uniform sampler2D _MainTex;
    20.              uniform sampler2D _StencilTex;
    21.              float _Ratio;
    22.                
    23.              struct v2f
    24.              {
    25.                 float2 uv : TEXCOORD0;
    26.                 float4 vertex : SV_POSITION;
    27.              };
    28.  
    29.              struct appdata
    30.             {
    31.                 float4 vertex : POSITION;
    32.                 float2 uv : TEXCOORD0;
    33.                 uint   id     : SV_VertexID;
    34.             };
    35.            
    36.              struct fragOut
    37.              {
    38.                 float4 color: SV_Target;
    39.                 float depth : SV_Depth;
    40.              };
    41.                    
    42.              v2f vert(appdata v)
    43.              {
    44.  
    45.                 v2f o;
    46.  
    47.                 float4 verts[4] = {
    48.                     float4(1, 1, 0, 1),
    49.                     float4(-1, 1, 0, 1),
    50.                     float4(1, -1, 0, 1),
    51.                     float4(-1, -1, 0, 1),
    52.                 };
    53.                 o.vertex = verts[v.id];
    54.                 o.uv = 1 - v.uv;
    55.  
    56.                 return o;
    57.  
    58.                 o.vertex = UnityObjectToClipPos(v.vertex);
    59.                 o.uv = v.uv;
    60.                 return o;
    61.              }
    62.  
    63.              fragOut frag(v2f i) {
    64.                 float2 uv = i.uv;
    65.                
    66.                 uv.y = 1.0 - uv.y; // flip the depth texture on Y
    67.  
    68.                 // rotate the depth texture by -90
    69.                 float x = uv.x;
    70.                 float y = uv.y;
    71.                 uv.x = y;
    72.                 uv.y = 1.0 - x;
    73.  
    74.                 // crop and scale on y (which will actually crop & scale on x in portrait)
    75.                 uv.y = (uv.y - ((1 - _Ratio) / 2)) / _Ratio;
    76.  
    77.                 fragOut o;
    78.                 o.color = float4(1,0,0,1); // debug purpose
    79.  
    80.                 // Cut out everything that's not a human using the stencil
    81.                 float stencil = tex2D(_StencilTex, uv).r;
    82.                 if (stencil < 0.9)
    83.                 {
    84.                 #if defined(UNITY_REVERSED_Z)
    85.                     o.depth = 0;
    86.                 #else
    87.                     o.depth = 1;
    88.                 #endif
    89.                 }
    90.                 else
    91.                 {
    92.                     float worldDepth = tex2D(_MainTex, uv).r;
    93.                     float nearPlaneZ = _ProjectionParams.y;
    94.                     float farPlaneZ = _ProjectionParams.z;
    95.                     float frustrumDepth = farPlaneZ - nearPlaneZ;
    96.                     float frustrumRelativeDepth = worldDepth / frustrumDepth;
    97.                     frustrumRelativeDepth = clamp(frustrumRelativeDepth, 0, 1);
    98.  
    99.                     #if defined(UNITY_REVERSED_Z)
    100.                         frustrumRelativeDepth = 1.0f - frustrumRelativeDepth;
    101.                     #endif
    102.  
    103.                     o.depth = frustrumRelativeDepth;
    104.                 }
    105.              
    106.                 return o;
    107.              }
    108.          ENDCG
    109.      }
    110.  }
    111.  }
    I'm not sure when and how to render the depth texture with this material. Any help would be appreciated !
     
    Last edited: Oct 25, 2019
  12. thibautvdumont

    thibautvdumont

    Joined:
    Oct 6, 2015
    Posts:
    25
    It hasn't been talked too much in this thread, but could you clarify this point ?
    On one hand, the depth buffer value is in meters [0, infinity), which led me to think it was expressed relative to the eye of the camera.
    On the other hand, you mention that a 0 value means a human occluding pixel at the near clip plane or a non-human pixel, which actually would imply that those metric values are expressed relative to the near plane.

    Could you kindly tells me which of those assertion is true ? This is the calculation I'm doing atm to get the ZBuffer value :

    Code (CSharp):
    1.                     float worldDepth = tex2D(_MainTex, uv).r;
    2.                     float nearPlaneZ = _ProjectionParams.y;
    3.                     float farPlaneZ = _ProjectionParams.z;
    4.                     float frustrumDepth = farPlaneZ - nearPlaneZ;
    5.                     float frustrumRelativeDepth = (worldDepth - nearPlaneZ) / frustrumDepth;
    6.                     frustrumRelativeDepth = clamp(frustrumRelativeDepth, 0, 1);
    7.  
    8.                     #if defined(UNITY_REVERSED_Z)
    9.                         frustrumRelativeDepth = 1.0f - frustrumRelativeDepth;
    10.                     #endif
    11.  
    12.                     o.depth = frustrumRelativeDepth;
     
  13. KzoNag

    KzoNag

    Joined:
    Nov 21, 2014
    Posts:
    4
  14. mayocoo

    mayocoo

    Joined:
    Apr 1, 2017
    Posts:
    7
    KzoNag likes this.
  15. AlterHaudegen

    AlterHaudegen

    Joined:
    Jul 8, 2013
    Posts:
    28
    Thanks for posting your code!

    This worked great for segmenting when I put my hand in front of things, but not so well for people walking into the scene. I tweaked the depth like this and it works well for people and a bit less well for hands (as in better for far, worse for close):

    Code (CSharp):
    1. float depth = (1.0 - _ZBufferParams.w * depthMeter) / (depthMeter * _ZBufferParams.z) / _depthFactor;
    at _depthFactor = 3.5 I had the best results.
     
    KzoNag likes this.
  16. raaden89

    raaden89

    Joined:
    Nov 16, 2019
    Posts:
    2
    Thanks for sharing code! one question.. there is any way for create the same things in arcore?
     
  17. KzoNag

    KzoNag

    Joined:
    Nov 21, 2014
    Posts:
    4
    Thank you for sharing your technique!! I do not understand exactry about transform between meter depth with [0...1] depth valu. Is my formula wrong? Or, is the meter depth from ARKit low accurency?
     
  18. KzoNag

    KzoNag

    Joined:
    Nov 21, 2014
    Posts:
    4
    Unfortunately, I do not know that is there any way to get depth information in ARCore.
     
  19. AbradolfLinkler

    AbradolfLinkler

    Joined:
    Feb 26, 2017
    Posts:
    26
    Hello Guys,
    Does People Occlusion (Occlusion Manager) work with Universal Render Pipeline? Anyone tried this?

    I am using AR Foundation 3.1.0, ARKit 3.1.0 and Unity 2019.3.0f6 with Universal Render Pipeline configured for AR Foundation - testing on an iPhone 8 Plus.

    AR foundation works fine with Universal render pipeline and its features, but when I add Occlussion Manager componen to the AR Camera it seems to have no effect at all (regardless of Fastest or Best settings).

    Is there some special workflow to make People Occlusion work properly with URP?
     
  20. todds_unity

    todds_unity

    Joined:
    Aug 1, 2018
    Posts:
    324
    ARKit people occlusion requires an A12/A12X bionic processor or later.

    People occlusion is not supported on the iPhone 8 Plus.
     
    unbreakable3D likes this.
  21. AbradolfLinkler

    AbradolfLinkler

    Joined:
    Feb 26, 2017
    Posts:
    26
    todds_unity likes this.
  22. hot-knife-digital

    hot-knife-digital

    Joined:
    Oct 22, 2013
    Posts:
    12
    Is it possible to use the shader graph with this instead?

    edit - nevermind I figured it out!
     
    Last edited: Mar 13, 2020
  23. MoyosaMedia

    MoyosaMedia

    Joined:
    Jul 20, 2017
    Posts:
    3
    Is it possible to do this with the front camera?
     
  24. todds_unity

    todds_unity

    Joined:
    Aug 1, 2018
    Posts:
    324
    No. The functionality is currently available only with the rear-facing camera.
     
  25. Prolantis

    Prolantis

    Joined:
    Jan 17, 2016
    Posts:
    5
    Is there a plan to make it work on the front camera any time soon?
     
  26. todds_unity

    todds_unity

    Joined:
    Aug 1, 2018
    Posts:
    324
    ARFoundation uses the functionality in ARKit. I am unaware of any plan to support front camera people occlusion in ARKit.
     
  27. SeokhyeonJeong

    SeokhyeonJeong

    Joined:
    Mar 18, 2020
    Posts:
    3

    I want to make the body appear without background in your project. I am not sure what to do.
     
  28. todds_unity

    todds_unity

    Joined:
    Aug 1, 2018
    Posts:
    324
    You could write your own background shader that uses the AROcclusionManger.humanStencilTexture to only render the human pixels.
     
  29. SeokhyeonJeong

    SeokhyeonJeong

    Joined:
    Mar 18, 2020
    Posts:
    3
    Isn't this the way people cannot be expressed in rgb?
     
  30. todds_unity

    todds_unity

    Joined:
    Aug 1, 2018
    Posts:
    324
    In your custom background shader, you would choose to render the pixel from the camera texture if the pixel from the AROcclusionManger.humanStencilTexture is set.
     
  31. SeokhyeonJeong

    SeokhyeonJeong

    Joined:
    Mar 18, 2020
    Posts:
    3
    thanks.
    I did it using a shader graph.
     
    Last edited: Jul 1, 2020
    TropicalCyborg likes this.
  32. Velcrohead

    Velcrohead

    Joined:
    Apr 26, 2014
    Posts:
    78
    Been trying to figure out how to rebuilt the people occlusion shader in Amplify Shader Editor with a moderate amount of success. Getting a weird parallax / depth issue with the stencil where the drawn material on top is not scaled correctly nor positioned correctly (and doesn't stick to the person when rotating the camera), which I'm not sure how to resolve really.

    Has anyone else tried this, or got any ideas?

    Cheers
     
  33. Velcrohead

    Velcrohead

    Joined:
    Apr 26, 2014
    Posts:
    78
    Sorry bump, gonna have to come back to this in a few days!