Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Issues with Single pass instanced HoloLens 2

Discussion in 'AR' started by sravinut, Apr 18, 2023.

  1. sravinut

    sravinut

    Joined:
    Feb 1, 2023
    Posts:
    4
    I have been working on a HoloLens 2 application that paints on the GameObjects by doing a ray cast from the hand finger tip. Think of it like splatoon, where the paint is retained on the objects.

    This works without issues with the multi pass approach.

    The above "painting" is achieved via the following shader script :
    Code (CSharp):
    1. Shader "Unlit/PaintShader"
    2. {
    3.     SubShader
    4.     {
    5.         Tags { "RenderType"="Opaque" }
    6.         ZTest  Off
    7.         ZWrite Off
    8.         Cull   Off
    9.         LOD 100
    10.         Pass
    11.         {
    12.  
    13.             CGPROGRAM
    14.  
    15.             #pragma vertex vert
    16.             #pragma fragment frag
    17.             #pragma multi_compile_instancing
    18.             #include "UnityCG.cginc"
    19.  
    20.             float _BrushSize;
    21.             sampler2D _MainTex;
    22.             float4 _BrushColor;
    23.             float _BrushOpacity;
    24.             float4x4 _ObjectToWorld;
    25.             float4 _PaintBrush;
    26.  
    27.             struct vertexIn
    28.             {
    29.                 float4 vertex : POSITION;
    30.                 float2 uv : TEXCOORD0;
    31.                 UNITY_VERTEX_INPUT_INSTANCE_ID
    32.             };
    33.  
    34.         struct vertexToFrag
    35.             {
    36.                 float4 vertex : SV_POSITION;
    37.                 float3 worldPos : TEXCOORD0;
    38.                 float2 uv : TEXCOORD1;
    39.                 UNITY_VERTEX_OUTPUT_STEREO
    40.                 UNITY_VERTEX_INPUT_INSTANCE_ID
    41.             };
    42.  
    43.             vertexToFrag vert (vertexIn v)
    44.             {
    45.  
    46.                 vertexToFrag o;
    47.  
    48.                 UNITY_SETUP_INSTANCE_ID(v);
    49.                 UNITY_TRANSFER_INSTANCE_ID(v, o);
    50.                 UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
    51.                 // The below is optional.
    52.                
    53.  
    54.                 float2 uvRemapped = v.uv.xy;
    55.  
    56.                 // transform for vertex
    57.                 // This is to ensure fragment shader renders into UV space instead of Clip Space.
    58.                 // moves from [0,1] -> [-1,1] thats is mandated by rasterizer.
    59.                 uvRemapped.y = 1. - uvRemapped.y;
    60.                 uvRemapped = uvRemapped *2. - 1.;
    61.  
    62.                 // using new uv remapped coordinates with depth always being 1 because its orthographic
    63.                 o.vertex = float4(uvRemapped.xy, 0., 1.);
    64.                 // transform to world cordinate system
    65.                 o.worldPos = mul(_ObjectToWorld, v.vertex);
    66.                 // passthrough for UV (only transform UV or vertex not both)
    67.                 o.uv = v.uv;
    68.  
    69.                 return o;
    70.  
    71.             };
    72.  
    73.  
    74.             fixed4 frag (vertexToFrag i) : SV_Target
    75.             {
    76.  
    77.                 UNITY_SETUP_INSTANCE_ID(i);
    78.  
    79.                 // color is the color of the current frag position of _MainTex
    80.                 float4 color = tex2D(_MainTex, i.uv);
    81.                 float size = _BrushSize;
    82.                 float brushOpacity = _BrushOpacity;
    83.  
    84.                 // f is the distance between the raycast hit point and the current frag
    85.                 float f = distance(_PaintBrush.xyz, i.worldPos);
    86.  
    87.                 // smoothstep(min, max, x), returns :
    88.                 // 0 if x < min | eg: min = 0.5, max = 0.7, x = 0.3 -> returns 0
    89.                 // 1 if x > max | eg: min = 0.5, max = 0.7, x = 0.8 -> returns 1
    90.                 // interpolates [0, 1] if x is in [min, max] | eg: min = 0.5, max = 0.7, x = 0.6 -> returns 0.5
    91.                 f = 1. - smoothstep(size*brushOpacity, size, f);
    92.                
    93.                 // 'f' from above becomes 0  if the pixel is greater than the brush size distance from the raycast hit point.
    94.            
    95.                 // lerp(color, _BrushColor, f * _PaintBrush.w) returns:
    96.                 // 'color' if 'f' = 0. This is how color retention happens, when the current fragment is outside the brush's influence.
    97.                 // '_BrushColor' if f = 1. The core of the brush always has '_BrushColor'.
    98.                 // linearly interpolated value between ('color', '_BrushColor') if 'f' is in (0, 1). This is used for applying brush fade.
    99.                 color = lerp(color, _BrushColor, f * _PaintBrush.w);
    100.    
    101.                 // This clamps the color range from 0 to 1 in case the lerp goes outside that range, as lerp can extrapolate.
    102.                 color = saturate(color);
    103.  
    104.                 return color;
    105.             };
    106.            
    107.             ENDCG
    108.         }
    109.     }
    110. }
    Note : The single pass specific commands (like UNITY_VERTEX_INPUT_INSTANCE_ID, UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o), #pragma multi_compile_instancing) in the shader script were used ONLY for the single pass instancing effort.

    To setup single pass instancing I :
    • Enable GPU instancing on the material that gets assigned to these objects.
    • Set Single pass instanced as the render Moder under OPEN XR settings.
    However with the single pass approach the following are the issues:

    1. When running in single pass mode stuff just renders into one eye (left eye only) WHILE actively painting on the objects.
    2. When not actively painting on the objects, stuff renders into both the eyes.
    3. The single pass specific commands in the shader seem to have no effect, even upon removing them (commenting out the likes of NITY_VERTEX_INPUT_INSTANCE_ID, UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o), #pragma multi_compile_instancing), I have the same experience as 1.
    My question :

    • Why is the single pass specific stuff in my shader seem to have no effect ?
    • Are there any other steps I am missing while building a single pass instanced application for the HoloLens 2 ?
    Thank you for your time.
     
  2. DevDunk

    DevDunk

    Joined:
    Feb 13, 2020
    Posts:
    4,969
    Follow the docs exactly.
    Stuff like UNITY_VERTEX_INPUT_INSTANCE_ID is double and should not be in v2f

    UNITY_TRANSFER_INSTANCE_ID(v, o); takes in the vertexIn in your code, while in the docs it takes in the v2f instead
     
  3. sravinut

    sravinut

    Joined:
    Feb 1, 2023
    Posts:
    4
    This issue is now resolved. This shader is purely responsible for painting the texture, so this renders to a non stereo render target, as a result the shader doesnt need any GPU instancing modifications.
    Since I was taking a command buffer approach for rendering, I needed two cameras, the main camera that uses the standard MRTK shader (hence it has support for GPU instancing to begin with) and a secondary camera responsible for painting, this renders to a non stereo target.
    All my C# scripts use this secondary camera.

    It was the usage of the secondary camera that was critical to fix the issue.
     
    thomas_key likes this.