Search Unity

Get left/right camera's textures offsets

Discussion in 'AR/VR (XR) Discussion' started by habolog, Feb 14, 2018.

  1. habolog


    May 17, 2017
    Hi guys!

    As we all know, VR engines makes some "shifts" of eye's cameras to achive stereoscopic effect. In very high level of abstractions it's means that the same image looks slightly shifted to the right in the left camera and to the left in the right camera.
    In my case, I would like to know the exact value of that shift in texture's pixels units.
    I work on HTC Vive with display size of 1512x1680 per eye, as well as rendered textures by default.
    Roughly measured by direct comparing of 2 saved RenderTexuture, that shift is about 40-60px.
    How to get that value programmatically?

    There are several open discussion about left/right camera positions which uses XR.InputTracking.GetLocalPosition(Rotation) and cameraToWolrdMatrix. But I don't really understand is it right way to get desired value of shift in pixels or should I use some different approach?

    Providing the visual example:
    ResultTexture_left_(945.7, 772.1)_(86.1, 1116.0, 1618.8)_1.png ResultTexture_right_(945.7, 772.1)_(86.1, 1116.0, 1618.8)_2.png
    The black boxes on this images are located on the same positions (in pixel coordinates of image 1512x1680), but they covers different areas of the same picture because VR engine shifted that pictures to left and right to get stereoscopic effect.
  2. bgolus


    Dec 7, 2012
    A shift is perhaps the wrong way to think about it. It's two camera views that are separated by ~65 cm (depending on your IPD settings). Additionally most VR devices have skewed projections such that "straight forward" isn't at the center of the rendered image.

    The result of this is the offset between the eyes isn't some constant value. The difference in position changes depending on how far away it is ... that's how stereoscopic version works. You would need to pick a position in 3d first and determine the "shift" from that by calculating the on screen positions.

    The XR position and rotation could get you part of the way, but what you're really going to need is the view matrix and projection matrix for both eyes.

    These two functions get you the necessary matrices.

    And you can use the functions from XRSettings to get the resolution in script if you aren't already.

    Then you need to take a world space position and multiply it by the view matrix, then the projection matrix, then apply the perspective divide to get the normalized screen position, and multiply that by the resolution.

    Psuedo code version:
    Vector4 worldPos = trans.position;
    worldPos.w = 1.0f;
    Vector4 viewPos = viewMatrix * worldPos;
    Vector4 clipPos = projMatrix * viewPos;
    Vector2 screenPos = Vector2(clipPos.x / clipPos.w, clipPos.y / clipPos.w);
    Vector2 pixelPos = Vector2(screenPos.x * eyeResWidth, screenPos.y, eyeResHeight);
    P_Jong and habolog like this.
  3. habolog


    May 17, 2017
    Thanks bgolus! It's sounds little bit complicated from my point of view, I expected more simple way on other level of abstraction. But anyway, thanks again, I'll try to sort it out somehow )
  4. evgen1580


    Apr 7, 2018
    Hi @bgolus. Thanks for your recommendation. But I have some difficulties with this task. Could you help me?
    I did what you said.

    Code (CSharp):
    1. struct appdata
    2.             {
    3.                 float4 vertex : POSITION;
    4.                 float2 uv : TEXCOORD0;
    5.             };
    7.             struct v2f
    8.             {
    9.                 float4 vertex : SV_POSITION;
    10.                 float4 screenPos: TEXCOORD1;
    11.             };
    13.             //simple vertex shader
    14.             v2f vert (appdata v)
    15.             {
    16.                 v2f o;
    18.                 o.vertex = UnityObjectToClipPos(v.vertex);
    19.                 o.screenPos = ComputeScreenPos(o.vertex);
    20.                 return o;
    21.             }
    23.             //world space fragment shader
    24.             fixed4 frag (v2f i) : SV_Target
    25.             {
    26.                 i.screenPos /= i.screenPos.w;
    28.                 //read none linear depth texture, accounting for
    29.                 float d = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,                          UnityStereoTransformScreenSpaceTex(i.screenPos)); // non-linear Z
    31.                 //pick one of the passed in projection/view matrices based on stereo eye selection
    32.                 float4x4 proj, eyeToWorld, proji, eyeToWorldi;
    34.                 if (unity_StereoEyeIndex == 0)
    35.                 {
    36.                     proj = _LeftViewFromScreen;
    37.                     eyeToWorld = _LeftWorldFromView;
    38.                     proji =_RightScreenFromView;
    39.                     eyeToWorldi = _RightViewFromWorld;
    40.                 }
    42.                 else
    43.                 {
    44.                     proj = _RightViewFromScreen;
    45.                     eyeToWorld = _RightWorldFromView;
    46.                     proji = _LeftScreenFromView;
    47.                     eyeToWorldi = _LeftViewFromWorld;
    48.                 }
    50.                 float2 uvClip = i.screenPos.xy ;
    51.                 float4 clipPos = float4(uvClip, d, 1.0);
    53.                 float4 viewPos = mul(proj, clipPos); // inverse projection by clip position
    54.                 viewPos /= viewPos.w; // perspective division
    55.                 float4 worldPos = mul(eyeToWorld, viewPos);
    56.                 float4 worldPoss = float4(,1);
    58.                 float4 viewPosi = mul(eyeToWorldi, worldPoss);
    59.                 float4 clipPosi = mul(proji,viewPosi);
    61.                 float res = abs(-clipPos.x+ clipPosi.x/clipPosi.w);
    63.                 return float4(res,0,0,0);
    But I can't get the correct shift. What am I doing wrong?
  5. bgolus


    Dec 7, 2012
    P_Jong likes this.