Search Unity

  1. Unity 2019.1 is now released.
    Dismiss Notice

Implementing locatable camera shader code

Discussion in 'Windows Mixed Reality' started by just_another_username, Jul 21, 2016.

  1. just_another_username


    Jul 21, 2016
    What I'm trying to do is take a photo, pick a pixel location in the photo to place a hologram, then actually place the hologram in the corresponding 3D location. I found a tutorial for this here under "Pixel to Application-specified Coordinate System": However, the code for the translation between the pixel coordinate to real world coordinate is in shader code. I created a custom shader and a material for that shader, but my question is how does the shader code snippet provided in that link actually fit into the shader file? Also, to what gameobject should the material be attached?

    As a bonus, it'd be great if anyone could explain why this is done in shader code. This thread has clues on what I want to accomplish, but the shader code is not included.
  2. BrandonFogerty


    Unity Technologies

    Jan 29, 2016
    Hi @just_another_username

    I recently posted a thread on how to do the reverse of what you are trying to do. The following link will provide you source code that demonstrates how to blend a captured photo with your physical environment using the PhotoCapture API. Please read through the code and let me know if it makes sense to you.

    You are essentially trying to go in the opposite direction. I don't believe you need a shader at all for what you are trying to do unless you want to highlight the selected pixel on the captured image. The code example listed on Microsoft's locatable camera documentation is not necessarily intended to be shader code.

    I put together the following code snippet to serve as an overview of what you would need to do assuming you are placing your captured image on one of Unity's quad game objects.

    Code (CSharp):
    1. public class PhotoPixelToWorldExample:MonoBehaviour
    2. {
    4.     public GameObject Canvas;
    5.     PhotoCaptureFrame photoCaptureFrame;
    7.     void Update()
    8.     {
    9.         // In Editor we can test more easily with the mouse.
    10.         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
    12.         // On the HoloLens you can replace the above line with
    13.         //Ray ray = new Ray(Camera.main.transform.position, Camera.main.transform.forward);
    15.         RaycastHit[] hits = Physics.RaycastAll(ray,1000.0f);
    16.         if(hits == null || hits.Length <= 0)
    17.         {
    18.             return;
    19.         }
    21.         RaycastHit hit = hits[0];
    22.         if(hit.collider.gameObject != Canvas)
    23.         {
    24.             return;
    25.         }
    27.         // First we need to figure our where our intersected pixel is
    28.         // in our virtual camera's clip space.
    30.         // Transform our intersection point into the object space of
    31.         // the quad/canvas game object.  This means that the quad
    32.         // will now be located at the origin (0,0,0) and our intersection point
    33.         // will be offset from that location by the same amount that it currently is.
    34.         // The intention behind this is to simplify the problem of transforming
    35.         // the intersected pixel coordinate into the -1 to 1 range.
    36.         // After this transformation, we can easily do that by just
    37.         // sampling the x and y coordinates of the transformed intersection point.
    38.         // In Unity, the quad primitive will range from -0.5 to 0.5 for both its width
    39.         // and height in object space.
    40.         Vector3 ImagePosProjected = Canvas.transform.worldToLocalMatrix * hit.point;
    41.         // Map the -0.5 to 0.5 domain to a -1.0 to 1.0 range.
    42.         ImagePosProjected.x *= 2.0f;
    44.         // From here you can follow the Microsoft documentation
    46.         Matrix4x4 cameraToWorldMatrix;
    47.         photoCaptureFrame.TryGetCameraToWorldMatrix(out cameraToWorldMatrix);
    48.         Matrix4x4 worldToCameraMatrix = cameraToWorldMatrix.inverse;
    50.         Matrix4x4 projectionMatrix;
    51.         photoCaptureFrame.TryGetProjectionMatrix(out projectionMatrix);
    53.         // Transform the projected clip space position back into real world
    54.         // web camera space
    55.         // Vector3 CameraSpacePos = UnProjectVector(projectionMatrix,
    56.         // new Vector3(ImagePosProjected.x, ImagePosProjected.y,1.0f));
    58.         // Transform the captured image's pixel location that is in the
    59.         // real world space of the physical web camera back into our
    60.         // virtual world space.
    61.         // Vector3 WorldSpaceRayPoint = cameraToWorldMatrix * CameraSpacePos;
    62.     }
    63. }
    Now for bonus points!

    The Holographic Blend shader, that I posted in the above link, essentially transforms the uvs of the quad mesh so that we sample pixels from the capture imaged texture in a way that makes it blend in with our real world environment. Since the captured image was captured by the physical web camera, and not our virtual camera, we need transform the uvs by the physical web camera's world to camera and projection matrices. This is a per pixel operation. So while we *could* do this on the CPU in a script, it would be very slow. The GPU however is much more optimized for handling per pixel operations.

    I hope that helps!
    behram and Unity_Wesley like this.
  3. just_another_username


    Jul 21, 2016
    Thanks for the help! I've learned a lot through your examples and I'm sure this will help someone else down the road!
    BrandonFogerty likes this.
  4. Waterine


    Jul 27, 2016
    Hi Brandon,

    Thank you for your examples and explanations, they are very helpful!

    Now if I want to do this outside a shader, what would be the best way to implement UnProjectVector()? The projection matrix from locatable camera is not invertible. What I am trying to do is to map image analysis results (OCR, face recognition, etc.) back to 3D world by raycasting onto spatial mapping layer.

    I started with using main camera and ScreenPointToRay, which gave me pretty good approximation when the capture resolution is 1280x720, but for best accuracy I want to use the locatable camera information. Then I tried to create a secondary camera just for raycasting, set its transform to the locatable camera's transform. As soon as I set the raycasting camera's projection matrix to be the same as the locatable camera's, ScreenPointToRay stopped working.

    Could you advise what's the best way to make use the projection matrix in my scenario?
  5. BrandonFogerty


    Unity Technologies

    Jan 29, 2016
    Hi @Waterine

    The projection matrices for the physical web camera on the HoloLens and your virtual main camera will be quite different. Therefore they are not interchangeable. Microsoft released a code snippet for demonstrating how you can unproject a vector. Their Unproject method is meant to be used in a Unity script. You can find their example below.

    I hope that helps!
    Waterine likes this.
  6. Waterine


    Jul 27, 2016
    @BrandonFogerty Thanks! I put that into my code and it works perfectly.
  7. travelhawk


    Jan 13, 2017
    @BrandonFogerty I tried the code from the "Locatable Camera" article to do the same. For testing purposes I get the projection and cameraToWorldMatrix from a virtual camera in my scene and try to compute a world position after clicking on a quad with a render texture with this camera applied. However the world position is super far away and not positioned correctly. Does that mean there is something wrong or can it still work on the HoloLens though to the fact that the projection and camera to world matrix is different?