Implementing locatable camera shader code

What I'm trying to do is take a photo, pick a pixel location in the photo to place a hologram, then actually place the hologram in the corresponding 3D location. I found a tutorial for this here under "Pixel to Application-specified Coordinate System": https://developer.microsoft.com/en-us/windows/holographic/locatable_camera. However, the code for the translation between the pixel coordinate to real world coordinate is in shader code. I created a custom shader and a material for that shader, but my question is how does the shader code snippet provided in that link actually fit into the shader file? Also, to what gameobject should the material be attached?

As a bonus, it'd be great if anyone could explain why this is done in shader code. This thread http://forum.unity3d.com/threads/locatable-camera-in-unity.398803/ has clues on what I want to accomplish, but the shader code is not included.

 

Hi @just_another_username

I recently posted a thread on how to do the reverse of what you are trying to do. The following link will provide you source code that demonstrates how to blend a captured photo with your physical environment using the PhotoCapture API. Please read through the code and let me know if it makes sense to you.

http://forum.unity3d.com/threads/holographic-photo-blending-with-photocapture.416023/

You are essentially trying to go in the opposite direction. I don't believe you need a shader at all for what you are trying to do unless you want to highlight the selected pixel on the captured image. The code example listed on Microsoft's locatable camera documentation is not necessarily intended to be shader code.

I put together the following code snippet to serve as an overview of what you would need to do assuming you are placing your captured image on one of Unity's quad game objects.

Code (CSharp):

1. public class PhotoPixelToWorldExample:MonoBehaviour
2. {
3.  
4.     public GameObject Canvas;
5.     PhotoCaptureFrame photoCaptureFrame;
6.  
7.     void Update()
8.     {
9.         // In Editor we can test more easily with the mouse.
10.         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
11.  
12.         // On the HoloLens you can replace the above line with
13.         //Ray ray = new Ray(Camera.main.transform.position, Camera.main.transform.forward);
14.  
15.         RaycastHit[] hits = Physics.RaycastAll(ray,1000.0f);
16.         if(hits == null || hits.Length <= 0)
17.         {
18.             return;
19.         }
20.  
21.         RaycastHit hit = hits[0];
22.         if(hit.collider.gameObject != Canvas)
23.         {
24.             return;
25.         }
26.  
27.         // First we need to figure our where our intersected pixel is
28.         // in our virtual camera's clip space.
29.  
30.         // Transform our intersection point into the object space of
31.         // the quad/canvas game object.  This means that the quad
32.         // will now be located at the origin (0,0,0) and our intersection point
33.         // will be offset from that location by the same amount that it currently is.
34.         // The intention behind this is to simplify the problem of transforming
35.         // the intersected pixel coordinate into the -1 to 1 range.
36.         // After this transformation, we can easily do that by just
37.         // sampling the x and y coordinates of the transformed intersection point.
38.         // In Unity, the quad primitive will range from -0.5 to 0.5 for both its width
39.         // and height in object space.
40.         Vector3 ImagePosProjected = Canvas.transform.worldToLocalMatrix * hit.point;
41.         // Map the -0.5 to 0.5 domain to a -1.0 to 1.0 range.
42.         ImagePosProjected.x *= 2.0f;
43.  
44.         // From here you can follow the Microsoft documentation
45.  
46.         Matrix4x4 cameraToWorldMatrix;
47.         photoCaptureFrame.TryGetCameraToWorldMatrix(out cameraToWorldMatrix);
48.         Matrix4x4 worldToCameraMatrix = cameraToWorldMatrix.inverse;
49.  
50.         Matrix4x4 projectionMatrix;
51.         photoCaptureFrame.TryGetProjectionMatrix(out projectionMatrix);
52.  
53.         // Transform the projected clip space position back into real world
54.         // web camera space
55.         // Vector3 CameraSpacePos = UnProjectVector(projectionMatrix,
56.         // new Vector3(ImagePosProjected.x, ImagePosProjected.y,1.0f));
57.  
58.         // Transform the captured image's pixel location that is in the
59.         // real world space of the physical web camera back into our
60.         // virtual world space.
61.         // Vector3 WorldSpaceRayPoint = cameraToWorldMatrix * CameraSpacePos;
62.     }
63. }

Now for bonus points!

The Holographic Blend shader, that I posted in the above link, essentially transforms the uvs of the quad mesh so that we sample pixels from the capture imaged texture in a way that makes it blend in with our real world environment. Since the captured image was captured by the physical web camera, and not our virtual camera, we need transform the uvs by the physical web camera's world to camera and projection matrices. This is a per pixel operation. So while we *could* do this on the CPU in a script, it would be very slow. The GPU however is much more optimized for handling per pixel operations.

I hope that helps!

 

Thanks for the help! I've learned a lot through your examples and I'm sure this will help someone else down the road!

 

Hi Brandon,

Thank you for your examples and explanations, they are very helpful!

Now if I want to do this outside a shader, what would be the best way to implement UnProjectVector()? The projection matrix from locatable camera is not invertible. What I am trying to do is to map image analysis results (OCR, face recognition, etc.) back to 3D world by raycasting onto spatial mapping layer.

I started with using main camera and ScreenPointToRay, which gave me pretty good approximation when the capture resolution is 1280x720, but for best accuracy I want to use the locatable camera information. Then I tried to create a secondary camera just for raycasting, set its transform to the locatable camera's transform. As soon as I set the raycasting camera's projection matrix to be the same as the locatable camera's, ScreenPointToRay stopped working.

Could you advise what's the best way to make use the projection matrix in my scenario?

 

Hi @Waterine

The projection matrices for the physical web camera on the HoloLens and your virtual main camera will be quite different. Therefore they are not interchangeable. Microsoft released a code snippet for demonstrating how you can unproject a vector. Their Unproject method is meant to be used in a Unity script. You can find their example below.

https://developer.microsoft.com/en-us/windows/holographic/locatable_camera

I hope that helps!

 

@BrandonFogerty Thanks! I put that into my code and it works perfectly.

 

@BrandonFogerty I tried the code from the "Locatable Camera" article to do the same. For testing purposes I get the projection and cameraToWorldMatrix from a virtual camera in my scene and try to compute a world position after clicking on a quad with a render texture with this camera applied. However the world position is super far away and not positioned correctly. Does that mean there is something wrong or can it still work on the HoloLens though to the fact that the projection and camera to world matrix is different?