Resolved Projection Matrix with PhotoCapture on Hololens 1 not available

Hello everyone,

I have developed a small application to test the PhotoCapture API of Hololens to capture a screenshot with location matrix. The screenshot is correctly taken and adding as texture on a gameobject, but the methods TryGetProjectionMatrix and TryGetCameraToWorldMatrix return always false. Despite hours of research on the Web, I do not find why the location data are not accessible. I have tried on the play mode of unity (computer) and on the Hololens directly, but the result is the same.

I need them because I must extracted the pixels of the real environment behind the virtual elements.

Unity version: 2019.4.27f1
Hololens version: 1
Hololens SDK: 10.0.17763.0
Capabilities activated: InternetClient, WebCam, Microphone, SpatialPerception, GazeInput

You can find below the script associated to the main camera under MixedRealityPlayscape.

I hope that someone can help me. Best regards, Maxime

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using System.Linq;
4. using UnityEngine.Windows.WebCam;
5.  
6. public class AutomaticColorAdaptation : MonoBehaviour
7. {
8.     PhotoCapture photoCaptureObject = null;
9.     Texture2D targetTexture = null;
10.  
11.     // Start is called before the first frame update
12.     void Start()
13.     {
14.         Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
15.         targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);
16.  
17.         // Create a PhotoCapture object
18.         PhotoCapture.CreateAsync(false, delegate(PhotoCapture captureObject) {
19.             photoCaptureObject = captureObject;
20.             CameraParameters cameraParameters = new CameraParameters(WebCamMode.PhotoMode);
21.             cameraParameters.hologramOpacity = 0.0f;
22.             cameraParameters.cameraResolutionWidth = cameraResolution.width;
23.             cameraParameters.cameraResolutionHeight = cameraResolution.height;
24.             cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
25.        
26.             // Activate the camera
27.             photoCaptureObject.StartPhotoModeAsync(cameraParameters, delegate(PhotoCapture.PhotoCaptureResult result) {
28.                 // Take a picture
29.                 photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
30.             });
31.         });
32.     }
33.  
34.     // Update is called once per frame
35.     void Update()
36.     {
37.        
38.     }
39.  
40.     void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
41.     {
42.         Debug.Log(result.success); // Display True
43.         Debug.Log(photoCaptureFrame.hasLocationData); // Display False
44.  
45.         Matrix4x4 m1 = new Matrix4x4();
46.         Matrix4x4 m2 = new Matrix4x4();
47.  
48.         Debug.Log(photoCaptureFrame.TryGetCameraToWorldMatrix(out m1)); // Display False
49.         Debug.Log(photoCaptureFrame.TryGetProjectionMatrix(out m2)); // Display False
50.  
51.         Debug.Log(m1); // Display Identity matrix
52.         Debug.Log(m2); // Display Identity matrix
53.  
54.         // Copy the raw image data into our target texture
55.         photoCaptureFrame.UploadImageDataToTexture(targetTexture);
56.  
57.         // Create a gameobject that we can apply our texture to
58.         GameObject quad = GameObject.CreatePrimitive(PrimitiveType.Quad);
59.         Renderer quadRenderer = quad.GetComponent<Renderer>() as Renderer;
60.         quadRenderer.material = new Material(Shader.Find("Unlit/Texture"));
61.  
62.         quad.transform.parent = this.transform;
63.         quad.transform.localPosition = new Vector3(0.0f, 0.0f, 3.0f);
64.  
65.         quadRenderer.material.SetTexture("_MainTex", targetTexture);
66.  
67.         // Deactivate our camera
68.         photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
69.     }
70.  
71.     void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
72.     {
73.         // Shutdown our photo capture resource
74.         photoCaptureObject.Dispose();
75.         photoCaptureObject = null;
76.     }
77. }

 

I have created a new project with Unity 2019.4.27f1 and MRTK 2.7.2. This time, I have not cloned the MRTK profile to disable the CPU performer bar. It's working on my Hololens 1. My problem is solved.