Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct here to familiarize yourself with the rules and how to post constructively.

  2. Unity 2022.1 is now available as the latest Tech release.
    Dismiss Notice
  3. Improve your project's performance with our new guide on profiling in Unity.
    Dismiss Notice
  4. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Resolved Projection Matrix with PhotoCapture on Hololens 1 not available

Discussion in 'AR' started by mcauz, Oct 27, 2021.

  1. mcauz

    mcauz

    Joined:
    May 12, 2021
    Posts:
    2
    Hello everyone,

    I have developed a small application to test the PhotoCapture API of Hololens to capture a screenshot with location matrix. The screenshot is correctly taken and adding as texture on a gameobject, but the methods TryGetProjectionMatrix and TryGetCameraToWorldMatrix return always false. Despite hours of research on the Web, I do not find why the location data are not accessible. I have tried on the play mode of unity (computer) and on the Hololens directly, but the result is the same.

    I need them because I must extracted the pixels of the real environment behind the virtual elements.

    Unity version: 2019.4.27f1
    Hololens version: 1
    Hololens SDK: 10.0.17763.0
    Capabilities activated: InternetClient, WebCam, Microphone, SpatialPerception, GazeInput

    You can find below the script associated to the main camera under MixedRealityPlayscape.

    I hope that someone can help me. Best regards, Maxime

    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using System.Linq;
    4. using UnityEngine.Windows.WebCam;
    5.  
    6. public class AutomaticColorAdaptation : MonoBehaviour
    7. {
    8.     PhotoCapture photoCaptureObject = null;
    9.     Texture2D targetTexture = null;
    10.  
    11.     // Start is called before the first frame update
    12.     void Start()
    13.     {
    14.         Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
    15.         targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);
    16.  
    17.         // Create a PhotoCapture object
    18.         PhotoCapture.CreateAsync(false, delegate(PhotoCapture captureObject) {
    19.             photoCaptureObject = captureObject;
    20.             CameraParameters cameraParameters = new CameraParameters(WebCamMode.PhotoMode);
    21.             cameraParameters.hologramOpacity = 0.0f;
    22.             cameraParameters.cameraResolutionWidth = cameraResolution.width;
    23.             cameraParameters.cameraResolutionHeight = cameraResolution.height;
    24.             cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
    25.        
    26.             // Activate the camera
    27.             photoCaptureObject.StartPhotoModeAsync(cameraParameters, delegate(PhotoCapture.PhotoCaptureResult result) {
    28.                 // Take a picture
    29.                 photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
    30.             });
    31.         });
    32.     }
    33.  
    34.     // Update is called once per frame
    35.     void Update()
    36.     {
    37.        
    38.     }
    39.  
    40.     void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
    41.     {
    42.         Debug.Log(result.success); // Display True
    43.         Debug.Log(photoCaptureFrame.hasLocationData); // Display False
    44.  
    45.         Matrix4x4 m1 = new Matrix4x4();
    46.         Matrix4x4 m2 = new Matrix4x4();
    47.  
    48.         Debug.Log(photoCaptureFrame.TryGetCameraToWorldMatrix(out m1)); // Display False
    49.         Debug.Log(photoCaptureFrame.TryGetProjectionMatrix(out m2)); // Display False
    50.  
    51.         Debug.Log(m1); // Display Identity matrix
    52.         Debug.Log(m2); // Display Identity matrix
    53.  
    54.         // Copy the raw image data into our target texture
    55.         photoCaptureFrame.UploadImageDataToTexture(targetTexture);
    56.  
    57.         // Create a gameobject that we can apply our texture to
    58.         GameObject quad = GameObject.CreatePrimitive(PrimitiveType.Quad);
    59.         Renderer quadRenderer = quad.GetComponent<Renderer>() as Renderer;
    60.         quadRenderer.material = new Material(Shader.Find("Unlit/Texture"));
    61.  
    62.         quad.transform.parent = this.transform;
    63.         quad.transform.localPosition = new Vector3(0.0f, 0.0f, 3.0f);
    64.  
    65.         quadRenderer.material.SetTexture("_MainTex", targetTexture);
    66.  
    67.         // Deactivate our camera
    68.         photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
    69.     }
    70.  
    71.     void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
    72.     {
    73.         // Shutdown our photo capture resource
    74.         photoCaptureObject.Dispose();
    75.         photoCaptureObject = null;
    76.     }
    77. }
     
  2. mcauz

    mcauz

    Joined:
    May 12, 2021
    Posts:
    2
    I have created a new project with Unity 2019.4.27f1 and MRTK 2.7.2. This time, I have not cloned the MRTK profile to disable the CPU performer bar. It's working on my Hololens 1. My problem is solved.
     
unityunity