Holographic Photo Blending with PhotoCapture

Some have asked for an example of capturing a holographic photo that blends in with the physical environment. You can do this by using the PhotoCapture API along with the projection and world to camera matrices that are included with the captured image data.

Attached is a C# script and a shader. The C# script captures an image using the web camera on the HoloLens whenever you do the airtap gesture. The C# script will upload the captured image to the GPU so that the shader can access the image data. The shader will calculate what part of the image should be shown based on where the photo was taken.

HoloLensSnapshotTest.cs

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using System.Collections.Generic;
4. using UnityEngine.VR.WSA.WebCam;
5. using UnityEngine.VR.WSA.Input;
6.  
7. public class HoloLensSnapshotTest:MonoBehaviour
8. {
9.     GestureRecognizer m_GestureRecognizer;
10.     GameObject m_Canvas = null;
11.     Renderer m_CanvasRenderer = null;
12.     PhotoCapture m_PhotoCaptureObj;
13.     CameraParameters m_CameraParameters;
14.     bool m_CapturingPhoto = false;
15.     Texture2D m_Texture = null;
16.  
17.     void Start()
18.     {
19.         Initialize();
20.     }
21.  
22.     void SetupGestureRecognizer()
23.     {
24.         m_GestureRecognizer = new GestureRecognizer();
25.         m_GestureRecognizer.SetRecognizableGestures(GestureSettings.Tap);
26.         m_GestureRecognizer.TappedEvent += OnTappedEvent;
27.         m_GestureRecognizer.StartCapturingGestures();
28.  
29.         m_CapturingPhoto = false;
30.     }
31.  
32.     void Initialize()
33.     {
34.         Debug.Log("Initializing...");
35.         List<Resolution> resolutions = new List<Resolution>(PhotoCapture.SupportedResolutions);
36.         Resolution selectedResolution = resolutions[0];
37.  
38.         m_CameraParameters = new CameraParameters(WebCamMode.PhotoMode);
39.         m_CameraParameters.cameraResolutionWidth = selectedResolution.width;
40.         m_CameraParameters.cameraResolutionHeight = selectedResolution.height;
41.         m_CameraParameters.hologramOpacity = 0.0f;
42.         m_CameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
43.  
44.         m_Texture = new Texture2D(selectedResolution.width,selectedResolution.height,TextureFormat.BGRA32,false);
45.  
46.         PhotoCapture.CreateAsync(false,OnCreatedPhotoCaptureObject);
47.     }
48.  
49.     void OnCreatedPhotoCaptureObject(PhotoCapture captureObject)
50.     {
51.         m_PhotoCaptureObj = captureObject;
52.         m_PhotoCaptureObj.StartPhotoModeAsync(m_CameraParameters,true,OnStartPhotoMode);
53.     }
54.  
55.     void OnStartPhotoMode(PhotoCapture.PhotoCaptureResult result)
56.     {
57.         SetupGestureRecognizer();
58.  
59.         Debug.Log("Ready!");
60.         Debug.Log("Air Tap to take a picture.");
61.     }
62.  
63.     void OnTappedEvent(InteractionSourceKind source,int tapCount,Ray headRay)
64.     {
65.         if(m_CapturingPhoto)
66.         {
67.             return;
68.         }
69.  
70.         m_CapturingPhoto = true;
71.         Debug.Log("Taking picture...");
72.         m_PhotoCaptureObj.TakePhotoAsync(OnPhotoCaptured);
73.     }
74.  
75.     void OnPhotoCaptured(PhotoCapture.PhotoCaptureResult result,PhotoCaptureFrame photoCaptureFrame)
76.     {
77.         if(m_Canvas == null)
78.         {
79.             m_Canvas = GameObject.CreatePrimitive(PrimitiveType.Quad);
80.             m_Canvas.name = "PhotoCaptureCanvas";
81.             m_CanvasRenderer = m_Canvas.GetComponent<Renderer>() as Renderer;
82.             m_CanvasRenderer.material = new Material(Shader.Find("AR/HolographicImageBlend"));
83.         }
84.  
85.         Matrix4x4 cameraToWorldMatrix;
86.         photoCaptureFrame.TryGetCameraToWorldMatrix(out cameraToWorldMatrix);
87.         Matrix4x4 worldToCameraMatrix = cameraToWorldMatrix.inverse;
88.  
89.         Matrix4x4 projectionMatrix;
90.         photoCaptureFrame.TryGetProjectionMatrix(out projectionMatrix);
91.  
92.         photoCaptureFrame.UploadImageDataToTexture(m_Texture);
93.         m_Texture.wrapMode = TextureWrapMode.Clamp;
94.  
95.         m_CanvasRenderer.sharedMaterial.SetTexture("_MainTex",m_Texture);
96.         m_CanvasRenderer.sharedMaterial.SetMatrix("_WorldToCameraMatrix",worldToCameraMatrix);
97.         m_CanvasRenderer.sharedMaterial.SetMatrix("_CameraProjectionMatrix",projectionMatrix);
98.         m_CanvasRenderer.sharedMaterial.SetFloat("_VignetteScale", 1.0f);
99.  
100.         // Position the canvas object slightly in front
101.         // of the real world web camera.
102.         Vector3 position = cameraToWorldMatrix.GetColumn(3) - cameraToWorldMatrix.GetColumn(2);
103.  
104.         // Rotate the canvas object so that it faces the user.
105.         Quaternion rotation = Quaternion.LookRotation(-cameraToWorldMatrix.GetColumn(2),cameraToWorldMatrix.GetColumn(1));
106.  
107.         m_Canvas.transform.position = position;
108.         m_Canvas.transform.rotation = rotation;
109.  
110.         Debug.Log("Took picture!");
111.         m_CapturingPhoto = false;
112.     }
113. }
114.  

HolographicImageBlendShader.shader

Code (CSharp):

1. Shader "AR/HolographicImageBlend"
2. {
3.     Properties
4.     {
5.         _MainTex ("Texture", 2D) = "white" {}
6.         _VignetteScale ("Vignette Scale", RANGE(0,2)) = 0
7.     }
8.     SubShader
9.     {
10.         Tags { "RenderType"="Opaque" }
11.         LOD 100
12.  
13.         Pass
14.         {
15.             CGPROGRAM
16.             #pragma vertex vert
17.             #pragma fragment frag
18.  
19.             #include "UnityCG.cginc"
20.  
21.             struct appdata
22.             {
23.                 float4 vertex : POSITION;
24.             };
25.  
26.             struct v2f
27.             {
28.                 float4 vertexPositionInProjectionSpace : SV_POSITION;
29.                 float2 uv : TEXCOORD0;
30.                 float4 vertexInProjectionSpace : TEXCOORD1;
31.             };
32.  
33.             sampler2D _MainTex;
34.             float4x4 _WorldToCameraMatrix;
35.             float4x4 _CameraProjectionMatrix;
36.             float _VignetteScale;
37.  
38.             v2f vert (appdata v)
39.             {
40.                 v2f o;
41.                 o.vertexPositionInProjectionSpace = mul(UNITY_MATRIX_MVP, v.vertex);
42.  
43.                 // Calculate the vertex position in world space.
44.                 float4 vertexPositionInWorldSpace = mul(unity_ObjectToWorld, float4(v.vertex.xyz,1));
45.                 // Now take the world space vertex position and transform it so that
46.                 // it is relative to the physical web camera on the HoloLens.
47.                 float4 vertexPositionInCameraSpace = mul(_WorldToCameraMatrix, float4(vertexPositionInWorldSpace.xyz,1));
48.  
49.                 // Convert our camera relative vertex into clip space.
50.                 o.vertexInProjectionSpace = mul(_CameraProjectionMatrix, float4(vertexPositionInCameraSpace.xyz, 1.0));
51.  
52.                 return o;
53.             }
54.  
55.             fixed4 frag (v2f i) : SV_Target
56.             {
57.                 // Transform the vertex into normalized coordinate space.  Basically
58.                 // we want to map where our vertex should be on the screen into the -1 to 1 range
59.                 // for both the x and y axes.
60.                 float2 signedUV = i.vertexInProjectionSpace.xy / i.vertexInProjectionSpace.w;
61.  
62.                 // The HoloLens uses an additive display so the color black will
63.                 // be transparent.  If the texture is smaller than the canvas, color the extra
64.                 // area on the canvas black so it will be transparent on the HoloLens.
65.                 if(abs(signedUV.x) > 1.0 || abs(signedUV.y) > 1.0)
66.                 {
67.                     return fixed4( 0.0, 0.0, 0.0, 0.0);
68.                 }
69.  
70.                 // Currently our signedUV's x and y coordinates will fall between -1 and 1.
71.                 // We need to map this range from 0 to 1 so that we can sample our texture.
72.                 float2 uv = signedUV * 0.5 + float2(0.5, 0.5);
73.                 fixed4 finalColor = tex2D(_MainTex, uv);
74.  
75.                 // Finally add a circular vignette effect starting from the center
76.                 // of the image.
77.                 finalColor *= 1.0-(length(signedUV) * _VignetteScale);
78.  
79.                 return finalColor;
80.             }
81.             ENDCG
82.         }
83.     }
84. }
85.  

 

Have you enabled both WebCam and Mic in the capabilities settings?

 

Hi cyberstorm5076!

You can create a game object in the editor like so,



You can also create the sphere game object dynamically in code like so,

Code (CSharp):

1. GameObject sphereGameObject = GameObject.CreatePrimitive(PrimitiveType.Sphere);
2. sphereGameObject.name = "Sphere";

The following is a good tutorial to get you up and running with writing shaders.

Shaders are just GPU programs that do certain things like modify your vertices, generate vertices, or calculate what the final pixel color should be. A Material is an asset that your artist typically creates. Think of it this way. If shaders are GPU programs, then materials are the command line arguments that you pass into the shader. The material also stores a reference to the shader that it uses.

The following code sample is a Color Fader shader and script I wrote. The Color Fader shader will morph the color of a game object between 2 different colors based on whatever the current color morph value is. The color morph value is a variable that must between the numbers 0.0 and 1.0. When the morph value is 0.0, the color of the game object will be whatever Color0 is. When the morph value is 1.0, the color of the game object will be whatever Color1 is.

The ColorController script dynamically creates the material and shader and then feeds in your values to the shader. In this script, I dynamically create the material and shader however you could also assign the material as a public property as well.

ColorController.cs

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. [RequireComponent(typeof(MeshRenderer))]
5.  
6. [ExecuteInEditMode]
7. public class ColorController : MonoBehaviour
8. {
9.     public Color Color0 = Color.red;
10.     public Color Color1 = Color.blue;
11.     [Range(0.0f,1.0f)]
12.     public float ColorMorphValue = 0.0f;
13.  
14.     private Material myMaterial = null;
15.  
16.     void Start()
17.     {
18.         Renderer gameObjectRenderer = this.transform.GetComponent<Renderer>();
19.         myMaterial = new Material(Shader.Find("MyCustomShader/ColorFader"));
20.         gameObjectRenderer.sharedMaterial = myMaterial;
21.     }
22.  
23.     void Update ()
24.     {
25.         if(myMaterial == null)
26.         {
27.             return;
28.         }
29.  
30.         // Send custom parameters from the CPU to the GPU shader.
31.         myMaterial.SetColor("_Color0", Color0);
32.         myMaterial.SetColor("_Color1", Color1);
33.         myMaterial.SetFloat("_ColorMorphValue", ColorMorphValue);
34.     }
35. }
36.  

ColorFader.shader

Code (CSharp):

1. Shader "MyCustomShader/ColorFader"
2. {
3.     Properties
4.     {
5.         _Color0 ("Color 0", COLOR) = (1.0, 1.0, 1.0, 1.0)
6.         _Color1 ("Color 1", COLOR) = (0.0, 0.0, 0.0, 1.0)
7.         _ColorMorphValue ( "Color Morph Value", FLOAT) = 0.0
8.     }
9.     SubShader
10.     {
11.         Tags { "RenderType"="Opaque" }
12.  
13.         Pass
14.         {
15.             CGPROGRAM
16.             #pragma vertex vert
17.             #pragma fragment frag
18.      
19.             #include "UnityCG.cginc"
20.  
21.             struct appdata
22.             {
23.                 float4 vertex : POSITION;
24.                 float2 uv : TEXCOORD0;
25.             };
26.  
27.             struct v2f
28.             {
29.                 float4 vertex : SV_POSITION;
30.                 float3 color : COLOR;
31.             };
32.      
33.             float3 _Color0;
34.             float3 _Color1;
35.             float _ColorMorphValue;
36.  
37.             v2f vert (appdata v)
38.             {
39.                 v2f o;
40.                 o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
41.                 o.color = (1.0-_ColorMorphValue) * _Color0 + (_ColorMorphValue * _Color1);
42.                 return o;
43.             }
44.      
45.             float4 frag (v2f i) : SV_Target
46.             {
47.                 return float4(i.color, 1.0);
48.             }
49.             ENDCG
50.         }
51.     }
52. }
53.  

If you would rather your ColorController script reference a premade material asset, then you will need to create the material asset like so,


Then you will need to assign the correct shader to the material like so,


Finally you will need to drag and drop the material onto your game object.
Then you will need to drag and drop the modified ColorController script onto your game object.

ColorController.cs - References an Assigned Material Asset

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. [RequireComponent(typeof(MeshRenderer))]
5.  
6. [ExecuteInEditMode]
7. public class ColorController : MonoBehaviour
8. {
9.     public Color Color0 = Color.red;
10.     public Color Color1 = Color.blue;
11.     [Range(0.0f,1.0f)]
12.     public float ColorMorphValue = 0.0f;
13.  
14.     void Update ()
15.     {
16.         Renderer myRenderer = this.GetComponent<Renderer>();
17.  
18.         // Send custom parameters from the CPU to the GPU shader.
19.         myRenderer.sharedMaterial.SetColor("_Color0", Color0);
20.         myRenderer.sharedMaterial.SetColor("_Color1", Color1);
21.         myRenderer.sharedMaterial.SetFloat("_ColorMorphValue", ColorMorphValue);
22.     }
23. }
24.  

In order to use the HoloLensSnapshotTest script, just drag and drop it onto your camera game object.

I hope that helps!

 

Hi @yjlin5210

I am glad I could help! Anytime Mixed Reality Capture is used on the HoloLens, the fps will automatically drop to 30 fps as the price of doing business. However I was unaware that it is dropping lower that 30 fps. Do you have a minimum repro project that I could take a look at? Either way, I will look into this further. Thanks for bringing this to my attention!

 

Hi @yjlin5210,

Have you tried using the PhotoCapture api to take photos?

 

Hi @acylum,

The Holographic Image Blend shader requires that the mesh it is applied to contain texture coordinates. However the spatial mapping mesh only contains vertices and indices. That is why your spatial mapping mesh appears white. You can add your own uvs to the spatial mapped mesh but that may be a bit complicated depending on what you are trying to do.

 

Shaders won't be included in a build if nothing directly references them in the scenes used for the build. To get around this, use the "Always-included Shaders" section of the Graphics Settings (https://docs.unity3d.com/Manual/class-GraphicsSettings.html).

 

Hi Mlotek -

I'm afraid I'm unable to puzzle out what exactly you're trying to do, and don't understand the problem you're running into either. Would you mind going into more detail, possibly pasting in some code and screen captures?

 

I don't think that's an assumption you can make in the context of photo capture. The PhotoCapture API exposes a set of resolutions supported by the photo camera - you basically just pick one of them and fill out the cameraResolutionWidth and cameraResolutionHeight on the CameraParameters object you pass to StartPhotoModeAsync (see https://docs.unity3d.com/ScriptReference/VR.WSA.WebCam.PhotoCapture.html). Hope that helps.

 

Are you running on the HoloLens or in the editor?