Check if user tap when gaze point to spatial mesh

Hi,
I'm learning how to use the HoloToolkit and I'm using the GestureManager to send the OnSelect message at the object pointed by gaze when the user tap. It works with object witch I can attach script where I define the OnSelect function, but I don't understand if this is the right way to do the same if the user tap on the spatial mesh, because I don't know how to refer to it.
I don't know how the spatial mesh is added to the scene because it is added by the engine automatically, because I don't know if an Object of some type is created or if the mesh data is saved in some data structure.
So, what is the best way to check if the user is tapping Spatial Mesh? I saw that the object collided by the raycast is called like "Surface-\d\d\d\d", where \d is a number.
Thank you!

 

Hi dshinigami,

As you said, the engine creates and updates spatial surfaces when HoloLens reports updates. The engine will also remove surfaces whenever the device thinks a surface has been removed.

There are a few ways you can detect if you are selecting a surface mesh using the SpatialMappingCollider component.

1. The SpatialMappingCollider component will automatically create child game objects with the name prefix "spatial-mapping" on the gameobject that has the SpatialMappingCollider component. The suffix id in the name represents the internal Unity id of the surface. Every so often you could iterate all the child objects of your SpatialMappingCollider component. Then you could apply your input handler script to those child objects whose name begins with "spatial-mapping". You would want to make sure you add your input handler script only if the child object doesn't already have the input handler script. While this may be the easiest solution to implement, it isn't very efficient.

2. A better way would be to create a new script that derives from SpatialMappingCollider. Then override the "SurfaceObserver_OnDataReady" method and attach your input handler script to the newly created spatial mesh gameobject. The following is an example.

Code (csharp):

1. using UnityEngine;
2. using UnityEngine.VR.WSA;
3.  
4. public class CustomSpatialMappingCollider : SpatialMappingCollider
5. {
6.     protected override void SurfaceObserver_OnDataReady(
7.     SurfaceData bakedData,
8.     bool outputWritten,
9.     float elapsedBakeTimeSeconds)
10.     {
11.         if (bakedData.outputMesh != null)
12.         {
13.             if(bakedData.outputMesh.gameObject.GetComponent<MyInputHandler>() == null)
14.             {
15.                 bakedData.outputMesh.gameObject.AddComponent<MyInputHandler>();
16.             }
17.         }
18.  
19.         base.SurfaceObserver_OnDataReady(bakedData, outputWritten, elapsedBakeTimeSeconds);
20.     }
21. }
22.  

You need to be aware that the way GazeManager detects selections is via a RayCast. The GazeManager has a public property called "RaycastLayerMask" which tells the GazeManager
which kinds of objects it should try to raycast against. In Unity, each GameObject is assigned to a layer. I am assuming your UI gameobjects are assigned to a specific UI layer. You probably want to assign the spatial surfaces to their own layer. You can do this by setting the SpatialMappingCollider or your CustomSpatialMappingCollider's "MeshLayer" property to another layer like "SpatialSurfaces". For more information about creating custom layers, take a look here.

http://docs.unity3d.com/Manual/Layers.html

If you want the GazeManager to detect input selections on both spatial surfaces and UI gameobjects, you will need to assign both your spatial surface and UI layer to the GazeManager's "RaycastLayerMask" property.

3. If you prefer to perform the raycast yourself, without using the GazeManager, you will need to derive your own custom SpatialMappingCollider as I suggested in the previous solution.
You will also want to make sure that you assign the the SpatialMappingCollider's "RaycastLayerMask" property to whatever layer you want the surface meshes to be on. Finally, in your own script you will do the following to determine which spatial surfaces the user is selecting.

Code (csharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. public class CustomLayerCollision : MonoBehaviour
5. {
6.     // Update is called once per frame
7.     void Update ()
8.     {
9.         // When the user presses the left mouse button,
10.         // Do a collision test.  You could fire the
11.         // DetectCollisions based on a gesture event.
12.         if(Input.GetMouseButtonDown(0))
13.         {
14.             DetectCollisions();
15.         }
16.     }
17.  
18.     void DetectCollisions()
19.     {
20.         // Raycast against all game objects that are on either the
21.         // spatial surface or UI layers.
22.         int layerMask = (1 << LayerMask.NameToLayer("SpatialSurface")) | (1 << LayerMask.NameToLayer("UI"));
23.  
24.         // We use ScreenPointToRay to create a ray whose origin is the
25.         // main camera's position and direction is from the position of the main
26.         // camera to the position of where the mouse position would be in world space.
27.         RaycastHit[] hits = Physics.RaycastAll(Camera.main.ScreenPointToRay(Input.mousePosition), float.MaxValue, layerMask);
28.         if(hits.Length > 0)
29.         {
30.             foreach(RaycastHit hit in hits)
31.             {
32.                 Debug.Log(string.Format("Hit Object \"{0}\" at position \"{1}\"", hit.collider.gameObject, hit.point));
33.             }
34.         }
35.         else
36.         {
37.             Debug.Log("Nothing was hit.");
38.         }
39.     }
40. }
41.  

I hope that helps.