ARKit Remote Touch Inpu

Hi there folks,

I've been doing a lot of searching on how to get the ARKit remote connection (which is by and large great) to register touch inputs in the editor. The following code works perfectly when running on the actual ipad I'm using, using easytouch as my touch/gesture library:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.XR.iOS;
5. using HedgehogTeam.EasyTouch;
6.  
7. namespace Nesta.DigitalBlack
8. {
9.     public class InputHelper : MonoBehaviour
10.     {
11.  
12.         public static ARHitTestResult _arTouchHitTestResult;
13.         public static ARHitTestResult _arCenterScreenHitTestResult;
14.  
15.         public static Vector3 arHitTestPosition;
16.  
17.         public static string AR_PLANE_LAYER = "ARKitPlane";
18.  
19.         public static bool TryARTouchHit() {
20.        
21.             if (EasyTouch.current != null) {
22.                 if(EasyTouch.current.type == EasyTouch.EvtType.On_TouchDown) {
23.                     var viewPosition = Camera.main.ScreenToViewportPoint (EasyTouch.current.position);
24.  
25.                     ARPoint point = new ARPoint {
26.                         x = viewPosition.x,
27.                         y = viewPosition.y
28.                     };
29.  
30.                     // prioritize reults types
31.                     ARHitTestResultType[] resultTypes = {
32.                         ARHitTestResultType.ARHitTestResultTypeExistingPlaneUsingExtent,
33.                         // if you want to use infinite planes use this:
34.                         ARHitTestResultType.ARHitTestResultTypeExistingPlane,
35.                         ARHitTestResultType.ARHitTestResultTypeHorizontalPlane,
36.                         ARHitTestResultType.ARHitTestResultTypeFeaturePoint
37.                     };
38.  
39.                     foreach (ARHitTestResultType resultType in resultTypes) {
40.                         List<ARHitTestResult> hitResults = UnityARSessionNativeInterface.GetARSessionNativeInterface ().HitTest (point, resultType);
41.                         if (hitResults.Count > 0) {
42.                             foreach (ARHitTestResult hitResult in hitResults) {
43.                                 _arTouchHitTestResult = hitResult;
44.                                 arHitTestPosition = UnityARMatrixOps.GetPosition (InputHelper._arTouchHitTestResult.worldTransform);
45.                                 return true;
46.                             }
47.                         }
48.                     }
49.  
50.                 }
51.             }
52.  
53.             return false;
54.         }
55.     }
56. }

However, when I use this in the editor with ARKit remote, it won't register any hits. The bizarre thing is that the output from easy touch is registering the same values in the editor as on the device, so I know the touch input data itself is good. Is there some kind of issue that is preventing UnityARSessionNativeInterface.GetARSessionNativeInterface ().HitTest(point, resultType) from working in the editor?

 

It's worth noting that I tried the techniques in "EditorHitTest.cs" to no avail.

 

The EditorHitTest example does work in its own scene but the same technique applied to my script is no bueno... very frustrating.

 

HitTest only works on device - you should simulate hit test with what EditorHitTest does - it casts a ray and collides against the generated planes. Are you using the GeneratePlanes GO in the scene?

 

That's just the thing! I tried adapting the EditorHitTest code for my game and no dice... maybe I need to actually draw the debug rays to see what's going on.

 

Check that the collider on your debugPlane prefab has a Layer set that is the same as the collisionLayerMask on EditorHitTest or change one or both to make them the same

 

This thread helped me to get it to work.. it isn't immediately clear that the collisionLayerMask is required, but this did solve it for me!

One additional question I had that remains unsolved... is it correct that the HitTest can't come from my mobile device's screen, and only by interacting with the editor? I can't seem to get taps on the screen to register, which makes sense since the camera feed doesn't display there either. Maybe there is a way around this?

 

Ok, so I've come back with a screen shot and a snippet to hopefully figure out what's going on here. This is the code I'm running in my update loop:

Code (CSharp):

1. //in-editor test
2.             if (Input.GetMouseButtonDown (0)) {
3.                 Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
4.                 RaycastHit hit;
5.  
6.                 if (Physics.Raycast (ray, out hit, 200f,  LayerMask.NameToLayer("ARKitPlane"))) {
7.                     Debug.Log("Hit ARPlane");
8.                     Debug.DrawRay(ray.origin, hit.point, Color.green, 2f);
9.                     return true;
10.                 } else {
11.                     Debug.Log ("Hit nothing");
12.                     Debug.DrawRay(ray.origin, ray.GetPoint(100), Color.red, 2f);
13.                 }
14.             }

And yet when I try and hit my planes, I get a red ray even though it's passing right through it, per the attached image. The layer masks match as well so I know that's not it. Any thoughts?

 

Hey nesta, have you tried adding a rigidbody and collider to the object you're testing ray hits against? I can't tell if the object in your screenshot is the one you're trying to test interaction with but I think those components are needed for a raycast to return a hit.

 

There is a collider on the child object but not a rigid body. I didn't thinking a rigid body was necessary for Ray casting?

 

I just added a rigidbody and still no dice. I uploaded a pic of the inspector view of the child object with collider, mesh, and rigidbodies on it.

This is the entire C# file... I am calling my method statically instead of having it be an instance of a MonoBehaviour... think that could be it?

 

Finally I have found some success! So ultimately, I had to create a public LayerMask variable and set its value in the editor... for some reason, using LayerMask.NameToLayer("ARKitPlane") didn't work for the layer parameter in the raycast call. Any ideas what's up with that?

 

I'm new to Unity and I can't get this to work. Taking UnityARKitScene as an example, what I did was:

1. Opened up the debugPlanePrefab
2. Made a new layer 'foo' and set the prefab's layer to it
3. Click Apply
4. Applied EditorHitTest to HitCube
5. Set CollisionLayerMask on that script on HitCube to 'foo' in the inspector

Am I missing anything?

This is super key for AR workflow. Would be amazing to get some more documentation here, a youtube video just setting this up would be awesome. Seems like there are two cases maybe - when you want to make hits on arkit planes, and when you want to make hits on game objects. Do each of those need to be handled differently?

 

Okay I guess in hindsight this is obvious, but I didn't really understand what the UnityARKitScene did, so: you have to wait until GeneratePlanes draws a plane onscreen before you can click on it to simulate a hit.

Then I started getting hits, but I would get this error on hit:

UnassignedReferenceException: The variable m_HitTransform of EditorHitTest has not been assigned.
You probably need to assign the m_HitTransform variable of the EditorHitTest script in the inspector.

I set Hit Transform in the inspector to ARCameraManager and now I'm getting values logged in the console.