Grab/Rotate/Pull/Push Objects with XR Ray Interactor?

I'm trying to setup a grab interaction like the one in the Oculus Rift Home environment - where you can use the ray interactor to grab an object at a distance, pull it towards you or away from you, rotate around an axis.

In Unity's new VR XR tutorial there is a prefab XR Rig with a Magic Tractor Beam, which is almost what I want, but you can only pull the object toward you, not away from you, and you can't rotate it.

Has anyone tried something like this yet?

 

Just curious if you have found if there is a built in way of doing this or is it something that will have to be written? I've seen drag functionality for AR but nothing for VR. Been through aPI docs and doesn't seem to be anything from what I can find, but thought I'd see if you had any luck yet.

 

No luck yet, probably because I'm very new to C#. I did see that as part of the new VR Beginner Tutorial on learn.unity.com that they also had an AxisDragInteractable script, that I didn't see before. I haven't tested it out yet though as I've been busy failing at other tasks as well

 

Here's my temporary solution (it's not elegant - I'm waiting for them to remove some of the private/internal restrictions on fields that should be public so I can write this more cleanly), but it's working for me:

https://forum.unity.com/threads/xr-interaction-toolkit-preview-release.795684/page-6#post-5577148

 

Great, thanks! I'm going to try and see if I can get this to work

 

So with your snippet of code from this link - is this a new script that you then attach to the game object which you want to grab, in addition to adding the XRGrabInteractable script?

 

Almost. It's attached to the controller, not to the grabbables (I wanted to attach it once only, in one place ). The dependencies in the code snippet are right at the top: it expects that its on a GameObject with an XRController and an XRBaseInteractor (i.e. any of the XRIT interactors - they all extend that class).

But you could easily attach a similar script to the grabbles with only small changes. Instead of:

Code (CSharp):

1. interactor.selectTarget

to get the Grabbable from the Interactor, you would use:

Code (CSharp):

1. (XRGrabInteractable).selectingInteractor

to get the Interactor from the Grabbable.

 

I like your idea better - I'd prefer to keep it one of my XRController GameObjects. I'm getting an error in the script with 'IntelligentXRExtensions - it says the name does not exist in the current context. Do you know what I may have done wrong? Here's what I've got...

Code (CSharp):

1. using UnityEngine;
2. using System;
3. using UnityEngine.XR;
4. using UnityEngine.XR.Interaction.Toolkit;
5. using System.Reflection;
6.  
7. public class AdjustOffset : MonoBehaviour
8. {
9.     public void Update()
10.     {
11.         var interactor = GetComponent<XRBaseInteractor>();
12.         var controller = GetComponent<XRController>();
13.  
14.         XRGrabInteractable grabbable = interactor.selectTarget as XRGrabInteractable;
15.         if (grabbable != null && grabbable.attachTransform != null)
16.         {
17.             InputDevice device = InputDevices.GetDeviceAtXRNode(controller.controllerNode);
18.  
19.             Vector2 thumbDirection = IntelligentXRExtensions.AmountMoved(device, CommonUsages.primary2DAxis);
20.             float moveAwayAmount = thumbDirection.y;
21.  
22.             Vector3 vectorToMoveObjectAlong = grabbable.attachTransform.parent.InverseTransformVector(controller.transform.forward);
23.             grabbable.attachTransform.localPosition += moveAwayAmount * vectorToMoveObjectAlong;
24.  
25.             MethodInfo unity_UpdateInteractorLocalPose = typeof(XRGrabInteractable).GetMethod("UpdateInteractorLocalPose", BindingFlags.NonPublic | BindingFlags.Instance);
26.             unity_UpdateInteractorLocalPose.Invoke(grabbable, new object[] { interactor });
27.         }
28.  
29.     }
30. }
31.  

 

IntelligentXRExtensions is one of my local classes - I'm away from desktop rgight now but will copy paste it here later

 

It was at the end of the original post, easy to miss, but:

 

I'm not sure what bits of code I'm supposed to cut out and how exactly (InputDevice obj).TryGetFeatureValue(moveable, out Vector2 movement) is supposed to be inserted.

Sorry to keep asking, I'm very new to C# and programming languages in general.

 

It's OK. I was using shorthand. So, for the start, I put brackets round the first two words as a semi-conventional thing to say "this is the type that goes here" (because brackets are used for type-casting in C#, Java, etc).

i.e. in longhand:

1. Whatever-your-Input-Device-object is (in the code you pasted, it's named "device"
2. ... invoke the method TryGetFeatureValue( ... )
3. ... and for the parameters/arguments, I assumed you'd guess this: use the CommonUsages value from Unity's API that is the thing you want to react to. In my case, I wanted to track the thumb-joystick, so I used: "CommonUsages.primary2DAxis"

As per Unity's docs (https://docs.unity3d.com/Manual/xr_input.html#AccessingInputFeatures), TryGetFeatureValue has multiple versions, which automatically send out the right data based on context.

If you pass in CommonUsages.primary2DAxis, you will get a Vector2 back from Unity.

So, overall, something like this:

Code (CSharp):

1. device.TryGetFeatureValue( CommonUsages.primary2DAxis, out Vector2 movement );
2.  
3. // this will read the value of the thumbstick, and output it as a new Vector2 variable called 'movement'

 

Although ... in the code snippet above, the line of code that comes after is expecting a variable called "thumbDirection", so you'd want to have the replacement line end in "out Vector2 thumbDireciton );" instead

 

I think my problem is trying to learn C# for VR Apps before starting in traditional keyboard/mouse 3d games and the difficulty I'm having just trying to access the buttons and thumbsticks on the Oculus Quest is making it hard to move forward and get more experience in C#. I've tried reading the docs that you mentioned but nothing seems to work, I think I'm just missing really obvious bits. This is what I tried to do with your response;

Code (CSharp):

1.             Vector2 thumbDirection = device.TryGetFeatureValue( CommonUsages.primary2DAxis, out Vector2 thumbDirection );
2.             float moveAwayAmount = thumbDirection.y;

And I didn't make anymore changes to the previous code I posted. There error I get is "Can't implicitly convert type bool to UnityEngine.Vector2. I'm guessing the error might be that I just wrote "device" but I'm not sure what I'm supposed to have there, am I supposed to direct the code to the oculus controller? Isn't that what you already had in the previous line of code, which came before what I wrote above, which is;

Code (CSharp):

1.  InputDevice device = InputDevices.GetDeviceAtXRNode(controller.controllerNode);

 

The keyword "out" in C# means "this variable is being created by the method I'm calling, and it will provide a value".

The idea is that normally you send data "in" to a method (that's your parameters/arguments). In some programming langauges, every parameter has to be explicitly marked "in". But sometimes you can send data "out" from a method. Before the "return" statement was invented, this was the only way methods could return data: by having one or more of their arugments marked as "out".

So ... you are currently trying to declare the same variable twice in that line. If you fixed your current bug, the next one would be something about how you can't declare the same variable because you've already declared it.

Separately: the error message tells you exactly what it means. You are trying to convert something of type bool to something of type Vector2. Look in that line and see where that's happening (if you're using VSCode or Rider, you can hover the mouse over a method name or a variable name and it should popup a thing telling you what type that is, or what type it returns).

 

I finally figured this out! Watched a walk-through video of some similar code implementation by the youtuber Dilmer Valecillos and was able to apply it to your code.

The only thing is the direction seems flipped to me, when I move the thumbstick down, the object moves away from me and when it's up, the object moves towards me. Here's what I have;

*First, on the grabbable object, I set it's attach transform to my GameObject with this code/XR Controller*

Code (CSharp):

1. public class TractorBeam : MonoBehaviour
2. {
3.  
4.     public void Update()
5.     {
6.         var interactor = GetComponent<XRBaseInteractor>();
7.         var controller = GetComponent<XRController>();
8.  
9.         XRGrabInteractable grabbable = interactor.selectTarget as XRGrabInteractable;
10.  
11.         if (grabbable != null && grabbable.attachTransform != null)
12.         {
13.             InputDevice device = InputDevices.GetDeviceAtXRNode(controller.controllerNode);
14.  
15.             device.TryGetFeatureValue(CommonUsages.primary2DAxis, out Vector2 thumbDirection);
16.                    
17.             float moveAwayAmount = thumbDirection.y;
18.  
19.             Vector3 vectorToMoveObjectAlong = grabbable.attachTransform.parent.InverseTransformVector(controller.transform.forward);
20.             grabbable.attachTransform.localPosition += moveAwayAmount * vectorToMoveObjectAlong;
21.  
22.             MethodInfo unity_UpdateInteractorLocalPose = typeof(XRGrabInteractable).GetMethod("UpdateInteractorLocalPose", BindingFlags.NonPublic | BindingFlags.Instance);
23.             unity_UpdateInteractorLocalPose.Invoke(grabbable, new object[] { interactor });
24.         }
25.  
26.     }
27. }

 

Sooo .... multiply the y value by -1 every time you use it .

 

Yup that'll do it!

 

Have you tried implementing a rotation as well, like with the thumbstick moving left and right by any chance?

 

That's going to be quite a bit harder. In theory its easy, but in practice it requires you to understand exactly what XRIT is doing with all the relative positions and rotations. Because if you just "rotate" the object, it will rotate it around the VR controller in 3D space, making it appear to fly off into the air - you need to rotate around the local-center of the objevt - but with XRIT, the local center isn't at the center any more, so it's a bit tricky to work out the sequence of rotations / translations to make that work.

In the end it's probably only an extra 2-3 lines of code, but you could do a lot of head-scratchign figuroing out exactly what those lines will be.

 

(I'm mkaing a new asset for the assetstore that will take care of all this, but I haven't done the in-air rotations yet. I'm still deciding what I want the user-interface to be for that - still experimenting with different combinations of thumbstick, button, etc to find out what feels "right")

 

Then it will for sure be more tricky for me! I was trying to do it something like this;
grabbable.attachTransform.transform.RotateAroundLocal
But I wasn't getting anywhere.

I'll for sure buy it from the asset store when you have it up!

Thanks!

 

I see your Snap and Plug asset in the store - but this one isn't for using VR in game right?

 

Yeah, sorry - the VR stuff is taking some time to finish . Lots of things to fix, and lots of features I want to get working smoothly in VR (some of which I've already got working, others are "sort of" working but don't feel good for the user, some I haven't started yet). Probably another month, maybe six weeks, before it's ready for early adopters/testers.

 

No problem, I'll for sure get it when you release it!

I've also had difficulty incorporating socket interactions properly. I think in one of your videos you attached one wheel to another wheel. I'm working on a film set building sim and the sockets are behaving oddly when I try to attach one part of a wall to another. In order to make the whole 'tractor beam' code work (which I attached to my right hand controller) I had to make the right hand controller the Attach Transform on the XR Grabbable Script on my Wall Game Object. So when I go to release the grip button over the socket, it's position is way off!

 

This stuff is currently quite difficult to get just right with XR - I've been coding in Unity for almost 10 years, and it still took me a couple of weeks to get all the attach-detach bits working as I wanted!

I think it'll get much easier in a future update, probably sometime in June/July - I think the Unity team are adding a lot of improvements so it's easier for us to do things like this. But they don't have a public roadmap, so we can only guess.

...I'll hurry up and keep working on my implementation, try to get it ready ASAP .

 

PS: in my videos, I'm using my pre-existing socket-management code from SnapAndPlug, it's custom-built and a lot more detailed/powerful than the socket implemetnation in XR. I can make it do things that the XR sockets just can't do .

(NB: the public version of SnapAndPlug doesn't integrate with XR yet, so don't buy it if you want XR integration today. I'm going to update it at the same time as I release my new vr-building system, upgraded so that it works well with XR. But I'm testing/debugging/fixing both of them in parallel, so none of this code is available yet)

 

Now I don't feel so bad! I'll wait for the XR Integrated version, no rush! I think I'll start experimenting with implementing UIs now anyways

 

@a436t4ataf
I know this is a relatively old thread, but I had a question:
I used your code example to work with my own SteamVR system, but I can't think of a way to stop the held object from moving past the controller. It works fine for pushing and pulling, but If I hold the touchpad/joystick down (y= -1) it will just keep going past me. Is there a way to clamp the movement so it doesn't go behind the controller, and not past a certain distance?
Thanks!

 

Clamping is very easy - you've got the vector, so measure the distance that you want to clamp at (either programmatically by looking up the world position of the controller and finding where that point projects to along the line - Unity has APIs that do that trig for you, or you can find plenty of examples on the web ... or by a setting you put in the inespector that you never want it getting closer than distance X) and clamp it.

I've recently rewritten my core library from the ground up, and I'm about to rewrite all the VR parts and package them up for a new asset, so I'll be revisiting this stuff very soon -- if I think up a better approach I'll follow up here.

 

Thank you for the reply! Unless I'm misunderstanding something, Mathf.Clamp() only works on single values, not Vector3/Transforms. This is what I have now: My system is a little different, but the line will make sense.

Code (CSharp):

1. // The object I want to move lerps to the position of objectControlPoint
2. objectControlPoint.transform.position += amountToMove * -objectControlPoint.forward;

Any specific Ideas would be greatly appreciated!

 

clamp() is literally two lines of code - 1 if you try hard enough! - there's no reason you should use Unity's one instead of writing it yourself .

But I was suggesting you clamp the *distance* (which is a single float - 'amountToMove' in your example) anyway, before doing the multiply.

 

Sorry for asking so many dumb questions, but I don't think that will work. Here's a little more of the script for some context. Clamping the amountToMove wouldn't make sense here. It's basically the same as your code above, just working with SteamVR instead of UXR.
Thanks so much for taking the time to reply, I really appreciate it!

Code (CSharp):

1.  
2. float amountToMove = touchpadInput.y * 0.2f; // multiplying it because the raw value is too fast
3. if (touchpadInput.y > 0.4f || touchpadInput.y < -0.4f) // To have a less sensitive deadzone
4. {
5.        objectControlPoint.transform.position += distance * -objectControlPoint.forward;
6. }
7.  

 

Right, so ...

0. Decide the line you're going to move along, and pick a start point so that all the distance calculations below are working in the same space (maybe use a Unity Ray object)
1. Figure out the current position of the controller (its world space)
2. Figure out how far along the line that position is (gives you a distance-along-line)
3. Figure out how far along the line the current position is (a different distance)
4. Figure out the proposed NEW diestance along the line (a third distance)
5. Clamp 4 above to being greater than 2 + (however far from the controller you want the minimum to be)

Unity has methods for finding the nearest point on a line, for finding the place where a line crosses a plane, etc. (but there are copy/pasteable versions on the web too).

I would do 2 above by creating a Unity Plane object at the controller, using the direction as the plane's normal, then asking Unity to tell me where the intersection of the plane and the direction is (that gives me the essence of "where on the line is the controller").

 

Lol, I'm still totally lost, I always get confused with vector math. Maybe this is a little too complicated for me. Thank you anyway for your time!

 

Yeah, no worries. I hope to have a pre-pacakged solution for people very soon - each little bit isn't that complex, but putting it all together is a lot of effort/time/debugging.

 

I just wanted to share, that in this example project from Unity, they already implemented this exact mechanic!
https://github.com/Unity-Technologies/XR-Interaction-Toolkit-Examples