Scripting controls that change dynamically based on the positions of object and camera.

Hi guys, I have an augmented reality game that lets users move around a scene with their mobile' device's camera. There is one joystick that controls a vehicle, up on the joystick is "forward" for the vehicle and back is "reverse." Left and right turn the car left and right respectively. This works well when the camera is directly behind the car, but when the car is facing any other direction in respect to the camera things get wonky. I want the controls to change dynamically so that no matter which way the car is facing in respect to the camera, the joystick always register's "forward" when it is pulled in the direction the car is facing.

I've posted what I've got so far below, which as I'm sure all the coders more experienced than I will quickly see, does not work. It works fine for when the car faces away from the camera or towards it, but I'm having trouble with the perpendicular situation. The thing is, I'm sure I'm on the wrong track here because I'm not looking for a "perfect" perpendicular situation. I want it to change smoothly so that say, if the car is pointing 45 degrees away and to the right of the camera, then pushing half way between the "up" and "right" values on the joystick will register as "forward."

As I said before the game is targeting mobile devices. The joystick is touch-based and feeds a value called "relativeVector" that provides a vector3 that ranges in the x and y between -1 for down/left values and 1 for up/right values. The Z axis is always 0.0.

Here's the code, I really appreciate any help I can get making this functionality work.

Code (CSharp):

1.     void mobileInput (Vector3 relativeVector){
2.         //find the forward vectors of camera and car
3.         Vector3 cameraForward = aRCamera.transform.TransformDirection (Vector3.forward);
4.         //cameraForward.y = 0.0f;
5.         //cameraForward.Normalize ();
6.         Vector3 carForward = this.gameObject.transform.TransformDirection (Vector3.forward);
7.         //carForward.y = 0.0f;
8.         //carForward.Normalize ();
9.  
10.         Vector3 cameraRight = aRCamera.transform.TransformDirection (Vector3.right);
11.         //cameraRight.y = 0.0f;
12.        // cameraRight.Normalize ();
13.         Vector3 carRight = this.gameObject.transform.TransformDirection (Vector3.right);
14.         //carRight.y = 0.0f;
15.         //carRight.Normalize ();
16.  
17.         //change controls to reflect orientation of object in relation to camera
18.         if (Vector3.Dot (cameraRight, carRight) > 0 && Vector3.Dot (cameraForward, carForward) == 0) {
19.             print ("Car is facing camera right.");
20.             horizontal = relativeVector.y;
21.             throttle = relativeVector.x;
22.                 }
23.         else if (Vector3.Dot (cameraRight, carRight) < 0 && Vector3.Dot (cameraForward, carForward) == 0) {
24.             print ("Car is facing camera left.");
25.             horizontal = -relativeVector.y;
26.             throttle = -relativeVector.x;
27.         }
28.         else if (Vector3.Dot (cameraForward, carForward) > 0 ) {
29.             print ("Camera facing car's back.");
30.             horizontal = relativeVector.x;
31.             throttle = relativeVector.y;
32.                 }
33.         else if (Vector3.Dot (cameraForward, carForward) < 0) {
34.             print ("Camera facing car's front.");
35.             horizontal = -relativeVector.x;
36.             throttle = -relativeVector.y;
37.                 }
38.  
39.         //horizontal = relativeVector.x;
40.         //throttle = relativeVector.y;
41.             }

 

Perhaps a simpler way of describing this problem would be: What would you do to create a simple, top-down car game where the car automatically turns and drives in the direction the joystick is being pressed? It's not quite what I'm looking for but it would get me closer to the solution I'm trying to find.

 

It would probably be easier to use raycasts, and then move the car towards the raycast hit point, then for the cameras you could either put it inside of the object (which will rotate the camera with the car automatically) or set up a script to have the camera follow the car.
Note: void OnMouseDown(){} is also an inherited raycast behind the scenes.

Heres a link to what I'm talking about:
http://answers.unity3d.com/questions/586745/get-an-object-to-facemove-to-raycast-hitpoint.html
or
http://answers.unity3d.com/questions/238579/moving-objects-to-point-of-touch-on-iphone.html

 

Hey I appreciate the help! Unfortunately the user needs to be free to move the camera at will (the game is being played in the real world, using the smartphone/tablet's camera). I want the player to be able move around the scene just the way he would in real life, so I pretty much can't tie the camera onto anything else in the scene. Forgive me if I missed something, I'll look further into raycasts as a possible alternative for movement, but I'd like to try to get this joystick working. I'm afraid this doesn't appear to be quite the solution to my problem.

 

So you want to move the car using the joystick, and the camera using something else?
Considering you said Augmented reality I'm assuming you want to be able to drive a car through a world by holding the joystick forward, and moving the phone using the gyro/accelerometer Yes?

Here's a tutorial for the joystick which I literally just posted somewhere else:

And for the accelerometer and gyro
Acc:
http://unity3d.com/learn/tutorials/modules/beginner/platform-specific/accelerometer-input
Gyro:
http://docs.unity3d.com/ScriptReference/Gyroscope.html

you could put those two together, have the joystick control the cars movement foward or backward, left or right, and the accelerometer and gyro control both the vision inside the car (ie looking at the dashboard, out the windows, etc)
OR you could have the joystick control the movement forward and backward, and then the gyro and accelerometer to dictate which way you are driving.

It's really hard to answer the questions without exactly what you want to do.
If you're afraid of letting your idea out, you can send me a personal message and I'll see if I can help you out more.

 

Sorry about being unclear, I'm not concerned with intellectual property or anything, I just didn't want to confuse the matter with unnecessary details but it seems I went too far the other direction.

My goal is just to let someone drive a little car on their living room floor. It's on mobile devices so people can see the car on their screen through the use of the device's inbuilt camera. The game finds a tracking marker on the floor and then spawns a car on top of it. The car is controlled using an onscreen single joystick which at the moment is mapped so that up and down control throttle, and left and right control the horizontal motion.

I'm using the Vuforia augmented reality plugin which has a feature called "extended" tracking which I believes uses gyro and accelerometer data to keep the Unity scene live, even after its lost track of the tracking marker. I've had mixed results with that feature so far. So although using that data as a method of steering is pretty acceptable in your typical mobile racing game, it doesn't really fit for a game whose whole selling point is being able to move around the scene at will.

So the vehicle is being driven third person with the onscreen joystick, and as long as people stay roughly behind the vehicle, the joystick controls work great as mapped. But since the spirit of the thing is letting people explore the scene, I want them to be able to walk around the car. The thing is, when people walk to the other side of the car, the joystick now feels inverse. Pressing the "up" on the joystick causes the car to move towards their feet, as opposed to away from them. So my goal is just trying to create a control scheme that feels natural, no matter where the user is standing in respect to the car.

I hope that clears things up. I'd be happy to provide any further details you feel might help.

 

The camera perspective I guess might be a bit closer to something like the early grand theft auto games? Since generally people are standing and looking down at their floors they are more or less above the car at all times. Although, that's not quite precise either because it's not isometric, the camera is just floating around, completely dictated by wherever the user is standing in relation to the car. The tricky bit here is the camera is completely dictated by the user's hand positions, because the game takes place in the real world. Does that make sense? I just need to find a way so that "up" on the joystick always causes the car to want to move the direction the camera is facing (in the x and z, we can completely discount the y).

Edit: I've attached a photo of it running on my webcam. The piece of paper is my tracking marker and the car is obviously part of scene in unity, sitting on top of an invisible plane. The hand is my hand.

 

Alright, so what you're going to want to do is to get the forward vector of the car in screen space, and then do some operations on that. Using dot products is a good idea, but I think there's a simpler way.

First of all, I think we should start with TransformDirection and its inverse. Transform the car's forward into world space, then use camera.transform.InverseTransformDirection to transform that direction vector into screen space. Full disclosure, I've never used those functions that way, but I think it will work. (There might be a bit of inaccuracy in the rotation because a camera's view frustum and its transform matrix are not quite the same thing, but I don't think the inaccuracy will be enough to notice, at least until the truck is at the very edge of the screen.)

So, you have the car's forward vector in screen space. What now? You're going to want to flatten it in the Z direction first, I think. And.... okay, I'm going to go off on a bit of a sidetrack here.

Do you want the interface to indicate to the user what's going on here? IMO it should, and a good way to do that would be to actually rotate the entire widget onscreen. If you agree, then this will be a bit easier. If not, you could probably accomplish the same thing with an invisible rotating thing overtop the main one. OK? OK. Moving on...

...so, now, you need a rotation angle, which Mathf.Atan2 can kindly provide for you. Rotate the joystick onscreen so that it matches the desired direction. So now, when the user taps and holds on the joystick, you should be able to (assuming you're using a 3D collider on the joystick) raycast onto the joystick, and then use joystickTransform.InverseTransformPoint on the raycast's hit point.

And that vector will contain an X and a Y coordinate, which should be the values you want for your truck movement. Yeah, there's an awful lot of if and it should work on this plan, but, uh.... it should work.

 

Mother of god. Alright! This is going to stretch the limits of my pitiful grasp of this engine to the breaking point, but damn if I won't give it the old college try. Thanks, I'll get to work chiseling away at this first thing in the morning.

I'll keep you updated. This might be a struggle so I'll come running back if I get lost.

Unfortunately right off the bat there was one problem. I can't quite grasp what exactly you're asking I would or would not want to communicate to the user? Do you mean rendering some sort of arrow or gizmo above the car that communicates the cars..um...heading?

I have the feeling we're in the jungle now.

 

We are definitely in the jungle, but we are jungle men aren't we?

So.. question for you before anything, logically, if you're standing at the car, and you press "Up" which is literally "forward" for the car to the user, that would make the car go forward no matter what direction you are. And back would always be reverse, left left for the car, and right right for the car, just like the old RC cars. It's something we are all used to, and I'm a pretty big fan of not reinventing wheels but re-purposing them instead. (no pun intended)

The thing is, once you actually code the rotation in, you might find it to be extremely disorienting for the user that the car is facing them, but going in reverse when they press the joystick up/forward, and I think generally when they actually want to drive it (not just spin around it with the camera cause its momentarily cool to do so) they will be behind the car "chasing" it, again like RC cars.

I think what star was saying about showing the user was to add something which shows the user what direction they are in relativity to the 3d space created by the augmentation.

If you do end up going the direction of rotating the joystick I think star's idea of putting a compass type object on the screen would be helpful for the user to catch their bearings. Something like the XYZ gizmo in every 3d program (including unity) really helps a person find themselves in a real/not-real 3d space.

 

Yeah, you're right. I might be over complicating this. I had a few playtests with people I know and they tended to want to stand stationary at the start point and control the car from there. Once the car was a distance away and turned around they had some trouble working the single joystick. Maybe if I just did two joysticks like a traditional RC car, one for throttle/reverse and one for the horizontal motion...I just had doubts that experience would translate well to a mobile device since there is no tactile feedback. For that reason I thought I'd try to find a new solution. I just had an instinct that this sort of set up I've been going for here would help make the single joystick more palatable and save precious screen real estate but you could be right.

Anyway, this might be beyond me. Just to make sure I've got the right idea about this though I've posted a picture. Is this what you guys are suggesting? Find the screen direction of the car and then rotating the base of the joystick to match it? Like in figures 1 and 2? Then communicating this orientation to the user with a compass, perhaps on the joystick itself like in figure 3?

I'm using CNJoystick from the asset store at the moment because it was easy to implement at the time, but it uses sprites so I may have to build one from scratch (perhaps using your tutorial!)

 

Yep, you've basically got what I was talking about, at least.

 

Hey, this might be madness, let me know if this doesn't make sense. But here's the function in the joystick script that creates the event that my car controller script is subscribed to, and through which passes the joystick x/y coordinates in the form of the variable "dragDirection."

Code (CSharp):

1.   void TweakJoystick(Vector3 desiredPosition)
2.     {
3.         //possibly rotating joystick base first base on relative position of the car
4.  
5.  
6.  
7.         // We convert our screen coordinates of the touch to local frustum coordinates
8.         ScreenPointToRelativeFrustumPoint(desiredPosition);
9.         // And then we find our joystick relative position
10.         Vector3 dragDirection = screenPointInUnits - invokeTouchPosition;
11.         // If the joystick is inside it's base, we keep it's position under a finger
12.         // sqrMagnitude is used for optimization, as Multiplication operation is much cheaper than Square Root
13.         if (dragDirection.sqrMagnitude <= joystickBaseRadius * joystickBaseRadius)
14.         {
15.             joystickTransform.localPosition = screenPointInUnits;
16.             dragDirection /= joystickBaseRadius;
17.         }
18.         // But if we drag our finger too far, joystick will remain at the border of it's base
19.         else
20.         {
21.             joystickTransform.localPosition = invokeTouchPosition + dragDirection.normalized * joystickBaseRadius;
22.             dragDirection.Normalize();
23.         }
24.  
25.         // If we're tweaking, we should dispatch our event
26.         if (JoystickMovedEvent != null)
27.         {
28.             JoystickMovedEvent(dragDirection);
29.         }
30.     }

Let's say I got the angle of my car in relation to my camera. (Haven't gotten to that bit yet.) My question is: is there a function/equation combination that would convert that angle into something that would affect the directional vector "dragDirection" in a way that recreates the general idea of rotating the whole joystick, but without actually having to physically rotate it?

Put another way:

dragDirection + angleAdjustment = adjustedJoystickValues

Does that make any sense? This seemed a lot simpler in my head.