Search Unity

Match Unity camera with iPhone camera

Discussion in 'iOS and tvOS' started by Mish, Mar 20, 2012.

  1. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
    Hi

    I am working on an augmented reality game, which has an unity scene as an overlay to the iPhone camera view. What I want to achieve is that when the phone is rotated up with the camera facing the celling or down towards the ground, the unity camera will follow realistically making it look like the object in the scene is where it would be expected.

    So far I am unity the iPhone gyroscope for the rotation.

    Code (csharp):
    1. newAngles.x += (Input.gyro.rotationRate.y * (180 / Mathf.PI)) * Time.deltaTime;
    However the rotation gets offset after a while due to the relative calculation of the rotation. What would be the best way to recalibrate the gyroscope?

    I tried to experiment with the attitude of the gyroscope

    Code (csharp):
    1. Gyroscope.attitude
    It gives a quaternion which I cant assign directly to the camera angles as the values seem a bit strange to me. As far as I know the values are suppose to be the yaw, pitch and roll. But it seems like some of the angles are dependent on each other. For instance when rotating around the X axis the value of the Y rotation suddenly switches so its inverted when using the euler angles representation. Im not sure why this happens?
     
  2. GameFreak

    GameFreak

    Joined:
    Oct 31, 2011
    Posts:
    13
    Okay I am looking for the exact same thing and will look into it but the place I am stuck is accessing iPhone camera from unity! :S

    How did you do that?
     
  3. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
  4. J_P_

    J_P_

    Joined:
    Jan 9, 2010
    Posts:
    1,016
    There's a gyro cam script somewhere on the forums that does exactly that. I'm using it for an augmented reality game myself :)
     
  5. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
  6. George Foot

    George Foot

    Joined:
    Feb 22, 2012
    Posts:
    399
    The gyro drifts over time, but it's great for detecting sharp movements. I made a script last week that smoothly corrects it based on compass data, which works quite well - it's a challenge because the compass itself is a bit naff. Still, I was happy with the result.

    On my Android tablet the compass is absolutely awful, though. :(

    Code-wise, the gist is to use gyro.attitude as a basic orientation with good high-frequency response, and also calculate the corresponding orientation based on the compass reading. Then calculate the difference, i.e. compassOrientation * Quaternion.Inverse(gyroOrientation), and maintain a correction quaternion that interpolates towards this difference (various algorithms to choose from depending on the accuracy-stability trade-off you're happy with). Finally, premultiply this smoothed correction factor to the gyro orientation, and you should get something which matches the compass when the device is stationary, but is more responsive when the device is rotated.

    I had to post-multiply gyro.attitude by a 90 degree rotation, too, before doing anything else - I think this is because the device was initialised in landscape mode.

    Here's the code, anyway.

    Code (csharp):
    1.  
    2. using UnityEngine;
    3.  
    4. public class MoveWithCompass : MonoBehaviour
    5. {
    6.     private double _lastCompassUpdateTime = 0;
    7.     private Quaternion _correction = Quaternion.identity;
    8.     private Quaternion _targetCorrection = Quaternion.identity;
    9.     private Quaternion _compassOrientation = Quaternion.identity;
    10.    
    11.     void Start()
    12.     {
    13.         Input.gyro.enabled = true;
    14.         Input.compass.enabled = true;
    15.     }
    16.    
    17.     void Update()
    18.     {
    19.         // The gyro is very effective for high frequency movements, but drifts its
    20.         // orientation over longer periods, so we want to use the compass to correct it.
    21.         // The iPad's compass has low time resolution, however, so we let the gyro be
    22.         // mostly in charge here.
    23.        
    24.         // First we take the gyro's orientation and make a change of basis so it better
    25.         // represents the orientation we'd like it to have
    26.         Quaternion gyroOrientation = Quaternion.Euler (90, 0, 0) * Input.gyro.attitude * Quaternion.Euler(0, 0, 90);
    27.    
    28.         // See if the compass has new data
    29.         if (Input.compass.timestamp > _lastCompassUpdateTime)
    30.         {
    31.             _lastCompassUpdateTime = Input.compass.timestamp;
    32.        
    33.             // Work out an orientation based primarily on the compass
    34.             Vector3 gravity = Input.gyro.gravity.normalized;
    35.             Vector3 flatNorth = Input.compass.rawVector -
    36.                 Vector3.Dot(gravity, Input.compass.rawVector) * gravity;
    37.             _compassOrientation = Quaternion.Euler (180, 0, 0) * Quaternion.Inverse(Quaternion.LookRotation(flatNorth, -gravity)) * Quaternion.Euler (0, 0, 90);
    38.            
    39.             // Calculate the target correction factor
    40.             _targetCorrection = _compassOrientation * Quaternion.Inverse(gyroOrientation);
    41.         }
    42.        
    43.         // Jump straight to the target correction if it's a long way; otherwise, slerp towards it very slowly
    44.         if (Quaternion.Angle(_correction, _targetCorrection) > 45)
    45.             _correction = _targetCorrection;
    46.         else
    47.             _correction = Quaternion.Slerp(_correction, _targetCorrection, 0.02f);
    48.        
    49.         // Easy bit :)
    50.         transform.rotation = _correction * gyroOrientation;
    51.     }
    52. }
    53.  
     
    PhilAR likes this.
  7. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
    Ah cool I was looking for this as well :D thanks for posting!

    Could you explain this part tho:



    Code (csharp):
    1.  // Work out an orientation based primarily on the compass
    2.  
    3.             Vector3 gravity = Input.gyro.gravity.normalized;
    4.  
    5.             Vector3 flatNorth = Input.compass.rawVector -
    6.  
    7.                 Vector3.Dot(gravity, Input.compass.rawVector) * gravity;
    8.  
    9.             _compassOrientation = Quaternion.Euler (180, 0, 0) * Quaternion.Inverse(Quaternion.LookRotation(flatNorth, -gravity)) * Quaternion.Euler (0, 0, 90);
    10.  
    11.            
    12.  
    13.             // Calculate the target correction factor
    14.  
    15.             _targetCorrection = _compassOrientation * Quaternion.Inverse(gyroOrientation);
    16.  
    How is the compass orientation calculated?
     
  8. runonthespot

    runonthespot

    Joined:
    Sep 29, 2010
    Posts:
    302
    Just a note: One of the reasons the various AR libraries exist (and they are pretty much all based on visual image marker tracking), is that gyroscope is just not accurate enough to keep the unity camera and real camera in sync.

    I know there is a QR code tracking libary somewhere on the forums, which @jasperstocker used to do an AR driving game in Unity a while back. Also, definitely worth looking at QCAR/Vuforia (free) or String (+$).
     
  9. George Foot

    George Foot

    Joined:
    Feb 22, 2012
    Posts:
    399
    You can use the magnetic or true heading fields of Input.compass, but I found calculating my own values worked better. So, I take the gravity vector from the gyro, and use it to flatten the magnetic flux vector from the compass sensor, giving me a horizontal "north" vector, relative to the device. This whole calculation is in portrait-device-relative coordinates.

    Then I make an orientation quaternion from that. Quaternion.LookRotation gives a rotation from relative-to-north-vector to relative-to-portrait-device - but I want the opposite, so I run it through Quaternion.Inverse. Then, I post-multiply by a Z rotation, because I told Unity in the project settings that the device would be landscape and it has aligned the main camera that way. I think this would not be necessary if I told Unity to use portrait mode.

    I also pre-multiply by a 180-degree X rotation, just because that seemed to be necessary too... I didn't think about it much, I just observed what was going on and added the rotations that seemed necessary to make the camera behave in a stable way.

    Once you've got _compassOrientation, you can assign it straight into transform.rotation if you like and experiment. It only updates infrequently, so things will be jerky, but you can still see whether it's giving the right orientation. This is how I determined whether the device responded to rotations in the right axes (if not, change the post-multiplied quaternion) and after that whether, overall, "north" and "up" were in the right direction (if not, change the pre-multiplied quaternion). Just put distinctive objects in the Unity scene above, below, and in the four cardinal directions from the camera, then see if they're in the right place in "real world" space.

    _targetCorrection is calculated as the thing I need to premultiply by in order to exactly turn gyroOrientation into _compassOrientation. I want X such that X * G = C. Post-multilpy by Ginv, and you get X = C * Ginv, so that's what I calculate here.
     
  10. George Foot

    George Foot

    Joined:
    Feb 22, 2012
    Posts:
    399
    Sure - Vuforia is very nice.

    It's not really true that the gyroscope is not accurate enough though - I was very impressed with the result you can get from it, but you need to be aware of the limitiations - you only get directional information. As soon as you want to do anything translational, you hit serious roadblocks. GPS is not accurate, many devices don't have it, and the accelerometers are not accurate enough to integrate out into real world movements, even in a crude fashion. It was fun to try though. :)

    So with this code you could fairly easily do something like Nintendo's 3DS face shooting game, as it's all directional, but if you want to tie game objects to real world objects and let the user move around them, you need image processing. Vuforia has several options here, and looks really powerful. I haven't looked at the others.
     
  11. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
    Thanks a lot for the explanation! ill try to experiment with it :)
     
  12. Swordfish90

    Swordfish90

    Joined:
    Mar 22, 2012
    Posts:
    1
    Amazing... Exactly what i was looking for... Thanks to everyone!!
     
  13. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
    Im trying to get the compass correction by using the trueHeading. But cant seem to get the rotations right, I tried several things but without luck. This is the current solution:

    Code (csharp):
    1. gyroOrientation = Quaternion.Euler (90, 0, 0) * currentAttitude * Quaternion.Euler (0, 0, 90);
    2.        Quaternion yDiff = Quaternion.Euler(0, heading, 0) * Quaternion.Inverse(gyroOrientation);
    3.        transform.localRotation = yDiff * gyroOrientation;
    Any suggestions?
     
  14. George Foot

    George Foot

    Joined:
    Feb 22, 2012
    Posts:
    399
    I used to derive compassOrientation from magneticHeading - here's the code:

    Code (csharp):
    1.  
    2.             _compassOrientation = Quaternion.Euler(
    3.                 Mathf.Rad2Deg * Mathf.Asin(-Input.gyro.gravity.z),
    4.                 Input.compass.magneticHeading,
    5.                 Mathf.Rad2Deg * Mathf.Atan2(Input.gyro.gravity.y, -Input.gyro.gravity.x));
    6.  
    You should probably use this, but switch Input.compass.magneticHeading with Input.compass.trueHeading, or whatever it's called. This then slots in to the rest of the code I posted the other day, in place of the flatNorth stuff.

    Oh how I wish Unity used radians for angles, like sane people do! Or at least make the whole UnityEngine.Mathf library use degrees, for consistency.
     
    D3m0n likes this.
  15. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
    Thanks for quick reply. But I get a similar problem as before, when holding the phone in landscape mode and tilting on the Z axis. Thats is down to the left or right, the whole scene rotates by a huge amount. Did that also happen to you?
     
  16. George Foot

    George Foot

    Joined:
    Feb 22, 2012
    Posts:
    399
    Post-multiply by whatever I was multiplying by for the gyro orientation - Quaternion.Euler(0, 0, 90) or something like that.

    I should really have moved that to the end of the function, rather than having it get mixed up in the rest of the calculation.

    The earlier thread that was linked before has a more extensive list of corrections for different orientations:

    http://forum.unity3d.com/threads/98...-camera-on-iPhone-4?highlight=gyro+cam+script
     
  17. andershartzen

    andershartzen

    Joined:
    Feb 17, 2012
    Posts:
    2
    Hey all

    A very interesting thread I must say! This is also something I am looking for in my current project.

    I am trying to just use "Input.compass.trueHeading" as MikeWG is as well.

    George Foot, to calculate the rotation you basically do this (when disregarding your check if the targetcorrection is more than 45 degrees away from the correction of last frame):

    _targetCorrection = _compassOrientation * Quaternion.Inverse(gyroOrientation);
    transform.rotation = _targetCorrection * gyroOrientation;

    I am just wondering why you need to multiply _compassOrientation with the inverse gyroOrientation? I am having trouble visualizing what _targetCorrection is other than a composite rotation of the Compass rotation and the inverse gyro rotation (which is its conjugate [-g_rotX, -g_rotY, -g_rotZ, scalar] because of its unit length). How does this composite rotation help us?

    If you rewrite the above code to this:
    transform.rotation = _compassOrientation * Quaternion.Inverse(gyroOrientation) * gyroOrientation;

    Don't you just end up with this?:
    transform.rotation = _compassOrientation * 1;

    Or have I got it wrong (which I may well have - quaternions are not my strong suit ;-) )

    Many thanks beforehand
     
  18. George Foot

    George Foot

    Joined:
    Feb 22, 2012
    Posts:
    399
    That's exactly right, but the whole point is that we don't actually multiply the correction factor by the gyro orientation - we multiply a smoothed version of the correction factor. So in the long term, if you hold the device still then it does end up just returning the compass orientation, but in the short term it trusts the gyro more closely than the compass.

    One reason for this is that the compass updates are low frequency; another is that they're not particularly accurate. The degree of smoothing I ended up using was very large - about 10 seconds I think to completely reach equilibrium.

    Also note that the correction target is only updated when the compass has changed. So "Quaternion.Inverse(gyroOrientation)" is really "Quaternion.Inverse(gyroOrientationLastTimeTheCompassWasUpdated)". It cancels out on that particular frame, but on subsequent frames the gyro will have moved, and you can read the whole thing as "_compassOrientation * changeInGyroOrientation" which might be more intuitive? That is still ignoring the smoothing.
     
    Last edited: Mar 25, 2012
  19. Mish

    Mish

    Joined:
    Apr 23, 2010
    Posts:
    96
    In order to fix the rotations when tilting I tried applying the tilt amount to the heading. So when the device goes from landscape to portrait mode it would subtract 90 degrees from the original heading.
    This works, but sometimes still gives some jitter.

    Code (csharp):
    1.  
    2. float zAngle = Mathf.Rad2Deg * Mathf.Atan2 (Input.gyro.gravity.y, -Input.gyro.gravity.x);
    3. usedHeading + zAngle;
    4.  
    I guess the most optimal calculations would be to calculate your own heading vector like above?
     
    D3m0n likes this.
  20. ikriz

    ikriz

    Joined:
    Dec 3, 2009
    Posts:
    85
    I'm running this on an Android device (Galaxy Tab) but i notice when the device flips towards facedown (looking upwards at a building) my compass flips north to south and everything rotates 180 degrees... how can i prevent this? btw fixed the application to landscape mode.
     
    Last edited: Jun 3, 2012
  21. D3m0n

    D3m0n

    Joined:
    Nov 11, 2014
    Posts:
    96
    No permanent solution?
     
  22. madclouds

    madclouds

    Joined:
    Nov 6, 2013
    Posts:
    30
    Just wanted to help out anyone who lands here. I made this script to control the rotation of the camera with a gyroscope. Attach this script to the camera you want to control with the gyroscope:

    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. public class GyroCamera : MonoBehaviour {
    5.  
    6.     Quaternion initialRotation;
    7.     Quaternion gyroInitialRotation;
    8.     bool gyroEnabled;
    9.  
    10.     void Start () {
    11.         initialRotation = transform.rotation;
    12.         Input.gyro.enabled = true;
    13.         gyroInitialRotation = Input.gyro.attitude;
    14.     }
    15.  
    16.     void Update() {
    17.         if(gyroEnabled){
    18.         #if !UNITY_EDITOR
    19.             Quaternion offsetRotation = ConvertRotation(Quaternion.Inverse(gyroInitialRotation) * Input.gyro.attitude);
    20.             transform.rotation = initialRotation * offsetRotation;
    21.         #else
    22.             //for unity editor contorl
    23.             float speed = 2.0f;
    24.             transform.Rotate(Input.GetAxis("Mouse Y") * speed, Input.GetAxis("Mouse X") * speed, 0);
    25.         #endif
    26.         }
    27.     }
    28.  
    29.     public void AlignGyro() {
    30.         gyroEnabled = false;
    31.         transform.rotation = Quaternion.identity;
    32.     }
    33.  
    34.     public void StartGyro() {
    35.         initialRotation = transform.rotation;
    36.         gyroInitialRotation = Input.gyro.attitude;
    37.         gyroEnabled = true;
    38.     }
    39.  
    40.     private static Quaternion ConvertRotation(Quaternion q)
    41.     {
    42.         return new Quaternion(q.x, q.y, -q.z, -q.w);  
    43.     }
    44. }
     
    codegasm and softrare like this.
  23. rafalpast

    rafalpast

    Joined:
    Sep 22, 2016
    Posts:
    1
    @George
    First of all, wow, amazing job, this works like charm!
    There's one thing I'm struggling with though - I want to align the game object with the real world, that is - invert the orientation and point north.
    An example would be a compass needle tilted along the x axis in the opposite direction than the phone, so when I look at my phone from the top - is see the needle from the top, when I tilt the phone and have it in front of me, positioned vertically - I see the back of it.
    How should I approach that?
     
  24. passerbycmc

    passerbycmc

    Joined:
    Feb 12, 2015
    Posts:
    1,470
    @George Foot really nice code, one of the smoothest ways of doing this that i have seen, but i am struggling trying to get the correct rotations for using this in portrait mode on a ipad. Does anyone have any ideas.