Search Unity

  1. Unity 2018.3 is now released.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. Our response to Improbable’s blog post (and why you can keep working on your SpatialOS game). Read more here
    Dismiss Notice
  4. Want to provide direct feedback to the Unity team? Join the Unity Advisory Panel.
    Dismiss Notice
  5. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice

Google ARCore and VR combined?

Discussion in 'ARCore' started by JDMulti, Aug 30, 2017.

  1. JDMulti

    JDMulti

    Joined:
    Jan 4, 2012
    Posts:
    381
    I was wondering if it would be possible at some point to combine ARCore and Google VR in one app? This way you can use ARCore for position tracking and VR for immersive applications. This would be a perfect situation for both technologies combining there strengths together.

    Is this even possible or are both technologies putting a lot of pressure on the device hardware and do we only see this after a couple of years? And if it would be possible, would we see Unity3D publish settings to be able to do this in short terms?

    The reason asking this is because I'm pretty excited about this company and there GearVR positional tracking and for now it seems to be possible to combine Google ARCore and VR tech and archive the same results (hopefully): http://univrses.com/site/

    I also found someone who mixed Apple ARKit with Google Cardboard, it sounds amazing tbh! =D
    https://github.com/andrewnakas/ARKit-Cardboard-VR


    Sorry if the question is also a bit VR related, but I'm waiting for such AR / VR combo to happen at some point. =)
     
    Last edited: Aug 30, 2017
    Robert-Strohmaier likes this.
  2. mdurand

    mdurand

    Unity Technologies

    Joined:
    Aug 1, 2016
    Posts:
    18
    Coexistence of ARCore and Daydream/Cardboard is not currently supported.

    -Best,
    Mike Durand
     
  3. JDMulti

    JDMulti

    Joined:
    Jan 4, 2012
    Posts:
    381
    It's not currently supported, but is it even possible to be supported at a later moment or is it totally out of scope and can't mobile devices handle this?
     
  4. JDMulti

    JDMulti

    Joined:
    Jan 4, 2012
    Posts:
    381
  5. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    223
    Robert-Strohmaier likes this.
  6. IMAGINE360

    IMAGINE360

    Joined:
    Apr 15, 2016
    Posts:
    3
    Hello
    Hello, do you mind sharing how you managed to make it work together ?
     
  7. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    223
    For the position tracking, it "simply work",
    We had to set "UnityEngine.XR.InputTracking.disablePositionalTracking" to true, as the headsets has little position changes.
    There's no way to disable the rotation of the scene's camera, so we set the ARCore object not to rotate. Then we follow the position of the ARCore object with our own scene's camera(disabled the ARCore camera).

    The main issue was with the webcam and the background texture, we had to edit the shader that shows the webcam texture in the background.
    But a better solution would be to add more API calls from the Java ARCore API, there you have an option to get the texture from the camera, and you'll have more option to use it in the way that you want.
     
    Arkade and IMAGINE360 like this.
  8. MaeL0000

    MaeL0000

    Joined:
    Aug 8, 2015
    Posts:
    23
    We set up ARCore + Cardboard and found that the objects weren't aligning correctly with the "real world" anymore, did you encounter this too and do you know of a fix?
     
  9. reedny

    reedny

    Joined:
    Mar 4, 2017
    Posts:
    56
    Yes, I'm using ARCore and GVR together and see the same thing. I now track the ARCore Y rotation and the VR Camera Y rotation, and when the differ by about 5 degrees (make it easily configurable) I rotate the Player go (under which the camera resides) and then the VR world aligns with the AR world and you can walk around accurately.

    This works well with both a VR world (careful walking around) and with the camera texture that ARCore provides so you are in AR world walking around, but also can place VR objects (in my case painting stuff).

    And yes, the CPU requirements are substantial -- I get around 10-20 fps in a VR world with AR positional tracking, down noticeably from the 60 fps that I normally get.

    This is using a Daydream headset modified to have the back removed so the phone camera can see out.
     
  10. Rbhaniwal

    Rbhaniwal

    Joined:
    Jan 23, 2015
    Posts:
    33
    Is this also working with google cardboard.
    Am also trying this gear + arcore working fine on my end too.
    But cardboard + arcore not working. getting build failure issue again and again.
    If it working fine your side could you please guide me.
    Thanbks in advance.
     

    Attached Files:

  11. reedny

    reedny

    Joined:
    Mar 4, 2017
    Posts:
    56
    Hi @Rbhaniwal - the error you see is because there is a Java package (com.google.android.aidl.BaseProxy) that is in both ARCore and Cardboard. I had the same issue with ARCore and GVR. You need to remove the package from one of the aar files. I chose arcore_client.aar from ARCore. And then (there must be an easier way to do this) :
    1. rename arcore_client.aar to arcore_client.zip
    2. look at arcore_client.zip in Windows explorer and find the classes.jar file inside.
    3. Copy classes.jar out of the zip
    4. Rename classes.jar to classes.zip
    5. Look inside classes.zip in Windows explorer for the above package and delete the .class files (I had 3 of them)
    6. Rename classes.zip to classes.jar
    7. Copy back into arclient_core.zip
    8. Rename arclient_core.zip to arclient_core.aar
    9. Copy back to the ARCore asset folder where you found it.
    10. That should give you a working build
     
    De-Panther and christougher like this.
  12. Rbhaniwal

    Rbhaniwal

    Joined:
    Jan 23, 2015
    Posts:
    33
    @reedny - Thanks a lot.
    It works for me.
     
  13. christougher

    christougher

    Joined:
    Mar 6, 2015
    Posts:
    343
    Also worked for me, thx! Was driving me crazy and I never would have found it on my own!
     
    Last edited: Mar 21, 2018
  14. param_appm

    param_appm

    Joined:
    Jun 14, 2017
    Posts:
    25
    Hello,

    @reedny @christougher We setup the ARCore + Google Cardboard and was able to track surface and load the terrain on detection.

    But we are facing a issue, that terrain keeps moving with our Head Movement. It don't stick to the surface in VR Mode.

    Have anyone of you faced such issues.
     
  15. MaeL0000

    MaeL0000

    Joined:
    Aug 8, 2015
    Posts:
    23
  16. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    223
    Regarding the rotations,
    Instead of waiting for the values of the ARCore Y rotation and the VR Camera Y rotation, to differ by more than few degrees, I'm setting the Parent GO rotation of the VR Camera, to be the ARCamera rotation multiply by the Inverse of the head rotation.
    Something like that:
    transform.rotation = arCamera.rotation * Quaternion.Inverse(InputTracking.GetLocalRotation(XRNode.Head));
     
  17. reedny

    reedny

    Joined:
    Mar 4, 2017
    Posts:
    56
    @De-Panther, that looks good, I'll try that. Do you also rotate the other VR GOs around the Parent GO of the VR camera so they keep the same position relative to the player? My VR world has a room that the user is in plus 3D paint where they have painted, so all of that has to rotate too to keep the AR rotation and the VR rotation consistent.
     
  18. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    223
    I think that I lost you... (Too many VR GO :D )

    I have a GO with the ARCamera(it gets position and rotation)
    I have a GO that follows the position of the ARCamera and get the rotation as I wrote. This GO is also the parent of my MainCamera/VRCamera.

    The other GOs positions depends on the game...
     
  19. ChristophGeske

    ChristophGeske

    Joined:
    Apr 26, 2015
    Posts:
    5
    ARCore and VR seem to work together by now but I discovered that the ARCore tracking frame rate drops by half when booth ARCore and VR are enabled compared to running them separately on the S7. I am currently working on fixing this issue and will publish my results here: https://github.com/ChristophGeske/ARCoreInsideOutTrackingGearVr/graphs/traffic
    as soon as my fix is working. I haven't tested the S8 or S9 maybe they run smooth with 60 FPS even when VR and AR are enabled together but the S7 does not. Has anyone discovered this issue before and knows of an easy fix or if this issue is also present on other ARCore supported devices?

    Update: My current understanding is that as soon as you enable VR support the performance drops significantly. The ARCore team knows about this bug but they do not fix it maybe they are not interested in investing time since VR is not what ARCore is meant to be used for. My current approach is building the VR camera myself so I do not have to enable VR support.
    The framerate of the ARCore is always 30fps not 60fps and can drop further if the performance load is to much. Interpolation can be used to bring the framerate to 60fps.
     
    Last edited: Jan 13, 2019 at 2:19 PM
    christougher and Arkade like this.
  20. rob_ice

    rob_ice

    Joined:
    Nov 11, 2016
    Posts:
    105
    Sounds like this combination could be very taxing on the device - not even considering when thermal throttling kicks in after running for a while as phones usually get warm enough when simply running a cardboard app when I was working on this a while ago