Google ARCore and VR combined?

I was wondering if it would be possible at some point to combine ARCore and Google VR in one app? This way you can use ARCore for position tracking and VR for immersive applications. This would be a perfect situation for both technologies combining there strengths together.

Is this even possible or are both technologies putting a lot of pressure on the device hardware and do we only see this after a couple of years? And if it would be possible, would we see Unity3D publish settings to be able to do this in short terms?

The reason asking this is because I'm pretty excited about this company and there GearVR positional tracking and for now it seems to be possible to combine Google ARCore and VR tech and archive the same results (hopefully): http://univrses.com/site/

I also found someone who mixed Apple ARKit with Google Cardboard, it sounds amazing tbh! =D
https://github.com/andrewnakas/ARKit-Cardboard-VR


Sorry if the question is also a bit VR related, but I'm waiting for such AR / VR combo to happen at some point. =)

 

Coexistence of ARCore and Daydream/Cardboard is not currently supported.

-Best,
Mike Durand

 

It's not currently supported, but is it even possible to be supported at a later moment or is it totally out of scope and can't mobile devices handle this?

 

This morning I saw a tweet about someone who got a Google Daydream setup working with ARCore.
https://twitter.com/ambocclusion/status/904483272771678208

As you can see, there is a major jitter but it does work. =)

 

We got it to work with both Cardboard and GearVR.
But it's not something that is consumer ready - as the device get hot really fast, and you need to restart the app from time to time.

A video with the GearVR demo:
https://www.facebook.com/DePanther/videos/vb.570342724/10155087959392725/

 

Hello

Hello, do you mind sharing how you managed to make it work together ?

 

For the position tracking, it "simply work",
We had to set "UnityEngine.XR.InputTracking.disablePositionalTracking" to true, as the headsets has little position changes.
There's no way to disable the rotation of the scene's camera, so we set the ARCore object not to rotate. Then we follow the position of the ARCore object with our own scene's camera(disabled the ARCore camera).

The main issue was with the webcam and the background texture, we had to edit the shader that shows the webcam texture in the background.
But a better solution would be to add more API calls from the Java ARCore API, there you have an option to get the texture from the camera, and you'll have more option to use it in the way that you want.

 

We set up ARCore + Cardboard and found that the objects weren't aligning correctly with the "real world" anymore, did you encounter this too and do you know of a fix?

 

Yes, I'm using ARCore and GVR together and see the same thing. I now track the ARCore Y rotation and the VR Camera Y rotation, and when the differ by about 5 degrees (make it easily configurable) I rotate the Player go (under which the camera resides) and then the VR world aligns with the AR world and you can walk around accurately.

This works well with both a VR world (careful walking around) and with the camera texture that ARCore provides so you are in AR world walking around, but also can place VR objects (in my case painting stuff).

And yes, the CPU requirements are substantial -- I get around 10-20 fps in a VR world with AR positional tracking, down noticeably from the 60 fps that I normally get.

This is using a Daydream headset modified to have the back removed so the phone camera can see out.

 

Is this also working with google cardboard.
Am also trying this gear + arcore working fine on my end too.
But cardboard + arcore not working. getting build failure issue again and again.
If it working fine your side could you please guide me.
Thanbks in advance.

 

Hi @Rbhaniwal - the error you see is because there is a Java package (com.google.android.aidl.BaseProxy) that is in both ARCore and Cardboard. I had the same issue with ARCore and GVR. You need to remove the package from one of the aar files. I chose arcore_client.aar from ARCore. And then (there must be an easier way to do this) :

1. rename arcore_client.aar to arcore_client.zip
2. look at arcore_client.zip in Windows explorer and find the classes.jar file inside.
3. Copy classes.jar out of the zip
4. Rename classes.jar to classes.zip
5. Look inside classes.zip in Windows explorer for the above package and delete the .class files (I had 3 of them)
6. Rename classes.zip to classes.jar
7. Copy back into arclient_core.zip
8. Rename arclient_core.zip to arclient_core.aar
9. Copy back to the ARCore asset folder where you found it.
10. That should give you a working build

 

@reedny - Thanks a lot.
It works for me.

 

Also worked for me, thx! Was driving me crazy and I never would have found it on my own!

 

Hello,

@reedny @christougher We setup the ARCore + Google Cardboard and was able to track surface and load the terrain on detection.

But we are facing a issue, that terrain keeps moving with our Head Movement. It don't stick to the surface in VR Mode.

Have anyone of you faced such issues.

 

@param_appm I think https://forum.unity.com/threads/google-arcore-and-vr-combined.491631/#post-3398033 is what you're looking for.

 

Regarding the rotations,
Instead of waiting for the values of the ARCore Y rotation and the VR Camera Y rotation, to differ by more than few degrees, I'm setting the Parent GO rotation of the VR Camera, to be the ARCamera rotation multiply by the Inverse of the head rotation.
Something like that:
transform.rotation = arCamera.rotation * Quaternion.Inverse(InputTracking.GetLocalRotation(XRNode.Head));

 

@De-Panther, that looks good, I'll try that. Do you also rotate the other VR GOs around the Parent GO of the VR camera so they keep the same position relative to the player? My VR world has a room that the user is in plus 3D paint where they have painted, so all of that has to rotate too to keep the AR rotation and the VR rotation consistent.

 

I think that I lost you... (Too many VR GO )

I have a GO with the ARCamera(it gets position and rotation)
I have a GO that follows the position of the ARCamera and get the rotation as I wrote. This GO is also the parent of my MainCamera/VRCamera.

The other GOs positions depends on the game...

 

ARCore and VR seem to work together by now but I discovered that the ARCore tracking frame rate drops by half when booth ARCore and VR are enabled compared to running them separately on the S7. I am currently working on fixing this issue and will publish my results here: https://github.com/ChristophGeske/ARCoreInsideOutTrackingGearVr
as soon as my fix is working. I haven't tested the S8 or S9 maybe they run smooth with 60 FPS even when VR and AR are enabled together but the S7 does not. Has anyone discovered this issue before and knows of an easy fix or if this issue is also present on other ARCore supported devices?

Update: My current understanding is that as soon as you enable VR support the performance drops significantly. The ARCore team knows about this bug but they do not fix it maybe they are not interested in investing time since VR is not what ARCore is meant to be used for. My current approach is building the VR camera myself so I do not have to enable VR support.
The framerate of the ARCore is always 30fps not 60fps and can drop further if the performance load is to much. Interpolation can be used to bring the framerate to 60fps.

 

Sounds like this combination could be very taxing on the device - not even considering when thermal throttling kicks in after running for a while as phones usually get warm enough when simply running a cardboard app when I was working on this a while ago

 

I am in desperate need of some help. I am completely new to unity and am trying to make an portal using ARCore and then combine it with Cardboard. I am using ARCores tracked plane to place the portal in the space and as you can see in the videos it stays in place on its own, but when I add the VR support under player settings it starts to distort the position of the portal.

I tried putting in the transform.rotation = arCamera.rotation * Quaternion.Inverse(InputTracking.GetLocalRotation(XRNode.Head)); you recommended @De-Panther but am not sure if I am doing it correctly.
I'm assuming that the GO that needs the edit you wrote would be the ARController?? I have attached screenshots of the hierarchy and ARController code (without your edit)