Resolved Does AR Foundation support ARKit 3 Simultaneous Front and Back Camera?

Hi I wonder does the latest AR Foundation support ARKit 3 Simultaneous Front and Back Camera feature?

I checked RearCameraWithFrontCameraFaceMesh scene from AR Foundation samples, but it doesn't demonstrate how to use front camera to enable face tracking and at the same time using device full orientation and position information.

Does Unity plan to create a demo like that?

Also I am curious whether user can get human depth and stencil texture from front camera, just like the human occlusion demo, but depth is from front camera instead of back camera. I tried to attached both ARHumanBodyManager and ARFaceManager on ARSessionOrigin, and it seems like as long as ARHumanBodyManager is turned on, it's using back camera, not front.

 

up up

 

I"m having the same issue, the RearCameraWithFrontCameraFaceMesh sample isn't doing exactly what I want, and seems to just switch back and forth between front and rear camera. I can't tell what it's tracking. What I want to know how to do is get the blend shape data from the face (in the form of variables) while doing rear camera AR with plane tracking etc...

 

built a little scene to test the issue, when you turn on face manager, it automatically switches to the front camera, which is NOT what I want to happen, did anyone at Unity actually test this? ARKit3.0 is supposed to be able to do this.

Stats:
Unity 2019.1.9f1
ARFoundation 2.1.1
ARKit Face Tracking 1.0.1
ARKit XR Plugin 2.1.1
Xcode 11.0 beta 4
iOS 13 beta 4

 

Any update?

 

dear lord someone help us

 

Arkit 3 support iPad 2018 11 inches?

When we use both cameras on in our app, the front camera works fine with blendshape (morphing) but with the back camera, if we enable AR face script on the 3D model, it does not augment. And if we disable AR Face script it augments ok, but blend shape does not work properly.

I used RearCameraWithFrontCameraFaceMesh sample but it's not working.
Does any sample provide for this?

Unity 2019.2
Xcode 11.0 beta 5
ARFoundation 2.2 preview.3
ARKit Face Tracking 1.1.0 preview.4
ARKit XR Plugin 2.2.0 preview.4

 

Hi,
Thanks for the reply.
I have below iPad
https://www.amazon.com/Apple-iPad-11-inch-Wi-Fi-256GB/dp/B07K3BZSNK
I test HumanBodyTracking3D its working fine.

 

Just tested this with all proper packages / versions (as listed above) it's still not working! It still switches to render face cam automatically as soon as ARFaceManager is enabled.

 

This is not a version issue and I'm testing on an iPhone XR. All of the versions I was using before were supposed to support ARKit 3.0 as well. It said as much in the documentation.

This issue is simple: the camera being rendered is switching to the selfie cam when you turn on the face manager, this should not happen with ARKit 3.0 and the dev should be in control of which camera is shown.

The code making this happen is somewhere in the bowels of ARFoundation/ARKIt/ARFaceTracking plugins. An engineer at Unity needs to fix this.

 

Are you checked RearCameraWithFrontCameraFaceMesh sample scene which provides in unity sample?

 

Yes I can see that the sample can show how many faces are being tracked while using the plane manager and showing the rear camera.

However, there is no clear example of how to get blend shape data from a face while using the plane manager and showing the rear camera. As far as I can tell, I need to use the Face Manager to spawn a face prefab to get access to the blend shape data. Is there a way I can get access to blend shape data without the Face Manager?

 

Kindly find the attachment for sample unity project (RearCameraWithFrontCameraFaceMesh1.scene) & Xcode
https://www.dropbox.com/sh/1kx4mdqzmgj9yu9/AACBc08HPqcBq5jkaDMbjb9Ua?dl=0


As per the image attached, if we put off the highlighted options then we are able to augment the 3D object but blend shape doesn’t work. And it’s vice versa.

 

I totally agree with @edwon, Please provide one sample to avoid issues

 

I already found the example you are referencing, but it requires the face manager to be enabled, and a spawned ARFace prefab in the scene, but, turning the face manager on switches the camera to the front facing camera, as I said before! Are you actually testing this stuff out?

You should really have a specific sample in the ARFoundation-Samples git repo that specifically shows blend shapes being appled to a character that is tracked and shown in the rear camera.

P.S. it's been more than 2 weeks since I posted in this thread and it's still unresolved

 

After many hours of debugging, found out that this line is causing the camera to switch to front camera if ARFaceManager is on:

arPlaneManager.subsystem.Stop();

this re-enforces my point that the dev should have explicit control of which camera is shown at any time, debugging this was hell

 

Any update?

 

Same isuue

 

Bumping as well. NEED THIS!

@jimmya @todds_unity @davidmo_unity

Tossing my src here. Im doing a facial analytics so I know expressions people make when certain things are on the screen or navigating ui elements in rear AR experiences.

Im rockin arfoundation3/arkit3/URP. Nothing on front&rear in this proj yet. Hoping all the above gets solved at some point.

https://drive.google.com/open?id=194d49IgNR1IPG1lvVV92P78ewdls0KIQ

 

What issue are you having exactly? I was actually able to get front and back camera working simultaenously by following the example Rear Camera with Front Facing Camera scene in https://github.com/Unity-Technologies/arfoundation-samples

My only issue was that it kept switching to render the front camera (even though it was still tracking both) but it was actually only happening because of me stopping/starting the plane manager for other reasons, which was resetting the AR session and causing the camera to flip.

I was able to fix my specific issue this way, but it's still a major problem that AR foundation doesn't provide an explicit way to switch between the cameras. It should not be automated!

 

@edwon
plane anchor with face blendshape not working in back camera.can you share your working sample?
It's very helpful to me slove above isuue.

 

For me the example scene in the AR-Foundation-Samples project on GitHub worked from the beginning. It just wasn't working in my project for reasons mentioned above. Remember you do an iPhone or iPad that supports ARKit 3.0 and it must have iOS 13 beta on it, and you need to use the latest Xcode beta when building.

 

Great, explicit camera control is much needed, it should not be automatically turning on and off based on which subsystems you're using.

Also a better way to check for feature compatibility would be great. Right now you have to explicit check each device via the Device.iPhoneX || Device.iPhoneXR etc.. it's a major pain. Would be great to have like a Device.iOS.ARKit.3.0Compatible flag or bool to check against.

 

Hello,
Having an easy way to select front or back camera is a great addition, I sure would like to be able to use different features regardless of whether the video source is coming from the front or back camera.

Is there any ETA for these features to be available ?
Thank you,
Guy

 

Any Update on this?

 

We are still working on this.

 

Hi,

As shown here that you can use the front camera (on specific devices) to detect faces as well as the world tracking by simply enabling the isWorldTrackingEnabled bool:
https://developer.apple.com/documentation/arkit/arfacetrackingconfiguration

Does this mean that we can also detect HumanBodies with a TrueDepth front camera?
If yes, how can I achieve this while using Unity ARKit Plugin (part of AR Foundation)? I want to use the device's front camera to detect both the face and the HumanBody (Joints and all that)?

 

Body tracking on ARKit only works with the rear-facing camera.

 

Thank you for your prompt reply.

I have one more question. When I connect my phone with a TV and position it in a way that the phone's back camera can see me, the TV screen shows everything inverted horizontally. Is there an easy way to invert the frame being received from the camera using the ARKIT - ARFoundation Unity, so that I do not see myself inverted on the TV screen?

 

I want to know how to turn off isWorldTrackingEnabled in arfoundation.

 

How do I modify isWorldTrackingEnabled value

 

These words may be redundant, but I want to make sure.The Ipad's front-facing camera with the A12 chip doesn't do the job of motion capture。Yes？

 

hi ! The methods you mentioned are all from apple , I want to learn Unity for ARfoundation.

 

Correct. Only the rear-facing camera supports motion capture.

 

Any future plans for providing human depth and stencil texture from front-facing camera?

 

That is functionality Apple would have to build into their ARKit SDK.

 

Hey everyone,

I have a game in which I'm trying to achieve this behaviour:

- First, with the rear camera and the plane manager, I'm detecting the planes
- Then, the user touches the screen to place an object to the ground (let say a cube)
- Then, he touches a play button and an animation starts to play
- The animation, will frequently swap the cameras and use face detection to add something on the user face (let say a hat) and swap back to show the object again.

The behaviour runs normally but the rear tracking seems off :
It can swap the worlds, it show the hat as desired but the object I placed on the ground has issues staying where it should be.

It starts by levitating, then I can show it on the face 'world' then, it can be everywhere in the world; surely because the phone moves while in face tracking. My guess is that the object is not tracked anymore on this mode.

How can I ask my device to continue using the rear camera (without showing it) to keep the object to the right place ?
I haven't understood how to achieve this reading the code, the samples, the documentations and this thread.
Is it at least possible ?

Best regards,
Sylafrs

 

Hi,

Was wondering if its possible to get light direction and ambient spherical harmonics from the True Depth camera on the front of the iPhone when busy using the back camera with world tracking?

Regards
Ryan

 

@RyanJVR Hi, Ryan did you ever get an answer on this? Know this thread is old but I'm trying to do exactly the same thing and can't find a good example. Many thanks.

 

Here is the example scene. But, unfortunately, I wasn't able to run it on my iPad Pro 2020 and AR Foundation 4.1.1.
https://github.com/Unity-Technologi...g/WorldCameraWithUserFacingFaceTracking.unity

 

I'm at my wits end. I've been through every forum post I can find including this one and get close to an answer but never quite there. I am trying to have both cameras active on a new iPhone 12 Pro where the user facing camera will read the Main Light Direction value and the back facing camera will track planes and anchors for object placement. The value for the lighting direction will then set the light in the scene to match the environment lighting on all the 3D objects. Seems like the obvious thing that everyone wants to do. How the !@#$ do you get Unity to do that? I know it's in ARKit 4. I've looked at all the GitHub samples including the one above but that doesn't solve this. I've looked at possibly using the ConfigurationChooser to do this but don't have any idea how to use it for this particular situation. Can anyone please help and provide a sample scene or script? Thank you.

 

Thanks @davidmo_unity. Ahhh that's a bummer. I thought this was possible. So what is the best way in Unity to switch between those two configurations and also to set the two flags you mentioned (userFaceTrackingEnabled/isWorldTrackingEnabled)? Should we using ConfigurationChooser or setting camera facing direction on ARCameraManger?

 

And also @davidmo_unity wouldn't we be able to hit the user facing camera, grab the light direction, and then switch to the back back camera for tracking and apply the main light direction we found from the user facing camera? Or are the coordinate systems going to be messed up?