Resolved ARKit facemesh not working when the camera background is removed

I'm trying to create something akin to the iOS animated Memoji's and for that I want to have the face information but I don't want to render the camera feed. And it seems that whatever I do, the face tracking only works if the ARCameraBackground script is attached to the camera (and rendering).

Without the camera background the face mesh only appears when the app regains focus but stays static after that. I tried the FaceMesh example from the 5.0 branch in the arfoundation-samples repository using Unity 2021.3.13f ARFoundation 5.0.2 in combination with URP 121.7, and tried:

• Removing the ARCameraBackground component
• Disabling the ARCameraBackground component
• Set the clear flags for the camera to skybox instead of solid color
• Use a custom camera background material in the ARCameraBackground that just returns a solid color
• Remove the ARBackgroundRenderer feature from the render pipeline settings
• Disable the ARBackgroundRenderer feature from the render pipeline settings

But the behavior stays the same: face tracking only works reliable with the camera background rendering..

Is this a bug? Or is this intended behaviour?

 

Yes, all those systems are enabled and running, I have a

ARFaceMeshVisualizer

(from the ARFoundation package) component on the face object.

Interestingly if I enable script debugging the build crashes on startup (this happens to me regardless of the camera background) with a threading error, so I can't break in the faces updated method:



With logging though I see a couple (between 12 and 6) frames of updates coming in before the face mesh freezes. Another interesting side effect I noticed just now is that the green camera use indicator turns off as well when the face freezes. No logged errors or warnings sadly.

All this is with the camera background component disabled in the inspector, running on an iPhone Xr and 12.

I'm now working around the issue by using a customised ARKitBackground.shader with this as the fragment function as a custom background material:

Code (CSharp):

1.  
2.             half4 _tint;
3.            
4.             fragment_output frag (v2f i)
5.             {
6.                 fragment_output o;
7.                 o.color = _tint;
8.                 o.depth = 0.0f;
9.                 return o;
10.             }
11.  

 

Allright, updating both Unity and ARKit / Foundation has fixed the issue. Thanks for checking