Freeze/Hangs with Arkit, sometimes full freeze with crash reports

Hi guys ,

We are currently running an iOS app with arkit 2.0 in a museum environment on 24 iPad Pros (the new model with depth cam).
The app involves face tracking, where the user can wear different virtual hats, collect items and play mini games. Everything works quite ok, but now that the app is live, running all day and being used by hundreds of visitors, we are seeing some weird issues daily.

Every now and then, we saw the video feed, and the entire app, freeze/ hang for a couple of seconds.
This would last for 4-10 secs, and then the app would continue to run as normal.
It seemed to happen totally random, and thus not easily reproducible at will.
At times like these, the xCode console would print errors like: execution of the command buffer failed, with IOAF code 5.
After searching the internet for possible solutions, these kind of errors seem very hard to debug. We tried things like disabling environment light, turning off anti-aliasing, only using standard shaders etc. Nothing really worked.
What we recently did was change the frame rate of arkit from 60fps to 30fps.This seemed to shorten the freezes significantly to about 2 secs. But still they occur.
Since the arkit video shader uses a command buffer we expect somethings wrong there, but we don't really know what.

We don't know if this is all related, but besides this, sometimes we discover the app has frozen entirely.
It generated crash logs then. See the text below.
There are two different type of crashlogs, but they all seem to end with shader/render related stuff and a semaphore wait-trap.
The difference is how they begin. One case seems to stem from our own CrystalManager (which loads and instantiates some prefabs from resources), the other case seem to stem from the camera that does a custom render operation of some sort?

Some questions we have:
- Does anyone have an idea where these freezes are coming from?
- Is there a difference in origin of the short freeze and the full freezes?
- Any measures we could take to avoid them?
- Is there a substantial difference between the crashlogs involving the crystal manager and the resources loading, and the ones with the camera render? Or do they all point to the same problem in the end?

We are running on Unity 2018.3.12f1
Built with cloud build using xCode 10.1, for iOS 12.0, running on iPad Pro's (with depth cam) on iOS 12.2

 

Some additional info I found. Not sure if this is related.

https://bitbucket.org/Unity-Technologies/unity-arkit-plugin/issues/4/facetracking-app-freeze
This used to be an issue with arkit causing freezes, but was thought to be solved.

https://issuetracker.unity3d.com/is...shader-initialization-freezes-the-application
Also an older issue from 2017, but again maybe this could be related to what we have, as we are using database polling in our app. Maybe the www calls are interfering with shader initialisation?
On that note, is there any way we can check if shader initialisation has completed, or is there a safe timewindow when we can start the polling?

 

Can you solve this problem?