Question Unity Camera Jittering When Using Coroutines Instead of Executing Every Frame

I am detecting faces with BlazeFace using the repository below:

https://github.com/keijiro/BlazeFaceBarracuda

It actually works quite well and it makes the inference in the LateUpdate method. I have a third person controller (using the new unity starter asset) and everything is smooth when I am playing with the character and also detecting faces.

Code (CSharp):

1. void LateUpdate()
2. {
3.     if (_webcam != null) RunDetector(_webcam.Texture);
4. }

However, I don't need to detect faces in each frame, lets say every 0.5 seconds is enough. So I used a
Coroutine and executed it in the Start method.

Code (CSharp):

1. IEnumerator Example()
2. {
3.     while (true)
4.     {
5.         yield return new WaitForSeconds(0.5f);
6.         if (_webcam != null) RunDetector(_webcam.Texture);
7.     }
8. }

As expected my fps increased dramatically but I have started to have jittering with the Unity camera when I am moving my character around. I have also tried doing it with a timer but ended up with the same jittery result.

Code (CSharp):

1. if (Time.time > nextActionTime)
2. {
3.     nextActionTime += period;
4.     // Webcam test: Run the detector every frame.
5.     if (_webcam != null) RunDetector(_webcam.Texture);
6. }

What am I missing here? When I detect faces in every frame, Unity camera is smooth however I am wasting my performance since I don't need to detect faces in every frame. I try to detect faces in every x seconds, my fps increases a lot but I end up with jittery Unity camera.

 

Hi @Afavar, it's hard to say what could be causing your jitteriness. I'd say it would be best to use remote profiling to see what is happening when the spike occurs. Could you try that and report back?

 

Also, a colleague of mine mentioned that on it might help to set the following:
_webcam.requestedFPS = 30;

 

Hi @amirebrahimi_unity , thank you for your suggestions. I have tried setting the requestedFPS to 15 and used
WebCamTexture.didUpdateThisFrame method to process the image. The result was def better. However, I have noticed that I experienced this behaviour while I am in the play mode. I tried to take a build and strangely everything was smooth.

 

Odd that it stutters in play mode, but glad to hear it is working correctly on the device.

 

Yeah I am glad it works quite well. The bigest issue I had after that was to realize texture colors are messed up when using the Linear color space. (which urp and hdrp uses by default) I haven't realized that until I had a look at the output texture of the tensor. Hopefully I have found a Unity blog post regarding on style transfer and thank god they also used linear color space for that demo. It took some time but I managed to use the shader there to fix the color issue. I hope this would be fixed with the later Barracuda releases by default.

 

Isn't this the standard Unity issue that asynchronous updates are processed really poorly by the editor? Coroutines are executed asynchronously in a separate thread and the unity editor just doesn't handle this well - it's the reason why loading screen progress bars only work properly in build.