Question Dealing with Physics, framerate, in Unit tests

I've been setting up some regression tests for a 2D game using physics. Stuff like "make sure the player ledge grabs if they hit a ledge going at speed" and such.

What I find is that if I just load the test scene and run the tests, the first one I run tend to fail at times, seemingly at random. If I instead make sure that I at least wait a second before the first test runs, my tests pass (unless I've introduced a bug!)


What's probably happening is that during load, there's some outlier cases with regards to the length of FixedUpdate, or things just spawning, or whatever, that's causing some of my tests to fail. Since it's not consistent, and I don't have any randomness in there, it's probably got to do with frame rate variance.

Now, what I'm wondering is how much I should worry. The code I'm testing will never actually run in the first few frames of booting the game, so dealing with this probably fixes some edge-cases that are only actually visible in tests.

At the same time, it hints that our code's doing something wrong with regards to delta times and fixed update and whatnot, which might mean that our game will have bugs at uncommon framerates. But then I want to test uncommon framerates that might happen, not "Unity is a bit laggy as it boots the game in the editor".

Does Application.targetFrameRate work well with tests? Is it a good idea to set it to a bunch of different values, and have the game run?

Ideally, I'd love to be able to override the game loop when running tests, so the game can pretend (with regards to Update/DeltaTime/FixedUpdate) to run at a rock-steady 60fps, or with a reproducible varied framerate, so I can get failures that happen every time rather than when I'm lucky with the framerate.

Any advice/experience?

 

That looks interesting!

I'll give it a go running the tests with that set to a few different levels, see what it achieves.

 

Yup, looks like the results are consistent now. As an added benefit - in our test scenes, the game actually runs at 600+ fps. Setting Time.captureFramerate to 60 means that the game runs at 10x speed, with the same results, which allows the tests to run a lot faster.

Thanks!

 

After having tried this for a while, it doesn't give 100% reproducibility.

I've got a test set up where I essentially test that the player is able to stick to the ground when moving over sloped terrain. The test uses this geometry:



It spawns the player at the large, white square, holds the stick right, and stops when the player reaches the small, white square, asserting that it's always grounded. Here's a simplified version:

Code (csharp):

1. player = spawnPlayer();
2.  
3. inputTester.Set(gamePad.leftStick, Vector2.right);
4.  
5. while(true) {
6.     yield return null;
7.     Assert.IsTrue(player.IsGrounded());
8.  
9.     if (player.position.x > testEndPoint();
10.         Debug.Log("Done on frame count " + Time.frameCount + ", fixed time " + Time.fixedTime + ", time: " + Time.time);
11.         break;
12.     }
13. }
14.  

I've got Time.CaptureFrameRate set to 60.

With my current implementation, the test very occasionally fails. The output from the Debug.Log also varies slightly from run to run:

Code (csharp):

1.  
2. Done on frame count 162, fixed time 2.683333, time: 2.684735
3. Done on frame count 162, fixed time 2.683333, time: 2.685117
4. Done on frame count 162, fixed time 2.675,    time: 2.682912
5. Done on frame count 162, fixed time 2.683333, time: 2.68481
6. Done on frame count 162, fixed time 2.666667, time: 2.672486
7. Done on frame count 162, fixed time 2.658334, time: 2.664814
8. Done on frame count 162, fixed time 2.658334, time: 2.664776
9.  

It's interesting that the frame count always is consistent, but both the readout of Time.fixedTime and Time.time varies. Since my movement is 2D physics based, I assume that it's the variance in how many fixed updates has run that causes the variation in my results.

Is this expected? I set Time.CaptureFrameRate in [OneTimeSetup], but if the time gets to run a varying amount before that, it makes sense that both the value of time and the value of fixedTime would vary due to that.