Determinism (PhysX)

I am starting a new thread about physics determinism (especially when using PhysX and 3D) to discuss and summarize the current state of it. There are lots of old threads about it, the best I found being this where @RedGirafeGames and @MelvMay solve an issue: https://forum.unity.com/threads/can...nd-manual-simulation-of-physics-scene.958540/

Personally, I do have lots of experience on the topic from various platforms and I am painfully aware of how hard it is to achieve determinism, especially if using floats. If any operations are performed in different order or math library is updated - or for a thousand other reasons - the determinism may be lost.

My game (currently in beta, https://store.steampowered.com/app/1478340/Turbo_Sliders_Unlimited/) online MP is based on clients predicting the future based on updates coming from host and known controls already made. This generally works smoothly (with fixed time steps and enhanced determinism set on), at least when prediction times are less than 200 ms, implying that things are quite deterministic. Now that I also implemented game recording and playback, I would like to optimize the amount of full state updates needed. I noticed that warping caused by non-determinism becomes visible with intervals as low as 500 ms, especially after collisions. If I start my game, record a game, exit, and then restart game and play back the recording, it works correctly. But if I then restart the recording without exiting the game, determinism is lost, especially after each collisions. But it seems that it is always lost the same way. If I keep restarting the playback, the warping seems to always look the same.

I understood from previous discussions that this is most likely because the colliders are in different order somewhere inside PhysX, because the exact order of initializing stuff has not been the same. In my case, it means first creating scene, then creating vehicles, then moving them, then destroying them and recreating them in the original positions. I noticed that if I query all the colliders in the scene at this point with physicsScene.OverlapSphere, I get the same colliders but in different order, which probably explains the non-determinism. I am not sure though if this is the only problem there is, though.

So, right now I am wondering whether there are any low-hanging fruits to improve the determinism as much as is possible, still understanding that full determinism is probably never possible. So, some questions to anyone who feels like answering:

1. PhysX is said to be deterministic on one platform if the same operations are performed in the same order (and enhanced determinism is on). But what kind of operations can break this? Can just moving an object around (from A to B and back to A instead of just not moving at all) cause it to break or is it more about the order of initialization? This question is relevant in online MP where clients frequently move objects around based on server updates and prediction ticks. If just simple movement via a different point can break determinism, there is not much hope to achieve full determinism.

2. Is there some way to reset the physics state to a deterministic state that is fully derived from the current physics objects in the scene? Could it work if I just disable all physics objects so that there is nothing there, and then reactivate all objects in a deterministic order? Or is there some hidden state inside PhysX that so that even that might not work? It would be so nice to have some kind of Reset function that would make PhysX's internal state be deterministic based on the objects it currently has but I guess that is too much to ask?

3. My scene currently has hundreds of colliders, most of which are terrain colliders. Do you see that using a separate PhysicsScene could somehow help solve the issues here? The API to do that seems just very limited so I don't know how that could be done. It also seems that some external plugins I am using (at least EasyRoads3D) assume that the default physics scene works and I didn't notice there being any way to create a new physics scene and making that the default. What I could try is create a new scene after the first initialization has been done. It is quite a big change to make so before trying to do it, I would like to get an opinion whether it would in theory help. To clarify, that would mean creating a separate empty physics scene, then movíng every object (including terrain and roads) with a collider there in a deterministic order whenever a new game/playback is started. Restarting game/playback would then mean moving objects back to main scene, destroying the temporary scene, and then creating a new physics scene, and moving objects there again. This seems like lots of hassle (and possibly break something else) and I am not sure if it is worth the effort but is the best idea that could be tried?

 

I tried to test creating a new empty scene with its own physics scene and move all objects from the original scene there when game starts to have a unified order in which PhysX scene gets initialized. However, I soon realized that apparently, RaycastCommand.ScheduleBatch only works with the current active physics scene and there is no way to change the active physics scene. So this seems to be a showstopper for me unless I am missing something.

So is it really so that RaycastCommands can only be used in the default physics scene and there is no way to change the default physics scene? (Asking the same question here: https://forum.unity.com/threads/raycastcommand-on-other-physics-scene.882901/)

 

To prefix this: "determinism" here is defined as getting identical results between different simulations with the same input. "Identical" means that every value in the simulation is identical down to the individual bit. If even one bit changes, it's not deterministic enough for applications requiring determinism. When situations do not require determinism (networking where state is manually replicated over the server), we do still care about consistency between clients, just not to the bit level.

Unity loads objects in a different order when you "cold load" the scene versus "hot load" (can see here). Not sure if this is the cause to the above.

PhysX is guaranteed deterministic on same machine, same builds with Enable Enhanced Determinism checked. It is not deterministic between different CPU vendors, like Intel vs. AMD (likely due to SIMD differences), and definitely not deterministic between platforms (x86 vs ARM). This determinism is broken when using client side prediction, since clients are predicting and rolling back different amounts from one another.

You could in theory make an entirely deterministic game if you stuck to a single CPU vendor, but I'll explain below why this is a bad idea.

No. PhysX maintains internal state that is not accessible by Unity. Manually disabling/enabling objects does clear some of this state, but breaks some features of PhysX, notably stacking (setting the Rigidbody.position/rotation manually also clears the cache).

tl:dr; PhysX has issues with creating consistency between server and clients for networking purposes. These kinds of divergences get worse over time, as you saw, but also can happen over very short timespans when many bodies are colliding. You can try to make things more consistent by force setting the positions of rigidbodies on both client and server, but this will lead to less stable behaviour. PhysX as well maintains internal state that is not accessible by Unity and cannot be modified. However, you can still use it for networking, which is what I am doing for my game with Photon Fusion, which has client side prediction.

Bonus: Unity itself is not deterministic, and there are all sorts of little oddities with Unity that make predicting PhysX annoying. An alternative to this would be using a physics engine built for networking, like the one Photon uses for Quantum or Unity Physics (I did a monoB wrapper prototype for it here).




(On determinism: if you need actual determinism, where things stay in sync to the bit level, it's almost essential to be able to check for desyncs every single tick. You can do this by iterating over every byte that makes up the state of your game and creating a checksum that the server can send to clients for them to compare against. Note this doesn't really apply here, since PhysX is not cross platform deterministic anyways.)

 

Thank you for the long and good reply!

Unfortunately, there is not much to be done anymore then, but the good side is that I don't have to spend time on something that won't work anyway and the current system does work quite well anyway. It is just that recordings will be much bigger than needed and there might be small warping. Changing to a completely new physics engine is out of scope at the moment but I'll remember that option for future projects...

 

@Andefob, Hi, I was your game on Steam. Very nice. I have a question which Network Backend you are using? And are you using dedicated servers or one player as a host?
-

 

Hi! Nice to hear you like the game!

I am using Steam sockets and doing most of the synchronization manually. The model is based on an authoritative server, client backtracking and predicting the future based on past controls whenever an update from server is received. So a game must have a server but any player can host a game (acting as a server and playing at the same time). And yes, anyone can also run a dedicated server, which is basically a headless version of the game just automatically running in host mode and without local players: https://www.turbosliders.com/help/dedicated-servers