How to diagnose desync issues?

I'm developing a multiplayer turn-based strategy game where numbers of individual AI-controlled units are spawned for each player, and it's suffering from some desynchronization issues.

Most unit-controlling and turn-controlling behavior is in a FixedUpdate() block. What are some common causes of desynchronization in such a situation?

What are non-synchronous methods? I assume I shouldn't be using yield WaitForSeconds, but what about messing with the timescale?

Code (csharp):

1.  
2. function FixedUpdate()
3. {
4. // turnStarted is set to true or false by RPC
5.     if (turnStarted)
6.         time.timeScale = 1;
7.    
8.     if (turnEnded)
9.         time.timeScale = 0;
10. }
11.  

What are other things that might be causing my desync issues? To the best of my knowledge, everything should be deterministic... but apparently it's not.

 

One thing that comes to my mind spontaneously are the interpolation and collision detection settings for rigidbodies:
http://docs.unity3d.com/Documentation/Components/class-Rigidbody.html
As far as I understand it, these settings can influence synchronicity for fast-moving and/or colliding rigidbodies. But I may be wrong.
I am rather a newbie concerning network sync details, but I am facing similar issues in my mutliplayer network game, and I am still searching for possible clues. So I'm looking forward to following this thread!

 

Apart from what I posted before, I assume that everything strongly depends on how your networking is implemented, So it's hard to give a generic answer.

Are you using network views and the OnNetworkViewSerialize function?
Are you using custom coded RPC calls?
Or rather a mixture of both.

The latter is what I've been doing so far (due to a huge lack of know-how and experience). It seems that it may be a part of my problem. I think it's not the best idea to mix up both concepts without thinking, as you can quickly run into horribly unreadable code.

 

Depending on the platform you are using, pretty much everything which is not built to be deterministic can de-sync. Everything from random number generators, floating point imprecision, non-deterministic physics, etc. can cause de-syncs.

The general rule of using Unity when it comes to determinism is this: If you have not built it yourself and specifically engineered it to be deterministic, it is not.

 

For random number generation I'm currently using the "world" game object to generate random numbers. It's seeded via RPC at the start of the game, so both clients are generating numbers with the same seed.

I've noticed they seem to get the same number 95% of the time, but not always. I'm wondering if I could improve it by pre-generating a list of 1000 random 3-digit numbers, and then grabbing the next random number from the pre-determined list (using modulus, or dividing by 1000f, or whatever.)

 

That sounds really weird, that you are getting different numbers despite the seed being identical. Are you using System.Random or UnityEngine.Random?

 

I'm thinking I may have been mistaking the symptom with the cause. (A previous desync causing a slight difference in the situation resulting in different random intervals.)

After two straight days of doing nothing but looking into the cause, it seems like it has something to do with the timescale... timescale is set to 1.0f after both players have submitted their moves for the turn, but for some reason sometimes greater than 0.02 seconds (the fixed timestep interval) will pass between the two submissions (according to a Debug.Log statement), even though the timescale is 0 until that second submission. It's driving me crazy, because it seems like it shouldn't be possible.

 

There's still a few minor issues I'm trying to track down, but all of the major ones were caused by minor changes not being included in a FixedUpdate block, and using Yield WaitForSeconds(). (Hint: Don't do it.)

I wrote a blog post at http://rabbitgamesca.blogspot.com/2013/06/unity3d-and-multiplayer-synchronization.html with a longer explanation in case that helps anyone else who's having issues.

 

Thanks for posting/blogging your experience!
As a first impression: From what I read between your lines, I understand that in your game you have to rely on syncing everything exactly to the nearest millisecond. I am wondering whether in your case an authoritative server approach would be more effective than having to fumble with lots of FixedUpdates. Just a thought... it all depends on what your game exactly does. I guess that authoritative server coding is hard stuff and can slow down things, but I've also read somewhere that coding everything into FixedUpdate can also cause a performance loss (because FixedUpdate is called more than once per frame). Correct me if the latter should be wrong.

[EDIT] In my game, I've gotten used to coding only rigidbody related operations into FixedUpdate(), bone-specific movement into LateUpdate(), and everything else into regular Update() functions. Seems to work fine so far, but my game doesn't need 100% exact syncing in everything.