Quantifying Server Client Sync with Selected RigidBodies in Server’s Physics Step

In building a relatively fast paced multi player game, I came across all of what I think are common and great references on lag compensation, like this, this, this, and this. I often saw phrases like “but try different techniques to see which is best“. What I could not find were suggestions on how to quantify which one worked best for any particular game. In this post I will attempt to communicate such a reporting system, and how I’m using it to try and optimize the game for both multi player and single player.

Quick background, the game is an authoritative server/client model, Unity 2019.2.17f, initially targeting Steam via the Facepunch.Steamworks API. Network messaging is done utilizing this bit compression library’s safe version. (I’d love to use the unsafe for speed, but there is a reported bug). Auto simulation is disabled and I use Physics.Simulate and two physics scenes, one for client and one for server (in single player mode).

Multi-player and single-player modes use the same code, and I hijack the single player mode to perform this optimization testing using artificial lag which will go away in production. The client and server each have a reference to each other in the scene, so at message send time they can just call the other’s OnP2PData method directly.

Code (CSharp):

1.  
2. // send the message
3. if (!m_bIsSinglePlayer)
4. {
5.        m_fpClient.Networking.SendP2PPacket(m_steamIDGameServer, bytes, bytes.Length);
6. }
7. else
8. {
9.     if (ARTIFICIAL_LAG_SEC > 0) {
10.        m_uiListSPOutgoingMsgIdxs.Add(m_players[m_iPlayerIndex].m_stMyClient2ServerPlayerUpdate.m_uiMessageIdx);
11.         IEnumerator aCoRo = SendP2PSinglePlayerClient2ServUpdateWithLag(0, bytes, m_players[m_iPlayerIndex].m_stMyClient2ServerPlayerUpdate.m_uiMessageIdx);
12.         StartCoroutine(aCoRo);
13.     } else
14.     {
15.         m_SinglePlayerServer.OnP2PData(0, bytes, bytes.Length, 0);
16.     }
17. } // if ! single player
18.  

To avoid a race condition with two enum’s finishing at the same time (which I found from testing), I also use the list of outgoing message indicies and only send when times up and our message is next, like this:

Code (CSharp):

1. // For characterization testing only, hijack single player with artificial lag
2. IEnumerator SendP2PSinglePlayerClient2ServUpdateWithLag(ulong _asteamid, byte[] buffer, uint _msgidx)
3. {
4.     float StartTime = Time.time;
5.     while (Time.time < StartTime + ARTIFICIAL_LAG_SEC || m_uiListSPOutgoingMsgIdxs[0] != _msgidx)
6.         yield return null;
7.  
8.     m_SinglePlayerServer.OnP2PData(_asteamid, buffer, buffer.Length, 0);
9.     m_uiListSPOutgoingMsgIdxs.RemoveAt(0);
10.     yield return null;
11. }
12.  

So now I can run in single player mode, using all of the messaging, client side prediction, and server side reconciliation for testing without having to host a server and client separately each time and then connect. I’m sure there will be differences, especially when I go to actually host on the steam cloud, but I figure it’s the best way to test things out in full multiplayer mode as one person with minimal effort.

The game for single player also has bots, which I hijack to simulate the load on the server of other players, which was kind of where this really useful reference ended for me, with this bit:

“I'll also point out that this example only has one client which simplifies things, if you had multiple clients then you'd need to either a) make sure you only had one player cube enabled on the server when calling Physics.Simulate() or b) if the server has received input for multiple cubes, simulate them all together.”

I went with option B here, but there is a big gotcha. From my perspective, you should only simulate clients who have input messages pending when the server does its physics step. Additionally, all inputs should be applied BEFORE a single server physics step is run. I’ll explain why I think so, but first a quick aside on how I collect the data to base that decision on.

This reference caught my eye with its graph of server and client positions to report on how their technique worked. My way to perform such reporting basically consists of rigging the game code up with streamwriter Write & WriteLn statements, triggered with static Boolean switches, easily referenced from any class so I can set the reporting flag and recompile.

As things can vary a bit, I wanted to have a standardized and repeatable test to collect data on prediction and lag performance. The test consists of the player moving from a defined transform to a collider. The input message is hijacked and the “forward” flag is set every frame, so he keeps going until he hits. The scene looks like this:



First I wanted to collect and sum up prediction and lag error for one forward walk from start to finish and produce a single number (scalar), then run it 10 times and see the spread.

To make life easier, I define a characterization test class and list of tests, the run through them automatically, teleporting the player from the end back to the starting transform. For some tests I need to stop and re-start the game in the editor, as creating bots has to happen when the server is first launched (right now).

Code (CSharp):

1. public class CharacterizationTest
2.     {
3.         public string m_sName;
4.         public bool m_bPauseAtEnd;
5.  
6.         public CharacterizationTest(string _name, bool _pause)
7.         {
8.             m_sName = _name;
9.             m_bPauseAtEnd = _pause;
10.         }
11.     }
12.  

To start I wanted to compare error between just one player and 20, the max in the game. I defined two tests:

Code (CSharp):

1. public static CharacterizationTest[] CHARACTERIZATION_TESTS = new CharacterizationTest[] {
2.         new CharacterizationTest("0 bots", true),
3.         new CharacterizationTest("19 bots", false)
4.     };
5.  

And a static method to set various Boolean flags for testing optimization, here I compare 1 player to 20 (me + 19 bots):

Code (CSharp):

1. // set our various flags to a given test condition
2. public static void SET_TEST_CONDITIONS(int _TestNo)
3. {
4.     switch (_TestNo)
5.     {
6.         case 0:
7.             ARTIFICIAL_LAG_SEC = 0.1f;
8.             SERVER_NUM_BOTS = 0;
9.             Characterization.TEST_FORWARD_RUN = false;
10.             Characterization.TEST_FORWARD = true;
11.             Characterization.TEST_ROTATE = false;
12.             break;
13.         case 1:
14.             SET_TEST_CONDITIONS(0);
15.             SERVER_NUM_BOTS = 19;
16.             break;
17.     }
18. }
19.  

In order to capture prediction error, I hijack the error computed between server and client positions for the same message index when client receives a server update, like this:

Code (CSharp):

1. // For characterizing testing with no client side corrections, we gotta grab the server update position and compare to current to get pos error
2. Server2ClientPlayerUpdate_t serverPlayerUpdateMsg = new Server2ClientPlayerUpdate_t();
3.  
4. // extract it
5. serverPlayerUpdateMsg.FromByteArray(m_Server2ClientUpdate.m_baServer2ClientUpdates, m_Server2ClientUpdate.GetPlayerUpdateByteArrayOffset(m_iPlayerIndex));
6.  
7. // latest server state
8. Vector3 lastProcServerPos = serverPlayerUpdateMsg.m_v3Position;
9. Vector2 lastProcServerRot = serverPlayerUpdateMsg.m_v2EulerAngles;
10.  
11. // client prediction input and state history at the same message index as the server last processed
12. Vector3 lastProcClientPos = m_InputHistoryAndState[idx].m_Server2ClientPlayerMsg.m_v3Position;
13. Vector2 lastProcClientRot = m_InputHistoryAndState[idx].m_Server2ClientPlayerMsg.m_v2EulerAngles;
14.  
15. // errors - difference between server and client local state - use later for Client Smoothing
16. m_v3PositionError = lastProcServerPos - lastProcClientPos;
17. m_v2RotationError = lastProcServerRot - lastProcClientRot;
18.  
19. m_fPositionError = m_v3PositionError.magnitude;
20. m_fTotIntegratedPosError += m_fPositionError;
21.  

Then at the end of each test, I dump fully normalized test settings and results, like this:

Code (CSharp):

1.  
2. public static void DUMP_SCALAR_REPORT()
3. {
4.    if (DUMP_SCALAR_REPORT)
5.    {
6.        // Scalar characterization test output
7.        StreamWriter sw = new StreamWriter("Characterization.txt", true);
8.  
9.        // write the settings
10.        DumpTestSettings("", sw, false);
11.  
12.        // write the output values - correction count and tot pos error
13.        sw.WriteLine(m_iCorrectionCount + // future threshold based correction
14.            "," + m_fTotIntegratedPosError + // total integrated prediction position error
15.            "," + m_fTotIntegratedPosError  /
16.                CHARACTERIZE_TEST_CLIENT_SERVUPDATE_COUNT +  // prediction position error per server update
17.            "," + m_iTestNoInputStepCount + // # of server steps without player input
18.            "," + m_fTotIntegratedRotError + // total integrated prediction rotation error
19.            "," + m_fTotIntegratedRotError /
20.                CHARACTERIZE_TEST_CLIENT_SERVUPDATE_COUNT + // prediction rotation error per server update
21.            "," + m_fTotIntegratedLagPosError + // total integrated lag error
22.            "," + m_fTotIntegratedLagPosError /
23.                CHARACTERIZE_TEST_CLIENT_SERVUPDATE_COUNT // lag error per server update
24.            );
25.  
26.        // close the file
27.        sw.Close();
28.  
29.    } // dump scalar report
30. }
31.  
32. public static void DumpTestSettings(string _prefix, StreamWriter _sw, bool _includeHeaders)
33. {
34.    // write our conditions as columns
35.    _sw.Write(_prefix +
36.        CHARACTERIZATION_TESTS[CURRENT_CONDITION_TEST].m_sName + "," +
37.        AVERAGE_FPS_ACHIEVED + "," +
38.        CHARACTERIZE_TEST_CLIENT_SERVUPDATE_COUNT + "," +
39.        SERVER_NUM_BOTS + "," +
40.        ARTIFICIAL_LAG_SEC + "," +
41.    );
42. }
43.  

When each test is run 10 times it produces a csv file that looks like this:



To plot the results I used python’s matplotlib and colored markers by test, here is the file reading and plotting code, formatted as c#

Code (CSharp):

1. import pandas as pd;
2. import matplotlib.pyplot as plt;
3.  
4. legendSize = 6
5.  
6. # create subplots
7. fig, (ax1,ax2) = plt.subplots(1,2)
8.  
9. #===========================================
10. # read the scalar files
11. sclr = pd.read_csv('Characterization_players1vs20.csv');
12. sclrGrps = sclr.groupby('TestNum')
13.  
14. # plot the results
15. for name, group in sclrGrps:
16.     ax1.plot(group.PosErrPerServUpdate, marker='.', linestyle='', ms=7, label='Pos Err ' + name)
17.  
18. # plot the # of server updates on a 2nd y axis
19. ax1_y2 = ax1.twinx()
20. ax1_y2.plot(sclr.ServUpdateCount, marker='', linestyle='dashed', label='Server Update Count')
21.  
22. # titles, labels, grid lines, legend
23. ax1.set_title('Prediction Pos Err / Server Update')
24. ax1.set_xlabel('Test Replicate')
25. ax1.set_ylabel('Prediction Pos Err / Server Update')
26. ax1.grid()
27. ax1.legend(prop={'size': legendSize})
28.  
29. #show the graphs
30. fig.show()
31.  

And here is the resulting graph:


This clearly shows that with just one player, the error is very low and consistent and driven primarily by the lack of full precision being sent in messages. (As a separate check, I ran it again but sending full precision and the error goes to zero). But when 19 more players are added, the error shoots up, the server is more bogged down as shown by the dotted line which is the number of server updates experienced in each test. Even though updates go down by half, the error is still way higher.

I wanted to see more detail on these two tests, so I rigged up additional streamwriter statements to have the client and server write the current message index, player position, and errors per message (client tick) to a csv file. I won’t go into code details, but the logic is similar to the scalar file shown above, except it’s a timelapse. Here is the graph of the player’s z-position on the client vs the server for both 1 and 20 players:


The server outpacing the client can be seen clearly by comparing it’s slope above to that of the 1 player case. The goal is to have the mechanics of the game as independent as possible to the number of players. i.e., the single player slope is “ground truth”. My reasoning for why the server is outpacing the client is because the server loop performing updates runs a physics step *inside* the client input loop, which will result in more physics steps on the server side than the client with additional players.

Here is code that is called once per FixedUpdate on the server to process collected client input messages, showing the physics step inside the input loop:

Code (CSharp):

1. public void ProcessPlayerUpdates()
2. {
3.     // if we have players and input messages, process them
4.     if (m_iPlayerCount > 0 && m_ClientUpdateMessages.Count > 0)
5.     {
6.  
7.         // loop through our pending messages and update the players based on each
8.         int t_currMessageLen = m_ClientUpdateMessages.Count;
9.  
10.         // ------------------------------
11.         // Player input message loop
12.         for (int idx = 0; idx < t_currMessageLen; idx++)
13.         {
14.             // Retrieve the client player to server update embedded in the client to server update
15.             SSClient2ServerPlayerUpdate_t clientUpdateMsg = new Client2ServerPlayerUpdate_t();
16.             clientUpdateMsg.FromByteArray(m_ClientUpdateMessages[idx].m_bClientUpdateMsg);
17.  
18.             // make sure it's a valid player slot
19.             if (m_PlayerSlotsFilled[clientUpdateMsg.m_iPlayerArrayIdx])
20.             {
21.                 // call the player update from message
22.                 m_Players[clientUpdateMsg.m_iPlayerArrayIdx].UpdatePlayerFromClient2ServerUpdate(clientUpdateMsg, m_eGameState);
23.  
24.                 // Update our most recent message index processed
25.                 if (clientUpdateMsg.m_uiMessageIdx > m_accdClientConnectionData[clientUpdateMsg.m_iPlayerArrayIdx].m_uiLastMessageIdxProcessed)
26.                 {
27.                     m_accdClientConnectionData[clientUpdateMsg.m_iPlayerArrayIdx].m_uiLastMessageIdxProcessed = clientUpdateMsg.m_uiMessageIdx;
28.                 }
29.  
30.                 // step the physics scene
31.                 m_serverPhysicsScene.Simulate(Time.fixedDeltaTime);
32.  
33.             } // valid player slot filled
34.  
35.         } // message loop
36.  
37.         // now go through and remove messages that are processed
38.         for (int idx = t_currMessageLen - 1; idx >= 0; idx--)
39.         {
40.             m_ClientUpdateMessages.RemoveAt(idx);
41.         }
42.  
43.         // update and send our all player update to all clients
44.         SendAllClientsUpdate();
45.  
46.     } // if player count > 0 && msg_count > 0
47.  
48. } // Process Player Update
49.  

In researching others who have encountered this kind of thing, I came across this great post, specifically this snippet of a reply from the author to a comment:

You cannot do clientside prediction with rigidbodies, since you can't manually step a rigidbody.

I think this post was written before physics scenes were available in unity, but that doesn’t completely solve the issue. You can step a rigidbody in theory, but it might require a separate physics scene for each one, and from the references post above that can bog things down, plus if your game has interactions and such with other players or game static things, that adds additional overhead.

Instead of manually stepping an individual rigidbody, the idea here is to *prevent* an individual rigidbody from being updated during a physics.simulate call. “Locking” a player who has no input during the server step would in theory bring the 20 player position curve above towards the single player line, assuming that the physics step is now outside the input message loop.

The player script has a new method to implement this locking and unlocking:

Code (CSharp):

1. // player class
2. // ...
3. public Vector3 m_v3PreLockPosition;
4. public Vector3 m_v3PreLockVelocity;
5. public bool m_bIsLocked;
6.  
7. // Lock
8. // enable or disable the player from being affected by physics updates
9. public void Lock(bool _lockMe)
10. {
11.     // make sure we aren't locking to the same thing again
12.     if ((_lockMe && m_bIsLocked) || (!_lockMe && !m_bIsLocked))
13.         return;
14.  
15.     // store or restore player velocity & position
16.     switch (_lockMe)
17.     {
18.         case true:
19.             // Cache rigidbody velocity and position
20.             m_v3PreLockVelocity = m_rb.velocity;
21.             m_v3PreLockPosition = transform.position;
22.          
23.             //mark locked
24.             m_bIsLocked = true;
25.             break;
26.  
27.         case false:
28.             // Restore rigidbody velocity & position from cache
29.             m_rb.position = m_v3PreLockPosition;
30.             m_rb.velocity = m_v3PreLockVelocity;
31.  
32.             // mark unlocked
33.             m_bIsLocked = false;
34.             break;
35.     }
36. }
37.  

Two new test cases are added, with a new Boolean flag “SERVER_STEP_INSIDE_INPUTS_LOOP”:

Code (CSharp):

1. public static CharacterizationTest[] CHARACTERIZATION_TESTS = new CharacterizationTest[] {
2.         new CharacterizationTest("1 player SSI T", true),
3.         new CharacterizationTest("20 players SSI T", true),
4.         new CharacterizationTest("1 player SSI F", true),
5.         new CharacterizationTest("20 players SSI F", true)    
6.     };
7.  
8. // set our various flags to a given test condition
9. public static void SET_TEST_CONDITIONS(int _TestNo)
10. {
11.     switch (_TestNo)
12.     {
13.         case 0:
14.             ARTIFICIAL_LAG_SEC = 0.1f;
15.             SERVER_NUM_BOTS = 0;
16.             SERVER_STEP_INSIDE_INPUTS_LOOP = true;
17.             Characterization.TEST_FORWARD_RUN = false;
18.             Characterization.TEST_FORWARD = true;
19.             Characterization.TEST_ROTATE = false;
20.             break;
21.         case 1:
22.             SET_TEST_CONDITIONS(0);
23.             SERVER_NUM_BOTS = 19;
24.             break;
25.         case 2:
26.             SET_TEST_CONDITIONS(0);
27.             SERVER_STEP_INSIDE_INPUTS_LOOP = false;
28.             break;
29.         case 3:
30.             SET_TEST_CONDITIONS(0);
31.             SERVER_NUM_BOTS = 19;
32.             SERVER_STEP_INSIDE_INPUTS_LOOP = false;
33.             break;
34.     }
35. }
36.  

The server method to process all pending player input messages gets updated to incorporate the new test switch and locking strategy, like this:

Code (CSharp):

1. public void ProcessPlayerUpdates()
2. {
3.     // if we have players and input messages, process them
4.     if (m_iPlayerCount > 0 && m_ClientUpdateMessages.Count > 0)
5.     {
6.  
7.         // loop through our pending messages and update the players based on each
8.         int t_currMessageLen = m_ClientUpdateMessages.Count;
9.  
10.         // ------------------------------
11.         // Player input message loop
12.         for (int idx = 0; idx < t_currMessageLen; idx++)
13.         {
14.             // Retrieve the client player to server update embedded in the client to server update
15.             SSClient2ServerPlayerUpdate_t clientUpdateMsg = new Client2ServerPlayerUpdate_t();
16.             clientUpdateMsg.FromByteArray(m_ClientUpdateMessages[idx].m_bClientUpdateMsg);
17.  
18.             // make sure it's a valid player slot
19.             if (m_PlayerSlotsFilled[clientUpdateMsg.m_iPlayerArrayIdx])
20.             {
21.                 // unlock the player if !SERVER_STEP_INSIDE_INPUTS_LOOP
22.                 if (!SERVER_STEP_INSIDE_INPUTS_LOOP)
23.                 {
24.                     m_Players[clientUpdateMsg.m_iPlayerArrayIdx].Lock(false);
25.                 }
26.              
27.                 // call the player update from message
28.                 m_Players[clientUpdateMsg.m_iPlayerArrayIdx].UpdatePlayerFromClient2ServerUpdate(clientUpdateMsg, m_eGameState);
29.  
30.                 // Update our most recent message index processed
31.                 if (clientUpdateMsg.m_uiMessageIdx > m_accdClientConnectionData[clientUpdateMsg.m_iPlayerArrayIdx].m_uiLastMessageIdxProcessed)
32.                 {
33.                     m_accdClientConnectionData[clientUpdateMsg.m_iPlayerArrayIdx].m_uiLastMessageIdxProcessed = clientUpdateMsg.m_uiMessageIdx;
34.                 }
35.  
36.                 // step the physics scene if SERVER_STEP_INSIDE_INPUTS_LOOP
37.                 if (SERVER_STEP_INSIDE_INPUTS_LOOP) {
38.                     m_serverPhysicsScene.Simulate(Time.fixedDeltaTime);
39.                 }
40.                  
41.             } // valid player slot filled
42.  
43.         } // message loop
44.  
45.         // Physics step outside if !SERVER_STEP_INSIDE_INPUTS_LOOP
46.         if (!SERVER_STEP_INSIDE_INPUTS_LOOP) {
47.          
48.             // step the physics scene outside the input loop
49.             m_serverPhysicsScene.Simulate(Time.fixedDeltaTime);
50.          
51.             // re-lock all players who were unlocked, optimize this
52.             for (int i = 0; i < m_Players.Length; i++)
53.                 if (!m_Players[i].IsLocked())
54.                     m_Players[i].Lock(true);
55.         }    
56.      
57.         // now go through and remove messages that are processed
58.         for (int idx = t_currMessageLen - 1; idx >= 0; idx--)
59.         {
60.             m_ClientUpdateMessages.RemoveAt(idx);
61.         }
62.  
63.         // update and send our all player update to all clients
64.         SendAllClientsUpdate();
65.  
66.     } // if player count > 0 && msg_count > 0
67.  
68. } // Process Player Update
69.  

After running the new test cases 10 times each to gather errors, here is the result:



And here is the graph of server vs client position:



As the data shows, the locking of players combined with the server step happening *outside* the input loop runs closer to the “ground truth” of one player with the server step *inside* the input loop. Combined with a lower prediction error this is the setup moving forward for multi player for my game.

Up next is characterizing rotation combined with moving forward, to simulate actual gameplay a bit more.