Search Unity

Resolved How to detect duplicate ICommandData on the Server?

Discussion in 'NetCode for ECS' started by vectorized-runner, Apr 6, 2021.

  1. vectorized-runner

    vectorized-runner

    Joined:
    Jan 22, 2018
    Posts:
    398
    I'm sending a single ICommandData when the player clicks, but the Server receives it 1-2-3 times randomly causing my system to not work properly. This is not a problem for movement but I'm trying to do damage to the enemies when player clicks and it needs to be more reliable. It doesn't work even if I add cooldown for damaging enemies. Do I need to use Rpc's for this kind of behaviour?

    Screenshot_1.png
     
  2. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    900
    What you mean by
    ?
    Are you calling AddCommandData only if the player click a button? In that case you need to add some extra logic.

    CommandData are a continuous stream of input sent from client to server. Is like your pad/keyboard is connected to the remove end. As a result, you should call AddCommandData and update your command stream every client frame.
    The server usually need the client input to simulate the current tick. If it doesn't find one, it will use the last received command from the client as a "guess" (this is done internally by GetDataAtTick).

    The GetDataAtTick return value is always true in practice (apart when there are not commands inside), but you can check if that command is old by comparing the Tick property and eventually doing nothing in that case.

    Another important aspect to consider is that the client and server may runs at different frame rate. Clients in particular are using a variable one.
    For that reason it is possible for a client to runs multiple "partial" ticks for a given predicted server tick.
    The commands recorded for the partial ticks are all using the same slot, so they are overwritten every time you call AddCommandData. Only the "last" one for the full tick is sent to the server.
    As an example: Server run at 30hz, client run at 120hz. B means button pressed

    Tick N, (predicting Tick N+1)
    N_1
    N_2 B <----- where you pressed the button
    N_3 -------> This is the last processed command, that says no button pressed
    Tick N+1

    The client will run 4 partial ticks (N, .. N_3), each time restoring the ghosts state to the last full tick (N) and applying the current recorded command. Without any special logic, the last run (N_3), that is also the one the server is going to process, will say No Button Pressed.

    For reliably check a single click condition (like you described) there are a couple of possible solutions, but the easiest one to start with is probably to check the button pressed (or the logic that inflict the damage) only for the full tick (ServerTickFraction == 1 or close to it).

    Another solution, could be to change how you record the commands when processing the input.
    For example you can retrieve the command for the current tick, check if the tick match, and:
    - if they don't, reset it to the default.
    - if they do, avoid to reset the click field if already set.

    Also is probably necessary to inflict the damage only if the ServerTick and the Tick property of the command match, to avoid (especially on the client) to re-apply the damage multiple time.
     
    vectorized-runner likes this.
  3. vectorized-runner

    vectorized-runner

    Joined:
    Jan 22, 2018
    Posts:
    398
    I'm always calling AddCommandData, on mouse click I send meaningful input and otherwise I send empty input

    Doesn't GhostPredictionSystemGroup.ShouldPredict do this already? I'm confused about this.

    Thanks for the explanation, though Client click seems to be working fine now. Problem is that Server is receiving 1-3 non-empty inputs for a single click.


    That really confused me that it's so hard to send reliable input for one key down. So do I need to check for ServerTickFraction on both Client and Server worlds, and for all my systems that require one key down detection, or does only Prediction systems need this?
     
  4. vectorized-runner

    vectorized-runner

    Joined:
    Jan 22, 2018
    Posts:
    398
    Also here's my client and server input code:

    Code (CSharp):
    1. using Unity.Entities;
    2. using Unity.NetCode;
    3. using UnityEngine;
    4.  
    5. namespace Code
    6. {
    7.     [UpdateInGroup(typeof(ClientSimulationSystemGroup))]
    8.     public class SamplePlayerInputSystem : SystemBase
    9.     {
    10.         protected override void OnCreate()
    11.         {
    12.             RequireForUpdate(GetEntityQuery(typeof(Player)));
    13.             RequireSingletonForUpdate<NetworkIdComponent>();
    14.             RequireSingletonForUpdate<RaycastResult>();
    15.         }
    16.  
    17.         protected override void OnUpdate()
    18.         {
    19.             var commandEntity = GetSingleton<CommandTargetComponent>().targetEntity;
    20.             if(commandEntity == Entity.Null)
    21.             {
    22.                 return;
    23.             }
    24.          
    25.             var playerInput = new PlayerInput();
    26.             var serverTick = World.GetExistingSystem<ClientSimulationSystemGroup>().ServerTick;
    27.  
    28.             playerInput.Tick = serverTick;
    29.             var playerInputBFE = GetBufferFromEntity<PlayerInput>();
    30.          
    31.             var raycastResult = GetSingleton<RaycastResult>();
    32.             var hitEntity = raycastResult.Hit.Entity;
    33.  
    34.             if(Input.GetMouseButtonDown(0) && hitEntity != Entity.Null)
    35.             {
    36.                 var enemyComponentData = GetComponentDataFromEntity<Enemy>(true);
    37.                 var ghostComponentData = GetComponentDataFromEntity<GhostComponent>(true);
    38.  
    39.                 if(enemyComponentData.HasComponent(hitEntity))
    40.                 {
    41.                     playerInput.Action = PlayerAction.Attack;
    42.                     var ghostId = ghostComponentData[hitEntity].ghostId;
    43.                     playerInput.TargetGhostId = ghostId;
    44.                     Debug.Log($"Send Attack Command {ghostId}");
    45.                 }
    46.                 else
    47.                 {
    48.                     playerInput.Action = PlayerAction.Move;
    49.                     playerInput.Position = raycastResult.Hit.Position;
    50.                     Debug.Log($"Send Move Command: {playerInput.Position}");
    51.                 }
    52.             }
    53.  
    54.             playerInputBFE[commandEntity].AddCommandData(playerInput);
    55.         }
    56.     }
    57. }
    Code (CSharp):
    1. using System;
    2. using System.Diagnostics;
    3. using Unity.Collections;
    4. using Unity.Entities;
    5. using Unity.NetCode;
    6. using Unity.Transforms;
    7. using Debug = UnityEngine.Debug;
    8.  
    9. namespace Code
    10. {
    11.     [UpdateInGroup(typeof(GhostPredictionSystemGroup))]
    12.     public class PlayerInputReceiveSystem : SystemBase
    13.     {
    14.         protected override void OnUpdate()
    15.         {
    16.             var group = World.GetExistingSystem<GhostPredictionSystemGroup>();
    17.             var tick = group.PredictingTick;
    18.             var worldName = new FixedString32(World.Name);
    19.             var random = new Unity.Mathematics.Random((uint)Stopwatch.GetTimestamp());
    20.          
    21.             Entities
    22.                 .WithAll<Player>()
    23.                 .ForEach((ref AttackTarget attackTarget, ref Destination destination, ref Translation translation, in DynamicBuffer<PlayerInput> inputBuffer, in PredictedGhostComponent prediction) =>
    24.                 {
    25.                     if(!GhostPredictionSystemGroup.ShouldPredict(tick, prediction))
    26.                     {
    27.                         return;
    28.                     }
    29.  
    30.                     if(!inputBuffer.GetDataAtTick(tick, out var input))
    31.                     {
    32.                         return;
    33.                     }
    34.                  
    35.                     switch(input.Action)
    36.                     {
    37.                         case PlayerAction.None:
    38.                         {
    39.                             break;
    40.                         }
    41.                         case PlayerAction.Move:
    42.                         {
    43.                             destination = new Destination
    44.                             {
    45.                                 IsReached = false,
    46.                                 Position = input.Position
    47.                             };
    48.                          
    49.                             Debug.Log($"Receive Move Command on {worldName}");
    50.                             break;
    51.                         }
    52.                         case PlayerAction.Attack:
    53.                         {
    54.                             attackTarget = new AttackTarget
    55.                             {
    56.                                 GhostId = input.TargetGhostId
    57.                             };
    58.  
    59.                             Debug.Log($"Receive Attack Command on {worldName}");
    60.                             break;
    61.                         }
    62.                         default:
    63.                             throw new ArgumentOutOfRangeException();
    64.                     }
    65.                 })
    66.                 .Schedule();
    67.         }
    68.     }
    69. }
     
  5. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    900
    Probably the explanation was confusing :) No you don't need anything like that!

    We have an example for something similar of what are you doing in one our sample, check the LagCompensation one.
    Also checking for tick fraction it may be possible in your case, since the action logic is inside the PlayerInputReceiveSystem (that is the one determining the outcome of the raycast).

    Looking at your code, one thing that catch my eyes is that the SamplePlayerInputSystem is not running in the GhostInputSystemGroup. If you ran in the ClientSimulationSystemGroup (not even inside the GhostSimulationSystemGroup), given the name, the system will always run after the GhostSimulationGroup, that will give you one more frame latency.

    And again, the only thing that can happen (regardless of the position) is that sometime you overwrite the command data and so on the client you will log Attack but not on the server.
    I tried doing something similar and I never received multiple command on the server (but I'm using the latest version of NetCode not yet available for you)

    The only case I can think of for witch you may receive multiple Attack event might be if the client skip one frame (that can happen since it is always trying to be ahead of the server and depending on the latency setting this can be more or less visible)

    Ex:
    Predicting Tick Pressed Server Tick Cmd Sent (comprise the buffer for the packet loss compensation)
    ..
    8.3 n 9
    8.7 n 9
    9.6 p 10 > nnnn (9, 8, 7, 6)
    11.1 n. 12 > ppnn (11, 10, 9, 8)

    Since there is an hole (missing frame 11), the GetDataAtTick on the client will reuse the last one (so the one for tick 11).
    This is the only case I can think off on top of my head.
    You can check what it is going on by print the current tick the data is processed on the client and on the server (inside the PlayerInputReceiveSystem.
     
  6. vectorized-runner

    vectorized-runner

    Joined:
    Jan 22, 2018
    Posts:
    398
    I moved my sample system to GhostInputSystemGroup as you said, then took more detailed logs, also realized that send/recv delay on PlayMode Tools is causing this.

    0ms delay: 1 send, 1 recv
    Screenshot_7.png

    100ms delay: 1 send, 2 recv
    Screenshot_8.png

    500ms delay: 1 send, 3 recv
    Screenshot_6.png


    It looks like server is also updating the next frame with previous input when there is lag

    Edit: So would it be wise for me to add this line for the system to run properly?
    Code (CSharp):
    1. if(isRunningOnServer && input.Tick < lastReceivedInputTick)
    2.     return;
    Edit: Here is another screenshot where server also prints input.Tick
    Screenshot_2.png
     
    Last edited: Apr 7, 2021
  7. vectorized-runner

    vectorized-runner

    Joined:
    Jan 22, 2018
    Posts:
    398
    I also checked the LagCompensation sample and rewrote my code, but still I don't know why it's behaving like this

    Code (CSharp):
    1. using Unity.Entities;
    2. using Unity.NetCode;
    3. using UnityEngine;
    4.  
    5. namespace Code
    6. {
    7.     [UpdateInGroup(typeof(GhostInputSystemGroup))]
    8.     public class SendPlayerInputSystem : SystemBase
    9.     {
    10.         protected override void OnCreate()
    11.         {
    12.             RequireForUpdate(GetEntityQuery(typeof(Player)));
    13.             RequireSingletonForUpdate<NetworkIdComponent>();
    14.             RequireSingletonForUpdate<RaycastResult>();
    15.         }
    16.  
    17.         protected override void OnUpdate()
    18.         {
    19.             var commandEntity = GetSingleton<CommandTargetComponent>().targetEntity;
    20.             if(commandEntity == Entity.Null)
    21.             {
    22.                 return;
    23.             }
    24.  
    25.             var playerInput = new PlayerInput();
    26.             var serverTick = World.GetExistingSystem<ClientSimulationSystemGroup>().ServerTick;
    27.             playerInput.Tick = serverTick;
    28.  
    29.             var hit = GetSingleton<RaycastResult>().Hit;
    30.             var worldName = World.Name;
    31.             var time = (float)Time.ElapsedTime;
    32.             var inputBuffer = EntityManager.GetBuffer<PlayerInput>(commandEntity);
    33.  
    34.             if(Input.GetMouseButtonDown(0) && hit.Entity != Entity.Null)
    35.             {
    36.                 var enemyCDFE = GetComponentDataFromEntity<Enemy>(true);
    37.                 var ghostCDFE = GetComponentDataFromEntity<GhostComponent>(true);
    38.  
    39.                 if(enemyCDFE.HasComponent(hit.Entity))
    40.                 {
    41.                     playerInput.Action = PlayerAction.Attack;
    42.                     var ghostId = ghostCDFE[hit.Entity].ghostId;
    43.                     playerInput.TargetGhostId = ghostId;
    44.                
    45.                     Debug.Log($"Send AttackCommand, ServerTick: {serverTick} Time: {time} Target: {ghostId} World: {worldName}");
    46.                 }
    47.                 else
    48.                 {
    49.                     playerInput.Action = PlayerAction.Move;
    50.                     playerInput.Position = hit.Position;
    51.                
    52.                     Debug.Log($"Send MoveCommand, ServerTick: {serverTick} Time: {time} Position: {playerInput.Position} World: {worldName}");
    53.                 }
    54.             }
    55.             else
    56.             {
    57.                 // (NetCode LagCompensation sample)
    58.                 // If there is no input currently, and a input for this tick already exists, don't overwrite it by adding empty input this frame, simply return.
    59.                 if(inputBuffer.GetDataAtTick(serverTick, out var otherInput) && otherInput.Tick == serverTick)
    60.                 {
    61.                     return;
    62.                 }
    63.             }
    64.  
    65.             inputBuffer.Add(playerInput);
    66.         }
    67.     }
    68. }
    Code (CSharp):
    1. using System;
    2. using Unity.Collections;
    3. using Unity.Entities;
    4. using Unity.NetCode;
    5. using Debug = UnityEngine.Debug;
    6.  
    7. namespace Code
    8. {
    9.     // I'm not sure if this system should be predicted or not
    10.     [UpdateInGroup(typeof(GhostPredictionSystemGroup))]
    11.     public class ReceivePlayerInputSystem : SystemBase
    12.     {
    13.         protected override void OnUpdate()
    14.         {
    15.             var predictionGroup = World.GetExistingSystem<GhostPredictionSystemGroup>();
    16.             var isFinalTick = predictionGroup.IsFinalPredictionTick;
    17.             var predictingTick = predictionGroup.PredictingTick;
    18.             var worldName = new FixedString32(World.Name);
    19.             uint serverTick;
    20.             var serverSystem = World.GetExistingSystem<ServerSimulationSystemGroup>();
    21.             var isServer = serverSystem != null;
    22.             var time = (float)Time.ElapsedTime;
    23.        
    24.             if(isServer)
    25.             {
    26.                 serverTick = serverSystem.ServerTick;
    27.             }
    28.             else
    29.             {
    30.                 serverTick = World.GetExistingSystem<ClientSimulationSystemGroup>().ServerTick;
    31.             }
    32.        
    33.             Entities
    34.                 .WithAll<Player>()
    35.                 .ForEach((ref AttackTarget attackTarget, ref Destination destination, in DynamicBuffer<PlayerInput> inputBuffer, in PredictedGhostComponent predictedGhost, in GhostComponent ghost) =>
    36.                 {
    37.                     if(!GhostPredictionSystemGroup.ShouldPredict(predictingTick, predictedGhost))
    38.                     {
    39.                         return;
    40.                     }
    41.                     if(!inputBuffer.GetDataAtTick(predictingTick, out var input))
    42.                     {
    43.                         return;
    44.                     }
    45.                
    46.                     switch(input.Action)
    47.                     {
    48.                         case PlayerAction.None:
    49.                         case PlayerAction.Test:
    50.                         {
    51.                             break;
    52.                         }
    53.                         case PlayerAction.Move:
    54.                         {
    55.                             destination = new Destination
    56.                             {
    57.                                 IsReached = false,
    58.                                 Position = input.Position
    59.                             };
    60.                        
    61.                             Debug.Log($"Recv MoveCommand, SentTick: {input.Tick} PredTick: {predictingTick} ServerTick: {serverTick} Time:{time} GhostId: {ghost.ghostId} World: {worldName}");
    62.                             break;
    63.                         }
    64.                         case PlayerAction.Attack:
    65.                         {
    66.                             // (NetCode LagCompensation sample)
    67.                             // (This check is only for client, since on Server ServerTick is always equal to PredictionTick)
    68.                             // Don't perform attack when rolling back, only when simulating latest tick
    69.                             if(!isFinalTick)
    70.                             {
    71.                                 return;
    72.                             }
    73.                             // (NetCode LagCompensation sample)
    74.                             // This check probably works for both Client and Server
    75.                             // Makes sure the input is executed exactly once (I hope?)
    76.                             if(input.Tick != predictingTick)
    77.                             {
    78.                                 return;
    79.                             }
    80.                        
    81.                             attackTarget = new AttackTarget
    82.                             {
    83.                                 GhostId = input.TargetGhostId,
    84.                                 WantsToHit = true
    85.                             };
    86.  
    87.                             Debug.Log($"Recv AttackCommand, SentTick: {input.Tick} PredTick: {predictingTick} ServerTick: {serverTick} Time: {time} GhostId: {ghost.ghostId} World: {worldName}");
    88.                        
    89.                             break;
    90.                         }
    91.                         default:
    92.                             throw new ArgumentOutOfRangeException();
    93.                     }
    94.                 })
    95.                 .Schedule();
    96.         }
    97.     }
    98. }
    and when running on 200ms send/recv delay:

    Screenshot_2.png

    Why did server receive ICommandData with Tick 542? That makes no sense to me
     
    Last edited: Apr 8, 2021
  8. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    900
    Is hard from your log to understand exactly what is the problem.
    If you field on your input and write there the tick at witch you pressed the button is making it easier to understand probably.

    But.. can I ask you what is your current frame rate? is above 60? or are you running somewhat near 30 (or lower).
    if you are below 60fps, then that can clarify everything I see in your log (and I can repro it as well).

    The default setting for the server is to run at fixed rate of 60hz. So if in the editor you are under that threshold, for each editor frame the server may run multiple ticks . So in your case:

    Client: Send data for tick 541 -> attack
    ..
    ..
    Server: (runt 2 ticks that frame)
    541 -> attack (as the player input say)
    542 -> no data -> re-use old command -> attack again

    That may justify what I see.

    I tried myself using the lag compensation and if put the same log and try with different latency and I never had any problem. But, it is running pretty fast, 120-230fps.
    If I set 200 ms latency and I try to run as slow as I can (by enabling job debugger and men leak) then I go below 20 ms.. And this is were I start seeing:

    client pressed : tick:130 fire:130
    client prediction: s:False t:130 fire:130
    ..
    ..
    server t:130 fire:130
    server t:131 fire:130 <-- that because it is running multiple time per frame
    server t:132 fire:130 <-- that because it is running multiple time per frame

    Is not your code fault in that case. Is just not the way the server would run in a real-case scenario.
    We don't have yet any safeguard against or way to avoid that in editor right now.
    So to be robust against this problem (if that is the problem) adding an extra field for the key press with the stamp on when it was pressed and check that in the prediction may solve the problem.

    Also, another question is: what version of net code are you using?
     
  9. vectorized-runner

    vectorized-runner

    Joined:
    Jan 22, 2018
    Posts:
    398
    On my last screenshot I also logged the "SentTick", is that what you want? I've seen on the LagCompensation sample they used another "lastFire" field, but isn't that always the same as ICommandData.Tick field when you're sending non-empty inputs?

    Yes but in my case "SentTick" also seems to be changed, if the Server is using old input why Tick is also increased?

    Indeed I'm running around 30fps, well I checked the profiler and it shows 29-30 calls for a frame for ReceivePlayerInputSystem (and other prediction systems) on the ClientWorld0, I don't know why it's that high but that's why I'm running so slow (is this expected?)

    I'm using 0.6.0-preview.7

    Anyway maybe I should try to repo this with a simpler example, this turned out to be more complicated then I thought :p
     
  10. vectorized-runner

    vectorized-runner

    Joined:
    Jan 22, 2018
    Posts:
    398
    Ok, so I used another field to store PressedTick on my ICommandData (which is only set to serverTick when I click), that is indeed different from the ICommandData.Tick variable (I guess another system configures this after I set it?).

    So your assumption seems to be correct, as PressedTick is the same when the Server runs twice.

    I'm thinking of storing another component on the player (on server) which holds LastAppliedPressedTick, hope this solves it. Or I need to make sure the Server can run on 60hz.

    I don't know if I need to do extra things on Client though, also maybe this system shouldn't be predicted at all? So confused rn
     
  11. StickyTommie

    StickyTommie

    Joined:
    Oct 23, 2019
    Posts:
    13
    This also seems to happen outside of the Editor. Anytime a client's FPS is below the Server's Simulation Rate, the server will run multiple frames with the latest CommandData from that Client and will, in this case fire 3 frames in a row on the server.

    Why is this?