Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Please make BufferedLinearInterpolator public

Discussion in 'Netcode for GameObjects' started by Skanou_Dev, Apr 14, 2022.

  1. Skanou_Dev

    Skanou_Dev

    Joined:
    Jun 22, 2017
    Posts:
    7
    Hi,

    I'm working on VR multiplayer project and because I need to send a lot of transform, I'm trying to implement my own network transform with compressed position and quaternion (and maybe pack them with finger data by player).
    I also want to interpolate my data because and avoid sending them every frame.

    In Unity.Netcode NetworkTransform, they use a BufferedLinearInterpolator class for interpolation. I would love to use it to interpolate my data, the problem is that BufferedLinearInterpolator is internal and can't be used.

    BufferedLinearInterpolator doesn't seems to rely on any other internal class of Unity.Netcode, so why is it internal ? I mean, in networking there is potentially a lot of thing you want to interpolate. I don't understand why this can't be accessed.

    It's not the first time I encouter something internal that I don't understand (ie, NetworkObject.GlobalObjectIdHash which is required in the Approval Callback).
    I know Netcode for GameObjects is focused is a high level API, but something it feels more like a blackbox with too many things burried and not accessible.
     
  2. Draener

    Draener

    Joined:
    Dec 17, 2021
    Posts:
    7
    Yeah, I agree with this statement. (Especially NetworkObject.GlobalObjectIdHash). Like your building this for people to just drag and drop on to objects... but these people aren't going to be digging into the internal structure of these things anyways, so why are you making it so no one else can use / modify them?

    All you're doing at that point is preventing people from using your code that MOSTLY solves their issue, and changing the part that doesn't quite do what you want. And I don't see the benefit in that.
     
  3. ep1s0de

    ep1s0de

    Joined:
    Dec 24, 2015
    Posts:
    168
    What is the problem!?

    Code (CSharp):
    1. using System;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4.  
    5. /// <summary>
    6. /// Solves for incoming values that are jittered
    7. /// Partially solves for message loss. Unclamped lerping helps hide this, but not completely
    8. /// </summary>
    9. /// <typeparam name="T">The type of interpolated value</typeparam>
    10. [Serializable]
    11. public abstract class BufferedLinearInterpolator<T> where T : struct
    12. {
    13.     internal float MaxInterpolationBound = 3.0f;
    14.     private struct BufferedItem
    15.     {
    16.         public T Item;
    17.         public float time;
    18.  
    19.         public BufferedItem(T item, float timeSent)
    20.         {
    21.             Item = item;
    22.             time = timeSent;
    23.         }
    24.     }
    25.  
    26.     /// <summary>
    27.     /// There's two factors affecting interpolation: buffering (set in NetworkManager's NetworkTimeSystem) and interpolation time, which is the amount of time it'll take to reach the target. This is to affect the second one.
    28.     /// </summary>
    29.     public float MaximumInterpolationTime = 1f;
    30.  
    31.     private const float k_SmallValue = 9.999999439624929E-11f; // copied from Vector3's equal operator
    32.  
    33.     private T m_InterpStartValue;
    34.     private T m_CurrentInterpValue;
    35.     private T m_InterpEndValue;
    36.  
    37.     private float m_EndTimeConsumed;
    38.     private float m_StartTimeConsumed;
    39.  
    40.     private readonly List<BufferedItem> m_Buffer = new List<BufferedItem>(k_BufferCountLimit);
    41.  
    42.     // Buffer consumption scenarios
    43.     // Perfect case consumption
    44.     // | 1 | 2 | 3 |
    45.     // | 2 | 3 | 4 | consume 1
    46.     // | 3 | 4 | 5 | consume 2
    47.     // | 4 | 5 | 6 | consume 3
    48.     // | 5 | 6 | 7 | consume 4
    49.     // jittered case
    50.     // | 1 | 2 | 3 |
    51.     // | 2 | 3 |   | consume 1
    52.     // | 3 |   |   | consume 2
    53.     // | 4 | 5 | 6 | consume 3
    54.     // | 5 | 6 | 7 | consume 4
    55.     // bursted case (assuming max count is 5)
    56.     // | 1 | 2 | 3 |
    57.     // | 2 | 3 |   | consume 1
    58.     // | 3 |   |   | consume 2
    59.     // |   |   |   | consume 3
    60.     // |   |   |   |
    61.     // | 4 | 5 | 6 | 7 | 8 | --> consume all and teleport to last value <8> --> this is the nuclear option, ideally this example would consume 4 and 5
    62.     // instead of jumping to 8, but since in OnValueChange we don't yet have an updated server time (updated in pre-update) to know which value
    63.     // we should keep and which we should drop, we don't have enough information to do this. Another thing would be to not have the burst in the first place.
    64.  
    65.     // Constant absolute value for max buffer count instead of dynamic time based value. This is in case we have very low tick rates, so
    66.     // that we don't have a very small buffer because of this.
    67.     private const int k_BufferCountLimit = 100;
    68.     private BufferedItem m_LastBufferedItemReceived;
    69.     private int m_NbItemsReceivedThisFrame;
    70.  
    71.     private int m_LifetimeConsumedCount;
    72.  
    73.     private bool InvalidState => m_Buffer.Count == 0 && m_LifetimeConsumedCount == 0;
    74.  
    75.     /// <summary>
    76.     /// Resets interpolator to initial state
    77.     /// </summary>
    78.     public void Clear()
    79.     {
    80.         m_Buffer.Clear();
    81.         m_EndTimeConsumed = 0.0f;
    82.         m_StartTimeConsumed = 0.0f;
    83.     }
    84.  
    85.     /// <summary>
    86.     /// Teleports current interpolation value to targetValue.
    87.     /// </summary>
    88.     /// <param name="targetValue">The target value to teleport instantly</param>
    89.     /// <param name="serverTime">The current server time</param>
    90.     public void ResetTo(T targetValue, float serverTime)
    91.     {
    92.         m_LifetimeConsumedCount = 1;
    93.         m_InterpStartValue = targetValue;
    94.         m_InterpEndValue = targetValue;
    95.         m_CurrentInterpValue = targetValue;
    96.         m_Buffer.Clear();
    97.         m_EndTimeConsumed = 0.0f;
    98.         m_StartTimeConsumed = 0.0f;
    99.  
    100.         Update(0, serverTime, serverTime);
    101.     }
    102.  
    103.     // todo if I have value 1, 2, 3 and I'm treating 1 to 3, I shouldn't interpolate between 1 and 3, I should interpolate from 1 to 2, then from 2 to 3 to get the best path
    104.     private void TryConsumeFromBuffer(double renderTime, double serverTime)
    105.     {
    106.         int consumedCount = 0;
    107.         // only consume if we're ready
    108.  
    109.         //  this operation was measured as one of our most expensive, and we should put some thought into this.
    110.         //   NetworkTransform has (currently) 7 buffered linear interpolators (3 position, 3 scale, 1 rot), and
    111.         //   each has its own independent buffer and 'm_endTimeConsume'.  That means every frame I have to do 7x
    112.         //   these checks vs. if we tracked these values in a unified way
    113.         if (renderTime >= m_EndTimeConsumed)
    114.         {
    115.             BufferedItem? itemToInterpolateTo = null;
    116.             // assumes we're using sequenced messages for netvar syncing
    117.             // buffer contains oldest values first, iterating from end to start to remove elements from list while iterating
    118.  
    119.             // calling m_Buffer.Count shows up hot in the profiler.
    120.             for (int i = m_Buffer.Count - 1; i >= 0; i--) // todo stretch: consume ahead if we see we're missing values due to packet loss
    121.             {
    122.                 var bufferedValue = m_Buffer[i];
    123.                 // Consume when ready and interpolate to last value we can consume. This can consume multiple values from the buffer
    124.                 if (bufferedValue.time <= serverTime)
    125.                 {
    126.                     if (!itemToInterpolateTo.HasValue || bufferedValue.time > itemToInterpolateTo.Value.time)
    127.                     {
    128.                         if (m_LifetimeConsumedCount == 0)
    129.                         {
    130.                             // if interpolator not initialized, teleport to first value when available
    131.                             m_StartTimeConsumed = bufferedValue.time;
    132.                             m_InterpStartValue = bufferedValue.Item;
    133.                         }
    134.                         else if (consumedCount == 0)
    135.                         {
    136.                             // Interpolating to new value, end becomes start. We then look in our buffer for a new end.
    137.                             m_StartTimeConsumed = m_EndTimeConsumed;
    138.                             m_InterpStartValue = m_InterpEndValue;
    139.                         }
    140.  
    141.                         if (bufferedValue.time > m_EndTimeConsumed)
    142.                         {
    143.                             itemToInterpolateTo = bufferedValue;
    144.                             m_EndTimeConsumed = bufferedValue.time;
    145.                             m_InterpEndValue = bufferedValue.Item;
    146.                         }
    147.                     }
    148.  
    149.                     m_Buffer.RemoveAt(i);
    150.                     consumedCount++;
    151.                     m_LifetimeConsumedCount++;
    152.                 }
    153.             }
    154.         }
    155.     }
    156.  
    157.     /// <summary>
    158.     /// Call to update the state of the interpolators before reading out
    159.     /// </summary>
    160.     /// <param name="deltaTime">time since last call</param>
    161.     /// <param name="renderTime">our current time</param>
    162.     /// <param name="serverTime">current server time</param>
    163.     /// <returns>The newly interpolated value of type 'T'</returns>
    164.     public T Update(float deltaTime, float renderTime, float serverTime)
    165.     {
    166.         TryConsumeFromBuffer(renderTime, serverTime);
    167.  
    168.         if (InvalidState)
    169.         {
    170.             throw new InvalidOperationException("trying to update interpolator when no data has been added to it yet");
    171.         }
    172.  
    173.         // Interpolation example to understand the math below
    174.         // 4   4.5      6   6.5
    175.         // |   |        |   |
    176.         // A   render   B   Server
    177.  
    178.         if (m_LifetimeConsumedCount >= 1) // shouldn't interpolate between default values, let's wait to receive data first, should only interpolate between real measurements
    179.         {
    180.             float t = 1.0f;
    181.             float range = m_EndTimeConsumed - m_StartTimeConsumed;
    182.             if (range > k_SmallValue)
    183.             {
    184.                 t = (float)((renderTime - m_StartTimeConsumed) / range);
    185.  
    186.                 if (t < 0.0f)
    187.                 {
    188.                     // There is no mechanism to guarantee renderTime to not be before m_StartTimeConsumed
    189.                     // This clamps t to a minimum of 0 and fixes issues with longer frames and pauses
    190.  
    191.                     Debug.LogError($"renderTime was before m_StartTimeConsumed. This should never happen. {nameof(renderTime)} is {renderTime}, {nameof(m_StartTimeConsumed)} is {m_StartTimeConsumed}");
    192.  
    193.                     t = 0.0f;
    194.                 }
    195.  
    196.                 if (t > MaxInterpolationBound) // max extrapolation
    197.                 {
    198.                     // TODO this causes issues with teleport, investigate
    199.                     t = 1.0f;
    200.                 }
    201.             }
    202.  
    203.             var target = InterpolateUnclamped(m_InterpStartValue, m_InterpEndValue, t);
    204.             m_CurrentInterpValue = Interpolate(m_CurrentInterpValue, target, deltaTime / MaximumInterpolationTime); // second interpolate to smooth out extrapolation jumps
    205.         }
    206.  
    207.         m_NbItemsReceivedThisFrame = 0;
    208.         return m_CurrentInterpValue;
    209.     }
    210.  
    211.     /// <summary>
    212.     /// Add measurements to be used during interpolation. These will be buffered before being made available to be displayed as "latest value".
    213.     /// </summary>
    214.     /// <param name="newMeasurement">The new measurement value to use</param>
    215.     /// <param name="sentTime">The time to record for measurement</param>
    216.     public void AddMeasurement(T newMeasurement, float sentTime)
    217.     {
    218.         m_NbItemsReceivedThisFrame++;
    219.  
    220.         // This situation can happen after a game is paused. When starting to receive again, the server will have sent a bunch of messages in the meantime
    221.         // instead of going through thousands of value updates just to get a big teleport, we're giving up on interpolation and teleporting to the latest value
    222.         if (m_NbItemsReceivedThisFrame > k_BufferCountLimit)
    223.         {
    224.             if (m_LastBufferedItemReceived.time < sentTime)
    225.             {
    226.                 m_LastBufferedItemReceived = new BufferedItem(newMeasurement, sentTime);
    227.                 ResetTo(newMeasurement, sentTime);
    228.                 // Next line keeps renderTime above m_StartTimeConsumed. Fixes pause/unpause issues
    229.                 m_Buffer.Add(m_LastBufferedItemReceived);
    230.             }
    231.  
    232.             return;
    233.         }
    234.  
    235.         if (sentTime > m_EndTimeConsumed || m_LifetimeConsumedCount == 0) // treat only if value is newer than the one being interpolated to right now
    236.         {
    237.             m_LastBufferedItemReceived = new BufferedItem(newMeasurement, sentTime);
    238.             m_Buffer.Add(m_LastBufferedItemReceived);
    239.         }
    240.     }
    241.  
    242.     /// <summary>
    243.     /// Gets latest value from the interpolator. This is updated every update as time goes by.
    244.     /// </summary>
    245.     /// <returns>The current interpolated value of type 'T'</returns>
    246.     public T GetInterpolatedValue()
    247.     {
    248.         return m_CurrentInterpValue;
    249.     }
    250.  
    251.     /// <summary>
    252.     /// Method to override and adapted to the generic type. This assumes interpolation for that value will be clamped.
    253.     /// </summary>
    254.     /// <param name="start">The start value (min)</param>
    255.     /// <param name="end">The end value (max)</param>
    256.     /// <param name="time">The time value used to interpolate between start and end values (pos)</param>
    257.     /// <returns>The interpolated value</returns>
    258.     protected abstract T Interpolate(T start, T end, float time);
    259.  
    260.     /// <summary>
    261.     /// Method to override and adapted to the generic type. This assumes interpolation for that value will not be clamped.
    262.     /// </summary>
    263.     /// <param name="start">The start value (min)</param>
    264.     /// <param name="end">The end value (max)</param>
    265.     /// <param name="time">The time value used to interpolate between start and end values (pos)</param>
    266.     /// <returns>The interpolated value</returns>
    267.     protected abstract T InterpolateUnclamped(T start, T end, float time);
    268. }
    269.  
    270. /// <inheritdoc />
    271. /// <remarks>
    272. /// This is a buffered linear interpolator for a <see cref="float"/> type value
    273. /// </remarks>
    274. [Serializable]
    275. public class BufferedLinearInterpolatorFloat : BufferedLinearInterpolator<float>
    276. {
    277.     /// <inheritdoc />
    278.     protected override float InterpolateUnclamped(float start, float end, float time)
    279.     {
    280.         return Mathf.LerpUnclamped(start, end, time);
    281.     }
    282.  
    283.     /// <inheritdoc />
    284.     protected override float Interpolate(float start, float end, float time)
    285.     {
    286.         return Mathf.Lerp(start, end, time);
    287.     }
    288. }
    289.  
    290. /// <inheritdoc />
    291. /// <remarks>
    292. /// This is a buffered linear interpolator for a <see cref="Quaternion"/> type value
    293. /// </remarks>
    294. public class BufferedLinearInterpolatorQuaternion : BufferedLinearInterpolator<Quaternion>
    295. {
    296.     /// <inheritdoc />
    297.     protected override Quaternion InterpolateUnclamped(Quaternion start, Quaternion end, float time)
    298.     {
    299.         return Quaternion.SlerpUnclamped(start, end, time);
    300.     }
    301.  
    302.     /// <inheritdoc />
    303.     protected override Quaternion Interpolate(Quaternion start, Quaternion end, float time)
    304.     {
    305.         return Quaternion.SlerpUnclamped(start, end, time);
    306.     }
    307. }
     
  4. Grave188

    Grave188

    Joined:
    Dec 13, 2017
    Posts:
    10
    Indeed :)
     
  5. liambilly

    liambilly

    Joined:
    May 13, 2022
    Posts:
    154
    [Netcode] renderTime was before m_StartTimeConsumed. This should never happen. renderTime is 17.9305105655144, m_StartTimeConsumed is 17.9313524716347
    what causes this error
     
    vinaysharma, naeemh11 and codeBatt like this.