Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Bug [Resolved] Segmentation fault in Unity.Transport during stress testing - AWS - Linux

Discussion in 'Multiplayer' started by NWalker_FGOL_Ubisoft, Oct 2, 2020.

  1. NWalker_FGOL_Ubisoft

    NWalker_FGOL_Ubisoft

    Joined:
    Aug 10, 2015
    Posts:
    25
    NullRef & Crash:

    Inside a try/catch of my     RelayRoomUpdateJob.Execute() method:
    Code (CSharp):
    1.  
    2.  
    3. Caught fatal signal - signo:11 code:1 errno:0 addr:0x200000062
    4. System.NullReferenceException: Object reference not set to an instance of an object
    5.   at Unity.Networking.Transport.NetworkDriver+Concurrent.BeginSend (Unity.Networking.Transport.NetworkPipeline pipe, Unity.Networking.Transport.NetworkConnection id, System.Int32 requiredPayloadSize) [0x00188] in C:\Perforce\RainbowCore\rcos-unity-sdk\Library\PackageCache\com.unity.transport@0.4.1-preview.1\Runtime\NetworkDriver.cs:164
    6.   at Unity.Networking.Transport.NetworkDriver.BeginSend (Unity.Networking.Transport.NetworkPipeline pipe, Unity.Networking.Transport.NetworkConnection id, System.Int32 requiredPayloadSize) [0x00001] in C:\Perforce\RainbowCore\rcos-unity-sdk\Library\PackageCache\com.unity.transport@0.4.1-preview.1\Runtime\NetworkDriver.cs:918
    7.   at RCOS.NetCode.Server.Relay.RelayRoomUpdateJob.BroadcastLedgerToClients () [0x000a9] in C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayRoomUpdateJob.cs:1173
    8.   at RCOS.NetCode.Server.Relay.RelayRoomUpdateJob.Execute () [0x00050] in C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayRoomUpdateJob.cs:117
    9. UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
    10. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    11. UnityEngine.Logger:Log(LogType, Object)
    12. UnityEngine.Debug:LogError(Object)
    13. RCOS.NetCode.Server.Relay.RelayRoomUpdateJob:Execute() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayRoomUpdateJob.cs:122)
    14. Unity.Jobs.JobStruct`1:Execute(RelayRoomUpdateJob&, IntPtr, IntPtr, JobRanges&, Int32)
    15. Unity.Jobs.JobHandle:ScheduleBatchedJobsAndComplete(JobHandle&)
    16. Unity.Jobs.JobHandle:Complete()
    17. RCOS.NetCode.Server.Relay.RelayServerRoom:FinishUpdate(Double) (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayServerRoom.cs:454)
    18. RCOS.NetCode.Server.Relay.RelayServerCluster:TickInternal() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayServerCluster.cs:183)
    19. RCOS.NetCode.Server.Relay.RelayServerCluster:Tick() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayServerCluster.cs:150)
    20. RCOS.NetCode.Server.ServerBootstrapper:Update() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\ServerBootstrapper.cs:128)
    21. (Filename: C Line: 0)
    22. NullReferenceException: Object reference not set to an instance of an object
    23.   at Unity.Networking.Transport.NetworkDriver+Concurrent.BeginSend (Unity.Networking.Transport.NetworkPipeline pipe, Unity.Networking.Transport.NetworkConnection id, System.Int32 requiredPayloadSize) [0x00188] in C:\Perforce\RainbowCore\rcos-unity-sdk\Library\PackageCache\com.unity.transport@0.4.1-preview.1\Runtime\NetworkDriver.cs:164
    24.   at Unity.Networking.Transport.NetworkDriver.BeginSend (Unity.Networking.Transport.NetworkPipeline pipe, Unity.Networking.Transport.NetworkConnection id, System.Int32 requiredPayloadSize) [0x00001] in C:\Perforce\RainbowCore\rcos-unity-sdk\Library\PackageCache\com.unity.transport@0.4.1-preview.1\Runtime\NetworkDriver.cs:918
    25.   at RCOS.NetCode.Server.Relay.RelayRoomUpdateJob.BroadcastLedgerToClients () [0x000a9] in C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayRoomUpdateJob.cs:1173
    26.   at RCOS.NetCode.Server.Relay.RelayRoomUpdateJob.Execute () [0x00050] in C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayRoomUpdateJob.cs:117
    27. UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
    28. UnityEngine.DebugLogHandler:LogException(Exception, Object)
    29. UnityEngine.Logger:LogException(Exception, Object)
    30. UnityEngine.Debug:LogException(Exception)
    31. RCOS.NetCode.Server.Relay.RelayRoomUpdateJob:Execute() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayRoomUpdateJob.cs:123)
    32. Unity.Jobs.JobStruct`1:Execute(RelayRoomUpdateJob&, IntPtr, IntPtr, JobRanges&, Int32)
    33. Unity.Jobs.JobHandle:ScheduleBatchedJobsAndComplete(JobHandle&)
    34. Unity.Jobs.JobHandle:Complete()
    35. RCOS.NetCode.Server.Relay.RelayServerRoom:FinishUpdate(Double) (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayServerRoom.cs:454)
    36. RCOS.NetCode.Server.Relay.RelayServerCluster:TickInternal() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayServerCluster.cs:183)
    37. RCOS.NetCode.Server.Relay.RelayServerCluster:Tick() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\Relay\RelayServerCluster.cs:150)
    38. RCOS.NetCode.Server.ServerBootstrapper:Update() (at C:\Perforce\RainbowCore\rcos-unity-sdk\Assets\RainbowCore\RCOS Unity SDK\Network\Server\ServerBootstrapper.cs:128)
    39. (Filename: C Line: 0)
    40. NullReferenceException: Object reference not set to an instance of an object
    41.   at Unity.Collections.LowLevel.Unsafe.UnsafeUtility.WriteArrayElement[T] (System.Void* destination, System.Int32 index, T value) [0x00000] in <08e6e494a0bd4270b4c5600179125435>:0
    42.   at Unity.Collections.NativeList`1[T].set_Item (System.Int32 index, T value) [0x00001] in C:\Perforce\RainbowCore\rcos-unity-sdk\Library\PackageCache\com.unity.collections@0.12.0-preview.13\Unity.Collections\NativeList.cs:160
    43.   at Unity.Networking.Transport.NetworkEventQueue.Clear () [0x00012] in C:\Perforce\RainbowCore\rcos-unity-sdk\Library\PackageCache\com.unity.transport@0.4.1-preview.1\Runtime\NetworkEventQueue.cs:144
    44.   at Unity.Networking.Transport.NetworkDriver+ClearEventQueue.Execute () [0x00001] in C:\Perforce\RainbowCore\rcos-unity-sdk\Library\PackageCache\com.unity.transport@0.4.1-preview.1\Runtime\NetworkDriver.cs:639
    45.   at Unity.Jobs.IJobExtensions+JobStruct`1[T].Execute (T& data, System.IntPtr additionalPtr, System.IntPtr bufferRangePatchData, Unity.Jobs.LowLevel.Unsafe.JobRanges& ranges, System.Int32 jobIndex) [0x00001] in <08e6e494a0bd4270b4c5600179125435>:0
    46.   at (wrapper delegate-invoke) Unity.Jobs.IJobExtensions+JobStruct`1+ExecuteJobFunction[Unity.Networking.Transport.NetworkDriver+ClearEventQueue].invoke_void_T&_intptr_intptr_JobRanges&_int(Unity.Networking.Transport.NetworkDriver/ClearEventQueue&,intptr,intptr,Unity.Jobs.LowLevel.Unsafe.JobRanges&,int)
    47. (Filename: <08e6e494a0bd4270b4c5600179125435> Line: 0)
    48. Caught fatal signal - signo:11 code:1 errno:0 addr:0x15f
    49. Obtained 12 stack frames.
    50. #0  0x007fcf357ca7e0 in funlockfile
    51. #1  0x007fcf3387bda5 in Unity.Networking.Transport.NetworkDriver.GetConnection(Unity.Networking.Transport.NetworkDriver* this, Unity.Networking.Transport.NetworkConnection id)_25C4BC36650DED98
    52. #2  0x007fcf33879b7d in Unity.Networking.Transport.NetworkDriver.SendPacket(Unity.Networking.Transport.NetworkDriver* this, Unity.Networking.Transport.Protocols.UdpCProtocol type, Unity.Networking.Transport.NetworkConnection id)_622B87DB9D90A362
    53. #3  0x007fcf3387e9ee in f00744ec48cd0864aed85ed9873e2356_avx2
    54. #4  0x007fcf36db0609 in ExecuteJob(ManagedJobData*, void (*)(void*, void*, void*, void*, int), int, unsigned char*)
    55. #5  0x007fcf36daf863 in ForwardJobToManaged(ManagedJobData*)
    56. #6  0x007fcf36d92c23 in JobQueue::Exec(JobInfo*, long long, int, bool)
    57. #7  0x007fcf36d933d8 in JobQueue::ProcessJobs(JobQueue::ThreadInfo*, void*)
    58. #8  0x007fcf36d92351 in JobQueue::WorkLoop(void*)
    59. #9  0x007fcf36fd6308 in Thread::RunThreadWrapper(void*)
    60. #10 0x007fcf357c040b in start_thread
    61. #11 0x007fcf354fae7f in clone
    62. Segmentation fault
    63.  
    64.  
    Log taken from the Command Line window output, attached to the AWS relay process via ssh.

    Also seen this:
    Code (CSharp):
    1.  
    2. Caught fatal signal - signo:11 code:1 errno:0 addr:0x400000044
    3. Obtained 3 stack frames.
    4. #0  0x007ff8382fa7e0 in funlockfile
    5. #1  0x007ff8363a2db0 in bf6fd54bf0266f5618f961bdb0b0b941_avx2
    6. #2  0x000000404f93ac in (wrapper managed-to-native) object:wrapper_native_0x7ff8363cb170 (Unity.Networking.Transport.NetworkInterfaceSendHandle&,intptr,int)
    7. Segmentation fault
    8.  
    And this:
    Code (CSharp):
    1.  
    2. Caught fatal signal - signo:11 code:2 errno:0 addr:0x7f4840401000
    3. Caught fatal signal - signo:11 code:2 errno:0 addr:0x7f4840401000
    4. Caught fatal signal - signo:11 code:2 errno:0 addr:0x7f4840401000
    5. Obtained 5 stack frames.
    6. #0  0x007f48ad6097e0 in funlockfile
    7. #1  0x007f48ad3985fa in __memmove_avx_unaligned_erms
    8. #2  0x00000040f7a01b in (wrapper managed-to-native) Unity.Collections.LowLevel.Unsafe.UnsafeUtility:MemCpy (void*,void*,long)
    9. #3  0x00000040feed24 in RCOS.NetCode.Server.Relay.RelayRoomUpdateJob:DisconnectDeferredConnections ()
    10. #4  0x00000040fee050 in Unity.Jobs.IJobExtensions/JobStruct`1<RCOS.NetCode.Server.Relay.RelayRoomUpdateJob>:Execute (RCOS.NetCode.Server.Relay.RelayRoomUpdateJob&,intptr,intptr,Unity.Jobs.LowLevel.Unsafe.JobRanges&,int)
    11. Aborted
    12.  

    Details & Setup:
    • 100% repro on Linux AWS Transport 0.3.1 (Unity 2019.4.1f1) and 0.4.1 (Unity 2020.1.6f1), usually a couple of seconds after a few hundred clients attempt to join the server.
    • The server is stable when no players are in it.
    • 100% repro with Burst enabled and disabled.
    • 0% repro on Windows, running the same stress test, but via loopback. I can reach 4k players just fine.
    • 0% repro and no errors when running the same server stress test in editor, even with JobsDebugger and Leak Detection: Full. Tested both Burst enabled and disabled.
    • The main symptom seems to be that the NativeArrays inside the job get corrupted. E.g. GetUnsafePtr returns invalid addresses.
    Relay Server:
    • It's a custom relay server built on Unity transport.
    • The relay server build creates a few hundred Driver Listeners (each on a different port), each representing a room.
    • Each "room" supports 4 players.
    • Each client sends a message at 10hz. Each relay batches incoming messages into a large array, and sends them all down to every player.
    • We also handle host designation, and communicating player identities to other clients.
    • The server cluster is running on a c5.xlarge, Amazon Linux.
    • The client cluster is a windows build running on my local machine.
    • When testing a local Windows stress test (where client and server are two processes running via loopback) no crash. No crash in editor only.
    • Thus, I believe it is a Linux issue only, or an issue only present when not using loopback.
    • I've made a huge amount of changes since my last AWS test, so unfortunately I can't tell you exactly what I'm doing differently. My relay used to be able to handle over 1k players.
    Relevant Packages:
    Code (CSharp):
    1.  
    2.     "com.unity.burst": "1.3.7",
    3.     "com.unity.entities": "0.14.0-preview.19",
    4.     "com.unity.transport": "0.4.1-preview.1",
    5.  
     
    Last edited: Oct 2, 2020
    NWalker_FGOL_Ubisoft, Oct 2, 2020
    #1

  2. NWalker_FGOL_Ubisoft

    NWalker_FGOL_Ubisoft

    Joined:
    Aug 10, 2015
    Posts:
    25
    Another 2 unique exception stacks:

    
/proc/self/maps:
    
40f8a000-40f9a000 rwxp 00000000 00:00 0
    
41bf5000-41ce5000 rwxp 00000000 00:00 0
    
55da800e0000-55da800e1000 r-xp 00000000 103:01 8409477                   /home/ec2-user/rcos-relay-server/rcos-relay.x86_64
    
55da802e0000-55da802e1000 r--p 00000000 103:01 8409477                   /home/ec2-user/rcos-relay-server/rcos-relay.x86_64
    
55da802e1000-55da802e2000 rw-p 00001000 103:01 8409477                   /home/ec2-user/rcos-relay-server/rcos-relay.x86_64
    
55da814ce000-55da82ad9000 rw-p 00000000 00:00 0                          [heap]
    
7fa9d8000000-7fa9d809a000 rw-p 00000000 00:00 0
    
7fa9d809a000-7fa9dc000000 ---p 00000000 00:00 0
    
7fa9dc000000-7fa9dc021000 rw-p 00000000 00:00 0
    
7fa9dc021000-7fa9e0000000 ---p 00000000 00:00 0
    
7fa9e0000000-7fa9e0001000 rw-p 00000000 00:00 0
    
7fa9e0001000-7fa9e0aa5000 ---p 00000000 00:00 0
    
7fa9e0aa5000-7fa9f0000000 ---p 00000000 00:00 0
    
7fa9f0000000-7fa9f1002000 rw-p 00000000 00:00 0
    
7fa9f1002000-7faa00000000 ---p 00000000 00:00 0
    
7faa00000000-7faa00200000 rw-p 00000000 00:00 0
    
7faa00200000-7faa10000000 ---p 00000000 00:00 0
    
7faa10000000-7faa1009e000 rw-p 00000000 00:00 0
    
7faa1009e000-7faa14000000 ---p 00000000 00:00 0
    
7faa14000000-7faa1405a000 rw-p 00000000 00:00 0
    
7faa1405a000-7faa18000000 ---p 00000000 00:00 0
    
7faa18000000-7faa18027000 rw-p 00000000 00:00 0
    
7faa18027000-7faa1c000000 ---p 00000000 00:00 0
    
7faa1efc3000-7faa1f347000 rw-p 00000000 00:00 0
    
7faa1f347000-7faa1f356000 r-xp 00000000 103:01 322860                    /usr/lib64/libbz2.so.1.0.6
    
7faa1f356000-7faa1f555000 ---p 0000f000 103:01 322860                    /usr/lib64/libbz2.so.1.0.6
    
7faa1f555000-7faa1f556000 r--p 0000e000 103:01 322860                    /usr/lib64/libbz2.so.1.0.6
    
7faa1f556000-7faa1f557000 rw-p 0000f000 103:01 322860                    /usr/lib64/libbz2.so.1.0.6
    
7faa1f557000-7faa1f57c000 r-xp 00000000 103:01 322838                    /usr/lib64/liblzma.so.5.2.2
    
7faa1f57c000-7faa1f77b000 ---p 00025000 103:01 322838                    /usr/lib64/liblzma.so.5.2.2
    
7faa1f77b000-7faa1f77c000 r--p 00024000 103:01 322838                    /usr/lib64/liblzma.so.5.2.2
    
7faa1f77c000-7faa1f77d000 rw-p 00025000 103:01 322838                    /usr/lib64/liblzma.so.5.2.2
    
7faa1f77d000-7faa1f791000 r-xp 00000000 103:01 322828                    /usr/lib64/libz.so.1.2.7
    
7faa1f791000-7faa1f990000 ---p 00014000 103:01 322828                    /usr/lib64/libz.so.1.2.7
    
7faa1f990000-7faa1f991000 r--p 00013000 103:01 322828                    /usr/lib64/libz.so.1.2.7
    
7faa1f991000-7faa1f992000 rw-p 00014000 103:01 322828                    /usr/lib64/libz.so.1.2.7
    
7faa1f992000-7faa1f9a9000 r-xp 00000000 103:01 322870                    /usr/lib64/libelf-0.176.so
    
7faa1f9a9000-7faa1fba8000 ---p 00017000 103:01 322870                    /usr/lib64/libelf-0.176.so
    
7faa1fba8000-7faa1fba9000 r--p 00016000 103:01 322870                    /usr/lib64/libelf-0.176.so
    
7faa1fba9000-7faa1fbaa000 rw-p 00017000 103:01 322870                    /usr/lib64/libelf-0.176.so
    
7faa1fbaa000-7faa1fbae000 r-xp 00000000 103:01 322882                    /usr/lib64/libattr.so.1.1.0
    
7faa1fbae000-7faa1fdad000 ---p 00004000 103:01 322882                    /usr/lib64/libattr.so.1.1.0
    
7faa1fdad000-7faa1fdae000 r--p 00003000 103:01 322882                    /usr/lib64/libattr.so.1.1.0
    
7faa1fdae000-7faa1fdaf000 rw-p 00004000 103:01 322882                    /usr/lib64/libattr.so.1.1.0
    
7faa1fdaf000-7faa1fdfe000 r-xp 00000000 103:01 622018                    /usr/lib64/libdw-0.176.so
    
7faa1fdfe000-7faa1fffd000 ---p 0004f000 103:01 622018                    /usr/lib64/libdw-0.176.so
    
7faa1fffd000-7faa1ffff000 r--p 0004e000 103:01 622018                    /usr/lib64/libdw-0.176.so
    
7faa1ffff000-7faa20000000 rw-p 00050000 103:01 622018                    /usr/lib64/libdw-0.176.so
    
7faa20000000-7faa20100000 rw-p 00000000 00:00 0
    
7faa20100000-7faa30000000 ---p 00000000 00:00 0
    
7faa30000000-7faa35100000 rw-p 00000000 00:00 0
    
7faa35100000-7faa40000000 ---p 00000000 00:00 0
    
7faa4006a000-7faa400ae000 rw-p 00000000 00:00 0
    
7faa400ae000-7faa400fa000 r--p 00000000 103:01 28848                     /home/ec2-user/rcos-relay-server/rcos-relay_Data/Managed/Mono.Security.dll
    
7faa400fa000-7faa400fe000 r-xp 00000000 103:01 322884                    /usr/lib64/libcap.so.2.22
    
7faa400fe000-7faa402fd000 ---p 00004000 103:01 322884                    /usr/lib64/libcap.so.2.22
    
7faa402fd000-7faa402fe000 r--p 00003000 103:01 322884                    /usr/lib64/libcap.so.2.22
    
7faa402fe000-7faa402ff000 rw-p 00004000 103:01 322884                    /usr/lib64/libcap.so.2.22
    
7faa402ff000-7faa4030f000 r-xp 00000000 103:01 622023                    /usr/lib64/libnss_myhostname.so.2
    
7faa4030f000-7faa4050e000 ---p 00010000 103:01 622023                    /usr/lib64/libnss_myhostname.so.2
    
7faa4050e000-7faa40511000 r--p 0000f000 103:01 622023                    /usr/lib64/libnss_myhostname.so.2
    
7faa40511000-7faa40512000 rw-p 00012000 103:01 622023                    /usr/lib64/libnss_myhostname.so.2
    
7faa40512000-7faa40524000 r-xp 00000000 103:01 322659                    /usr/lib64/libresolv-2.26.so
    
7faa40524000-7faa40724000 ---p 00012000 103:01 322659                    /usr/lib64/libresolv-2.26.so
    
7faa40724000-7faa40725000 r--p 00012000 103:01 322659                    /usr/lib64/libresolv-2.26.so
    
7faa40725000-7faa40726000 rw-p 00013000 103:01 322659                    /usr/lib64/libresolv-2.26.so
    
7faa40726000-7faa40728000 rw-p 00000000 00:00 0
    
7faa40728000-7faa4072d000 r-xp 00000000 103:01 322652                    /usr/lib64/libnss_dns-2.26.so
    
7faa4072d000-7faa4092c000 ---p 00005000 103:01 322652                    /usr/lib64/libnss_dns-2.26.so
    
7faa4092c000-7faa4092d000 r--p 00004000 103:01 322652                    /usr/lib64/libnss_dns-2.26.so
    
7faa4092d000-7faa4092e000 rw-p 00005000 103:01 322652                    /usr/lib64/libnss_dns-2.26.so
    
7faa4092e000-7faa40939000 r-xp 00000000 103:01 322654                    /usr/lib64/libnss_files-2.26.so
    
7faa40939000-7faa40b38000 ---p 0000b000 103:01 322654                    /usr/lib64/libnss_files-2.26.so
    
7faa40b38000-7faa40b39000 r--p 0000a000 103:01 322654                    /usr/lib64/libnss_files-2.26.so
    
7faa40b39000-7faa40b3a000 rw-p 0000b000 103:01 322654                    /usr/lib64/libnss_files-2.26.so
    
7faa40b3a000-7faa40b40000 rw-p 00000000 00:00 0
    
7faa40b42000-7faa40da8000 rw-p 00000000 00:00 0
    
7faa40da8000-7faa40da9000 ---p 00000000 00:00 0
    
7faa40da9000-7faa40daa000 rw-p 00000000 00:00 0
    
7faa40daa000-7faa40db2000 ---p 00000000 00:00 0
    
7faa40db2000-7faa412b5000 rw-p 00000000 00:00 0
    
7faa412b5000-7faa4154c000 r--p 00000000 103:01 25057                     /home/ec2-user/rcos-relay-server/rcos-relay_Data/Managed/System.dll
    
7faa4154c000-7faa4164c000 r--p 00000000 103:01 28860                     /home/ec2-user/rcos-relay-server/rcos-relay_Data/Managed/System.Core.dll
    
7faa4164c000-7faa417f3000 rw-p 00000000 00:00 0
    
7faa417f3000-7faa417f4000 ---p 00000000 00:00 0
    
7faa417f4000-7faa41ff4000 rw-p 00000000 00:00 0
    
7faa41ff4000-7faa41ff5000 ---p 00000000 00:00 0
    
7faa41ff5000-7faa427f5000 rw-p 00000000 00:00 0
    
7faa427f5000-7faa427f6000 ---p 00000000 00:00 0
    
7faa427f6000-7faa42ff6000 rw-p 00000000 00:00 0
    
7faa42ff6000-7faa42ff7000 ---p 00000000 00:00 0
    
7faa42ff7000-7faa437f7000 rw-p 00000000 00:00 0
    
7faa437f7000-7faa437f8000 ---p 00000000 00:00 0
    
7faa437f8000-7faa43ff8000 rw-p 00000000 00:00 0
    
7faa43ff8000-7faa43ff9000 ---p 00000000 00:00 0
    
7faa43ff9000-7faa447f9000 rw-p 00000000 00:00 0
    
7faa447f9000-7faa447fa000 ---p 00000000 00:00 0
    
7faa447fa000-7faa44ffa000 rw-p 00000000 00:00 0
    
7faa44ffa000-7faa44ffb000 ---p 00000000 00:00 0
    
7faa44ffb000-7faa457fb000 rw-p 00000000 00:00 0
    
7faa457fb000-7faa457fc000 ---p 00000000 00:00 0
    
7faa457fc000-7faa45ffc000 rw-p 00000000 00:00 0
    
7faa45ffc000-7faa45ffd000 ---p 00000000 00:00 0
    
7faa45ffd000-7faa467fd000 rw-p 00000000 00:00 0
    
7faa467fd000-7faa467fe000 ---p 00000000 00:00 0
    
7faa467fe000-7faa46ffe000 rw-p 00000000 00:00 0
    
7faa46ffe000-7faa46fff000 ---p 00000000 00:00 0
    
7faa46fff000-7faa477ff000 rw-p 00000000 00:00 0
    
7faa477ff000-7faa47800000 ---p 00000000 00:00 0
    
7faa47800000-7faa48000000 rw-p 00000000 00:00 0
    
7faa48000000-7faa48021000 rw-p 00000000 00:00 0
    
7faa48021000-7faa4c000000 ---p 00000000 00:00 0
    
7faa4c000000-7faa4c044000 rw-p 00000000 00:00 0
    
7faa4c044000-7faa4c045000 ---p 00000000 00:00 0
    
7faa4c045000-7faa4c145000 rw-p 00000000 00:00 0
    
7faa4c145000-7faa4c146000 ---p 00000000 00:00 0
    
7faa4c146000-7faa4c946000 rw-p 00000000 00:00 0
    
7faa4c946000-7faa4c947000 ---p 00000000 00:00 0
    
7faa4c947000-7faa4d147000 rw-p 00000000 00:00 0
    
7faa4d147000-7faa4d148000 ---p 00000000 00:00 0
    
7faa4d148000-7faa4d948000 rw-p 00000000 00:00 0
    
7faa4d948000-7faa4d949000 ---p 00000000 00:00 0
    
7faa4d949000-7faa4e149000 rw-p 00000000 00:00 0
    
7faa4e149000-7faa4e14a000 ---p 00000000 00:00 0
    
7faa4e14a000-7faa4e14b000 rw-p 00000000 00:00 0
    
7faa4e14b000-7faa4e153000 ---p 00000000 00:00 0
    
7faa4e153000-7faa4e94a000 rw-p 00000000 00:00 0
    
7faa4e94a000-7faa4e94b000 ---p 00000000 00:00 0
    
7faa4e94b000-7faa4e94c000 rw-p 00000000 00:00 0
    
7faa4e94c000-7faa4e954000 ---p 00000000 00:00 0
    
7faa4e954000-7faa4f14b000 rw-p 00000000 00:00 0
    
7faa4f14b000-7faa4f14c000 ---p 00000000 00:00 0
    
7faa4f14c000-7faa4f14d000 rw-p 00000000 00:00 0
    
7faa4f14d000-7faa4f155000 ---p 00000000 00:00 0
    
7faa4f155000-7faa4f94c000 rw-p 00000000 00:00 0
    
7faa4f94c000-7faa4f94e000 r-xp 00000000 103:01 4210062                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/libnetwork.bindings.so
    
7faa4f94e000-7faa4fb4d000 ---p 00002000 103:01 4210062                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/libnetwork.bindings.so
    
7faa4fb4d000-7faa4fb4e000 r--p 00001000 103:01 4210062                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/libnetwork.bindings.so
    
7faa4fb4e000-7faa4fb4f000 rw-p 00002000 103:01 4210062                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/libnetwork.bindings.so
    
7faa4fb4f000-7faa4fb69000 r--p 00000000 103:01 4209116                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/lib_burst_generated.so
    
7faa4fb69000-7faa4fc1b000 r-xp 0001a000 103:01 4209116                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/lib_burst_generated.so
    
7faa4fc1b000-7faa4fc1c000 r--p 000cc000 103:01 4209116                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/lib_burst_generated.so
    
7faa4fc1c000-7faa4fc1d000 rw-p 000cd000 103:01 4209116                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/lib_burst_generated.so
    
7faa4fc1d000-7faa50000000 r--p 00000000 103:01 28850                     /home/ec2-user/rcos-relay-server/rcos-relay_Data/Managed/mscorlib.dll
    
7faa50000000-7faa50400000 rw-p 00000000 00:00 0
    
7faa50400000-7faa60000000 ---p 00000000 00:00 0
    
7faa60000000-7faa60400000 rw-p 00000000 00:00 0
    
7faa60400000-7faa70000000 ---p 00000000 00:00 0
    
7faa70000000-7faa70800000 rw-p 00000000 00:00 0
    
7faa70800000-7faa80000000 ---p 00000000 00:00 0
    
7faa80000000-7faa8a800000 rw-p 00000000 00:00 0
    
7faa8a800000-7faa90000000 ---p 00000000 00:00 0
    
7faa90000000-7faa90110000 rw-p 00000000 00:00 0
    
7faa90110000-7faa90200000 ---p 00000000 00:00 0
    
7faa90200000-7faa90240000 rw-p 00000000 00:00 0
    
7faa90240000-7faa90280000 ---p 00000000 00:00 0
    
7faa90280000-7faa902c0000 rw-p 00000000 00:00 0
    
7faa902c0000-7faa90300000 ---p 00000000 00:00 0
    
7faa90300000-7faa90340000 rw-p 00000000 00:00 0
    
7faa90340000-7faa90380000 ---p 00000000 00:00 0
    
7faa90380000-7faa90388000 rw-p 00000000 00:00 0
    
7faa90388000-7faa90390000 ---p 00000000 00:00 0
    
7faa90390000-7faa90398000 rw-p 00000000 00:00 0
    
7faa90398000-7faa903a0000 ---p 00000000 00:00 0
    
7faa903a0000-7faa903a8000 rw-p 00000000 00:00 0
    
7faa903a8000-7faa903b0000 ---p 00000000 00:00 0
    
7faa903b0000-7faa903b8000 rw-p 00000000 00:00 0
    
7faa903b8000-7faa903c0000 ---p 00000000 00:00 0
    
7faa903c0000-7faa903c8000 rw-p 00000000 00:00 0
    
7faa903c8000-7faa903d0000 ---p 00000000 00:00 0
    
7faa903d0000-7faa903d8000 rw-p 00000000 00:00 0
    
7faa903d8000-7faa903e0000 ---p 00000000 00:00 0
    
7faa903e0000-7faa903e8000 rw-p 00000000 00:00 0
    
7faa903e8000-7faa903f0000 ---p 00000000 00:00 0
    
7faa903f0000-7faa903f8000 rw-p 00000000 00:00 0
    
7faa903f8000-7faa90400000 ---p 00000000 00:00 0
    
7faa90400000-7faa90408000 rw-p 00000000 00:00 0
    
7faa90408000-7faa90410000 ---p 00000000 00:00 0
    
7faa90410000-7faa90418000 rw-p 00000000 00:00 0
    
7faa90418000-7faa90420000 ---p 00000000 00:00 0
    
7faa90420000-7faa90428000 rw-p 00000000 00:00 0
    
7faa90428000-7faa90430000 ---p 00000000 00:00 0
    
7faa90430000-7faa90438000 rw-p 00000000 00:00 0
    
7faa90438000-7faa90440000 ---p 00000000 00:00 0
    
7faa90440000-7faa90448000 rw-p 00000000 00:00 0
    
7faa90448000-7faa90450000 ---p 00000000 00:00 0
    
7faa90450000-7faa90458000 rw-p 00000000 00:00 0
    
7faa90458000-7faa90460000 ---p 00000000 00:00 0
    
7faa90460000-7faa90468000 rw-p 00000000 00:00 0
    
7faa90468000-7faa90470000 ---p 00000000 00:00 0
    
7faa90470000-7faa90478000 rw-p 00000000 00:00 0
    
7faa90478000-7faa90480000 ---p 00000000 00:00 0
    
7faa90480000-7faa90490000 rw-p 00000000 00:00 0
    
7faa90490000-7faa904a0000 ---p 00000000 00:00 0
    
7faa904a0000-7faa904e0000 rw-p 00000000 00:00 0
    
7faa904e0000-7faaa0000000 ---p 00000000 00:00 0
    
7faaa0000000-7faaa0401000 rw-p 00000000 00:00 0
    
7faaa0401000-7faab0000000 ---p 00000000 00:00 0
    
7faab0000000-7faab0802000 rw-p 00000000 00:00 0
    
7faab0802000-7faac0000000 ---p 00000000 00:00 0
    
7faac0000000-7faac0402000 rw-p 00000000 00:00 0
    
7faac0402000-7faad0000000 ---p 00000000 00:00 0
    
7faad0000000-7faad0102000 rw-p 00000000 00:00 0
    
7faad0102000-7faae0000000 ---p 00000000 00:00 0
    
7faae0000000-7faae0302000 rw-p 00000000 00:00 0
    
7faae0302000-7faaf0000000 ---p 00000000 00:00 0
    
7faaf0000000-7faaf0100000 rw-p 00000000 00:00 0
    
7faaf0100000-7fab00000000 ---p 00000000 00:00 0
    
7fab00002000-7fab0009b000 rw-p 00000000 00:00 0
    
7fab0009b000-7fab0009c000 ---p 00000000 00:00 0
    
7fab0009c000-7fab0009d000 rw-p 00000000 00:00 0
    
7fab0009d000-7fab000a5000 ---p 00000000 00:00 0
    
7fab000a5000-7fab0029c000 rw-p 00000000 00:00 0
    
7fab0029c000-7fab005f0000 r-xp 00000000 103:01 136100                    /home/ec2-user/rcos-relay-server/rcos-relay_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so
    
7fab005f0000-7fab007f0000 ---p 00354000 103:01 136100                    /home/ec2-user/rcos-relay-server/rcos-relay_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so
    
7fab007f0000-7fab007f7000 r--p 00354000 103:01 136100                    /home/ec2-user/rcos-relay-server/rcos-relay_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so
    
7fab007f7000-7fab007fd000 rw-p 0035b000 103:01 136100                    /home/ec2-user/rcos-relay-server/rcos-relay_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so
    
7fab007fd000-7fab00a19000 rw-p 00000000 00:00 0
    
7fab00a19000-7fab00a20000 r-xp 00000000 103:01 322661                    /usr/lib64/librt-2.26.so
    
7fab00a20000-7fab00c1f000 ---p 00007000 103:01 322661                    /usr/lib64/librt-2.26.so
    
7fab00c1f000-7fab00c20000 r--p 00006000 103:01 322661                    /usr/lib64/librt-2.26.so
    
7fab00c20000-7fab00c21000 rw-p 00007000 103:01 322661                    /usr/lib64/librt-2.26.so
    
7fab00c21000-7fab00c24000 r-xp 00000000 103:01 25118                     /usr/lib64/libdl-2.26.so
    
7fab00c24000-7fab00e23000 ---p 00003000 103:01 25118                     /usr/lib64/libdl-2.26.so
    
7fab00e23000-7fab00e24000 r--p 00002000 103:01 25118                     /usr/lib64/libdl-2.26.so
    
7fab00e24000-7fab00e25000 rw-p 00003000 103:01 25118                     /usr/lib64/libdl-2.26.so
    
7fab00e25000-7fab00fc6000 r-xp 00000000 103:01 25114                     /usr/lib64/libc-2.26.so
    
7fab00fc6000-7fab011c6000 ---p 001a1000 103:01 25114                     /usr/lib64/libc-2.26.so
    
7fab011c6000-7fab011ca000 r--p 001a1000 103:01 25114                     /usr/lib64/libc-2.26.so
    
7fab011ca000-7fab011cc000 rw-p 001a5000 103:01 25114                     /usr/lib64/libc-2.26.so
    
7fab011cc000-7fab011d0000 rw-p 00000000 00:00 0
    
7fab011d0000-7fab011e8000 r-xp 00000000 103:01 322656                    /usr/lib64/libpthread-2.26.so
    
7fab011e8000-7fab013e8000 ---p 00018000 103:01 322656                    /usr/lib64/libpthread-2.26.so
    
7fab013e8000-7fab013e9000 r--p 00018000 103:01 322656                    /usr/lib64/libpthread-2.26.so
    
7fab013e9000-7fab013ea000 rw-p 00019000 103:01 322656                    /usr/lib64/libpthread-2.26.so
    
7fab013ea000-7fab013ee000 rw-p 00000000 00:00 0
    
7fab013ee000-7fab01403000 r-xp 00000000 103:01 25068                     /usr/lib64/libgcc_s-7-20180712.so.1
    
7fab01403000-7fab01602000 ---p 00015000 103:01 25068                     /usr/lib64/libgcc_s-7-20180712.so.1
    
7fab01602000-7fab01603000 r--p 00014000 103:01 25068                     /usr/lib64/libgcc_s-7-20180712.so.1
    
7fab01603000-7fab01604000 rw-p 00015000 103:01 25068                     /usr/lib64/libgcc_s-7-20180712.so.1
    
7fab01604000-7fab01743000 r-xp 00000000 103:01 322272                    /usr/lib64/libm-2.26.so
    
7fab01743000-7fab01942000 ---p 0013f000 103:01 322272                    /usr/lib64/libm-2.26.so
    
7fab01942000-7fab01943000 r--p 0013e000 103:01 322272                    /usr/lib64/libm-2.26.so
    
7fab01943000-7fab01944000 rw-p 0013f000 103:01 322272                    /usr/lib64/libm-2.26.so
    
7fab01944000-7fab034fb000 r-xp 00000000 103:01 8409475                   /home/ec2-user/rcos-relay-server/UnityPlayer.so7fab034fb000-7fab036fb000 ---p 01bb7000 103:01 8409475                   /home/ec2-user/rcos-relay-server/UnityPlayer.so7fab036fb000-7fab0376c000 r--p 01bb7000 103:01 8409475                   /home/ec2-user/rcos-relay-server/UnityPlayer.so7fab0376c000-7fab037a1000 rw-p 01c28000 103:01 8409475                   /home/ec2-user/rcos-relay-server/UnityPlayer.so7fab037a1000-7fab0389c000 rw-p 00000000 00:00 0
    
7fab0389c000-7fab038c0000 r-xp 00000000 103:01 25107                     /usr/lib64/ld-2.26.so
    
7fab038c0000-7fab038c4000 rw-p 00000000 00:00 0
    
7fab038c4000-7fab038c8000 r--p 00000000 103:01 608755                    /home/ec2-user/rcos-relay-server/rcos-relay_Data/Managed/System.Runtime.dll
    
7fab038c8000-7fab038c9000 ---p 00000000 00:00 0
    
7fab038c9000-7fab038e9000 rw-p 00000000 00:00 0
    
7fab038e9000-7fab038eb000 r--p 00000000 103:01 4209115                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/liblz4.so
    
7fab038eb000-7fab03905000 r-xp 00002000 103:01 4209115                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/liblz4.so
    
7fab03905000-7fab03908000 r--p 0001c000 103:01 4209115                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/liblz4.so
    
7fab03908000-7fab03909000 r--p 0001e000 103:01 4209115                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/liblz4.so
    
7fab03909000-7fab0390a000 rw-p 0001f000 103:01 4209115                   /home/ec2-user/rcos-relay-server/rcos-relay_Data/Plugins/liblz4.so
    
7fab0390a000-7fab0390f000 rw-p 00000000 00:00 0
    
7fab0390f000-7fab0396e000 ---p 00000000 00:00 0
    
7fab0396e000-7fab03ab7000 rw-p 00000000 00:00 0
    
7fab03ab8000-7fab03abc000 rw-p 00000000 00:00 0
    
7fab03abc000-7fab03abd000 rw-s 00000000 00:13 58102                      /dev/shm/mono.5354
    
7fab03abd000-7fab03abf000 rw-p 00000000 00:00 0
    
7fab03abf000-7fab03ac0000 r--p 00023000 103:01 25107                     /usr/lib64/ld-2.26.so
    
7fab03ac0000-7fab03ac1000 rw-p 00024000 103:01 25107                     /usr/lib64/ld-2.26.so
    
7fab03ac1000-7fab03ac2000 rw-p 00000000 00:00 0
    
7ffe93b11000-7ffe93b19000 ---p 00000000 00:00 0
    
7ffe9429c000-7ffe94310000 rw-p 00000000 00:00 0                          [stack]
    
7ffe94379000-7ffe9437c000 r--p 00000000 00:00 0                          [vvar]
    
7ffe9437c000-7ffe9437e000 r-xp 00000000 00:00 0                          [vdso]
    
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
    
Native stacktrace:
    
Caught fatal signal - signo:11 code:2 errno:0 addr:0x7faaa0401000
    
Caught fatal signal - signo:11 code:2 errno:0 addr:0x7faaa0401000
    
Segmentation fault



    
Native stacktrace:
    
        /home/ec2-user/rcos-relay-server/rcos-relay_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so(+0xd7588) [0x7f30346d0588]
    
        /home/ec2-user/rcos-relay-server/rcos-relay_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so(+0x5b75f) [0x7f303465475f]
    
        /lib64/libpthread.so.0(+0x117e0) [0x7f3035d3f7e0]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(+0x11c5ecb) [0x7f3037667ecb]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(+0x5730ad) [0x7f3036a150ad]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(+0x56dc25) [0x7f3036a0fc25]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(+0x8c8b05) [0x7f3036d6ab05]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(_ZN4FMOD12DSPSoundCard4readEPvPj16FMOD_SPEAKERMODEij+0x62) [0x7f3038214432]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(_ZN4FMOD6Output3mixEPvj+0x357) [0x7f30381f84b7]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(_ZN4FMOD12OutputPolled10threadFuncEv+0x3d7) [0x7f30381f9447]
    
        /home/ec2-user/rcos-relay-server/UnityPlayer.so(_ZN4FMOD6Thread8callbackEPv+0x114) [0x7f3038197b44]
    
        /lib64/libpthread.so.0(+0x740b) [0x7f3035d3540b]
    
        /lib64/libc.so.6(clone+0x3f) [0x7f3035a6fe7f]
    
Debug info from gdb:
    
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb
    
=================================================================
    
Got a SIGSEGV while executing native code. This usually indicates
    
a fatal error in the mono runtime or one of the native libraries
    
used by your application.
    
=================================================================
    
Caught fatal signal - signo:11 code:1 errno:0 addr:0x400009
    
Caught fatal signal - signo:11 code:2 errno:0 addr:0x7f2fd1001000
    
Caught fatal signal - signo:11 code:2 errno:0 addr:0x7f2fd1001000
    
Caught fatal signal - signo:11 code:2 errno:0 addr:0x7f2fd1001000
    
Obtained 3 stack frames.
    
#0  0x007f3035d3f7e0 in funlockfile
    
#1  0x007f3035ace5f5 in __memmove_avx_unaligned_erms
    
#2  0x00000041352e4f in (wrapper managed-to-native) Unity.Collections.LowLevel.Unsafe.UnsafeUtility:MemCpy (void*,void*,long)
     
    Last edited: Oct 2, 2020
    NWalker_FGOL_Ubisoft, Oct 2, 2020
    #2

  3. Kmsxkuse

    Kmsxkuse

    Joined:
    Feb 15, 2019
    Posts:
    256
    Oh my, that's quite a chunk of error logs.

    Mind also including the relevant sections of server code?

    Specifically line 117 in RelayRoomUpdateJob.cs. Null reference exception might mean it's not the network code fault but something platform specific and variable assignment.
     
    Kmsxkuse, Oct 5, 2020
    #3

  4. NWalker_FGOL_Ubisoft

    NWalker_FGOL_Ubisoft

    Joined:
    Aug 10, 2015
    Posts:
    25
    @Kmsxkuse None of the code is platform specific. The null ref is a symptom of an underlying memory issue, and the NullRef doesn't always happen at that location.

    Saying that, just to be sure:
    Code (CSharp):
    1.  
    2.         private void BroadcastLedgerToClients()
    3.         {
    4.             if (Ledger.Length == 0)
    5.             {
    6.                 return;
    7.             }
    8.  
    9.             var ledgerAsArray = Ledger.AsArray();
    10.             // NW: Don't need to dispose as AsArray does not copy or re-alloc.
    11.  
    12.             if (ledgerAsArray.Length > NetworkUtils.MaxPacketPayloadSizeBytes)
    13.             {
    14.                 Debug.LogError($"NW: Cannot send payload larger than MTU! Upgrade to transport version 4.x to fix (it supports fragmentation), but that requires Unity 2020.1.2. Payload length: {ledgerAsArray.Length}, MTU: {NetworkUtils.MaxPacketPayloadSizeBytes}!");
    15.                 return;
    16.             }
    17.  
    18.             MarkerBroadcastLedgerToClients.Begin();
    19.             for (int index = 0; index < Connections.Length; index++)
    20.             {
    21.                 var c = Connections[index];
    22.                 if (c.IsCreated && Driver.GetConnectionState(c) == NetworkConnection.State.Connected)
    23.                 {
    24.                     var writer = Driver.BeginSend(ReliablePipeline, c, ledgerAsArray.Length);
    25.                     writer.WriteBytes(ledgerAsArray);
    26.                     Driver.EndSend(writer);
    27.                 }
    28.             }
    29.  
    30.             MarkerBroadcastLedgerToClients.End();
    31.         }
    Ledger is a NativeList<byte>. The Driver and ReliablePipeline is not null when execute is called. Connections is a NativeList<NetworkConnection>, and is completely valid up until the segmentation fault.

    In other words: The segmentation fault on one thread may also cause Null References in other threads as it shuts down, which may also explain the log file being somewhat jumbled.
     
    NWalker_FGOL_Ubisoft, Oct 6, 2020
    #4

  5. NWalker_FGOL_Ubisoft

    NWalker_FGOL_Ubisoft

    Joined:
    Aug 10, 2015
    Posts:
    25
    Resolved. Ended up just being an extremely rare out of range exception:

    Code (CSharp):
    1.         private bool TryRemoveConnectionFromPending(NetworkConnection connection)
    2.         {
    3.             bool didRemove = false;
    4.             for (int i = PendingConnections.Length - 1; i >= 0; i--)
    5.             {
    6.                 var pending = PendingConnections[i];
    7.                 if (pending.InternalId == connection.InternalId)
    8.                 {
    9.                     PendingConnections.RemoveAtSwapBack(i);
    10.                     didRemove = true;
    11.                 }
    12.                 if (pending.GetState(Driver) == NetworkConnection.State.Disconnected)
    13.                 {
    14.                     // NW: Also remove other dead pending connections.
    15.                     PendingConnections.RemoveAtSwapBack(i);
    16.                 }
    17.             }
    18.             return didRemove;
    19.         }
    Missing 
    else
    on the second 
    if
    caused repeated calls to RemoveAtSwapBack for the same index. I guess it never triggered on my local stress test because connections were never "lost" to network throttling.

    So, my bad! Appreciate you looking into it.
     
    NWalker_FGOL_Ubisoft, Oct 6, 2020
    #5
    Joe-Censored and ferverence like this.