NetworkConnection.send fails suddenly!

Hi,

Hoping somebody can help me out with the following problem.

We're running a multiplayer game that has both players on mobile and players via websockets (WebGL).
The server is hosted on AWS using their GameLift service.

I'm running two NetworkServerSimple objects, one that listens for UDP connections and one for Websockets. It works most of the time, but every now and then, something strange happens and people get disconnected.

I've pinned it down to one or more Websocket connections suddenly returning false when calling NetworkConnection.send(). The strange thing is, the connections report that they are connected but still fail.

Here's the code that sends messages:

Code (CSharp):

1. for (int index = 0; index < websocketServer.connections.Count; ++index)
2.             {
3.                 NetworkConnection connection = websocketServer.connections[index];
4.                 if (connection != null)
5.                 {
6.                     handled++;
7.  
8.                     bool success = connection.Send(msgType, msg);
9.                     if (!success)
10.                     {
11.                         // This is where it goes wrong!
12.  
13.                         Debug.LogWarning("Message not sent to " + connection.connectionId + " lastError: " + connection.lastError.ToString());
14.                     }
15.  
16.                
17.                     flag &= success;
18.                 }
19.             }

LastError doesn't report anything though.

My network config is as follows:
DisconnectTimeout = 5000;
PingTimeout = 2000;
FragmentSize = 900;
PacketSize = 1440;
ResendTimeout = 2500;
AllCostTimeout = 200;
AcksType = 128;

I've also tried to Disconnect and Dispose networkconnections that fail on .Send() immediately, but this doesn't seem to do anything. They keep on being connected and .Send() just returns false everytime. After a while it seems like they do disconnect though.

What could be reasons for NetworkConnection.send() to fail suddenly? Anything that can point me in the right direction will help. I know UNet is deprecated but I'm hoping I can still use this system reliably...

Thanks!

 

I should also mention I'm using two channels:

Channel 0 - Reliable
Channel 1 - Reliable Fragmented

Channel 1 is used to send the entire game state to somebody that connects.

This can be several KB's of data. I'm splitting it up in chunks of 4500 bytes (5 * fragmentsize) and sending it over channel 1.

Does this influence other connections?

All of the other communication is done over channel 0.

 

Are you seeing any errors in the server's log, such as "no free events for message in the queue"?

 

Nothing in the log. But I think I've fixed the problem with what's described in this thread:

https://forum.unity.com/threads/unet-webgl-websocket-clients-get-disconnected-from-server.389204/

After I've increased the receive buffer it looks like this problem also disappeared.

 

Well not exactly solved.

I am now getting this error (albeit rarely) on UDP connections. Seems like all connections are dropped when this happens. Already increased the receive buffer so that's not the problem here.

I don't know for sure if there is anything in the Unity console log, I have no access to that atm since it's a headless build running in the cloud and I'm not getting all Debug.log's. I'll put these in place and report back if it happens again. In the meantime if anybody can point me in the right direction that would be great.

EDIT: I'm also quite sure it's not due too some networking overflow because I'm running a websocket server next to it which did not fail at the time it happened.

 

Ok, got some logs:

ChannelBuffer buffer limit of 16 packets reached.

I've increased it to 64. Let's see what happens.

 

I realize I'm talking to myself here but I'm against the wall. We're already live with the game and this is a big issue for us.
Right now UDP connections sudddenly fail with the following message:

Channels not initialized sending on id '0

Have to restart the server in order for this to go away.

Only errors I get before this are:

Failed to send internal buffer channel:0 bytesToSend:1252 (number varies)
Send Error: WrongConnection channel:0 bytesToSend:1252

The above two messages happen from time to time but they don't always mess up the rest. Looks like something is not getting cleaned up and no more connections are being accepted.

All of the time there is a NetworkServerSimple using websockets running alongside the server that uses UDP which is working fine and is sending the exact same messages....

Anybody?!

 

I'm also getting

lastError = WrongOperation regularly right after a client has connected. What does this mean? Seems to happen randomly.