Handling "server-transfers"

This is obviously an MMO subject, but I thought I'd rather raise it sooner than later, even though it's not an issue for me right now because right now I'm putting everything into a single server. But that is just the beginning ;-)

The way I understand the current networking API of Unity, it's designed in a way that only supports a single server. That's fine and simple for most cases. However, I'm wondering how one could do a server transfer in an MMO-style game (I believe that "with a few hundreds" of players playing the same game, you'll already run into these kinds of things).

Obviously, when you want to support larger amounts of players, you'll need to partition the game space so that different servers can handle different areas. I'm not even sure if that should be called "massive", yet. There's a couple of different approaches handling that, but my understanding is that in many of them, there's the need to be connected to multiple servers during the transfer. At least, that'll be needed if you try to achieve any kind of "smooth" transition.

Furthermore, there's obviously things like "global chat" that would require being connected to a "chat server" as well as a game server. Finally, there's setups where the servers also need to be connected with each other.

Would this be currently possible? Since most network related stuff having to do with connections is static, I would assume it's not... That could possibly be a dead-end in the long run.

Maybe instead of using a lot of static stuff for networking, some factory pattern could be used? That'll work smoothly for the approach that most people will use (single server, few clients), but it can scale when things become more involved.


As mentioned before: This is not something I need right now, but what I'm doing right now is aimed towards that, and I would like to not run into a dead end on the way... Doing such significant API-changes can become quite painful when there's already a lot of games using those APIs, so the earlier it is changed (if a change is needed), the better ;-)

Any ideas / comments on this?

Jashan

 

That's great news!

 

We've just started work on a game that requires this exact thing.

Our current thoughts are this:

For a persistent world, you need a database backend anyways. So when a player transitions to a different world, the Unity server stores his state in the database and tells him where to go next. Client disconnects, connects to new server it's just been told, and hands it the login credentials. New server authenticates user and loads the player objects from the database.

At this point your entire distributed computing problem has become a distributed database problem, and that's pretty much a long-solved issue.

 

Hi Tom,

That works smoothly, when you have separate worlds. It does not when you want to create one seamless world where people hardly notice they're switching from one area to the next. Obviously, the latter comes with quite a few other catches, too, and I'm not sure if all of them can be solved with Unity (e.g. loading different terrains into one level probably won't work - but I haven't looked into this, yet; also not everything has to be terrains, and with "just geometry", I don't see much of a problem with doing this completely dynamically).

It also doesn't work when you want to have people communicating between those different areas. Of course, you could put the messages to the database and then distribute from there to all the servers...

Btw, I don't think that really would require a distributed database system. It's just a bunch of clients (at the same time acting as game servers) accessing a single database. The difficult problems here are still with the many game clients accessing the game server (but I'm getting your point: it's significantly less clients per server and significantly less hassles on server transfers, so it may in the end be the best route to go in most cases) ;-)

Sunny regards,
Jashan

 

True.

Depends on the size of your project.

 

True ;-)

 

We're writing our own network layer independent of Unity's in TCP - with the ultimate end goal of a P2P decentralized server setup (with minimal central switching - but no central unity client), but backed by a database. We're using the mono libraries.

If there are a a few people working on similar external networking projects, we can start a thread and pool knowledge.

 

Hi Brian,

well, what I've been doing recently is creating my own networking on top of the Unity networking. I think except for this "small issue" with only allowing a single server (and a few minor issues), it's a very nicely designed API and implementation (I'm not using the buffered stuff, though, and neither am I using the Network.Instantiate-stuff because I feel I want and need more control on how things are handled... basically, what I'm using are just RPCs ;-) ).

So, what I'm basically doing in my project is moving the "All" and "Others" RPCModes into my own code so that I can distribute messages only to the people they are relevant to. I think that's another thing that is really missing in the current API, but I don't mind implementing that myself too much (guess it gives me a little performance hit doing a lot networkView.RPC(..., player, ...) instead of a single call into the C++ stuff, but that shouldn't be too bad... ... ... I hope ;-)

I'm happy to share any experiences I'm having with this on the forums (see, for instance, Writing own network layer on top of Unity network layer). Maybe there's some intersection of common issues we might run into (on the more abstract levels) ;-)

Sunny regards,
Jashan

 

Is it really necessary to switch servers? If you're doing this to split network traffic then you might still have issues, unless your servers are on completely different network backbones. Eg. if your total outgoing bandwidth is 10mbit/s then 10 servers won't be able to send 100mbit/s of traffic, they'll send 1mbit/s each for a total of 10mbit/s (assuming even distribution). So you're still stuck in that respect.

I think the real benefit of having multiple servers is just splitting the game logic work that has to be done, not the traffic.

With that in mind, how about this design:

Code (csharp):

1.  
2.          / Logic Server \        / Client
3. Database - Logic Server - Router - Client
4.          \ Logic Server /        \ Client
5.  

Here all clients connect to a single server, the Router server (login server/chat server omitted, would be seperate).

The Router is just a dumb server that forwards traffic to the appropriate Logic Servers, and Clients. When the player has to move to a different Logic Server the Router simply points all the traffic from that Client to the new Logic Server, while the Client maintains the same connection to the Router. Hence no need for the Client to hop servers, it would be transparent to the Client.

This also has the added benefit of not having to authenticate the player when they hop servers. Authentication is usually a slow process, which involves extra security measures, you don't want to do it every time the user hops a server.

 

Hi Dafu,

that looks like an interesting approach. How would you implement the Router?

AFAICS, the router couldn't be implemented with Unity (but that would probably also be "kind of overkill" ;-) ). The router would only have to know which client IP needs to be mapped to which server. That's probably something the router could "discuss" with the servers kind of easily.

But I'm not sure you could even properly implement the servers based on Unity in such a setup... maybe it would work - given an understanding of how RAKKNET handles these things inside of Unity, and implementing the router in a way that makes it "more or less" transparent to the server that it's not talking directly to a client but to an intermediary.

It's true that authentication is somewhat of an issue, but unless people hop between the seams (which would be a clear indication that the level design wasn't done properly), it only happens "every once in a while", while all communication would always have to go through the router. Seems like that might become a bottleneck at some point in time, and if that happens, I couldn't imagine how to solve that...

Sunny regards,
Jashan

 

From what I understand Unitys implementation of RAKKNET is currently not entirely suited for MMOs. I was thinking in terms of .NET sockets for the client, and C/C++ sockets for the servers.

The router is definitely a bottleneck, but like I said if you're on a single network backbone the real bottleneck will be the network. You'll saturate the network bandwidth before you saturate the router.

In an environment where you have multiple backbones you could have a design like this:

Code (csharp):

1.  
2.  
3.            - Logic Server-\         / Client
4.           /- Logic Server--  Router - Client
5.           |- Logic Server-| /       \ Client
6. Database -|- Logic Server-|-
7.           |- Logic Server-| \       / Client
8.           \- Logic Server--  Router - Client
9.            - Logic Server-/         \ Client
10.        
11.  

(excuse my ascii

In this scenario you have two Routers, each Router sits on a seperate network backbone, so it has its own bandwidth. All Routers can route to all Logic Servers, but one Router is only responsible for a subset of the Clients.

When the Client first connects to the Login Server (not pictured) the server tells the Client which Router it can connect to (it picks one with least connected Clients), and the Client remains connected to the Router for the rest of the session.

Also, Routers and Logic Servers are on the same local network, so we're assuming "infinite" bandwidth between them.

I'm making this stuff up as I go along, but I do have some networking experience It's definitely an interesting problem, and like many developers I am also interested in MMOs, in the back of my head at least!

 

I'm developing a multiplayer game. It's essentially a FPS (trough designed to allow you to live a while and not be killed in the first 6 seconds you enter the world) with some RTS elements (meaning you can build structures). Whole world needs to be persistent, and also I can't allow multiple servers to handle everything. So essentially I need:
* a way to switch server in case of exit or net problems
* ways to save and load world state

Essentially I just need to send and receive basic unity types like vector3, int, float.
Would like soo much to see a persistent thread towards games more P2P than intented with the current Unity networking architecture.
Ah, and thank you for all your posts, very interesting.

 

All of this is possible with native raknet, I have som screen shots of multiple servers going talking to client somewhere in threads, am posting from iPhone so can't really go into detail. Pm me if you want more info.

Edit:
found the post, see the following thread:
http://forum.unity3d.com/viewtopic.php?t=15304&postdays=0&postorder=asc&start=0

To add to your ascii art, what Unity currently really is would be

Client\
Client --Server/Client -- (anything else you want to attach to the server)
Client/