Server architecture & framework for an online game (with lobbies / hubs, missions, chat)

Hi guys,
I am about to finish my master of informatics and I wanted to use my last semester break to think about how you would approach developing an online multiplayer game. While I definitely do not want to develop an MMO and invest my next 5 years into such a large project, I still find it very interesting to think about how you would approach building such a system. Unfortunately, during my studies I mainly focused on machine learning, AI and stuff like that and therefore I have very little knowledge about networking technologies. So hopefully, you guys can give me some insights, tips or links to other interesting resources!

So, lets make this more precise and talk about the game that I envision from the users point of view:
1. you are presented a login screen, have to type your name and password as usual
2. you are presented a character selection screen and can click on your existing character to continue playing (of course you created that character sometime earlier)
3. you press on start and find yourself in let's say a town. This acts like a lobby where you can see many people (lets say 30)
4. you invite up to 3 of your friends to join your group (the friends could for example also be in another lobby / town)
5. you click on let's say a portal and you and your friends are now teleported into a mission.
6. here you play the actual game and lets say kill some monsters (only you and your friends can enter this "mission" no other players are supposed to join you)

Well, I hope you get the basic idea. I think there are multiple games that work somehow like this, having lobbies and missions only for you and your friends. One example that comes to mind is Warframe.

Now, let me present you my thoughts on implementing such a game:
To realize...
1. we need a database that stores account information like name, password a unique account ID and maybe other stuff. Then we would need an authentication server that receives the name & password, tries to find that combination in the account db and finally knows the unique account ID for the player
2. we need a second database that stores character information like level, equipment, inventory... Also we need another "master" server that has access to this character db. So I would imagine that the authentication server tells the master server the unique account ID, the master server can look up the corresponding character information in the db and finally sends the character info back to the client.
3. the master server already knows about the logged-in player. Now he has to assign him to a lobby (town). Let's imagine there is already one lobby that is full (has 30 players). So the master server now has to create a new lobby (a second lobby) where the player can join. In my opinion, the lobby (town) is a unity scene, so for that the master server has to start up a new lobby server (a new unity application that acts as a lobby server)?
4. Since we want to chat & invite friends across lobbies, we would need another server that is responsible for managing global chat rooms and chat with friends. That would be the chat server.
5. + 6 Now the friends want to play a mission. Therefore, we need a game server that actually hosts the game, controls the enemies, decides what kind of loot drops etc... So the master server would be responsible for creating a new game server and connect the friends to it so they can play.

So in total, we would need 2 databases (one for account info, one for character info) as well as one authentication server, one master server, one chat server and several servers for lobbies (servers hosting just the town e.g. a unity scene that is a town) as well as even more servers for the missions / games (servers hosting a certain mission, e.g. a unity scene with enemies and so on).
When I say server this of course does not have to be a physical server. Theoretically, everything could run on a single machine and each "server" is just an application.

So to implement the authentication server, chat server and the databases I would not need to touch unity at all. But what about the master server, lobby servers and mission servers? In scenarios like this, do you really have a master server that is responsible for dynamically starting unity applications? Like starting a new unity application that represents the lobby and starting a new unity application that acts as the mission server?

How would you approach a game like this? And maybe even more importantly, do you know any frameworks that can help to program the "lobby" and "mission" unity servers / applications?

 

It also kinda depends on which networking solution you choose, for example if you would use Unity's new NetCode then you are using ECS which has the advantage that you have different worlds in your game each running on different memory. You usually have a server and client world, but you can also have multiple server worlds in one game, that way one game instance could run maybe 10 lobby/mission servers for you.

It also depends on how your game works, maybe in the lobby you can turn off physics and then don't use unity for the lobby server, you could write it in any language and just send position/actions updates, if your game also has no real physics like most MMO's then you also dont need a unity game as server for those.

Since you are planning to have a more complex architecture I would avoid unity instances as much as possible and rather create the server applications outside the engine, that way you could have one process run multiple game "servers" via multi threading which gives you the benefit of shared memory for inter server communication. If you have different processes you would have to use pipes for communication between servers.

 

I'm by no means an expert on unity specific networking, but I have a lot of experience in server admin, network design, etc. so I'll weigh in with my general networking knowledge and hopefully someone else can fill in for some Unity specific tips.

"We need a database that stores account information like name, password a unique account ID and maybe other stuff"

Yes, and for the love of all that is holy, please, please, please salt and hash passwords before sending them over the network.

"Then we would need an authentication server that receives the name & password, tries to find that combination in the account db and finally knows the unique account ID for the player"

The *hashed* password, yes.

"We need a second database that stores character information like level, equipment, inventory..."

I probably wouldn't use a database for this, instead I'd put this info in a json text file (one for each user) and just have a link to it in the first database, otherwise your server is going to suffer from some serious lag once your user count starts climbing and it's not easy at all to scale a database, with a flat file you get more breathing room before upgrade (less resource usage to read/write to).

"Also we need another "master" server that has access to this character db. So I would imagine that the authentication server tells the master server the unique account ID, the master server can look up the corresponding character information in the db and finally sends the character info back to the client."

That's not too far off. Typically you have what's known as an API - Application Programming Interface, this is essentially a server that sits in front of all of the "business" logic. The client sends a login request to the API server which then calls the authentication server and pulls in the session cookie or JWT token (most likely you would want to use a session cookie for this application) and returns it to the client. After that, the client signs every request with their cookie to authenticate themselves. You could also return the character information alongside the session cookie if you wanted.

You probably wouldn't want to store the session cookie or handle it on the game server itself though, in fact the best practice is to not allow a direct communication between a client and the actual server where your content is being hosted, but to use an API where you can better mitigate any damage from evil users and not waste precious resources taking care of administrative tasks.

"The master server already knows about the logged-in player. Now he has to assign him to a lobby (town). Let's imagine there is already one lobby that is full (has 30 players). So the master server now has to create a new lobby (a second lobby) where the player can join. In my opinion, the lobby (town) is a unity scene, so for that the master server has to start up a new lobby server (a new unity application that acts as a lobby server)?"

Oh, no-no-no-no, no, never, ever do this, you are wasting so much storage, bandwidth, and vital server resources. You create a client-side application (whether it be in a web browser or a game file they download) and make the users handle the hard part. What the game server is there for is to manage and allow for the different game sessions to communicate securely and fairly. Basically, each player is running their own version of the game, your server occasionally sends them updates on what other players are doing and they occasionally send you updates on what they're doing. The server makes sure what they're doing is legal/allowed and if it is it sends that updated information to the other players.

If you've ever had a game lag and suddenly found your character 10' back or something, this is why, your player coordinates were too far from their last known position to be a valid move.

"Since we want to chat & invite friends across lobbies, we would need another server that is responsible for managing global chat rooms and chat with friends. That would be the chat server."

Yes, and the same goes for audio communications, I'd recommend using a service like Twillio or SignalWire to handle that. Also, a lot of this stuff (e.g. authentication) should be off-loaded to a third party provider, yes it costs a little more than DIYing it, but you're also removing a lot of the legal liabilities from yourself as well.

"Now the friends want to play a mission. Therefore, we need a game server that actually hosts the game, controls the enemies, decides what kind of loot drops etc... So the master server would be responsible for creating a new game server and connect the friends to it so they can play."

That is one way to do it, yes.

 

Really nice write up, I agree mostly with you, but in my opinion a lobby could also be a unity instance, I don't see how a headless unity server build would waste storage or bandwidth, just memory consumption would be higher, but in return you can use unity features in your lobby like physics etc, plus you can use the same networking library that will be later used for game servers.

 

Great discussion!

I have approached this in a very similar way in our Master Server Kit (UNET) and Dedicated Server Kit (Mirror) assets. Conceptually, there is a master server that is responsible for the authentication of players and the management of the game lobby/matchmaking. This server spawns independent game server instances, where each instance represents a game room. While spawning an independent binary for every game has the potential to be expensive on a large scale, it is much easier to handle than using a single process for the entirety of games (e.g., a single game crashing does not have any effect on the rest of games). The master server is also the only entity able to access the database that contains the player data.

In Master Server Kit, even the master server itself was a Unity instance. For Dedicated Server Kit, we moved it to a Go binary and made the entire dedicated server API RESTful. This allows leveraging the extensive backend development ecosystem of Go and keeps the dedicated server layer engine-agnostic (so one can migrate to non-Unity-based game servers more easily).

 

If you won't take my word for it, here's the info from a unity server build in a blank project.

That's 5.4GB total (SWAP, cache, etc.) with 300MB of RAM being used for a headless server that does (or rather, should be doing) nothing.

Compare that to a webserver built with a very bloated framework (Ionic) served by nodeJS:

Less than 1GB total with 43MB of RAM currently being used.

Now let's compare both of these to a proper API written in Python/Flask following best practices:

Less than 0.2GB total with 44MB RAM being used.

Basically, it's bloated. If you're using it for LAN parties or something like AmongUs, it will probably work, but for an MMO or any other heavy project it's not going to work well and will end up costing you a lot more money in the long run.

Don't get me wrong, I love Unity, I just don't think their server is good for the task suggested. No sense in locking up 6GB of RAM in a VPS when you can get away with 180MB in scalable containers for a fraction of the cost.

 

For sure the overhead is bigger and if you have a simple game there is no point in using unity for the server, but a lot of interactive game lobbies require a physics system and thats included in a unity server build.

 

Which is why you pay for enterprise and buy the Unity source code so you can strip out what's not needed and build your customization on top of it.

AAA titles like Call of Duty routinely get 70+ million daily active players, let's say there will be 20 people in each match, that means you need 3.5 million unity server instances, if those instances require 6GB each of memory you're looking at 2,000TB or 20 Petabytes, that's going to cost you ~$7,000 per day, $2.5M/year just to have the instances running, we haven't even gotten into all the other parts, the databases, monitoring services, the firewalls, the walled-garden infrastructure to protect the game servers, etc. the costs are astronomical.

You reduce that 6GB down to 1GB by removing the overhead, you're looking at $1,260/day or ~$460,000/year saving you over $2 million each year. It's a worthwhile investment to make if you're expecting high server traffic, for prototypes, one offs, and indie games it's probably not necessary, but if someone developed a system that integrated with Unity it would definitely be worth the $50-100 they charge for it

 

I do not how you get to 6gb per unity instance, I have never seen any of my headess server instances use anything near that.

 

It's the rounded VIRT size of the unity server build (version 2020.2.0b4.3200) on my system, that's the total amount of memory space the program is using, be it cached files, SWAP, etc. Couldn't tell you why it's so high on mine. If I don't know what that VIRT memory is, I'm making sure I have enough RAM to cache it, because you never know when the program is going to try to pull it all in, last thing you want is an OOM error in production.

Your mileage may vary, and that's okay. OP asked for opinions on how to set something up, this is just how I would do it, I have very strong opinions on overhead because I've been hit with the bill before, and it really adds up quickly

 

Hey mate, could you please elaborate on how you create your dedicated server? Do you use any libraries like mirror? Do you have a non-unity dedicated server or does it actually run unity?