Dedicated Server: Planning and architecture

Good afternoon my fellow Unitarians!

Over the last 2 decades I have spent many years working on game engines, working with professional MMO development companies and dabling in my own indie game and asset development. It has been... going on 6 years or more since I've made a truely concerted effort to get back into game development. I generally look for pivot points in the market toolsets available for expediting and cutting development costs and lifecycle to see if it is worth coming back.

I am not a Unity novice, I did some indie work in in 5-6 years ago when it was first coming out. That being said, the explosion of assets (yes some better than others) in the community, the third party tools available and most importantly the ability to integrate with some of the most powerful hosting services (Azure and AWS most notably) have really got my attention. Any monies I can save up front I can use to pay contributing developers or buy additional content.

Based on my recent dabling in the mobile platforms, and the somewhat surprising marginal mainstream success for AlbionOnline I am going to dip my toes back into the development space and was hoping to get some answers from people much more experienced than myself on the right places to look for starting points.

I love server architecture and database design. I have been doing it professionally for over 25 years. I am a network/communications geek and my company is currently under contract to develop communications and VPN solutions for major energy sector networks across North America. What would be great is if someone could vette my old concepts, let me know if they are still on track and what, if any, changes I should plan for cloud enabled solutions.

Overview: (Not debating the merits or risks of these concepts they are simply quick and easy to develop)
- Instanced world zones
- Server side authentication
- Dedicated servers (No client side server activity of any kind)
- Account Management

Questions on Dedicated Servers: (ANSWERS: These are just what I have dug up from reading over the last day. Happy to hear disparate recommendations. I am simply filling them in for anyone else coming back with similar questions.)

1. Run out of a data center (What services are people using for indie dev startup?) Free is great, cheap just as good. ANSWER: Amazon Web Servers seems to have a fantastic offering with GameLift. I have not used it yet but will switch to this once my proof of concept servers are running and connecting locally on my internal network.) ADDITIONAL ANSWER: (See nxrighthere's post a few answers below) Scaleway cloud servers. Very solid pricing.
2. Run in non-graphical mode (Reduce CPU cycle)
   • Do you suggest Windiows/Unix with Unity and why? ANSWER: Going to setup on Windows as I am not a big unix guy but I suspect based on AWS charging for CPU, memory and bandwidth utilization I will do some testing to determine which way is more efficient. (It's all about the $$$ folks )
3. It sounds like Unity has taken some strides towards their own dedicated server APIs or am I mistaken and what are the pitfalls/benefits vs writing my own? ANSWER: UNET looks like a strong performer but is currently having a little issue with a memory leak. I am going to start working in ENet but will likely come back and visit UNet in the near future as the devs are working hard on the fix.

Questions on Architecture: (Are any of these areas no longer relevant or replaced with better process)

1. Client Authenticates to the dedicated Login Server
   • The Login Server is datacenter hosted and querries the DB server (I assume DB servers are still relevant vs localizing the DB to the Host/Login server?)
   • The Client is passed back a token and player/game information if login correct and passed to the player/game selection UI
2. Once Player selects character/game the selection is handed back to the Login Server who checks the Player token is correct (other security checks here like last IP etc) and then pulls the Instance host data for the characters zone and passes that back along with an access token that the Client application then uses to make a connection request for the account to that instance host
3. All actions are server side authoritave meaning when the client executes and action (example amovement or an ability) it will start to happen on the client (as in start to execute the animation) but the request to the action is sent to the instance server which replicates it across all connected clients within the sphere of influence. (Within range of graphical viewing or sphere of influence)
   • Example of this is players hacking the local client to equip the 'Archangels Staff of Hellfire'. Good for you little buddy, but all the server sees is you equiping the item from inventory slot 121 which the server checks in the data base and sees is a 'Cracked Staff' and your stats and abilities are set accordingly.
   • Example 2, Client tries to execute an ability they cant perform or hack a cooldown on the client side. Good for you little buddy but all the server sees is that you are trying to execute Ability.XYZ , checks Ability.XYZ.isReady, sees it is still on cool down and sends you the appropriate 'DENIED!' response.
4. Are there Unity based native tools now for 'Operations Monitoring' of dedicated servers or do I need to write my own special dashboard tool that connects to the Login/Master server to handle providing an overview of the game's current activity?
   • Think things you might see in a games Network Operations Center
     • Currently connected Users
     • IP Bannings/Blacklistings
     • Currently running instances
     • Highest concurrent user count in last 24 hours / last 7 days
     • Instance names
     • Population of Instances
     • Instance lockout buttons
     • Instance shutdown buttons
     • Graceful/Immediate server farm shutdown controls
     • Player management tools (Banning/Unbanning)

Questions on Middleware and Tools:

1. Any recommendations on more robust networking middleware for instancing 20-100 players in a zone? Is Unity networking truely strong enough to handle these types of heavy load, high responsiveness? Last I took a stab at this was with uLink and it was a great product.
2. Any recommendations on how to handle repositories. I used to use LAN installed repositories on my internal network but to enable collaboration with developers around the globe seems like code handling is better handled in the cloud. Free is best, cheap is great. Pros and Cons of your recommendations?
3. Any recommendations on hosting services (I know AWS was offering a free development platform for a while) and why you recommend them?

Thank you in advance for your patience in reading my long winded post and even more thanks for any contributing recommendations or corrections you can provide. All the best to you in your creative journies!

 

Welp based on some light reading tonight it looks like Amazon has some very solid game development offerings and I work with AWS professionally so seems like a good fit. HAvent been able to get along with MS data centers professionally for some years.

Sounds like there are no out of the box tools for server status/NOC monitoring so I'll write my own. (Would love some insite on how others have done this.)

Some thoughts on my server clusters:
- Login Server: authenticates accounts, creates/modifies accounts, presents player character information/creation options. Talks directly to the DB (although could be done through the master server) for authentication and account updates.

- Master server: (there are many flavours of how to do this, this approach is just how I have done this in the past. Really looking for insight on this) Master server keeps track of all of the instance servers health, spins up new instances when requested from a client hitting an unpopulated zone. Master Server talks directly to the DB server for authenticating clients to different instance servers.

- Instance Server: console based server that loads the instance for a scene and handles all ServerSideAuthentication requests and data replication among connected clients. Instance server is called by the Master Server when a client is in proximity to trigger a zoning event. In the past I have handled this by placing a trigger area within 10 seconds or so of a zoning point that confirms the next instance is running and active or calls to the master server to spin it up incase the client zones.

I am going to start writing the scripts for the login and master servers tonight. I am rusty so I am sure it will take a while. If anyone knows of any good tutorials on how Unity 2017 is best used at calling new "rooms", "Instances" or whatever the hell they are called now Feel free to post

Thanks in advance for any suggestions.

 

For anyone finding this thread later I am switching to ENET based on a current mem leak with UNET but the devs are very much aware and working hard on the leak. Just FYI, once fixed UNETs performance seems quite impressive. Likely I'll switch back. ENET has a super simple connection protocol which will make it super easy to call instances. So right now I'm just moving ahead.

ANSWER: ENet has a crystal clear tutorial on how the calling is done here: http://enet.bespin.org/Tutorial.html

I just have to figure out how to call a scene specific config file for the instance server with all of the dynamic details about the scene.

 

Just want to warn you that even if ENet works great, some fixes and improvements are still needed for such project.

 

Thanks nx. I figure most of the transport middlewares will require some tweaking. I am working under the goal of getting a proof of concept server farm running and then I will start doing some actual testing. Your work on the performance stress testing was just awesome to see. So many people dont appreciate the nuances of the network layer. I almost messaged you the other night with some questions but I will hold off those kinds of conversations until I actually get my farm up and communicating and some basic toolsets built.

Thank you for the heads up. Much appreciated.

 

Sure, drop me a message if you want to ask something.

Since you already found the answer to the first question about middleware and tools, for the last two I would recommend Fossil (you can use Chisel for it for free) and Scaleway bare-metal servers, it's very cheap.

 

We made an custom dynamic instance opened and closed by Master server dynamically depending on user requests and master server verifications. We find that it is very useful when for UNET memory leak to allow master server to dynamically open and shutdown instance, therefore we could give an instance a hour of usage, then not allow any users to go into the instance, and the instance will shutdown after it is empty for awhile. So that's current work around.