What I learned from (successfully) building an MMO in Unity.

Hi there, my name is Kristoffer Blasiak, and I'm the CEO and Lead Programmer at Rigid-Soft.

I just wrote a document, detailing what I learned after months of hard work on a project called "Lifeless", which is a full MMO. Fully made in Unity. It's actually gotten to the state where you can actually call it a MMO, because it's just that. Not a couple of demo scenes slapped together, no it's a fully functional MMO.

The original purpose of this document was internal use, but I figured that more people could learn from it than our internal teams, so I am posting it here for anyone to learn from.

The document covers a lot of topics, ranging from what our internal server architecture looks like, to what shortcomings we found in Unity, and how we fixed them, to the tools we used for things such as patching.

I'll post the Introduction here, and if you want the whole thing, you can find it here:
http://rigid-soft.com/LifelessOverview/

 

Since I appairently can't edit my post, here's a PDF version of the document:
http://rigid-soft.com/LifelessOverview/LifelessOverview.pdf

 

I've been writing an FPS, not an MMO, but this was still a nice little read.

Thanks for taking the time to write this up! I especially liked how you tried as hard as you could to make it look like you weren't completely dissing the standard Unity networking workflow, even though you were.

 

Respect for creating a full MMO, and sharing the ride with us developers. I'm sure this will illuminate the process alot.

Thanks.

 

Thank you I was interested in what you did for networking and you gave very nice detail. Good read, bookmarking for future.

Did you guys look into other networking solutions before writing your own on top of Unity? If so, can you elaborate? Did you look into photon networking etc? Thanks

Not making an MMO myself but interested in the topic

 

Thanks! Yeah I try to be fair to Unity, the groundwork is good, but once you get into more advanced topics, it gets a little different.

Thank you .

We evaluated other solutions, and no other solution really gave us the stability and scalability we wanted.
There are other very good solutions, but we felt that the idea of having a third party developer do something as important as the networking in the game itself was a too big of a risk for us to take. So we took the hit and made our own.

Furthermore, a lot of the networking solutions are closed libraries, and we don't want to fix someones elses library, and then still have to pay a license fee for it.

As well as the simple fact that some of the things we want to do, are not supported by these libraries.

I hope that answers your question .

 

Yep that answers my question, thanks

 

Fantastic! Eventually, you may run into issues with Dead Reckoning - the bobbling, jittering, warping you get with 3-5 updates per second. If you do, you might find my chapter, 'Believable Dead Reckoning for Networked Games' (in Game Engine Gems 2) useful.

Gigi

PS - PM me with email address if you want a PDF of the chapter

 

Thanks, kind of you to share this information and it is a really good read.

 

Sent to Kindle!

 

My man!

I have been experimenting with a lot of the same approaches as you guys have been through and technologies you are using, so it's a nice verification to see others use some of the same methods. Can't wait to get through the entire document and learn from it.

 

From what I can tell you haven't even released the games alpha yet. I think that makes it a little too early to say you have successfully built an MMO.

 

The game has been in closed Beta for a while now, they are adding some polish and doing the marketing ground work. Myself and Hippo have assisted in this project, whilst 99% of the work was done by the lifeless team it was good to see how it all came together and assist in any way I could.

They have done a cracking job.

 

Allow me to rephrase, the architecture presented seems to define something that would work for a large multiplayer FPS with some persistent state. I would be quite surprised if what presented can handle thousands of real players in the same persistent world.

I see use of JSON which is small compared to XML but massive compared to a custom binary format, I see a single MySQL server. How is the data shared between game servers? Is there a distributed cache of some kind? Surely a database round trip is too slow for gameplay decision. How do players move between servers, or does one Unity Game Server impose the limit on number of players per "world"? etc

I think maybe the definition of MMO changed in the last few years to mean anything with persistent state and more than few players.

EDIT: Not to say that this isn't a great achievement, just a personal peeve of mine about what gets called an MMO, particularly with all the people selling "MMO kits" that really aren't.

 

I'm surprised there isn't a decent MMO backend kit. Something with login, database, and client infastructure already established. I know there are services like Photon, but you still have to setup the networking for it. Considering the vast amount of other assets that cover game design, I figured it would only be a matter of time before some full scale solutions pop up.

 

Yes, so it's exactly as you stated, a Multiplayer FPS with persistent state, and the game servers hold around 25-100 players each .
But please keep in mind that a MMO doesn't neccessarily mean 1000 players per server, rather the fact that you can play with hundreds of other players, together, not that the hierarchy in itself has to handle 1000 players within the same exact area.

Once you join a Game Server, you play on it until you leave, you don't jump between them. Which is how the database question you asked is handled.
It has all the normal traits of a MMO you would expect, Quests, Achievements, Groups, etc. The only difference being that you play on smaller servers.

The only thing differentiating us between a traditional MMO (from a player perspective) is that you play on a slightly smaller scale. But you can switch servers any time you want, and your items and progress will load into any server you choose. So I would still argue that you can classify us as a MMO.

I would think this is a risky path (as the person selling), also you'll end up with too many people that expect you to make most of the game for them, and they just add the fun parts. So you'll end up with a bunch of 12 year olds that rage at you for making something too complex for them to understand .

 

Perfect answer, I'm still stuck in the old school of believing MMO to mean thousands of players interacting in the same world, and I don't want to change!

 

I have to say, that I actually agree .
We wish we could make something like that, we just don't have the knowledge and resources to pull it off (which we were well aware of, so we knew the scope of a project we could pull off, and did just that).

We understood how far we could push it, and still make it. So we decided on smaller world, and splitting up players into smaller chunks, to keep it more manageable. While keeping the internal structure open to actually support bigger worlds in the future .

 

Completely disagree. A lot of complex assets are on the store, including NGUI, A*, FPS Control, and more. The point is to make assets that reduce the time and work of other developers in exchange for money. If everyone was too afraid to publish because some people might email them, the asset store would be empty.

Will kids who haven't written a single line of code expect to make WoW v2? Sure. Asset publishers don't have to help them with anything outside the scope of the asset. If nobody understands how to use the asset, then it's a poor asset.

 

@Justei do you know if some of the tips you provide in your document could also apply to Photon Unity Networking?

I am currently working on a Multiplayer RPG MOBA-style Game using Photon Unity Networking. While the Game Sessions will Only consist of probably 3 vs 3 Players, I also have tons of NPCs (mobs) in a single world (game instance). I found that the one thing that was consuming the most resources and killing our FPS was the Movement Scripts of our NPCs. We have since Optimized it, but I believe there is probably much more we could do to optimize the updates. Such as not sending messages to Players about NPCs on the other end of the world, which I think currently that's still happening. I'm wondering if we could apply some of the same techniques from your document into our game. Note I didn't fully review it only briefly reviewed it just now and thought it was interesting and something we could learn from.

By the way... When is Lifeless going to be Released? Seems Awesome! I want to Play it! or Beta Test it, Sign me Up!

 

None of those assets are even in the same ballpark of complexity as even a simple MMO framework. Development costs themselves are going to run in to the hundred of thousands. Then add the support costs of the "12 year olds" as justei put it. You are certainly going to need to run a subscription model. Even a killer product will be hard pressed to get worthwhile ROI/ROR. Not saying it wont ever happen but its hard to justify the risk vs creating many small assets.

EDIT Attached a little spreadsheet for running some basic numbers. I can't see this being viable with any kind of realistic numbers. Of course this is revenue only, but even excluding infra and support costs it looks bad.

 

Did you even read the OP article?

He did it with less than $1000.

Your argument is invalid.

Also, why do you "need" a subscription model for a framework? The point is to have a package with efficient networking, socket handling, user management, and message sending already plotted out. It's not like they're running the servers.

It's not approaching the complexity of Unity, and Unity doesn't require a subscription.

 

Ahh, sure if you don't want to pay for your time you can build anything for free.

From the article: "7 days a week, 16 hours a day (minimum) for 9-12 months" times three people...

365 * 3 * $50.00 per hour ~ $875,000

Plus the time to make it usable for the wider world, documentation, etc (which in my experience is at least double the effort).

 

What exactly are they doing for 16 hours a day all week?

They already have the framework built. What are you suggesting they need to blow through $200,000 building?

 

You asked in general why someone hasn't done this, not why these guys don't do it. I was answering from the perspective of someone who has considered it, who has a number of profitable assets on the asset store, and has a background in development of similar systems. I thought that might be useful to you.

As to why these guys aren't doing its likely some or more of the following:

a) want to build and maintain their game which will be a full time job for some time.
b) they don't want to deal with all the support questions as mentioned
c) the profitability is unproven at best

But hey, they can answer themselves if they want

- John A

PS Previous post updated with more accurate numbers.

PPS As to why you need a subscription model. Setting up the servers will be complex, allowing people to do it themselves will be both a massive barrier to entry and also require a lot of support. It also means you rely solely on income from the asset store.

The cloud model provides you an additional source of income, reduces barriers to entry, and means you have to spend a lot less time doing support for customers (you need to keep one complex system running, but you don't need to help every user keep their own system running).

Or in other words, I think it would be hard to make a subscription model viable, I think it would be much much harder to make a standalone model viable.

 

You're also coming from a perspective of someone that doesn't already have the framework developed. If you went through $900,000 developing it for your game, then it's already paid for. Adding it to the asset store isn't a huge leap at that point. Not everything on the asset store has to be built from the ground up for the asset store.

And as I already pointed out, these people did not spend $900k on it. They spent less than $1000 on the entire game. That blows your financial requirement right out of the water.

I never said it was easy or trivial. I understand the time and difficulty of building these systems. But if they already have everything done...

 

Didn't I already answer that. You asked a general question, to which I provided a general answer. You said "Considering the vast amount of other assets that cover game design, I figured it would only be a matter of time before some full scale solutions pop up". I was providing my thoughts on why they hadn't.

As to why these guys don't do so (and maybe they will), I provided some thoughts but again I will leave it up to them to deliver more details if they want to.

I would add that I also disagree with this: "Adding it to the asset store isn't a huge leap at that point. Not everything on the asset store has to be built from the ground up for the asset store."

In my experience it is quite a bit of work to take even a simple system written in house for a game to a product on the store. Particularly if it wasn't written with that in mind from the start. Now I don't think it would take them anywhere near $900,000 worth of effort, but it WILL take quite a lot of effort. Documentation alone is likely many weeks of work.

 

^this^

no doubt. just trying to create some nice tools for your own use in Unity can be rather time consuming... imagine trying to make those tools now for others.

can't blame them for wanting to hold to some kind of IP / advantage... can't blame them at all.

regardless.. was a nice write up, left a lot of open space for the mind to wonder though


I was curious though if you had looked into using the version of RakNet available to build your customizations? I briefly looked at it, but the price structure seemed far out there...

 

This is where I disagree - I expect *none* of that from a MMO, I would expect those things from an *RPG*. Still reading through your pdf

Edit:

Looks to be a good effort at building a small scale MMO, well done!

 

Had a watch of the trailer and quite impressed with what you guys have achieved. Bookmarking the article for a read through later!

Well done

 

We actually did not consider it, as we feel that pretty much anything that cost us money is outside of our price range .
Of course in the future, we can change this, which is why the code structure is the way it is, to be able to be switched out with ease.

The biggest reason as to why we would not release on the Asset Store is that we feel the profits would be too low vs the maintaining the framework. As well as the fact that all MMO frameworks are tailor-made specifically for each game, meaning that our framework would be horrible in a traditional MMO the way it is currently implemented.

Furthermore, it contains a lot of software which is outside of Unity (Node.JS, etc) which would mean that it can not be bundled in with unity in the Asset Store.

Also the obvious fact that we still work the same hours (16+ hour days) to maintain everything that it takes, adding features, bug fixing (our bug tracker is quite extensive, haha), working with press, working with companies we cooperate with, etc.
So running a asset on the asset store would not make economical sense (and for it to do that, we would need to sell it for at least 500-1000usd, just to make sure we weed out the people that would take up too much support time).

I actually don't know if Photon do any type of culling in the way we do actually :/, I don't think so .
It's actually essential I have found, and I would really recommend that to be done for any networked game.

I can't tell you an exact date, but it's really not far from release .

Thank you! I appreciate the kind words .

 

I thought this might be relevant, in case anyone's google fu brings them here:

http://blogs.unity3d.com/2014/05/12/announcing-unet-new-unity-multiplayer-technology/

Kind of crushes the entire debate we just had.

 

Cheers for sharing!

 

I would still not put my money on their new solution just yet, their solution won't be relevant for a while .
Once they get into Phase 2, I'm very interested!

 

Great tech post-mortem Justei, nicely done! G'luck with your MMO.

 

I meant in relation to "it's only a matter of time before someone creates an asset".

Kind of a moot topic since Unity is working on official framework. Though who knows how many years it will take to see the light of day

 

Cmon Johnny.. how else was he going to sell this thread and get people to read it if 'MMO' wasn't in the subject line?

Good job Justei, and great write up, thanks for the insights. Did you guys ever consider uLink and PikkoServer, extremely powerful...

 

Yea, I can't say much for PUN but Photon Server SDK does not come with any built-in interest management system. It does however have a demo for it. I wouldn't say that that is too useful though as it's not too hard to implement.

@Kuroato What you're looking for is interest management. There are many academic papers on the topic related to MOGs/MMOs.

 

Yeah I agree with ya as well. Although I personally usually flee from MMO games because I dont want to get addicted. Dark Souls and the like are too additive as it is

 

We did, and we didn't feel they met our requirements . Not that they're not good, but we had some extremely specific use cases where we needed 100% control, and did not want to rely on a 3rd party developer.

Finally I have a name for it, hah. Time to read some papers on it! I wonder if my implementation is good or not (it works well, but in comparison).