Discussion Saving Data for LARGE Games

Hello guys, I've been wanting to start an MMO for a while, now obviously that's a HUGE project with TONS of data that needs to be saved. This will be my biggest project yet. As of right now for my games, I have used the Binary Formatter. I'll probably get roasted for saying that but my games have nothing that needs to be secure that's why I use it (simple things like coins or wins). My question is, what are some good ways to save a LARGE amount of data like that? Having a variable for every little piece of data seems like too much. It would be cool if I could almost just save a giant picture of the game.

 

Well, I‘m sorry to say this but when you have to ask this kind of question but you want to make an MMO (comparable to even the most limited title that carries the MMO tag) then you have to be thinking in terms of 2030-ish regarding a release date. If ever. And yes, of course, everything needs to be as secure as possible in an MMO. Even something as innocent as player chat because you wouldn‘t want the hacker to change little Annie‘s texts into offensive, rude language for everyone else. Etc etc etc

Having said that, it never hurts to try and learn as much as you can on the way. The question you are asking within the scope of an MMO knows only one answer: a database. And a server (search for „backend“). And clients that map all this data to code (search for „ORM“ as in object relation modelling). Of course you‘ve got to pay for the servers and the traffic.

 

MMOs are big but you can make one that is reasonable for a indie dev it doesn't have to be like the MMOs made by large companies a indie dev can make a smaller one to. Servers are reasonable for the amount of players a small indie dev is going to have like me. By the way to store data for larger games you didn't really answer me, no offense, I was more talking about maybe how to save it because, for example, JSON wouldn't be good cause I can just go in and change a file, binary formatter has pretty good security cause it's just a bunch of scrambled mess, but what I was wondering is, right now I have to write a string for every single data wins.dat coins.dat, I was wondering if there's a easier way, cause in a mmo that seems like to much.

 

Sigh. No, wrong, incorrect, bzzzt.... Binary data is no harder or easier to change than text data. In fact, it's the SAME THING. It's data. It's on the disk or in memory. Seriously. I don't know where these crazy ideas come from. Data on a computer you do not control can be changed by someone else. End of story.

If binary data was hard to change and hack, operating systems would be 100% secure. Instead we have thousands of vulnerabilities discovered every week. Most of those vulnerabilities involve changing data that the original designers thought "Oh nobody will be able to change THIS data, I've secured it as binary!" And they were WRONG every single time.

There's nothing special about "Data for LARGE Games" versus "Data for small games."

It's just data. Ones and zeroes. The only purpose of your program (and all programs) is to transform data from one form to another. Get to it! Make your game! If you're balking at the data structures involved, focus on learning how to manipulate that data until you're comfortable with the process.

Load/Save steps:

https://forum.unity.com/threads/save-system-questions.930366/#post-6087384

An excellent discussion of loading/saving in Unity3D by Xarbrough:

https://forum.unity.com/threads/save-system.1232301/#post-7872586

When loading, you can never re-create a MonoBehaviour or ScriptableObject instance directly from JSON. The reason is they are hybrid C# and native engine objects, and when the JSON package calls

new

to make one, it cannot make the native engine portion of the object.

Instead you must first create the MonoBehaviour using AddComponent<T>() on a GameObject instance, or use ScriptableObject.CreateInstance<T>() to make your SO, then use the appropriate JSON "populate object" call to fill in its public fields.

If you want to use PlayerPrefs to save your game, it's always better to use a JSON-based wrapper such as this one I forked from a fellow named Brett M Johnson on github:

https://gist.github.com/kurtdekker/7db0500da01c3eb2a7ac8040198ce7f6

Do not use the binary formatter/serializer: it is insecure, it cannot be made secure, and it makes debugging very difficult, plus it actually will NOT prevent people from modifying your save data on their computers.

https://docs.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide

 

I don't need to have secure data, i'm just saving things like coins and wins, it's not like i'm saving somebody's credit card data.

 

May be kinda weird but it's not like I can go and just change data on a large game owned by a big company.

 

You say this now...

... and yet a few hours ago you were concerned:

So you can be concerned or not, that's up to you. It's irrelevant.

Just know that ALL data on another computer is NOT secure.

 

Wat?! Why not? Have you heard of the game modding community? Have you heard of Cheat Engine?

 

Yes, but it's not easier, it's THE way to go when working with complex data sets.

First, you have to decouple that data from both storage and presentation/interaction. For a reasonably simple use-case like yours and assuming you have to work with a database that's accessible only to the server (for security & cheating reasons) you need to draft an architecture. Here's my thoughts on it:

• Server-side
  • Layer to read/write data to database
  • Client GET/SET layer to handle networking events
  • Data input validation - return errors if anything is fishy or passes data on to storage layer. You can assume that data FROM the database is correct but it might not be in case the database itself is corrupt, incorrectly versioned or got hacked.
  • Account sign-up with email address or other means like OAUTH via Google, Apple, Facebook, etc.
    
  • Client connection handling - is the client allowed to connect? account banned? not paid for? is client running a compatible version of the client? etc etc.
• Client-side
  • Server GET/SET layer to handle networking events
  • Presentation/interaction layer that references serializable data and all other requests sent to server (account creation, avatar creation, avatar stat updates, etc.)
  • all the usual client stuff (ie how the game looks and feels)
• Common layer
  • Data sets/tuples - for instance in the simplest case a struct containing: PlayerId, WinCounter, CoinCounter - but it may get more complex so you may want to have: Account information
  • These datasets know how to (de)serialize them to and from networkable data formats (ie JSON, binary, whatever).
  • A utility that wraps that data and makes it secure by adding a hash/checksum and possibly (de)compressing data sent over network.
• Database Model
  • Tables as follows:
    • Account: contains all information required to identify a player ie email and salted hash of password (NEVER store passwords directly, always get the hash with salt!)
    • Avatar: contains all infos related to account's Avatars (in case there can be multiple), in your case Coins, Wins
    • Ideally increase granularity of the data, ie Coins, Wins could be their own tables if you decide to add more information to them like for instance timestamps and what not.
  • ORM framework that eases the transformation of data stored in database vs data stored in memory. ORM also helps prevent making stupid mistakes.

Oh yes, you've got to have a test and a live server, too. And means of upgrading the live server from the test server (deployment), which means a way of putting the server temporarily offline, be able to test the live update yourself or with selected users before going live again. While the client handles all these messages and responds with "scheduled maintenance, we'll be back in x hours".

And

so

on.

And yes, you've got to have all this and certainly more if you want to call it an "MMO" that works like users expect MMOs to work at a minimum (of course you'll also have server crashes or overloads and need an action plan how to respond to those quickly even while you're asleep, on vacation, without Internet).

 

Ok thanks.

 

If all data is not secure then what's so bad about Binary Formatter, that means JSON is insecure as well.

 

Binary formatter makes the entire user's computer susceptible to malware. There's a full discussion available at the above links.

 

Ok valid point, but with JSON the file I bielieve has a ID of some sort and then a value of some sort so like 2, whereas binary formatter is just a mess in the file. A normal user would easily be able to overwrite a JSON file, however a normal user who isn't really techy wouldn't have a clue on how to change a binary file. Is there any alternatives other than json?

 

You talk about making a MMO but you want to save the data on the clients' computers?

 

I don't know to be honest, that's why I was asking this question. Obvoiusly people can cheat all the time, but a normal user shouldn't be able to go on their computer to change the data.

 

In a MMO data is saved on the server, usually in a database, not on files on the clients' machines.

 

Thanks.