Data Driven Development within Unity

I think it would be great if some folks could contribute techniques and ways to do DDD within Unity as looking around on the forums I didn't find much current about it.

So that we are on same page the definition for DDD I am using is where the object data and gameplay logic is stored external to the game engine, usually in flat files.

For my current project I am using Moonsharp Lua which is a C# implementation of Lua to load and process my data files. Keeping the interpreter in C# allows me to avoid the interop overhead of just embedding a native Lua VM. My primary driver for going this route was two fold:

- Moddability: Anyone can take my Lua files and spritesheets and modify or replace them, effectively modding the game. Since I am striving to decouple all gameplay from the engine they can effectively make a completely reskinned/different game easily.
- Portability: My eventual goal is to take this same gameplay layer and port it to a multiplayer framework that runs on a server. Since all the logic and objects are defined in Lua the only piece that has to be changed is the engine.

I'm in the early stages of this but I think it has promise. If you can see pitfalls or better ways of doing this(that don't involve users modifying the actual project) I am all ears.

 

I've been doing something similar in my projects. The biggest problem I've had, is opening up the Resources folder for modding - the Resources folder gets 'compiled' by Unity, which prevents users from being able to easily get to the assets in it, to make their own mods.

I haven't been making softcoded behavior via Lua or anything like that, instead I've been hardcoding behavior, and creating JSON 'scripts' that define objects, allowing me (or a modder) to compose specific game objects. This really helps me iterate my digital games quickly (I did this back on the Chicken game, probably my biggest success with this sort of technology).

 

This is the approach I took in my previous project which was a roguelike and it worked well. I am trying to take it a step further now and move the game logic into the data layer and for that I needed a real scripting language that can be interpreted or JIT'ed at runtime, thus the move from JSON to Lua.

edit: removing my example because I am not sure if I would do pathfinding in the engine or not yet. Things like input, graphics, etc. definitely would all have hooks.

 

One thing I know I haven't experimented with yet is the asset streaming setup. I think if I did that I could make an 'easy' modding setup, provided I make my framework amenable to that sort of thing.

 

You know I've never tried streaming assets folder. I'll give it a shot and see if it behaves like I expect and post a snippet here if it does.

 

StreamingAssets looks like a good place to store the files, thanks for that tip. Just did a quick proof of concept and works great.

Assume two lua scripts, in the one defined in code a = 1, in the one defined in StreamingAssets b =2.

Code (csharp):

1.  
2. using UnityEngine;
3. using System.Collections;
4. using System.Collections.Generic;
5. using MoonSharp.Interpreter;
6.  
7. public class LuaBridge : MonoBehaviour {
8.  
9.     // Use this for initialization
10.     void Start () {
11.  
12.         // Inline example
13.         string scriptCode = @"    
14.           a = 1";
15.  
16.         Script inlinescript = new Script();
17.         inlinescript.DoString(scriptCode);
18.         DynValue helloworld = inlinescript.Globals.Get("a");
19.         Debug.Log(helloworld);
20.  
21.         // Loading from StreamingAssets subfolder on desktop
22.         string modpath = "<my mod directory>";
23.         string luafile = Application.dataPath + "/StreamingAssets/" + modpath + "/helloworld.lua";
24.         Script.DefaultOptions.ScriptLoader = new MoonSharp.Interpreter.Loaders.FileSystemScriptLoader();
25.         Script externalscript = new Script();
26.         externalscript.DoFile(luafile);
27.         DynValue helloworld2 = externalscript.Globals.Get("b");
28.         Debug.Log(helloworld2);
29.     }
30. }
31.  

As expected it prints out 1, then 2.

 

I use the streamingassets folder and JSON files to store the data as well as wav files, textures, etc.

I would love to use lua to handle things such as movement, weapons, etc but I think it would be alot of effort.

 

The main concern is to decouple unity I/O with a good interface for User I/O and rendering/physics. There is a big difference between communicating with a set of utility classes, which effectively define a specialized/abstract subset of unity or to expose everything you can do with the engine.

That is why this discussion is interesting though. Personally I use the DDD concept to some extend at least, by handling (almost) all sideffects event based through FMSs and by exposing all custom configuration variables of my game objects. It is then a small step to move the statemachine structure and config vars to a different place and only keep the state transition and frame update code in the gameobjects. An external statemachine, for example on a server, could then directly invoke the state transitions (which are now exposed through a more generic interface). But I don't know if that is the way you guys are doing these things or if it isn't enough flexible that way.

 

Oh I am not suggesting exposing everything you can do with the engine. For something like that you would probably be better off just allowing user compiled .NET assemblies. I like Lua because it allows me to sandbox certain libraries in the language(don't want users distributing mods that could be destructive outside of the game folders, etc.)

The only parts of your Unity scene you need to expose are dependant on your game, you basically have to make an API for the game that you then build it upon. Ie. if you need to change out a tile's graphic you might have Game.setTileSprite(tileid, "newsprite.png") which calls some Unity C# code with the correct calls to the Unity 2D system.

If you move this Lua code to a different platform you just have to rewrite that setTileSprite() function in whatever engine you are using to do the right thing and can leave your Lua untouched.

 

I was trying to imply that. Exposing the whole engine was more like an extreme example to showcase the kind of the decisions that go with an architecture like this.

I have a question to you though: Did you ever think about generating C# code instead of interpreting from lua scripts directly? If you have a good API for your behaviour then this might be a feasable solution because you could use a purely declarative language (or DSL) to generate Unity classes directly, or server code.

 

Nope, I have not. Assuming you can transpile your DSL to C# code, the user would still need to compile it themselves correct? Then the game would load it as a runtime assembly?

 

Ok I know the problem here haven't thought this through. You would have to design a workflow which is secure in that regard, so generate it during runtime so the user never actually sees the generated code. I don't know if unity scripts can be hotdeployed into a running game though. It would surprise me actually if this would be possible. There might be a solution for this but I clearly didn't think this through since iam comming from an open source background where things like this don't matter.

edit: note that iam a new unity user.

 

I have seen some games like Cities: Skylines allow you to compile a C# program into a DLL which you can then include.

See this page: http://skylines-modding-docs.readthedocs.org/en/latest/modding/Getting-Started/

Seems pretty slick and they provide an interface much like we were discussing. Personally I think it is a very heavy way to do modding, making users fire up Visual Studio and compile it themselves but definitely would give the most flexibility, if you are not concerned about sandboxing the environment.

There is nothing to stop me from making a Cities Skyline mod that goes and deletes all the files on my hard drive for instance. Unless I am missing some control they have.

 

Just did abit of googling and apparatently another possibility would be to generate unityscript (what they call javascript but it isnt really) and use eval() at runtime. This way you, the end users and the modders would be safer from user errors or even hacks.

A thing that would need to be tested is the performance of eval() though. I have no idea how much it would take unity to eval() and you would need to do that every time you load in a mod.

In the case of interpreting Lua I would be especially concerned about the worst case which is malicious modders: giving a modder turing completeness, file system access yada yada in the case of letting them inject lua into your game. But I'am not very experienced with modder communities and secure modding solutions to be honest, so handling everything during runtime seems to be a good way to do this in any case. I assume that there are solutions for containing your Lua scripts since it is a language designed for embedding.

 

Lua is meant to be embedded so they have robust sandboxing to prevent stuff like that. Moonsharp Lua which is what I am using right now has full sandboxing support: http://www.moonsharp.org/sandbox.html so you can restrict access to things like filesystem, os, etc.

I am a bit confused sometimes when I see things like Cities: Skyline which allows full C# access because it means any user can create a malicious mod with full access to your computer. My field is network security so having a sandboxed interpreter is a must have for me.

I don't want to have to inspect all the mods for my game to make sure someone isn't doing malicious stuff. If they want to blow up the game that's fine, but there needs to be a boundary for anything that has my name attached to it.

 

Then your approach seems to be very straight forward and secure. Lua is also an accessible language so thats a big plus as well.

 

Yeah but it is just one approach among many. I am genuinely interested to know other approaches people might be taking and their experiences with them.

 

Can you explain why not? What kind of extermism can exposing all Unity API cause (considering we sandbox the code)?

 

The point is that you are basically just have a 1:1 mapping of what is already there, so the only thing you gain is a wrapper for a different language. For an API to be worth a damn it needs to be an abstraction layer. Else you could just use Unity itself.

One of my favorite books about programming in general, Structure and Interpretation of Computer Programs (its free), has a very good discussion about this (allthough I recommend anyone to read the whole book because its building up step by step):
https://mitpress.mit.edu/sicp/full-text/sicp/book/node29.html

My only two experiences worth mentioning are generating code from a declarative language which we discussed already and generating statemachines, but in this case not directly code but building them in runtime by composing state transitions based on data:

I feel like FSMs are a very good data model for virtually anything you want in most types of games. I generally try to enclose all event based sideeffects into state transitions. When the transitions are predefined then it is straight forward to generate the machines from a simple data structure like a JSON. The transitions are basicly the API of the system and the machine descriptions the data. This is easier done with a language which has a good support for first class functions and I'am not yet convinced of C#s "delegates" as I'am a new user of the language and they seem funny to me at first glance. The not event based but frame update based sideeffects could be handled in a similar way, by just defining the ones you need basicly. But since C# doesn't see code as data as much as other languages do I thought generating classes would be the way to go since that is in my experience the way how these things are done in Java which is similar in that regard.

 

How can one use Unity itself for this purpose?

 

I don't even understand the question. You just, you know... use it.

 

I haven't been able to get any lua require() calls to work from the StreamingAssets folder, has anyone accomplished this and would they mind posting an example?

Cheers and thanks.

 

I suppose my own project is an example of this. It is a turn-based "grand strategy" game (Diplomacy-based). Unity is just the front-end client -- display and input. The data is stored in SQL Server, there is an IIS HttpHandler layer of REST-style services which includes a homegrown database management and ORM library, and the client and server both share a set of libraries for reading and representing the game map files, caching database content for the current player and game (no direct DB dependency, so the client can use the same DLL), and several helper libraries as the rules are pretty complicated and often require reviewing the outcome of several previous turns. Oh and there is the order-processing ("judging") engine, which runs server-side but is also sufficiently stand-alone it could really run on the client, too. Client/server communication is about 99% HTTP header values for requests and maybe 75% JSON for the response (and UnityWebRequest is a real mess). Architecturally the whole thing is a pretty typical design for a distributed multi-tier application.

This is DDD in the sense that the map files (and some associated texture assets) are completely replaceable. One of the holy grails of Diplomacy implementations is supporting more than just the standard layout the original 1950's board game offered. It's hard to do because the rules are so complex and there are many map-specific special-case rules (which I have managed to represent abstractly through my map file format). The map reader knows how to load and represent maps in general. The order parser and judge know how to process orders based on the relationships of regions and powers described in the map file. The client knows how to set up different game types and numbers of players based on what the map file says that map can represent. Overall it's a pretty strict "separation of concerns" approach that keeps such a large and complicated system manageable by one guy.

I'm not sure yet if I'll allow just anyone to produce maps for it. There are complicated requirements for getting maps and their textures to work together correctly, and my map editor utilities are definitely "in-house" quality and not especially end-user-friendly. It can take quite awhile to debug a map (making sure adjacencies are correctly described, etc.) A single map file representing Western Europe (like the original Diplomacy board game) is over 1,000 lines of text data and requires six 4K textures, several of which are data stored in RGBA channels rather than "visual" textures.

I've also thought about building a plugin interface. Diplomacy is notoriously difficult for computers to play (the number of possible opening moves are many, many orders of magnitude higher than chess, and obviously the player-to-player negotiation aspect complicates things) but there is a small group out there who build Diplomacy AIs and play them against one another. I'm not sure it would be worth the effort apart from just being an interesting project to attempt.

Back when .NET was new (I think they were still calling it NGWS 2000) Microsoft had a sandbox server out there which let people upload .NET DLLs that implemented some sort of "ant farm" AI. It was something I always meant to play around with but never made the time before it went away. Awhile back I tried to find references to the project but I haven't had any success. Sandboxing isn't easy but it has been possible in .NET going back to the very start.