Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

powered by ECS: Spuds Unearthed - a mix of RTS/TD and god sim now in Early Access! (VR game)

Discussion in 'Entity Component System' started by mmankt, Jan 10, 2019.

  1. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49


    Hi!

    I am the lead programmer on Spuds Unearthed battle module. Our game studio is called Gamedust and we are based in Poznań, Poland.

    Wanted to share some info on game, our engine and thoughts on using ecs/jobs since we've just entered early access.

    The game is divided into two modes. The core of the gameplay is the battle module (our codename is ‘combat’) which is a mix of real time strategy, tower defense with a bit of shooter elements, all in VR.

    The second part is your home base where you build your army, upgrade your units exchange 'gold' on tokens which let you produce units/turrets etc.

    The whole game is set in a living galaxy of planets to conquer. It's basically an async multiplayer game where players conquer planets controlled either by zombie units or other player's defense systems.

    The player has many different means of battle at his hands.

    You have the titular Spuds that act as heroes in our game. Each Spud can be upgraded into a special class like a pilot, driver, heavy etc. Each class has a unique basic and special attack and a specific role on the battlefield.

    Spuds can also use vehicles. Right now we have a plane and a tank. Any spud can use them but a pilot and driver will add special abilities to your vehicles.

    You can also use turrets with unique VR interactions (anything from a regular revolver to a minigun powered by a crank ). The player can build those turrets by combining the base functionality with a firing mechanic - for example, you can attach a pump from a flamethrower to a minigun and it will turn into a shotgun etc.

    We also have a cannon fodder type of unit - small autonomous Goonbots that are strong in masses. The whole goal is to use all of the provided means of war to conquer planetary systems in our two game modes: Infestation and Conquest.

    In Infestation, you must fight on planets controlled by Zombuds - a kind of corrupted spuds. In all of the modes the goal is to destroy the enemy defense system (base) on the other side of the map. Once a planet is conquered it becomes yours and brings you money. Other players can attack your players and take them from you. This is when the second mode comes into action - Conquest.

    When you attack a planet which was conquered by an another player, you fight with an ai controlled army that the he had left there during his previous battle. Each planet has a fortress which will use artillery fire, deploy tanks, planes and goonbots. Each planet gives you more playdust (our gold/money ;) ) which lets you travel to other planets and to build new units.Once you've conquered a system or you want to try fresh, you can travel to a new one. The ultimate goal is to be the best ranked player in the whole galaxy.

    As the game is in early access we will be adding many new features such as player ranks, new vehicles, spud classes, unit upgrades, traps and new turrets.

    The combat module was developed by a self contained 3-man team and it’s code base is a whole separate module. We get all the crucial info that is needed to start a battle from the server and we just run our simulation.

    The core features of our engine are:

    The simulation is deterministic, lockstep, we use fixed-point math. (because of that all of the code in the sim is custom.)

    All the ground units are simulated in 2D. (gameplay is fully 3d).
    We have a custom 2d physics system (OOP) and a simple 3d physics system (turret projectiles and all the units you can throw into battle) (ECS). Some units have both 2d and 3d colliders.
    The terrain is represented as a height map for view and 3d collisions.
    We use influence maps for ai.
    Pathfinding is done on a prebaked flow field. We also have a dynamic, cost based pathfinding system.
    All of our data like heightmaps, walkable areas and flowfields is stored as a 1d grid.
    The simulation is separate from the view. The view can’t influence the sim.
    One monobehaviour as an entrypoint, everything in the simulation is pure c# in simple OOP systems-based architecture and pure ECS for all the units, ai, projectiles etc.
    We built custom tools to sort system execution order (posted it here), bake map data from unity to our fixed format and to bake flow fields.
    The player can influence the sim via inputs. So for example the player will take an object in VR and throw it - this will create an input command with all the needed data that will create a proper Entity with just the view ‘re-parented’ to it. All inputs are stored and serialized so they can be used to watch a replay of the battle (replays not implemented yet).
    We connect to the view via an index reference. We have all of your view objects stored with a proper id. Once the sim is done we collect all relevant flag components in view systems and send proper events with the view id in them. Everything that doesn’t need a skinned mesh or mono controllers for the view is rendered via an instanced rendering system (so positioning, rotation etc is done inside ecs).

    A lot of our solutions were inspired by Homeworld: Deserts of Kharak talk on Unite <3 and obviously tons of other rts engine materials from starcraft 2 etc.

    We wanted to be fully jobified but in our first tries we found that it was breaking our determinism. We also aren’t using burst at this moment btw. Since we were on a really tight schedule we had to focus on making everything work with the knowledge we had at that moment. We will definitely improve a lot of aspects in the future.

    We’re using Unity’s ecs right from it’s first release and our biggest problem was lack of documentation and the constant changes. We had to look through samples to see what kind of features we can use or how to use them.

    We decided on using ECS since it’s a perfect match for a rts with lots of units and we wanted a high performance engine right from the start in order to support a lot of units in 90fps stereo.

    From performance side we can have a battle of up to 1500 goonboots (single thread) and it runs at 300 fps in our tests. Obviously pretty graphics bring it down but right now we support pretty intense battles and we are very happy with the results.

    I love using ECS and even if it takes longer to develop some simple features it’s really great in the long run. We’ve learned a lot over the course of development and right now we’ll focus on speeding up the engine in crucial places to support even bigger battles. I’m working on some side projects in ECS and experimenting with fully jobified solutions. Really looking forward to the future of ECS. ( DOCS/summary in code PLZZZ! )
    Sorry if the engine description is a bit too chaotic. Just wanted to point out all the important aspects. I’m looking forward to any questions.

    Some fun job based experiments:



     
    Last edited: Jan 11, 2019
    GilCat, NotaNaN, Rewaken and 10 others like this.
  2. Singtaa

    Singtaa

    Joined:
    Dec 14, 2010
    Posts:
    492
    Awesome job! I'm not sure if it's applicable in your case, but were you able to check your fixed point math performance against using floats? I did a custom SoftFloat in the past to solve determinism. If I remember correctly the performance was 3 to 15 times slower than using floats (best to worse case).

    =D
     
    mmankt likes this.
  3. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    Yes I've tested it. We're using Fix64, (we'e been testing determinism on multiple platforms (pc/android/ps4) and all results are the same.) The perf hit seems to be around 30%. So it's ok. Can't wait till burst will be officially deterministic. I am making a brand new 3d rts engine at home right now which will be fully jobified and based on float and unity mathematics so can't wait to see the results. ;)
     
    Singtaa and Antypodish like this.
  4. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Definitely some smart stuff are going here :)
    The experiments demo are really cool.
    And of course main demo.
    Keep up.

    Just questions, if not mind:
    What 2d physics methods have you used in your second parallel 2D physics video?
    Anything spatial? Colliders, or anything else?

    Interestingly, how you bring homeworld inspiration into your game? I mean both are RTS, but other than that I can see little of similarities from demo. Can I assume, there are some mechanics, similar to homeworld?
     
    mmankt likes this.
  5. starikcetin

    starikcetin

    Joined:
    Dec 7, 2017
    Posts:
    340
    Looks good. The 2D physics demo also looks very interesting.

    I would love to read an article from you about using ECS in production.

    Cheers.
     
    mmankt likes this.
  6. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    1. the parallel demo 2df physics demo is based on force accumulation. I solve all the collisions that occur in a step for a body and just accumulated the resulting forces without changing anything on it. Once all bodies are processed the new velocities are applied so everything solves properly. It's less precise than a traditional solving loop but good enough for certain purposes. It's also the easiest way to solve in parallel.
    The galaxy demo is just simulating the gravity forces of each body on all other bodies, no collisions. Same idea as with the physics demo - I'm accumulating forces.

    2. what i meant about homeworld is about the engine solutions - watch the talk to go deeper but basically we're also using a heightmap, 2d physics etc.

    Cheers :)
     
  7. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Cool thx for explanations.
    Unless I am wrong, seams like your concept is similar to ECS Boid demo?
     
  8. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    We use hash maps to determine body locations etc so it's kinda similar.

    One thing I wanted to point out about working with ECS is that the performance in the editor is really terrible ( I don't remember exactly but from our tests it was around 6 times slower). I know it's due to all safety checks but even when I disable everything from the menu it still runs really bad.
     
    Last edited: Jan 11, 2019
  9. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Yep, many of us are aware about editor performance downgrade. Still it is much better in my experience, than same thing in OOP. Kind of good thing, since you always can expect extra boost after build. So I things work in editor well, you maybe nice and comfortable for even worse hardware :)

    Thx for sharing.
     
  10. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    It's not good when you need to test an actual demanding indie AA budget game and we play in slow motion :'D Builds of such projects usually take a while ;)
     
    Antypodish likes this.
  11. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Do you have burst enabled on those jobs that are slow or is it main thread code?
    The expectation is definitely that when using burst jobs there should be zero overhead relative to player when disabling the safety checks & job debugger. This is specifically for the time to execute the jobs. Scheduling overhead is still there a little bit in the editor.
     
  12. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    This happens even for regular, main thread, non burst, ComponentSystems. (as I said, pretty much all of our game loop is on the main thread right now. I use pure jobs (no burst enabled) for some heavy influence map analysis for AI and they're very slow in the editor. )
     
  13. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    But as mentioned, try convert at least something into burst. It will give you massive kick performance boost. Even starting from smaller less intensive bits of code.
     
  14. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    The funny thing is that my 2d physics demo which was 100% jobified, used floats and unity.mathematics, native multi hashmaps etc and was a really clean example of how to do well optimized code in ecs didn't gain a lot by enabling burst compilation. Probably there was a bottleneck on my side but overall the whole thing ran only about 10% faster. I will be doing some deep testing soon-ish.

    I have a small bullet hell demo with raycast commands which is also very clean and I get 100fps with 30k projectiles with burst and 60fps without. So very nice but that code is really straight forward.
     
  15. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Nice one. But weather is simple code, or complex.is less relevant, if you multiply by number of executions. For example 30k. If you utilise multithreads, that is already release some stress from main thread. Wouldn't be worth to embrace that through project, where possible?
     
  16. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    of course, the goal is to be fully multithreaded with burst. don't worry we will be improving all perf aspects of our game. we didn't go that route because it was breaking our determinism but we figured out the errors we made and we will look into making everything better ;) I don't know if you have any experience with actual game production but I hope that you get my point :)
     
    Antypodish likes this.
  17. zyc_dc

    zyc_dc

    Joined:
    May 11, 2018
    Posts:
    42
    Cool! But how do you make sure it's deterministic when you use floats?
     
  18. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    we're not using floats. we use fixed point math in spuds. that was a separate demo :) I had the same demo with fixed point math and it was 100% deterministic and a lot slower ;)

    small update:
    I started rewriting some heavy systems to jobs. Burst doesn't like our fixed point math so until I have time to look deeper into it it won't be enabled on our jobs. Noticed big speed ups as expected anyways.
     
    Last edited: Jan 23, 2019
    Antypodish likes this.
  19. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    Another update: I rewrote pretty much all of the heavy systems to jobs. The game is finally running well in the Editor :D
     
    NotaNaN likes this.
  20. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    Another update :

    We've been porting the game to PS4 (i'm responsible for the whole custom ecs-based combat part of the game)
    (we're using entities 19 in 2018.3.13f1 - will probably update to the last supported version, since inject is deprecated in 2019 i don't have time yet to re write over 130 systems to the new api)

    So far the results were mixed, the ps4 cpu is really slow but i've been optimizing the game to use all the cores in all of the heavy systems that were completely negligible on pc. Making a fully optimized il2cpp build reigns amazing results (no optimization - 28ms for 1 combat lockstep - full optimization - around 7ms) - for example our 2d physics loop is only 50% slower than on a 4.6ghz 7th gen intel i5 core (vs 1.6ghz amd bulldozer)

    To achieve big gains, i've dropped our fix math format in favor of float - this by itself improved things around 30 percent. since the platform is closed the sim is still deterministic though. Hopefully, ill be able to enable burst on our jobs since it didnt like our fixed math - now it's just wrapping a float so maybe we'll see some gains there.

    The game is nowhere near playable on ps4 since we get huge drops thanks to GC but i'm working on minimizing our allocs right now.

    I'm wondering if our game is the first playstation 4 game to use dots ;)
     
    FROS7 likes this.
  21. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Is all of the simulation code running bursted jobs?
     
  22. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    Almost none. Right now I'm enabling burst and testing one by one because I need to clean them up from any incompatibilities.

    Btw: I've been moving some c# arrays to nativearrays to reduce gc alloc but i noticed that it still shows in the profiler as gc alloc (nativiearray allocate with deep profile) is that supposed to happen?
     
  23. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Disable Leak detection in the editor and it will go away.
     
    mmankt likes this.
  24. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    oh, thx I had it disabled but I never noticed that it if I disable it before entering the playmode it turns on after i hit play.
     
  25. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    That has been fixed in 19.1
     
    mmankt and FROS7 like this.
  26. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    Check out a small tech demo i've prepared :)

    (il2cpp (burst in 2018.3 refuses to compile our systems so I can't use it, sorry @Joachim_Ante ;) ))

     
  27. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Looks fun.
    Just title thought, I would add different laser colors for each team / faction, or something to distinguish, which side is firing.
     
    mmankt likes this.
  28. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Looks great. Is it possible to upgrade to 2019.1? Almost all of our pro-active testing is done against 2019.1 since Unity.Entities releases now require that.
     
  29. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Is it a code stripping issue in il2cpp? If so you can look at making at linker xml file:
    https://docs.unity3d.com/Manual/IL2CPP-BytecodeStripping.html

    We still have some holes in the automatic generation of that, but those should be all addressed with new API's in 19.3. Until then sometimes you have to help the bytecode stripping to not strip out your code using the linker.xml file.
     
  30. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    @Joachim_Ante on that note, any chance link.xml can work from the Packages/ directory? From my last test they are ignored unless added to the Assets/ directory* and it'd be very handy to ship it with plugins/third party packages rather than having to instruct users to .

    * the documentation does say that, "creating a link.xml file and placing it into the Assets folder (or any subdirectory of Assets)" but it'd be very handy if it was changed to work from Packages folder as well.
     
  31. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    can't upgrade, after trying to switch to 2019.1 we loose all of our serialization. We simply don't have time to mess with that. I'm stuck with entities version that still uses inject (i upgraded the project to the latest before inject being removed but it didn't give me anything)

    I don't think it's code stripping because it even fails to compile in the editor while still in mono. I have a link file that preserves everything we want - it was our first wtf moment when testing il2cpp when it was stripping all ecs rendering so that was my first thing to do :)

    Our game is finishing it's development and we are very satisfied with the results. The ps4 port is almost fully optimized so i'm happy with our results so far. I hope that our next project will give me an opportunity to work in the newest version of dots and I will definitely push things as far as they go :)
     
  32. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    We added a mechansim to build the xml file from a callback in 19.3. So we can make up the rules in the entities package programatically and solve it properly.
     
    Erothez likes this.
  33. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    @Joachim_Ante good news! managed to enable burst on all of my jobs that support it in my version.
    Had time to think and made some crucial rewrites and optimizations to the code (like simplifying many calculations not to require allocating native arrays per Execute etc. Unfortunately some of my jobs do require command buffers so those are left as is.. :) I'll measure the improvement and report on it later.
     
  34. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Cool yeah. We have landed necessary changes for entitycommandbuffers to be burstable in 19.3.
     
    5argon, eizenhorn, optimise and 9 others like this.
  35. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    I'm happy to announce that the game has been finally released on pc! (psvr in november, hopefully)

    here's the release trailer:



    here's a VR tech demo because ECS and lots of units always go nice ;)

    il2cpp and burst when possible :)



    I think that's my last post about this game. Looking forward to questions and to my future projects with ECS :)
     
    florianhanke and Antypodish like this.
  36. RoughSpaghetti3211

    RoughSpaghetti3211

    Joined:
    Aug 11, 2015
    Posts:
    1,697
    That looks like so much fun , congratulations to you and your team
     
    msfredb7 and mmankt like this.
  37. mmankt

    mmankt

    Joined:
    Apr 29, 2015
    Posts:
    49
    tertle likes this.