Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Unity ECS and Job System in production

Discussion in 'Data Oriented Technology Stack' started by eizenhorn, Sep 26, 2018.

  1. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    I'm always on latest packages and beta. I'm using Rider.
     
  2. wechat_os_Qy0_X1ITy6VrZHYPJFd58JGlA

    wechat_os_Qy0_X1ITy6VrZHYPJFd58JGlA

    Joined:
    Nov 15, 2018
    Posts:
    62
    Thanks for your helps guys.

    I have fixed this problem, I updated my Burst and ECS version.
     
    Last edited: Oct 12, 2019
    MNNoxMortem likes this.
  3. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Just compilation of what we have now:)
    Btw if some one will go to DevGAMM Minsk 2019, come to say hello :) we’ll be showcase and I will be speaker again about our game and how we solved most popular RTS problems and things with DOTS. Also there will be @Fabrice_Lete with talk I’m looking for, don’t miss your chance to see him in person and ask about DOTS guts!
     
    phobos2077, Enzi, pm007 and 12 others like this.
  4. Karsten

    Karsten

    Joined:
    Apr 8, 2012
    Posts:
    109
    This is all nice, interesting and promising, but will it ever be "easy" ?
    For me DOTS feels much much more complex than pure OOP
    leading to much longer development time.
    It somehow makes me think "I'm building block towers in the air without foundation"
    Does any of you guys know an indie/commercial released game that has been build with a
    ~100% DOP approach (Unity or another engine) ?
    Dont get me wrong I don't want to say ECS/DOTS/DOP is generally "bad" I am just interested
    learning about its positive AND negative sides.
     
    Last edited: Nov 9, 2019
  5. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    7,381
    @Karsten this thread is not dedicated strictyl to DOTS, but applying DOTS with results in production.
    Title is a little misleading in this sense. Thread is more about project using DOTS.

    DOTS main purpose is performance.
    But it gets easier to use from month to month.
    Still nothing close as OOP, and lack of features in comparison.
    It does not means, it stops anyone from building games using DOTS.
     
  6. Micz84

    Micz84

    Joined:
    Jul 21, 2012
    Posts:
    299
    Blizzard have made ECS based engine for Overwatch. There are talks on GDC Vault where they are explaining the benefits of ECS approach for them.
     
  7. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    538
  8. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    We added dynamic to our game, our world become much more alive :)
    Animals system. Birds flying around, animals (chickens, rabbits, deers) living their own lives, avoiding obstacles, eating grass, reacting to player and enemy units (running away)


    Trees shaking a bit, reacting to wind.
     
    Last edited: Nov 15, 2019
    pm007 likes this.
  9. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,045
    Important to note that blizzard used ECS for its architectural benefits, and not for its easy ability to combine with data oriented design and performance improvement. They favoured ease of use, etc. over raw performance.
     
  10. Vacummus

    Vacummus

    Joined:
    Dec 18, 2013
    Posts:
    136
    Yup, they also did it for determinism. ECS made things a lot more predictable for them that allowed them to easily solve a lot of issues dealing with building a multiplayer game (such network latency issues).

    But I do want say that though DOD does allow you to easily write performant code, the very same reasons why it is able to do this are also the reasons why it makes it easier and more predictable to make games with. By simplifying and making it more predictable for the hardware to reason about your code is why DOD is performant. And by solving this problem at the root level (hardware), the benefits naturally cascade upwards to the devs that have to reason about the code as well. In contrast, OOP overcomplicates and makes it very unpredictable for the hardware to reason about your code, and this naturally cascades upwards as well.
     
  11. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Did you miss me?:D
    Our first step in to really epic battles :) Siege machines! :eek: Just first iteration, without any cool VFX, enjoy!


    EDIT:
    Just a bit improved trajectories, rotations and now trees flying too :)
     
    Last edited: Dec 7, 2019
    lclemens, phobos2077, SLASH24 and 7 others like this.
  12. Radu392

    Radu392

    Joined:
    Jan 6, 2016
    Posts:
    210
    That looks really cool. Been following this for a while but never posted anything. I’m doing something similar, only in 2D and another setting. A few questions.

    How many units can it handle on the screen at once? And on the map?

    Also, how many people are working on this project? Are you guys working full time?
     
  13. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Around 30-40k units total (army + enemy units + citizens + animals). Also on screen you see around 50k non static trees and 20k cliff tiles. On this screen you see around 15k, but I will improve this count, I have a room of places and ideas for optimizations, but currently we focused on gameplay features and stability, after that we will cleanup and prettyfy project and code.
    1 programmer (me), 1 artist, 1 game designer, and recently we hired one more programmer (as part timer) because I’m program not only gameplay logic but graphics to (shaders, etc.).
     
    Last edited: Dec 6, 2019
    lclemens, phobos2077, Jes28 and 3 others like this.
  14. June1111

    June1111

    Joined:
    Aug 11, 2017
    Posts:
    33
    Just thought I'd chime in for a second. Read your post back when I was playing around with ecs when you first posted it. Based on what I saw I decided to build my whole game on dots and it really really paid off. I won't ever go back to the old way of making games after learning dod.
    Just thought I'd give my thanks eizenhorn. Looking forward to playing your game someday.
     
  15. Radu392

    Radu392

    Joined:
    Jan 6, 2016
    Posts:
    210
    Once you go dots, you can’t go back.
     
    mkracik, Jes28 and MNNoxMortem like this.
  16. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    7,381
    @eizenhorn loving your flying trees.
    You need just explosion and flame :)

    Somehow deer animals stays safe, even under shadow of falling rocks from the sky.

    Well done ;).
     
    eizenhorn likes this.
  17. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Yep as I said it’s without vfx yet :) will be on next step. Animals already safe, just not in video, they running away from buildings player and enemy units correctly now, and they wouldn’t be in target area anymore :) Thanks :)
     
    Antypodish likes this.
  18. francois85

    francois85

    Joined:
    Aug 11, 2015
    Posts:
    1,349
    For the pathfinding and moving are you using steering with physical?
     
  19. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Not. Read on second page.
     
  20. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Just improved units collision :) Happy New Year and Merry Christmas :)
     
    BroOf, phobos2077, MookeeDev and 11 others like this.
  21. Micz84

    Micz84

    Joined:
    Jul 21, 2012
    Posts:
    299
    Looks cool. Are you using ECS Physics for that or some custom solution? And how big is your flow field cell relatively to unit size?
     
    Antony-Blackett likes this.
  22. Antony-Blackett

    Antony-Blackett

    Joined:
    Feb 15, 2011
    Posts:
    1,606
    You’re all giving me ideas for ecs driven games. This is dangerous as I’m only part way through my current project. XD
     
  23. thelebaron

    thelebaron

    Joined:
    Jun 2, 2013
    Posts:
    518
    how are you planning on tackling vfx? as more ecs systems or using unity particles, or something else?
     
  24. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Custom physics. One flow field - location, is 40x40 (standard Unity units) whole map is 25x25 locations, one flow field cell is 1x1, unit size ~2-3 units in radius (of course siege machines bigger)
     
    Micz84 likes this.
  25. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Write my own shaders and systems :)
     
  26. bugfinders

    bugfinders

    Joined:
    Jul 5, 2018
    Posts:
    203
    @eizenhorn I just wanted to say how inspiring it is to see this. While so much of me would love to see how you coded it all up in dots, I respect your game and obviously if you showed us all how its done some of that magic would be lost.

    I am given great inspiration that great things will come from dots games as it will allow us far more complexities and quantities than ever before
     
    eizenhorn likes this.
  27. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Thanks. You can see some code on my talks (2018-2019 years) from DevGAMM conference. Latest was in Minsk at the end of 2019 (soon will be on YouTube). Where I detailed showing structure of our pathfinding and code how it implemented (19 jobs) and how our fog of war\ health bars implemented on JobComponentSystems (collecting data) and post processing shaders (visualisation of fog and bars). I'll post it here when it will be public. It's in Russian, but all slides and code in English and you can use auto translation for YouTube, I have strong and clear voice, it should be translated fine :D
     
  28. Miroslav-Zeman

    Miroslav-Zeman

    Joined:
    Feb 11, 2014
    Posts:
    1
    @eizenhorn Hey! Just want to say its amazing how much you progress with DOTS! Hope the presentation from Minsk will be on youtube soon! Im so interested in seeing your DOTS flow! :D
     
    phobos2077 and bugfinders like this.
  29. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Just small preview from our first story trailer :)
    preview.jpg
     
    Opeth001, BroOf, kvfreedom and 15 others like this.
  30. LazyGameDevZA

    LazyGameDevZA

    Joined:
    Nov 10, 2016
    Posts:
    132
    Seeing how far this project has come in the 1.5 years I've been following along here it's really a testament to what can be done with DOTS. There's still loads of sharp edges developers can get caught out by, but if you're willing to do a little reinvention of the wheel the gains seem to be worth it. Keep up the good work @eizenhorn !
     
  31. bugfinders

    bugfinders

    Joined:
    Jul 5, 2018
    Posts:
    203
    @eizenhorn watching your game grow is very inspiring. I wish I had the time to devote to my game writing, but sadly my day job is what pays my bills
     
  32. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    387
    As PlayerLoop is out of experimental phase in 2019.3, I'd say that job system is ready to be used in production. Here's profiler timeline of mine AudioSource.GetSpectrumData replacement:
     
  33. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    As promised, my talk from latest DevGAMM Minsk 2019 "DOTS RTS Case Study" about our implementation of Fog of War, Progress Bars, Pathfinding. (In Russian, use subtitles for English :) )
     
    davenirline, Sarkahn, RaL and 8 others like this.
  34. jGate99

    jGate99

    Joined:
    Oct 22, 2013
    Posts:
    1,090
    cant wait for story trailer
     
  35. Kelevra

    Kelevra

    Joined:
    Dec 27, 2012
    Posts:
    82
    Hello @eizenhorn! Great talk and thank you for sharing!

    Noticed one thing in your FogOfWar fill system. Your algorithm is iterating over X coordinate and then over Y, considering your memory layout it's not very efficient from the point of processor cache. Because your index is always jumping between array indexes which leads to cache misses (x + y * MapSize). Try to swap cycles and iterate by Y and then by X it should give your some performance boost due to better utilisation of data from cache.

    Also, I wouldn't be so sure about "nothing bad" with accessing the same index from different threads because it will leads to "false sharing" and problems with cache coherence. Some times it could leads to performance drops.

    And you can switch from checking length to lengthsq :)
     
    Last edited: Feb 25, 2020
    eizenhorn and siggigg like this.
  36. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Yep I told about that on question time (not in video) I think I wrote this in a midnight, and just overlooked :) From time to time I'm founding this strange things and fixing that. But tbh even with this small ugly pieces - performance good and stable :)
    In this case it's fine :) it gives us full parallelization (without rethinking logic for calculation overlapped fog affectors) and false sharing cost not being so much (of course all is depends, but in our case and map size it's ok)
    Yes, good catch, absolutely agree that jumping through rows for every index it's not so good, yeah iteration over rows at first place it's much better, thanks! It's like with length lengthsq, in some places I'm following best pattern (rows first for good cache filling if accessing indicies) and sometimes I just forgot about that :)
    upload_2020-2-25_15-13-33.png
     
    Last edited: Feb 25, 2020
    Kelevra likes this.
  37. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Hey! Long time I was gone! Meanwhile, more work than ever! We decided to rewrite globally the core of our game (which was far from ideal) for better memory management and a smoother (for the user) boot process between screens. Now we use Addressables for absolutely all content in our game! They combine perfectly with DOTS! Memory management is now under full control, content loading is extremely fast and optimal, we got rid of all the excess junk by rewriting the kernel from scratch (instead of rewriting in the current version) thanks to the DOTS modularity, all other systems were transferred without any changes and they just work! A completely new Pure DOTS camera controller and with it a day and night cycle system have been written and all this works inside the systems and does not get out of a uniform code style (thanks AddHybridComponent)! Systems for serializing and deserializing the game world have been written, now saving the game works the way we want, preserving the fully current state of the game session up to commands for units, flying shells and even the state of animation continues exactly from the second on which it was saved. On the visual part - the cycle of changing day and night is accompanied by a very pleasant trifle - the lights in the buildings light up, which gives yet another highlight to our pleasant visual style. Сonstruction system has been redesigned, now it is more "saturated"!
    \
    And small secret frames of our first story trailer ;) This one finished but not yet released, and two more on their way! Soon we'll open our Steam page for wishlists.
    upload_2020-4-4_1-59-19.png
     
  38. bugfinders

    bugfinders

    Joined:
    Jul 5, 2018
    Posts:
    203
    Looking seriously awesome, my ECS/DOTS skill is still at the it looks like a 1980s game level, I cant work out so much..

    Ive been looking at trying to move to systembase, but, frankly that went hiddeously badly so I backed out of that
     
  39. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
  40. bugfinders

    bugfinders

    Joined:
    Jul 5, 2018
    Posts:
    203
    I searched on steam but it didnt find "Diplomacy is not an option" as a game :( help. (PS your facebook group is hidden enough i couldnt see that either :( )
     
  41. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Steam page on review. This post was just as teaser, that steam page will be opened soon :)
     
  42. June1111

    June1111

    Joined:
    Aug 11, 2017
    Posts:
    33
    build has to go through approval as well
     
  43. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    for wishlist you don't need build. It's not release, we just started build community around game :)
     
  44. Radu392

    Radu392

    Joined:
    Jan 6, 2016
    Posts:
    210
    Wow nice progress! Any idea on the release date?
     
  45. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    Hey! We finally got the Steam page! Add to wishlist and stay connected! More updates coming soon! I also plan to write a couple of threads about the use and how we developed with DOTS in the process of developing our project, like - why we decided rewrite every thing from scratch, how we got benefits of usage Addressables with DOTS, how we using multiple worlds etc. But this will only happen after I finally finish with refactoring :D Hope this will be interesting for someone.

    https://store.steampowered.com/app/1272320/Diplomacy_is_Not_an_Option
    Discord:
    https://discord.gg/U93MmGV
    prepare_wishlist.png
     
    Last edited: Apr 18, 2020
  46. RBogdy

    RBogdy

    Joined:
    Mar 6, 2019
    Posts:
    60
    Looking forward for threads and game :D
     
  47. marijnz

    marijnz

    Joined:
    Sep 20, 2012
    Posts:
    57
    Hi @eizenhorn. Cool to see the progress! It's looking great.

    I'd be interested to know on how your team dealt with the constant changes with ECS and how it relates to your decision to rewrite everything. Looking forward to blogposts :)
     
    bugfinders likes this.
  48. francois85

    francois85

    Joined:
    Aug 11, 2015
    Posts:
    1,349
    This is my type of game. @eizenhorn you should try and get on Mike Geig Unity podcast. He had is first one last week
     
    bugfinders likes this.
  49. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,061
    I't not so big for @Mike-Geig podcast :D
     
  50. bugfinders

    bugfinders

    Joined:
    Jul 5, 2018
    Posts:
    203
    I confess @eizenhorn I really am interested on how you did your health bars :)
     
unityunity