Search Unity

  1. How has 2019.2 and the beta been for you so far? Give us feedback in this thread.
    Dismiss Notice

Anyone know how UE's generated class from blueprint allocate and use data?

Discussion in 'Data Oriented Technology Stack' started by 5argon, Jun 1, 2019.

  1. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,231
    A little bit off topic, but I hope it is allowed to talk about here because it's still data oriented and kind of related to what we are going for. I would like someone with UE experience to tell me about internal data usage pattern of a class that was generated/nativized from the Blueprint.

    Is the engine just `new` instances of those generated class, or the engine knows how to order memory by-field for each instance? (As the user just use these normal looking methods which even return a pointer. But they are not a normal C++ `new` I don't know if any magic is in there.)

    The fact that they returned a pointer means you could access whatever field you want (contrary to Unity which the API guards the data very tightly, and provide specific iteration entry point) so how could the engine convert something like this to DOD at runtime? If that level of intelligence is possible it would be an equivalent of Burst.

    This is because lately, I have been talking a lot about the recent DOTS movement of Unity in game dev community (engine agnostic), and always someone would keep claiming that UE is already data oriented for long, he said "just that we programmed in OOP, but in runtime all the blueprints will be converted to a class which utilize memory in data oriented way." And the net don't have much information about this, the official docs is talking as deep as compilation but not at the level of how the memory is allocated and used.

    And I did Google with keywords involving memory, DOD, Unreal, Unity, etc. but all I could find are baseless claims. (There are both someone saying that UE is OOP behemoth, and someone saying UE is data-oriented) I hope this one show up the next time someone wondered.

    ps. I did some work by actually digging through the source too, so this time it is not as baseless. My deduction is that UE memory usage is like regular OOP's `new` where it allocates per instance incrementally. (No such pattern like pre-reserve a chunk to arrange data later, etc.) However I may missed something as I didn't actually use the software. So if we follow from NewObject

    Screenshot 2019-06-02 00.34.11.png

    Screenshot 2019-06-02 00.34.48.png

    Going to implementation of this StaticConstructObject_Internal, then likely this StaticAllocateObject

    Screenshot 2019-06-02 00.38.05.png

    After some certain checks, an object size is calculated along with required memory alignment, that goes into this

    Screenshot 2019-06-02 00.39.52.png

    In a new file I finally found a Malloc, but it seems to be just using the size of the whole object got from TotalSize earlier. In effect this is 1 object allocation and I could not see how the next object could be sliced and arrange in DOD style with the previous one. (Or did he actually just misunderstand from somewhere that UE is data oriented?)

    Compared to Unity's EntityManager.CreateManager which works even without malloc if the chunk is already there, just memory copy to the space we already own. And copy such that the data is spread out and make space for the next instance (entity) to come.

    Screenshot 2019-06-02 00.41.40.png
     
    Last edited: Jun 2, 2019
  2. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,231
    (The above post reached image limit.) I tried to see inside that custom malloc but all from this point seems to be just a cross-platform handling code that make it malloc memory correctly (?)

    Screenshot 2019-06-02 00.55.56.png

    And there is a pooling inside that malloc external which I couldn't understand. If something magically become DOD it has to be there, but I guess it is unlikely. Pooling may change and arrange memory location but it shouldn't change the fact that each one is the size of an object and thus OOP-styled. (?) Feel free to debunk in the case that someone with UE experience passed by here. Thank you!

    Screenshot 2019-06-02 01.24.11.png
     
    Last edited: Jun 1, 2019
    GilCat and francois85 like this.
  3. Josh4364

    Josh4364

    Joined:
    Aug 11, 2017
    Posts:
    22
    I dont have my old UE projects on hand but what I did back then to take a peek into blueprints, is make a simple one, go and enable blueprint nativization, build it and check the intermediate directory. I'm gonna take a peek and see if I can find one I saved, but it's basically just a c++ class calling internal oop methods all over the place. The basic structure is a class with a state machine running on a dynamic list of int next state with lots of fall though states.
    UE is not data oriented at all and tim said at one point they are watching unity to see how it turns out for them. I'll see if I can find that post as well. there was a project someone posted where they added a ECS system (single threaded) and saw large gains compared to creating lots of uActors to perform the same thing.
     
    5argon likes this.
  4. Josh4364

    Josh4364

    Joined:
    Aug 11, 2017
    Posts:
    22
  5. francois85

    francois85

    Joined:
    Aug 11, 2015
    Posts:
    528
    In my person opinion DOTS was a really good move by unity to future proof the system.


    My only concern is Unity had to take 3 steps back to be able to move forward and Epic is taking advantage of that especially in the animation and visual effect industry where there are huge potential for growth. I’m sure there are many Unity users here that would agree with me that the film industry at this point in time are desperately looking to reduce cost with real-time rendering and faster simulations. A single character render can take anywhere between 10-25 min a frame and simulation times of 2-14 min a frame :(


    The feeling on the floor is Epic is reaching out trying to establish themselves. Unity seems more reserves as if they are waiting , maybe waiting on DOTS to land idk. I hope DOTS lands before it’s too late, it would be nice to have a choice in software
     
  6. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,231
    Thanks a lot! Haven't seen that one yet. That thread's comment are some really nice UX study for programming languages. Lua is surprisingly popular.

    imo DOTS is fun to think about and to try to map every problem to become iteratable in order to use ECS's strength. It is awesome for more "data" stuff like server code, scoring in games, etc. I would love a job to code in DOTS and unit test to perfect them all day rather than having to deal with scene/game view.. actually I think this is one strength of DOTS where someone could outsource DOTS programmer to create a testable system in isolation and it would integrate cleanly with the game.

    But it's true as UE's CEO said that this is mostly unexplored in gameplay programming, it is difficult to think how to do typical game-things, that is, not just rendering 10000 of the same thing, or making particle systems where it is obvious that ECS is fitting. New physics, new UI, new whatever high level stuff should take care of that so we could just make a game like usual but it's a long way to go.
     
    Last edited: Jun 2, 2019
    psuong and Brendon_Smuts like this.
  7. Josh4364

    Josh4364

    Joined:
    Aug 11, 2017
    Posts:
    22
    Digging around in my compiler project today, I found the outputted file as a reference in the notes folder so here it is:
    https://pastebin.com/Nd9SVQ4Z
     
    5argon likes this.