Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

ECS and Voxel engine

Discussion in 'Entity Component System' started by RemiArturowicz, Feb 8, 2019.

  1. RemiArturowicz

    RemiArturowicz

    Joined:
    Nov 22, 2018
    Posts:
    9
    Hello. I'm starting to learn ecs approach. I have a problem with making a voxel engine in ecs style.

    First problem:
    I want to have entities called chunks(not ecs chunks - When I refer to chunk I mean entity with voxel component on it) on which I will have position component and VoxelsComponent. But my first problem is with VoxelComponent. How can I store array in it? I know the limitations on IComponentData and I know why it should have fixed bittable size but if I want create array with predefined const size this should be possible. If I know size at the compile time int[16] should be no different than 16 ints. How can I achive this?

    Second problem:
    Chunk entities are not independant of eachother. In most parts they are but for rendering purposes I need to access neighbours chunks(for face culling). What is the best approach for it? Iterating over all chunks and checking their positions seems as not very optimized way to do it. Storing entities in dictionary <position,chunk> would work but if I understand correctly this defeats the purpose of ecs and linear memory access. Theoreticaly if I have 16x16 chunks iterating over all of them should not be that much of a bottleneck but for 128x128 it starts to be unefficient.

    I'm new to the ecs concept so if I don't see something obvious I apologize :)
     
    Last edited: Feb 8, 2019
    NotaNaN likes this.
  2. JooleanLogic

    JooleanLogic

    Joined:
    Mar 1, 2018
    Posts:
    447
    You can use
    fixed int data[16];
    if you know the size of array but I think this only works for primitive types.
    For everything else you can use IBufferElementData which is a component type that can hold an array.

    As for second problem, SharedComponent might be a good candidate for spatially grouping chunks. I don't know much about voxel systems but there has been several discussions on it in this forum so I'd do a search and see what others have come up with already.
     
    RemiArturowicz likes this.
  3. RemiArturowicz

    RemiArturowicz

    Joined:
    Nov 22, 2018
    Posts:
    9
    Thanks for answer. Fixed mean unsafe. I guess in this case it isn't that big of a deal. I will look up forum and shared component. To clarify lets say we have chunk that have 10 voxels in a row and it is placed on position (0,0). Next chunk have the same size and it is placed on (10,0). To render first chunk I can do it without the second one but voxels on edge will be renderer fully because I don't know what is next so I can't decide if it will be visible. For example if on (10,0) voxel is air then voxel on (9,0) should have right face but if (10,0) is solid, I don't need to render (9,0) right face because it will not be visible. This example scale to x,y,z dimensions so every chunk needs to know its neighbours. I have problem with this not because I don't know how to achive this, the problem is with how to achive this in good terms with ecs in which I don't feel comfortable yet so I don't know in which direction to go. Key points are - there can be many chunks in my game so iteration is not the option. In oop implementation I stored chunks in dictionary so no matter the size of the world I could get chunks in the same time but dictionary with references to different chunks from what I understand is againts DOD. I will look up other threads about voxel engines but google didn't provided me with much informations so any help would be great.

    Edit. I need to store only primitive types in array so it is actually a great solution. Thanks :)
     
  4. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    You can have component per each voxel block (entity), which holds 6 entities in component data, left, right, front, back, up, down.
    Now you can rapidly access neighbour entities.

    Or If you store your chunk of blocks in an array, you can easily access neighbours by knowing index offset. Which probably is much cheaper, than having left right etc, per each block.
     
    GTLugo likes this.
  5. RemiArturowicz

    RemiArturowicz

    Joined:
    Nov 22, 2018
    Posts:
    9
    The problem isn't about accessing neighbour voxel in the same chunk(entity) but across multiple entities. Each enity has array of voxels and edge voxels needs to know other entity edge voxels. I can't store entities in an array because it is multiplayer game server. Multiple players can be in completly different locations. If I render 4 chunk(entities) for each player I can have chunks loaded for position (-1000,0,0) and for (xx,xx,xx) position. You can say that I should then treat each user chunks separatly and store them separatly but this won't work - what if 2 players have the same positions but each of them have his own representation of terrain under him? Like I said before. I could iterate over all chunks. Check their positions. If positions suggest that chunks are next to eachother then proceed with accessing this chunk voxel array but I need ecs way of handling this neighbour chunk searching. Each entity as a voxel is not optimal solution. It is too slow and memory consuming.
     
    Last edited: Feb 8, 2019
  6. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Doesn't matter with chunk with blocks you are accessing, when you store them in an array. Each chunk holds same structure of blocks. Weather is 2d or 3d. You select correct chunk, then calculate offset index to get block at the edge, then get adjacent chunk and calculate offset index of corresponding near block. And you can show hide faces if you like. In the end, you need store blocks in serialized form as array/database.
     
  7. RemiArturowicz

    RemiArturowicz

    Joined:
    Nov 22, 2018
    Posts:
    9
    But it would not be efficient. Chunks(entities) in array is impossible or completly overkill when we don't have a "smooth" world. We have for example 16 chunks at player position (-100,-100,0) and another 16 chunks at position (200,200,0). In beetwen them we dont have chunks because there is no player there so it is not needed. What good does it give me to store chunks in array if position in array doesnt mean they are next to eachother. The only scenario when they would be next to eachother in array like that is if I would create array big enough to store the whole world into it from position -10000 to 10000 on every axis flatten to 1d array which mean insanely huge array in memory. I have voxels in array so if I want to serialize it i save it as a chunk/part of the world into file. I need to have separate entities(chunks) with arrays to store my data because I want to create "infinite" terrain so I need to split world data into smaller chunks. The problem is again about finding neighbour chunk to another chunk - neighbour voxel array to another neighbour voxel array. I can have multiple groups of this voxel arrays and they can be as far away of eachother as possible. So in array we would have one chunk[100] here and another chunk[100000000] and the number increase depending on how far away players are from eachother. This would not scale to many players and world sizes. I don't have constant number of chunks. I have constant number of voxels in chunk. Chunk/entities are created dynamicly depending on players count and their view range. I had this implemented in oop. All chunk were stored in dictionary<position, chunk> so if chunk wanted another chunk it would simple do world.get(position + offset).
     
  8. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    What I mean is, having voxel blocks in a chunk, stored as 1d array.

    But if you want strictly find near chunks only, I would add neighbour component, as I mentioned earlier. But in this case, probably only 4 neighbours, left, right, front, back.
    Having 10k scattered chunks all over the world, won't break the memory bank at all for that reason. You pay significantly more in memory, for storing blocks (voxels) in each chunk.

    Then simply knowing in which chunk player is, you can get cheaply 9, 25 etc surrounding chunks in blink of eye.

    And will work for any player in world.
     
  9. RemiArturowicz

    RemiArturowicz

    Joined:
    Nov 22, 2018
    Posts:
    9
    I don't know if we understand eachother correctly. I will put this in non programming way. I'm looking for best approach in ecs to handle given case. I have book library that potentialy can store infinite number of books. I need them to be sorted by name the most efficent way. You gave my an array as a potential solution. There is a problem with it. If I already put books with letter K in my array and someone give me a book with letter A I need to shift the whole array to insert it. Also if my array can hold 1000 books and I just got 1001 what should I do? In oop I would create dictionary for fast access to books in different positions.

    Edit. Didn't see your next response. So having reference to neighbours is acceptable in ecs? And about array. I acctualy have 3d array flatten to 1d in my IComponentData attached to entity(chunk). What do you mean that entity + array of voxel is more memory consuming than entity + voxel. If I understand correctly in one case we have memory for one entity + voxel_array[100] and in the other one we have (memory for one entity + voxel) * 100. In numbers the example difference would be 1 + 1*100 or (1 + 1) * 100? Or I don't get it correctly?
     
    Last edited: Feb 8, 2019
  10. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Why not? Is just solution to the problem, like many others.

    If that are voxels blocks that is correct. You probably want fixed size of an array for simplicity.
    In the end you have blocks like air. Minecraft probably have some form of compression, of each chunk, where blocks groups are the same. I don't mean mesh. Just indexing. But otherwise, if you have fixed array size per chunk, you can find easy any block in that chunk.

    If you consider vertical chunks as well, then air blocks, or same blocks per chunk, could be significantly optimized, in terms of memory, as instead holding array of x voxels blocks. you could have single block type. At least until chunk's blocks are changed.

    What I meant, is that array of voxels blocks in each chunk takes significantly more memory, than memory taken, by chunk neighbors. You can have for example (3D) 10x10x10 = 1000 voxels blocks per chunk. While only 4 (or 6 with up/down) neighbors chunk (entities) per chunk entity. Hence cost is insignificant of storing neighbors. If that makes sense?

    Just to be aware, if storing entity, you store two ints. Entity index and version. Hence in fact, 4 (or 6) become 8 (or 12) ints.

    You could even storing neighbors optimize more, in terms of memory, but I don't want to get into that at this point, until above is clarified.
     
  11. RemiArturowicz

    RemiArturowicz

    Joined:
    Nov 22, 2018
    Posts:
    9
    I know it is solution but because I'm new with ecs I'm beeing careful not to borrow solutions from oop into ecs. Mental shift from one to other may push me into using oop techniques into ecs which would not be good. Mostly that is why I'm here. I could implement it in many ways but I want the "ecs friendly" way. It was like that when I was learning rust-lang. I was constantly trying to structure my code as oop because my only programming experience was with oop.

    Ok now I understand what you meant. Stroing neighbours per chunk = max of 6 neighbours per entity. I can survive with that. Thanks for helping. I have few more questions but I will ask them after I implement what I already know. Maybe my questions will answer themselfs after I write some code.
     
    NotaNaN and Antypodish like this.
  12. Myrault

    Myrault

    Joined:
    May 31, 2017
    Posts:
    10
    Hi, I'm trying to port my voxel engine to ecs and I'd like some advice on how to structure things.
    I store voxels in a flat array and access adjacent through their index.

    Code (CSharp):
    1. public struct Chunk : ISharedComponentData
    2. {
    3.     public int Index;
    4.     public NativeArray<Voxel> Voxels;
    5. }
    Jobs that only requires one chunk works great but I'm having trouble on how to write jobs that require multiple chunks. Jobs that need multiple chunks are for example:
    Occlusion Job, that hides faces that aren't seen.
    Pathfinding Job, that traverses the world.

    The Occlusion Job only requires that I know the six adjacent chunk of a given chunk. In this thread, Antypodish suggested that each chunk has a Neighbour Component.

    Code (CSharp):
    1. public struct ChunkNeighbour : ISharedComponentData
    2. {
    3.     public Chunk Back;
    4.     public Chunk Front;
    5.     public Chunk Right;
    6.     public Chunk Left;
    7.     public Chunk Up;
    8.     public Chunk Down;
    9. }
    This works for the Occlusion Job as I can access the adjacent chunks. Now my problem is on how to scale this solution. The Pathfinding Job would require more than just the six adjacent chunks and I'm lost on how to access them.
    My thought was to use this Neighbour Component to traverse the world like you would do in a linked list or a tree - but I can't access a ISharedComponentData in a job.

    Code (CSharp):
    1. public struct PathfindingJob : IJob
    2. {
    3.     public Entity Entity;
    4.     // N number of chunks that I work with
    5.     // (doesn't work)
    6.     public NativeArray<Voxel>[] SearchSpace;
    7. }
    I've tried with collections like DynamicBuffer<NativeArray<Voxel>> but it doesn't work since it's not blittable.
    My first thougt was to store the entire world inside a NativeArray<Voxel> and access chunks by an offset. But this must be terrible for job access and scheduling.

    One solution would be to copy all the worlds (or relevant chunks) voxels to a new NativeArray<Voxel> before the job. This could require me to copy an enormous amount of data each job.

    I also thought that it might be possible - to inside the job use a NativeArray<int>, where the ints are the pointers to each chunk, and access them in an unsafe way.

    So essentially the problems I have, are using either in a job:
    Accessing ISharedComponentData,
    A collection of NativeArray<Voxel> with unknown size

    Thank for you any thoughts or pointers!
     
  13. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    If you consider using chunk as entity, with component of 6 neighbor chunks, then when you start from chunk A, you run your path finding, as similar as in 2D grid. Only this one is in 3D. The main difference is, you don't use index in 2D/3D array, but index and version of neigbour entity.

    Also, if you consider highly scalable in any direction, you could use array, but either you need give it right capacity at initialization, or need resize, when more chunks are discovered, than current capacity. Otherwise, using chunk as entity will be suitable.

    In job, you just follow relevant direction. For example forward, then each every consecutive neighbor entity with neigbour component, you will read forward entity reference to next chunk. Repeat searching loop, until goal is reached. Or is failed.

    Sorry, this post may be a bit hasty, as I typed it quickly. Let me know, if you need clarification.
     
    Myrault likes this.
  14. FernandoDarci

    FernandoDarci

    Joined:
    Jan 25, 2018
    Posts:
    19
    In fact, you just need a method where you pass the voxel coordinate (any one of them) and them if returns the block type you have at this position. It returns na index that let´s you look in a 1d array of block types. Just need some calculations inside, like Perlin Noise 3D, or thresholds, like Sand in the 3rd first blocks on top, rock below, etc...

    Dont need to know the chunk the voxel is, just the position.
     
  15. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    That's all fine, as long you know size of voxel array.
    This approach become more complicated, when considering undefined size of voxel array.

    Imagine for example minecraft world.
    If looking at blocks per chunk, you know their position perfectly. Fixed array size works in such case very well.
    But looking at chunks in the world, they can be generated in any direction, for indefinite distance. Hence there isn't any logical ordering in the array. Other than referencing neighbours. Quadtree and octree can assist in such cases, as it dosn't care, where and what lies in such tree structure.
     
  16. FernandoDarci

    FernandoDarci

    Joined:
    Jan 25, 2018
    Posts:
    19
    N
    In my approaching of Voxel maps, I pass the world position to each voxel, summing the position to the coordinate, so my function to calculate the voxel block Works with absolute values. I use 16x16 chunks with a perlinNoise2d function to calculate Heights. As they use absolute world coordinates, every chunk is rendered at same position, but the mesh has the distance offset, and both functions can calculate any voxel in the world regardless the distance

    Code (CSharp):
    1. public int GetVoxelHeight(int x, int y)
    2.     {
    3.         //Low tercian height is the minimum height of a chunk, while High Tercian Height is the variance between 0 and the value
    4.         //of it added.
    5.         return biomes[CurrentBiome].LowTercianHeight +
    6.                     Mathf.FloorToInt(biomes[CurrentBiome].HighTercianHeight *
    7.                     Noise.GetPerlin2D(this, new Vector2Int(x, y), biomes[CurrentBiome].Offset, biomes[CurrentBiome].TerrainScale));
    8.  
    9.     }
    10.  
    11.  
    12. public byte GetVoxel(Vector3Int voxel)
    13.     {
    14.         if (voxel.y < 0 || voxel.y > GetVoxelHeight(voxel.x, voxel.z) - 1) return 0;
    15.         else if (voxel.y == 0) return 1;
    16.         else if (voxel.y > 0 && voxel.y < GetVoxelHeight(voxel.x, voxel.z) - 5) return 3;
    17.         else if (voxel.y >= GetVoxelHeight(voxel.x, voxel.z) - 5 && voxel.y < GetVoxelHeight(voxel.x, voxel.z) - 1) return 4;
    18.         else if (voxel.y == GetVoxelHeight(voxel.x, voxel.z) - 1) return 2;
    19.         else return 0;
    20.     }
    21.  
    22. public bool DrawFace(Vector3Int voxel, int face)
    23.     {
    24.         if (voxel.y < 0) return false;
    25.         switch (face)
    26.         {
    27.             case 0: return blocks[GetVoxel(new Vector3Int(voxel.x, voxel.y, voxel.z - 1))].IsSolid;
    28.             case 1: return blocks[GetVoxel(new Vector3Int(voxel.x, voxel.y, voxel.z + 1))].IsSolid;
    29.             case 2: return blocks[GetVoxel(new Vector3Int(voxel.x, voxel.y + 1, voxel.z))].IsSolid;
    30.             case 3: return blocks[GetVoxel(new Vector3Int(voxel.x, voxel.y - 1, voxel.z))].IsSolid;
    31.             case 4: return blocks[GetVoxel(new Vector3Int(voxel.x - 1, voxel.y, voxel.z))].IsSolid;
    32.             case 5: return blocks[GetVoxel(new Vector3Int(voxel.x + 1, voxel.y, voxel.z))].IsSolid;
    33.             default: return true;
    34.         }
    35.     }
    36.  
     
    Last edited: Dec 15, 2019
  17. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    @FernandoDarci I understand your approach. And indeed, your approach to render voxels is functional.

    My question however will be following, providing you haven't answered it yet:
    How do you store chunks of the world, and how you reference them? Lets say, you got chunks on both ends of the world? Or if you want render 24 chunks nearby?

    You can have also a situation, where some chunks are detached from the rest and can be located in any arbitrary position of the map.
     
  18. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    789
    A pure ECS Voxel terrain is not the best approach, simple the amount of data i to large. As example if you has a terrain with 64 * 64 * 16 chunks with 16*16*16 blocks, that makes 268,435,456 blocks. Each entity has two ints as value (Id, Version) that makes 2GB ram only Entity data, without any other data like Voxel type.
     
  19. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    @runner78 it is irrelevant weather is ECS or not. Described problem applies to any type of approach or language.

    The thing is, it matters how designer decides, how chunks are generated and allocated. You don't want typically use a fixed size grid for chunks, but only populate chunks, which were discovered. Unless is strictly predefined world from the beginning, with all pre-generates chunks. Which is typically not what you want, or need for sandboxes open world based games.

    However, it is not uncommon, for Minecraft worlds weight many GB of storage space. Special when traveling a lot.

    Compression and frequent resetting chunks techniques, can help ro reduce and free memory of unused chunks.
     
  20. MrArcher

    MrArcher

    Joined:
    Feb 27, 2014
    Posts:
    106
    One way to really cut down on your world filesize (assuming your generation algorithm is deterministic) is to flag chunks and voxels only once they're modified. When you load in a chunk, you check for modified voxels within the chunk first, and let the rest default back to the world generation algorithm.
     
    NotaNaN and Antypodish like this.
  21. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    789
    Normaly you have a view distance and so a fixed size of visible chunks and originshift with reuse of chunks.
     
  22. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    You are referring now to rendering techniques. Which is fine.

    But we were discussing about storing already generated chunks. Like saving to file for example. @MrArcher pointed out at smart resolution to the matter.

    Also, same default types of voxels in chunk, like air, don't need be saved. Lets say we got sky (air), above height 64. Anything between 64 and ceiling level (I.e 125), could be discarded in smart approach. It can reduce amount of data by much.
     
  23. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    789
    I have also recently experimented with ECS and a block voxel terrain. I store the voxels as simple 1 dimensional byte array, i also playing with octrees.
    My idea was to store the full voxel data in chunk is nessesary, like the chunk near the Player or for Mesh generation.
    For near the Player or interactive voxel you can use Entities.

    If the Worldgeneration change over time like in Minecraft, that not work.
     
  24. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Minecraft uses seed to generate world. So if resetting a chunk for example, it always will revert to same initial state.

    Hence, if chunk data never changed, like block never was destroyed or placed, then can easily just use generator, instead saving chunk to file. With DOTS it is pretty feasible and performant. And could optimize further down, to generate only visible parts and update on demand. I.e. Going into cave. Every player at anytime, will see such chunks in same way. So the only information to be stored is, if chunk is discovered.
     
  25. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    789
    Minecraft save the entire chunk. In the history of Minecraft, the worldgenerate has changed every time new biome added and newer Game version generate different Terrains with the same seed. I you play an old savegame with a newer version you can see the hard line between old chunks und new generated chunks.

    If you now, the game never change the wold generation, then you can use it, or you lock savegames to the game version, or ship the old generator with the game.
     
  26. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    It is correct, that same seed did generate different results, between major different minecraft versions. However, having same major version, and having seed, generated always same results. This is why people did share seeds over internet. For example spawning in village. Or finding specific ground formation (i.e. flying rocks), at specific location.

    Try this, for visualize seed concept (same seed will give you always same map)
    http://mineatlas.com/

    The reason minecraft saves each chunk, is just easier to handle data. No need regenerate, which is relatively computing expensive, and load all blocks. DOTS can easily mitigate that, if anyone desires to do it that way. As far I am aware, Java does not have privilege of burst.
     
  27. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    789
    In my experiment i use burst job for world generation and async/await for Mesh data (and for waiting is the job done). The only performence problem was the generations of the unity Mesh object.
     
  28. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Out of interest, did you use marching cube approach, or similar, for mesh simplification?
     
  29. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    789
    No, 2 tris per visible block-face.
     
  30. FernandoDarci

    FernandoDarci

    Joined:
    Jan 25, 2018
    Posts:
    19
    I save the mesh info (vertices, uvs and triangles calculations) of the chunk in a single binary file with the mesh (x,z) coordinates. So when I need to render it, I ask to the system if the file exists, and if so, I load the file info direct to the method that builds the mesh, if not, I run the rendering algorithm to the chunk.
    I reference each chunk by the x, z position towards the world origin, that in case is (0,0,0) (but can be any position).

    Both "ENDS" of the world is relative, since I dont define a world´s end. I can render chunks infinitely. What I do is put a limit to visible chunks, answering your second doubt, and iterate over the offsets using player position as pivot. So I render 5 chunks each side, and I have 25 rendered chunks around the player. So I repeat the process checking if the player moves, and if so, I disable the chunks, and iterate again, now checking if I need to calculate/load from disk the chunk or simply enable it again.

    As I said before, I just calculate the voxel position by the ABSOLUTE coordinate from the world. No chunks are really detached from one to another, you can have some chunks filled with "Air" separating two chunks (what allow the algorithm to render the sides of the chunk between them, and generate a empty file.)

    Just for completing, the "caching system" is the way minecraft saves the worlds you play, but since the algorithm used to it is unknown, I can´t say what is recorded there. My cache files are very small (Around 35k).
     
  31. FernandoDarci

    FernandoDarci

    Joined:
    Jan 25, 2018
    Posts:
    19
    I´ll try to explain the simplest way I can:
    First, I calculate the chunk using as parameters the (x,z) world coordinates.
    Second, After calculating the mesh, I save the mesh triangles, uvs and vértices in a cache file, normally with 35~40k.
    The file name have a pattern, like "Chunk_X_Z".

    You can´t get chunks in both ENDS of the world for a concept reason. Voxel worlds are virtually horizontally endless.

    If you want render N nearby chunks, just iterate for to get the x and z coordinates relative to pivot (the player in almost cases), and build the chunks inside the loops.

    Code (CSharp):
    1.  
    2. Dictionary<int2, Chunk> TerrainChunks = new Dictionary<int2, Chunk>();
    3. for (int xx = SpawnPosition.x - VisibleChunks.x; xx < SpawnPosition.x + VisibleChunks.x; xx++)
    4.             for (int yy = SpawnPosition.y - VisibleChunks.y; yy < SpawnPosition.y + VisibleChunks.y; yy++)
    5.             {
    6.                 int2 pos = new int2(xx * ChunkSize.x, yy * ChunkSize.y);
    7.                 if (!TerrainChunks.ContainsKey(pos))
    8.                 {
    9.                     Chunk chunkObject = new Chunk(this, pos);
    10.                     TerrainChunks.Add(pos, chunkObject);
    11.                 }
    12.                 else if (!TerrainChunks[pos].IsActive) TerrainChunks[pos].IsActive = true; ;
    13.             }
    Code (CSharp):
    1.  
    2. if (!Deserialize()) //Load the mesh data directly to the BuildMesh method
    3.         {
    4.                 vertices = new List<float3>();
    5.                 Uvs = new List<float2>();
    6.                 triangles = new List<int>();
    7.                 triangleIndex = 0;
    8.                 BuildChunk();
    9.         }
    10.  
    11.         BuildMesh();
    12.  

    I believe that SpawnPosition and VisibleChunks explains itself.


    Detached chunks are a simple, incorrect approach. The best approach in this case is to fill the gap between the chunks with "Air", that is just a dummy block. It don´t generate extra data, and don´t break the chain.
     
  32. FernandoDarci

    FernandoDarci

    Joined:
    Jan 25, 2018
    Posts:
    19
    Just to information, a "canonical" minecraft chunk is 16 * 16 * 256 (anvil approach) and the chunks are 2D over the x,z axis. Each chunk is treated like a unique mesh, and most commonly, just the "surface" blocks are used for drawing the faces.
     
  33. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    789
    So far i know, one Minecraft chunk has 16 sections to 16*16*16 blocks.
     
  34. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Minecraft doe not generate empty chunks, between two arbitrary fixed points in world.

    For example, when teleporting to position 200k, 300k.
    There is no need for such approach. Otherwise things would become very complex, when starting approaching position from different directions.

    Imagine teleporting to 10 random places in world, which haven't been discovered yet. Now following proposed approach, every location would need to be linked with each other, with air chunks. And considering multiple possible paths for each chain. That is definitely wrong approach and bring multiple other issues along side.
     
  35. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    Sounds like optimal solution, to reduce data usage. Also, it reduces requirement, to generate whole chunk at once.
     
  36. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    I hope you don't mind if I ask here. I also have some voxel data and a world split into chunks of voxels. Now, did you guys try accessing voxel data via Hashmap?

    The alternative being: Take voxel's world coordinates, calculate chunk coordinates, turn chunk coordinates into chunk index, get the chunk, get it's voxel buffer, calcualate local voxel coordinates, turn those into voxel index, now access any voxel data via index in that chunk.

    The second approach sounds kinda slow, but I assume hashmap would still be slower. Or not?
     
  37. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    @illinar
    I haven't used hashmap to store voxels, since linear array is most optimal in this case, and easy to check neighbors. Also, because of that, voxel blocks in chunks are ordered in most cases. But I would consider use hashmap, to store chunks, if anything, as these are most likely be none ordered anyway. Unless we have fixed size world. Then I would consider some array, to store chunks.
     
  38. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    My world consists of objects that are placed in a voxelized world. For example a wall is an object, a staircase is an object. I need to know which voxel in the world contains an object (it's Entity) and which is empty. So I'm considering NativeMultiHashMap so each object will have a set of int3 that point to it. So wondering about performance difference. about the described two ways of accessing the voxels. I'm guessing that if there is a difference, the simplicity of "direct" access via hashMap is still worth it. Plus this will take less memory than "3d" arrays of voxels since most of my world is air.
     
  39. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    @illinar exact performance may vary, depending on implementation.

    If you have simple 1D array, you have data linearly organized. But yes, it cost extra memory, in respect to hashMap.
    However, you don't need load every single chunk of the world at once. Only having some. Now when you move in the world and load new chunks, you need deal with checking unused keys of blocks, to avoid potential hashMap collisions. In Dictionary, at least we can override directly given key. Don't know, if we can do that, without deleting key element first in hashMap.

    But I may be missing some analysis here.

    The downside with hashMap, is that you are using Vector3 / float3, and calculating hashKey, if you want retrieve specific blocks. That adds a little overhead. Don't know how much exactly, in respect to calculating array index. Then risking potential collision in hashMap, for large data sets.

    Personally I like work on arrays, they are easier to jobify for me.
    Plus arrays / buffers, are easy to debug, which is big THUMB up.

    Here is some I think interesting reading
    https://blog.tedd.no/2018/01/13/speeding-up-unitys-vector-in-lists-dictionaries/

    But in the end, I suggest run test and profile results. I haven't done any extensive / stress tests.

    I would be interested in comparison, of lets say placing 1000 houses, on uneven terrain in 1000 chunks, using hashMap vs array.
    Of course using burst and with jobs.
     
    illinar likes this.
  40. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    @Antypodish I profiled and found some interesting results.

    First the simple hashmap with vector keys:
    HashMap<int3, Entity>() capacity: 1.000.000 contains: 100.000
    Trying 1mm keys: 14ms


    Then int64 keys with conversion to 1D index, and without for comparison.
    HashMap<long, Entity>() capacity: 1.000.000 contains: 100.000
    Trying 1mm keys with conversion of 3d index to 1d: 10ms
    Trying 1mm plain int keys: 2.3ms


    So I though the convertion of world coordinates to index takes 7.7ms. Bu that didn't seem right. It takes barely any time, but there are apparently hash collisions on those converted coordinates and they take up the bulk of access time.

    I would have to create my own optimized key type for this to perform better if that would work at all.

    Code (CSharp):
    1. Entities.ForEach((ref HashMapTest t) =>
    2.         {
    3.             for (int x = 0; x < 1; x++)
    4.             {
    5.                 for (int y = 0; y < 1000; y++)
    6.                 {
    7.                     for (int z = 0; z < 1000; z++)
    8.                     {
    9.                         long i = x + 100 * (y + 100 * z);
    10.                         t.b = hmI.TryGetValue(i, out var e);
    11.                     }
    12.                 }
    13.             }
    14.         }).WithName("LONG_HASH_TEST").WithReadOnly(hmI).Schedule(handle).Complete();
    I'll try buffers now. The problem with buffers is that I can't come up with a good API to accessing chunkEntity>chunkBuffer>voxelEntity>Data

    It's bulky as hell.

    GetVoxel(EntityManager, BufferFromEntity<ChunkBufferElement>, BufferFromEntity<VoxelBufferElement>, int3 position);
    ??

    And I need to get those buffer accessors first as well.
     
    andywatts and Antypodish like this.
  41. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    I forgot to try another solution. To try improving ratio between HashMap capacity and contained count. Previous ratio of 10 gave 10ms, 100 gives 3 and 1000 gives 2.

    But I can't allocate a HashMap bigger than 1000.000, it chrashed Unity. And I think I would need 10-100mm at least.
     
  42. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    @illinar interesting.
    Buffer maybe feel a bit bulky but worth a try.
    Is just going through accessor, using chunks, or do simpler IJobForEach, which is just based on chunks anyway, but you avoid accessor part. Providing I understood your case, as in per IJobChunk.

    I assume good time result, is based on multithreading, as job is spread across. But yes, I would be careful with collisions. Anyway I think hash got good timing for set of 100k.
     
  43. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,754
    There is a point, where you keep data on the drive in files per chunk, and load dynamically, as travelling through te world. At the same time unloading old chunk data and replacing with new.

    For that purpose, I am not convinced hashmap is good approach. While having entities per chunk, with buffer of voxels in that chunk, is much easier to manage loading and unloading data.

    Unloading is just deleting entity with whole its voxels buffer. Lets say when going far away from the chunk. Loading chunks, is just reading from file first. That is most expensive part, and loading to buffer afteward. Or otherwise to hashmap, providing there isn't collisions, to keep stuff efficient.

    I can imagine things go potentially south with hashmap, when playing for one hour and travelling a lot.
     
  44. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    Of course the world is still in chunks and loaded dynamically. But yes, the price of removing and adding keys should also be accounted for, and I'll have to profile that if (before) I decide to use HashMap. Will test and compare buffers first.
     
  45. Sarkahn

    Sarkahn

    Joined:
    Jan 9, 2013
    Posts:
    440
    I've been working on this too, trying to decide how to best divide and access chunks. As was mentioned earlier, in minecraft a "chunk" in the way it's seen visually inside the game is actually a stack of 16x16x16 sections of blocks. I imagined this would make it easy to reference/work with large areas of the world and ignore empty sections where it makes sense.

    So I divided my world into "regions", which are exactly what minecraft calls a chunk. In my game a chunk is a 16x16x16 section of blocks, so a region is a stack of chunks. In minecraft a "region" mean something completely different too but hey, I need to differentiate a subsection of blocks from a stack and the naming collision nightmare is already underway, let's keep it rolling.

    Anyways, in thinking about how to access regions from arbitrary systems I came up with this: A RegionProvider maintains an internal NativeHashMap<int2,Entity>. I don't want to have to worry about passing around job handles so when a system wants to access a region it creates a RegionsRequest entity, which is just an entity with a a dynamic buffer of indices of the regions it wants to access.

    While processing these Request entities the provider can generate/load any requested regions as needed and then fill a separate ResultEntities buffer on the Request. Once that's done the provider can tag the request as fulfilled.

    The original system can query for the completed request and handle it as needed. Here's the ProviderSystem:
    Code (CSharp):
    1. namespace BlockWorld.Regions
    2. {
    3.     public class RegionProviderSystem : JobComponentSystem
    4.     {
    5.         NativeHashMap<int2, Entity> _regionMap;
    6.  
    7.         EndSimulationEntityCommandBufferSystem _barrierSystem;
    8.  
    9.         EntityQuery _unitializedRegionsQuery;
    10.        
    11.         protected override void OnCreate()
    12.         {
    13.             _regionMap = new NativeHashMap<int2, Entity>(100, Allocator.Persistent);
    14.             _barrierSystem = World.GetOrCreateSystem<EndSimulationEntityCommandBufferSystem>();
    15.            
    16.             _unitializedRegionsQuery = GetEntityQuery(
    17.                 ComponentType.ReadOnly<UninitializedRegion>(),
    18.                 ComponentType.ReadOnly<RegionIndex>(),
    19.                 ComponentType.ReadOnly<ChunkEntityBuffer>()
    20.                 );
    21.         }
    22.  
    23.         protected override void OnDestroy()
    24.         {
    25.             _regionMap.Dispose();
    26.         }
    27.  
    28.         protected override JobHandle OnUpdate(JobHandle inputDeps)
    29.         {
    30.             var commandBuffer = _barrierSystem.CreateCommandBuffer();
    31.             var concurrentCommandBuffer = commandBuffer.ToConcurrent();
    32.  
    33.             var regionMap = _regionMap;
    34.  
    35.             // First update:
    36.             //  Process initial requests -
    37.             //    First we write all requested indices to a queue in parallel. We also tag
    38.             //    our requests as Processing so they can be processed properly in the next update.
    39.             //    From the queue we generate a list of unique indices from all requests.
    40.             //    From that list we check each index against our RegionMap to see which regions are requested
    41.             //    but haven't yet been generated, and set those up for creation via the command buffer.
    42.             #region First_Update
    43.            
    44.             NativeQueue<int2> allRequestedRegions = new NativeQueue<int2>(Allocator.TempJob);
    45.             var allRequestedRegionsWriter = allRequestedRegions.AsParallelWriter();
    46.  
    47.             // Push all requested region indices into a queue and tag our requests as processing.
    48.             inputDeps = Entities
    49.                 .WithAll<RegionsRequest>()
    50.                 .WithNone<RegionsRequestProcessing>()
    51.                 .WithNone<RegionsRequestFulfilled>()
    52.                 .ForEach((int entityInQueryIndex, Entity e, in DynamicBuffer<RegionsRequestIndices> indices) =>
    53.                 {
    54.                     for (int i = 0; i < indices.Length; ++i)
    55.                         allRequestedRegionsWriter.Enqueue(indices[i]);
    56.  
    57.                     concurrentCommandBuffer.AddComponent<RegionsRequestProcessing>(entityInQueryIndex, e);
    58.                 }).Schedule(inputDeps);
    59.            
    60.             NativeList<int2> uniqueRequestedRegions = new NativeList<int2>(Allocator.TempJob);
    61.  
    62.             // Build a list of all unique requested region indices from our queue
    63.             inputDeps = Job
    64.                 .WithCode(() =>
    65.                 {
    66.                     while (allRequestedRegions.Count > 0)
    67.                     {
    68.                         var index = allRequestedRegions.Dequeue();
    69.                         if (!uniqueRequestedRegions.Contains(index))
    70.                             uniqueRequestedRegions.Add(index);
    71.                     }
    72.                 }).Schedule(inputDeps);
    73.  
    74.             allRequestedRegions.Dispose(inputDeps);
    75.  
    76.             var deferredRequested = uniqueRequestedRegions.AsDeferredJobArray();
    77.  
    78.             // Generate requested regions that haven't yet been created
    79.             inputDeps = Job
    80.                 .WithReadOnly(regionMap)
    81.                 .WithCode(() =>
    82.             {
    83.                 for (int i = 0; i < deferredRequested.Length; ++i)
    84.                 {
    85.                     int2 regionIndex = deferredRequested[i];
    86.                     if (!regionMap.ContainsKey(regionIndex))
    87.                     {
    88.                         Entity e = commandBuffer.CreateEntity();
    89.                         commandBuffer.AddComponent<RegionIndex>(e, regionIndex);
    90.                         commandBuffer.AddBuffer<ChunkEntityBuffer>(e);
    91.                         commandBuffer.AddComponent<UninitializedRegion>(e);
    92.                     }
    93.                 }
    94.             }).Schedule(inputDeps);
    95.  
    96.             uniqueRequestedRegions.Dispose(inputDeps);
    97.            
    98.             #endregion
    99.  
    100.  
    101.             // Second update:
    102.             //   Write newly generated regions to the region map -
    103.             //     We can't write the newly generated entities into the region map during
    104.             //     the first update since the entities returned by CommandBuffer.CreateEntity
    105.             //     are only valid within the context of the barrier system. Once the barrier system
    106.             //     finishes at the end of the first update, we can query for our generated region entities
    107.             //     and add them to the map, removing their "uninitialized" tag
    108.             #region Second_Update
    109.            
    110.             // Write our generated region entities to the region map in parallel
    111.             var mapWriterParallel = regionMap.AsParallelWriter();
    112.             inputDeps = Entities
    113.                 .WithAll<UninitializedRegion>()
    114.                 .ForEach((int entityInQueryIndex, Entity e, in RegionIndex index) =>
    115.                 {
    116.                     mapWriterParallel.TryAdd(index, e);
    117.                     concurrentCommandBuffer.RemoveComponent<UninitializedRegion>(entityInQueryIndex, e);
    118.                 }).Schedule(inputDeps);
    119.  
    120.             // Now that we know for sure our regions will be generated in the barrier system, we can add the
    121.             // buffer to the request entity which will hold the requested region entities.
    122.             inputDeps = Entities
    123.                 .WithAll<RegionsRequestProcessing>()
    124.                 .WithNone<RegionsRequestResultEntities>()
    125.                 .ForEach((int entityInQueryIndex, Entity e) =>
    126.                 {
    127.                     concurrentCommandBuffer.AddBuffer<RegionsRequestResultEntities>(entityInQueryIndex, e);
    128.                 }).Schedule(inputDeps);
    129.  
    130.             #endregion
    131.  
    132.  
    133.             // Third update -
    134.             //    Fill the newly added buffers on our processing requests and tag the requests as fullfilled
    135.             #region Third_Update
    136.  
    137.             inputDeps = Entities
    138.                 .WithReadOnly(regionMap)
    139.                 .WithAll<RegionsRequestProcessing>()
    140.                 .ForEach((
    141.                     int entityInQueryIndex,
    142.                     Entity e,
    143.                     ref DynamicBuffer<RegionsRequestResultEntities> entityBuffer,
    144.                     in DynamicBuffer<RegionsRequestIndices> requestedIndices) =>
    145.                     {
    146.                         for (int i = 0; i < requestedIndices.Length; ++i)
    147.                             entityBuffer.Add(regionMap[requestedIndices[i]]);
    148.                         concurrentCommandBuffer.RemoveComponent<RegionsRequestProcessing>(entityInQueryIndex, e);
    149.                         concurrentCommandBuffer.AddComponent<RegionsRequestFulfilled>(entityInQueryIndex, e);
    150.                     }).Schedule(inputDeps);
    151.            
    152.             #endregion
    153.  
    154.            
    155.             _barrierSystem.AddJobHandleForProducer(inputDeps);
    156.  
    157.  
    158.             return inputDeps;
    159.         }
    160.     }
    161. }
    Of course this is completely separate from how you would actually access the blocks inside the chunks inside the regions...it's a lot for my tiny brain to handle.

    It feels a bit messy right now. I'm still really struggling with how to handle complex interactions like this in ECS. It would be nice to just have a "WriteToBlocks" job that you can call from anywhere and you can depend on it and it would magically write blocks to the world but I'm not sure how to make that work in an ecs and job-friendly way.
     
    NotaNaN and illinar like this.
  46. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    I'm looking for patterns as well. I don't think delayed access would work for me, since I won't know in advance which voxels I would need.

    I'm very happy with what I've came up with in terms of usability.

    Code (CSharp):
    1. using Unity.Entities;
    2. using Unity.Jobs;
    3. using Unity.Collections;
    4. using Unity.Mathematics;
    5.  
    6. public class CreateChunksSystem : JobComponentSystem
    7. {
    8.     public static int chunkSize = 128;
    9.     public static int mapSize = 1000;
    10.  
    11.     protected override void OnCreate()
    12.     {
    13.         EntityManager.CreateEntity(typeof(BufferTest));
    14.         EntityManager.AddBuffer<ChunkBufferElement>(EntityManager.CreateEntity());
    15.         var chunkBufferEntity = GetSingletonEntity<ChunkBufferElement>(); //just testing
    16.         for (int cx = 0; cx < 10; cx++)
    17.         {
    18.             for (int cy = 0; cy < 10; cy++)
    19.             {
    20.                 var chunkEntity = EntityManager.CreateEntity();
    21.                 var voxelBuffer = EntityManager.AddBuffer<VoxelBufferElement>(chunkEntity);
    22.                 for (int x = 0; x < chunkSize; x++)
    23.                 {
    24.                     for (int y = 0; y < chunkSize; y++)
    25.                     {
    26.                         for (int z = 0; z < chunkSize; z++)
    27.                         {
    28.                             voxelBuffer.Add(1);
    29.                         }
    30.                     }
    31.                 }
    32.                 var chunkBuffer = EntityManager.GetBuffer<ChunkBufferElement>(chunkBufferEntity);
    33.                 chunkBuffer.Add(chunkEntity);
    34.             }
    35.         }
    36.     }
    37.  
    38.     protected override JobHandle OnUpdate(JobHandle handle)
    39.     {
    40.         return handle;
    41.     }
    42. }
    43.  
    44. public class BufferTestSystem : JobComponentSystem
    45. {
    46.     protected override JobHandle OnUpdate(JobHandle handle)
    47.     {
    48.         var vp = this.GetVoxelProvider();
    49.         Entities.ForEach((ref BufferTest bt) =>
    50.         {
    51.             for (int x = 0; x < 100; x++)
    52.             {
    53.                 for (int y = 0; y < 100; y++)
    54.                 {
    55.                     for (int z = 0; z < 100; z++)
    56.                     {
    57.                         bt.i = vp.GetVoxel(x, y, z);
    58.                     }
    59.                 }
    60.             }
    61.         }).WithName("BUFFER_TEST").WithReadOnly(vp).Schedule(handle).Complete();
    62.  
    63.         return handle;
    64.     }
    65. }
    66.  
    67. public struct VoxelProvider : IComponentData
    68. {
    69.     private BufferFromEntity<VoxelBufferElement> VoxelBuffers;
    70.     private BufferFromEntity<ChunkBufferElement> ChunkBuffers;
    71.     private ComponentDataFromEntity<MapChunk> MapChunkComponents;
    72.     private Entity ChunkBufferEntity;
    73.  
    74.     public VoxelProvider(BufferFromEntity<ChunkBufferElement> chunkBuffers, BufferFromEntity<VoxelBufferElement> voxelBuffers, ComponentDataFromEntity<MapChunk> mapChunkComponents, Entity chunkBufferEntity)
    75.     {
    76.         ChunkBuffers = chunkBuffers;
    77.         VoxelBuffers = voxelBuffers;
    78.         MapChunkComponents = mapChunkComponents;
    79.         ChunkBufferEntity = chunkBufferEntity;
    80.     }
    81.  
    82.     public int GetVoxel(int x, int y, int z)
    83.     {
    84.         return GetVoxel(new int3(x, y, z));
    85.     }
    86.  
    87.     public int GetVoxel(int3 position)
    88.     {
    89.         var chunkPosition = position / Map.ChunkSize;
    90.         var voxelPosition = position - chunkPosition * Map.ChunkSize;
    91.         return VoxelBuffers[ChunkBuffers[ChunkBufferEntity][GetChunkIndex(chunkPosition)].ChunkEntity][GetVoxelIndex(chunkPosition)].Value;
    92.     }
    93.  
    94.     public static int GetVoxelIndex(int3 vector)
    95.     {
    96.         return GetVoxelIndex(vector.x, vector.y, vector.z);
    97.     }
    98.  
    99.     public static int GetVoxelIndex(int x, int y, int z)
    100.     {
    101.         return x + Map.ChunkSize * (y + Map.ChunkSize * z);
    102.     }
    103.  
    104.     public static int GetChunkIndex(int3 vector)
    105.     {
    106.         return GetChunkIndex(vector.x, vector.y, vector.z);
    107.     }
    108.  
    109.     public static int GetChunkIndex(int x, int y, int z)
    110.     {
    111.         return x + Map.Size * (y + Map.Size * z);
    112.     }
    113. }
    114.  
    115. public static class ComponentSystemExtensions
    116. {
    117.     public static VoxelProvider GetVoxelProvider(this ComponentSystemBase cs)
    118.     {
    119.         return new VoxelProvider(cs.GetBufferFromEntity<ChunkBufferElement>(), cs.GetBufferFromEntity<VoxelBufferElement>(), cs.GetComponentDataFromEntity<MapChunk>(), cs.GetSingletonEntity<ChunkBufferElement>());
    120.     }
    121. }
    122.  
    123.  
    124.  
    125.  
    126.  
    127.  
    128. public struct BufferTest : IComponentData
    129. {
    130.     public bool b;
    131.     public long i;
    132. }
    133.  
    134. [InternalBufferCapacity(0)]
    135. public struct ChunkBufferElement : IBufferElementData
    136. {
    137.     public static implicit operator Entity(ChunkBufferElement e) { return e.ChunkEntity; }
    138.     public static implicit operator ChunkBufferElement(Entity e) { return new ChunkBufferElement { ChunkEntity = e }; }
    139.  
    140.     public Entity ChunkEntity;
    141. }
    142.  
    143. [InternalBufferCapacity(0)]
    144. public struct VoxelBufferElement : IBufferElementData
    145. {
    146.     public static implicit operator int(VoxelBufferElement e) { return e.Value; }
    147.     public static implicit operator VoxelBufferElement(int e) { return new VoxelBufferElement { Value = e }; }
    148.  
    149.     public int Value;
    150. }
    I'm a little to tired atm to explain, but check the usage it's very simple, convenient and safe:
    Code (CSharp):
    1. public class BufferTestSystem : JobComponentSystem
    2. {
    3.     protected override JobHandle OnUpdate(JobHandle handle)
    4.     {
    5.         var vp = this.GetVoxelProvider();
    6.         Entities.ForEach((ref BufferTest bt) =>
    7.         {
    8.             for (int x = 0; x < 100; x++)
    9.             {
    10.                 for (int y = 0; y < 100; y++)
    11.                 {
    12.                     for (int z = 0; z < 100; z++)
    13.                     {
    14.                         bt.i = vp.GetVoxel(x, y, z);
    15.                     }
    16.                 }
    17.             }
    18.         }).WithName("BUFFER_TEST").WithReadOnly(vp).Schedule(handle).Complete();
    19.  
    20.         return handle;
    21.     }
    22. }
    Which is cruel for me to point out since now I'm gonna tell you that it is slow af.

    Must be this line:


    return VoxelBuffers[ChunkBuffer[GetChunkIndex(chunkPosition)].ChunkEntity][GetVoxelIndex(voxelPosition)].Value;


    With all DOTS safety features off it does 1,000,000 accesses in 12 ms. The hashMap does the same in 2 ms with int64 key 1000,000 capacity and 100,000 elements.

    Edit: There are some logical errors there, and I've improve
     
    Last edited: Dec 28, 2019
    NotaNaN, andywatts and Sarkahn like this.
  47. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    There was a logical error there. I fixed it. Now reading voxels from buffers takes 30ms. There still might be errors, but shouldn't be ones that affect performance.

    Next step is to avoid using buffers, and go straight for native arrays.
     
    NotaNaN likes this.
  48. Sarkahn

    Sarkahn

    Joined:
    Jan 9, 2013
    Posts:
    440
    I think that access chain qualifies as a Coding War Crime. Sorry but you're going to Coding Jail. :D

    Unless I'm missing something (and I may very well be, I am a newb when it comes to this stuff) delayed access is the only way to benefit from ecs and the job system. If you NEED immediate direct access (which I don't think you do in most cases) then you're probably better off just not using ecs. You could still get a lot of benefit from the job system doing it that way, but I feel like asynchronous events are part and parcel of creating scalable systems in ECS.

    The problem with your direct access method as you pointed out is the access chain. Doing all those ComponentDataFromEntity/BufferFromEntity calls in a row is of course going to be slow as hell compared to direct container access. The natural evolution of my RegionProvider system above is to eventually have BlockProvider Systems that let you create massive batch operations that can efficiently operate on chunks in parallel, and return the results to the caller once the operations are done, which is what you're losing out on with your direct access method.

    If you need to know what's in a chunk, you ask the provider to gather all the blocks efficiently and give you a nice container of what you're looking for. Once that's done you can schedule your write operations based on what you're told is there.
     
    NotaNaN likes this.
  49. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    863
    I really can't come up with a use case where your approach is beneficial (mb cos I'm tired). Can you give an example? My readonly "direct" access is still jobified and parallel or as random as it would be with any approach.

    The original was even better, btw: VoxelBuffers[ChunkBuffers[ChunkBufferEntity][GetChunkIndex(chunkPosition)].ChunkEntity][GetVoxelIndex(voxelPosition)].Value;

    I think I have a good solution coming. Couple more things I will test first.
     
    Last edited: Dec 28, 2019
    NotaNaN likes this.
  50. Sarkahn

    Sarkahn

    Joined:
    Jan 9, 2013
    Posts:
    440
    Hah, you're right. Sorry, I was misreading your loop and didn't see that it is in fact running in parallel. I guess since you are using CDFE/BFE ECS will still handle dependencies for you too, and it does give nice and simple random access.

    It's hard to me to come up with an example since I I haven't really written it out yet, your way may in fact just be objectively better. I think at the very least what I'm thinking of could help avoid the cdfe/bfe access chain on every single block access, which would make it significantly faster for batch operations. If you can define block operations for large areas of blocks the providers can read/write directly on their buffers as nativearrays, or some portion their buffers using slices/memcpy.
     
    NotaNaN likes this.