Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

World Building ProTiler - The Streaming 3D TileWorld Editor for Unity

Discussion in 'Tools In Progress' started by CodeSmile, Mar 23, 2023.

?

Like wuuuut?

  1. Finally!

    100.0%
  2. Meh.

    0 vote(s)
    0.0%
  1. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    ProTiler is the streaming TileWorld Editor for Unity designed specifically for 3D world design. Interested?
    I plan to submit it to the Asset Store in May 2023 for FREE (early access).
    Screenshot 2023-03-22 175736.png

    Please leave a comment to let me know what you're looking forward to having in a tile editor that's built specifically to support 3D worlds and focuses on design first, generating content later. That means: handcraft the design and gameplay quickly, then save tons of time detailing the world - rather than vice versa.

    Playable Runtime Example (WebGL)
    This is a (unfinished) world I built today as I added more features, most notably tile rotation and flipping. Even with zero optimizations the tile streaming works smoothly on WebGL. Graphics by PolyPerfect.
    Screenshot 2023-03-23 180641.png
    I will be updating these WebGL builds frequently ...

    Boundless Worlds
    Draw anywhere! Worlds can be any shape!

    There is nothing forcing you to draw tiles within a pre-sized rectangle, or a rectangle to begin with.

    Streaming Renderer
    Tiles are instantiated on the fly in a configurable visible area. I plan on making a callback that allows you to customize what "visible area" means for your game.

    While rendering currently relies on the established MeshRenderer, custom rendering is an option for anyone who wants to pursue it. Personally I wish to implement an alternative Entities Graphics renderer in the future.

    Tiles aren't GameObjects
    Tiles are not stored as GameObjects in the scene, thus the scene file remains reasonably small.

    Visible tiles are pooled game objects instantiated from prefabs. Everything that's in a tile prefab will work as you expect it to. No surprises there! Just consider that the lifetime of a tile instance is temporary.

    Responsive Editing
    You can create huge worlds without having a sluggish Unity Editor performance. I've had a million tiles with the editor scene view still reasonably responsive even in its unoptimized state that doesn't use spatial division tile storage (chunking) yet.

    If you did this with the each tile is a GameObject madness, Unity Editor would just succumb to the progressbar and your scene file size would be measured in gigabytes.

    Made with Experience

    I've been working with and on tile-map design tools and games since the late 1980's on Amiga games, then several GameBoy color games with its tiling hardware followed by RPG/RTS games in the 2000's using tile-based terrains. I've been using Unity since 2013 and really enjoy elevating 2D TileMaps to 3D TileWorlds.

    Live Development
    I publicly document the development of the editor, working almost daily on it now and for the foreseeable future. Feel free to join me:
     
    Skewbald_Mare, knxrb and cerestorm like this.
  2. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    I added rotation and flipping of tiles, and "fill rectangle" drawing and accordingly updated the WebGL demo.

    The new build includes far fewer "holes" in the world (thanks to rect drawing) and a performance fix where playback would freeze every now and then. This was caused by Mesh.Bake PhysX CollisionData according to the Profiler, and only during Activate/Deactivate of GameObjects (pooling). This 13-year old forum thread pointed me in the right direction.

    I also installed dotCover to Rider and love it. It lets you auto-run tests and provides more visible, immediate code coverage stats than Unity's Code Coverage package does. I no longer leave Rider to run those unit tests either which makes working with tests a breeze now. Very happy with that! :)
     
  3. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    It's been a week since my last update. I went back to the drawing board as I realized that after making the prototype, the actual implementation was still quickly becoming a drag to work in. I dug my head back into refactoring tasks, as well as researching tilemap tools (specifically Tiled and Unity 2D Tilemaps) and how they approach tasks.

    It does hurt a bit because I made an announcement, then gotten into both a tech and a mental drag, and now I'm back starting from scratch again. Although I know it's ultimately for the better, I just wish I had more visual progress to show.

    I do come out of that with a sound architectural design, as well as designs for some of the systems I want to have in place that no other tilemapping tool has. I have no word for it other than "masking", but it's a concept used by other apps like Photoshop, where you can take a layer of brush strokes and enable or disable each rather than making all edits in the target layer.

    For example, imagine you want to draw a network of rivers, roads, and railways with a regular tilemap. Decade-long conception has it that whenever you wanted to change a river, road or railway layout then ... you'll be clearing the tiles you don't want anymore and then draw new tiles elsewhere. That's a two step process that has you drawing over both existing and new tiles, making some drawing mistakes along the way too.

    But ... what if you wouldn't have to erase and draw any tiles at all to accomplish this?

    I wish to define an interconnected network of brushstrokes that persists. Each brush network has a tile type like river, road or railway assigned to it. This brush applies its tiles (and rules) to the map whenever either the brush or close-by tiles change.

    To change where this brush network is drawing, you would simply grab an edge of a path and drag it to either side, or drag a corner, or pull out another corner to make an intersection and add some more line-drawings to the network. You could even offset the entire brush.

    In a similar fashion, you could decorate an area (rectangle, circle or any selection of tiles) that applies brush strokes of "detailing" to it. This detailing could be (randomized) flowers, trees, rocks and so on, each with their own rules, so they won't draw over road or river tiles for instance. You can then change the actual tilemap underneath without losing any of these details, nor would they suddenly be on your roads. In the same way you can change the detail brush by adjusting its borders to cover more or less tiles as needed.

    Does that sound like a fun and productive way to edit tilemaps? :)
     
    Last edited: Apr 2, 2023
    Olipool likes this.
  4. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    Two weeks passed and nothing? Not really, I moved. :eek:

    And whenever I had some free time I felt compelled to set up CI (continuous integration) using GameCI and Github Actions.

    Now I can run automated tests and builds for specific (even multiple) Unity versions. Here's the output for the lowest Unity version ProTiler will support:
    upload_2023-4-18_10-41-24.png

    I'm targeting 2021.2 as the lowest supported version for ProTiler due to availability of the Overlay API. And to some extent EditorTool, UI Builder and C# 9 features that aren't essential but I can take advantage of since Overlay forces me to use 2021.2 anyway.

    This GDC 2023 talk by the LEGO Lead Platform Architect then prompted me to investigate project dependencies and use of packages, as well as enabling the most aggressive code/engine stripping settings to support those extremely effective app-size reduction measures asap.

    For me, the iOS build size is particularly concerning. Even after optimizing every build setting for size and enabling compression I only got it down from 960 MB to 680 MB. The additional size stripping down to 514 MB came entirely from removing unnecessary dependencies and packages and rewriting some code to be more conservative. Still, I wonder if there could be done more?

    CI helped a lot applying and testing these changes because I could just check in, have lunch and afterwards just compare the results with ease.

    I understand the build size isn't representing the installed app bundle size. But since Github only allows you to store 2 GB for the smallest paid plan I was trying to be able to at least store all artifacts of the latest run, possibly for two Unity versions.

    Right now, I'm building from the bottom up, putting the low-level classes under unit tests. I should have something more visual to show in another two weeks.
     
    mike2022unity likes this.
  5. mike2022unity

    mike2022unity

    Joined:
    Nov 29, 2022
    Posts:
    7
    upload_2023-5-24_16-33-10.png


    How is this implemented? Can you share the C# code thanks
     
  6. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    This is not mine, it’s [Graphy] and available for free on the asset store.
     
    Last edited: May 24, 2023
    mike2022unity likes this.
  7. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    Just to not be completely silent: Yes, I keep hacking away at ProTiler daily!

    The past two weeks I began a complete overhaul AGAIN ditching the first full-fledged design. I've now fully embraced the new goodies Unity provides rather than trying to avoid them because some might think about that as "bloat".

    That means I make heavy use of Collections and Mathematics to enable Burst and Jobs in the backend. I will also rely on the Unity.Serialization package even though that is only out of preview for Unity 2022.2. The preview versions work fine in 2021 though.

    The issue with serialization being that I cannot rely on Unity's built-in serialization, like [SerializeField] and such. I want to support runtime, thus I have to ensure I can serialize at runtime too. And make the Undo/Redo system work at runtime too. And streaming chunks.

    In fact, initially I wanted to "think about runtime after v1" but now supporting runtime is my primary focus because it's not that difficult to add if you design for it. If you don't, you'd have to rewrite it, or support two separate code paths.

    So right now I'm having fun building up the plan I laid out and testing out serialization. I hacked the entire class design, namespaces, call flows out in stub classes to see if I run into any issues and have me generate UML diagrams from the code using PlantUML:
    upload_2023-6-17_21-46-34.png

    Finally an optimal way to create UML! I always felt the other way around wasn't just a drag, it was dead wrong. :confused: UML never alerts you to compile errors that might invalidate the whole design - you know, stupid but understandable oversights such as a design built on structs with an "abstract base struct".:D
     
  8. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    It's such a hot day today I could not code. Instead, I started writing down how to work with the ProTiler grid/map data model since that is already very well fleshed out and supports any kind of (value-type) data in chunked maps, laid out in either linear collections (one instance for each coord) or sparse collections (only instances for coords that you set data to).

    Let me know what you think of the first draft, any questions or any feedback is very welcome! :)

    (some of the text is missing context, like there's more to be said about serialization)
    ((the entire doc is available online which is both my tech design, old and new, and starts with some completely unnecessary ProTiler history that I wrote so I won't forget))


    =============== Manual Draft Excerpt ================
    ProTiler Data Storage

    ProTiler’s data model allows storing any value-type for any grid coordinate, where the map is divided into custom-sized chunks.

    Storing Value-Type Data
    That means any simple type (int, float, bool, byte, enum) and any struct containing simple types. Even native collections are possible, such as BitArray.


    Example data struct:

    Code (CSharp):
    1. public struct MyData {
    2.     public int MyValue;
    3.     public byte MyFlags;
    4. }
    Serializing Data
    Note that you do NOT need to flag the struct as [Serializable] nor do you need to attribute the fields.


    What gets serialized, and how, is determined solely by the IBinarySerializable interface methods Serialize() and Deserialize().


    You do not normally need to implement those for simple value types or structs containing simple value types such as Vector3 or int4. But you can use it to, for example, serialize an int as ushort if you know that it will never exceed 65k.


    Most importantly, you can use the IBinarySerializable interface to serialize collection types that aren’t supported by ProTiler (yet), such as UnsafeBitArray or UnsafeHashSet. I do aim to support all collection types in the future by writing IBinaryAdapter implementations.


    Here’s a complete example how you would write serialization code manually. As you can see, it is far from difficult:


    Code (CSharp):
    1. public struct TestData : IBinarySerializable
    2. {
    3.     public int3 Coord;
    4.     public UInt16 Index;
    5.  
    6.     public unsafe void Serialize(UnsafeAppendBuffer* writer)
    7.     {
    8.         writer->Add(Coord);
    9.         writer->Add(Index);
    10.     }
    11.  
    12.     public unsafe void Deserialize(UnsafeAppendBuffer.Reader* reader, Byte dataVersion)
    13.     {
    14.         switch (dataVersion)
    15.         {
    16.         case 0:
    17.             Coord = reader->ReadNext<int3>();
    18.             Index = reader->ReadNext<UInt16>();
    19.             break;
    20.         default:
    21.             throw new SerializationVersionException(dataVersion);
    22.         }
    23.     }
    24. }

    Note the use of the unsafe keyword due to the use of the UnsafeAppendBuffer* pointers. The script must be in an assembly definition that has the “allow unsafe code” checkbox enabled and the method must include the unsafe keywords in its definition.


    Also take note of the dataVersion byte. When serializing, you will be able to pass in the “current” version. When you do make a breaking change but you want to keep the old data still deserializable you can use the dataVersion byte to deserialize previous formats so that the designers don’t have to trash their worlds and start from scratch.


    A breaking change is when you modify the struct by:

    • removing a field

    • changing a field’s Type

    • adding a new field which must not have its value set to “default”

    Then, when reading the data and the version is not the current-most version, you would:

    • read the data of a removed field, but discard its value

    • read the data of a field using its previous Type, then convert it to its current Type

    • compute/assign a non-default value to the new field that must not have a “default” value

    The next time this data is serialized it will be serialized with the current dataVersion and can then be considered as “upgraded”.


    Since supporting many previous versions of data will become cumbersome you should remove previous version support from time to time. For example, you could read and write all data once to ensure it is up to date, then remove all previous version support and reset the current version to zero.


    Do not worry about the byte not supporting more than 256 binary versions. Rather, be happy about not having to test, debug and support this many backward versions. If you still feel this is restrictive, you can always introduce an entire new data map with a separate struct. You may want to do so anyway because data should be tightly coupled by use. Thus it’s perfectly fine to model your data with only maps of simple types like int, float, Vector3 but you lose expressiveness in the code: What was that Vector3 data chunk about again?

    Data Chunks
    The grid map is divided into chunks of X/Z dimension with the minimum chunk size 2x2 (*) and no limit on the upper bounds. Chunk sizes need not be square. Data chunks are structs for burstability and jobbifiability (yes, those are actual words but you are right, I made them up) reasons and thus cannot be subclassed and may only contain value types themselves.


    *: The minimum size of 2x2 is to prevent collisions of the chunk coord hash function.


    There are two kinds of data chunks:

    Linear Data Chunks
    Linear chunks always keep one value-type item for each coordinate in the chunk. Chunk data can be accessed and iterated in a linear fashion like an array.


    Data is laid out in X/Z layers, one per Y. Initially, a chunk reserves no memory until data is set to it. Then it will expand its size to include the topmost Y layer such that setting data to grid coord (0,3,0) will allocate 4 layers of data (array length: 4*X*Z).


    Linear data is most useful for data that needs to be iterated over in a linear fashion, possibly in parallel. For example, voxels in a voxel map will benefit from a linear data alignment.

    Sparse Data Chunks
    Sparse chunks store data per coordinate. This is meant for data that is sparse, eg randomly and rather rarely set to some of the coordinates in a chunk. For example, flagging some coordinates as spawn points or pathfinding vertices.


    Internally, a hash map is used to look up whether there is data for a given coordinate.

    Custom Data Chunks
    For advanced uses, creating custom data chunks is possible. Any unsafe Unity.Collections collection can be used to store data, and they can be nested too.


    An example custom data chunk could be a specially optimized linear chunk for a voxel map where it is beneficial to performance to lay out tiles in vertical stripes rather than in XZ layers. An example usage could be a “falling sand” game where most of the time “pixels” are moving along incrementing or decrementing Y axis and the algorithms that work with that game mechanic will more often look up grid data above or below the current coord.


    Custom data chunks require writing a custom binary serialization adapter however, so it is recommended to familiarize yourself with Unity.Serialization and binary serialization concerns such as versioning data. The provided (Linear/Sparse)DataMap(Chunk)BinaryAdapter implementations will provide implementation guidance.

    Data Maps
    A data map contains a collection of chunks of a given kind. So there are two main DataMap types:

    • LinearDataMap contains LinearDataMapChunk instances

    • SparseDataMap contains SparseDataMapChunk instances

    All data map classes must inherit from the abstract base class DataMapBase. You can of course make a CustomDataMap for your CustomDataMapChunk type by following the example of one of the above two classes. Again, this is for advanced users and requires custom serialization code, but it isn’t too hard and I will support you in writing that.

    The Grid
    Lastly, the GridBase class stores LinearDataMap and SparseDataMap instances in separate lists. That is, even if you only use one of the two data maps you will have lists for both but the unused list will not allocate memory of course.


    Naturally, when creating a CustomDataMap you will want to subclass GridBase and follow its template in extending it to also support CustomDataMap instances.


    GridBase also handles all serialization concerns and has a list of binary serialization adapters. By default, you will not need to look into this, but if you want to create custom chunks, custom maps or otherwise have needs to customize serialization, this is the class to add your custom IBinaryAdapter classes to.


    In normal operation, a user would simply create a concrete subclass of GridBase and add whatever linear or sparse data maps are needed to store grid data. Initially, ProTiler will ship with Tilemap3D and VoxelMap concrete GridBase subclasses that handle rendering of 3D tilesets and colored / textured cubes.
     
    Last edited: Jun 19, 2023
  9. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    Now things start to fall in place ... :)

    I haven't seen this coming: there's plenty of grid-based 'modular' 3d asset packs out there except they aren't the kind of "3d tiles" that I expected - I wasn't specifically looking for 'modular' and did not notice their grid-based nature until recently.

    These modular packs provide the building blocks and so much more which is wonderful because it really lends itself to making custom tiles: walls, ceilings, floors at the lowest level for greatest flexibility. Custom blocks (cells) at the mid-level for uniqueness. And entire rooms at the high-end as complex but reusable playspaces.

    This is my first attempt at using Synty Studios' modular Dungeon Pack (it's a far cry from their demo but hey, that's a week worth of work and at most 20% of the time spent designing prefabs):
    upload_2023-7-16_19-52-36.png

    From a user's perspective, this will be 95% about building prefabs of tiles, cells, and rooms with the content you have, then lay out the play area on a 3d grid really fast. Ultimately, I want to be able to create interior spaces like Synty Studio's demo scenes showcase, except in a fraction of the time than with Unity's grid, and still either faster or far less repetitive than what other design tools can do.

    I honestly did not want to end up in the "Dungeon Maker" niche but it does provide a good starting point for working with a grid. Of course, I made a Dungeon Crawler character controller to support this.

    I call it: GridPersonCharacterController. :D

    Tomorrow (Monday 3PM GMT+2) I will review the last week and plan the next live on Twitch - following SCRUM methodology by myself but streaming it to feel commited has been nothing but fantastic!
     
    Last edited: Jul 16, 2023
  10. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    I mostly worked on the GridPersonController this week to bring it up to Grimrock standards, fully customizable.
    upload_2023-7-22_18-1-3.png

    Here's a WebGL demo, try to find the door.
    upload_2023-7-22_18-2-9.png
     
  11. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    Yay! :)
    upload_2023-7-25_16-41-36.png
     
    Razmot likes this.
  12. canxerian_

    canxerian_

    Joined:
    Jan 29, 2016
    Posts:
    17
    Impressive thread! updates!
     
    CodeSmile likes this.
  13. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    I have a new demo online mainly for testing lookdev with synty assets and how well postprocessing performs in webgl. I get 120 fps on the start position in Chrome (less in Firefox) which I totally did not expect. This is probably URP magic. :)

    https://play.unity.com/mg/other/webgl-builds-361601
    upload_2023-7-28_18-22-23.png
     
    Last edited: Jul 28, 2023
  14. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    For the Dungeon Crawler dogfood project I'll go for KISS. Right now, the editor is simply multiple components that allow switching meshes, materials and allow rotation. I'll add more features as I go, and later have editor tools that drive the interaction rather than having to go through the Inspector .. which, nevertheless, is already quite the improvement over doing all this manually!

    upload_2023-7-30_17-17-33.png
     
  15. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    I made just the minimal amount of tooling to change meshes, materials and rotate tiles. Eventually these will be driven by editor tools.
     
  16. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
  17. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    5,129
    As I stream my daily work I tend to forget making updates here or on my blog. :oops:
    So here's a quick recap:

    I just returned from vacation in Switzerland and I'm now looking forward to jumping back into development. :)

    I started ProTiler as a general purpose 3D tile editing tool. Eventually I realized that I'm lacking a use case to model the editor towards, and that's where I began working on ProCrawler, a classic first-person grid-based dungeon crawler much like Legend of Grimrock.

    The crawler controller is currently in review for the Asset Store. I understand that this kind of game is extremely niche however. And the dungeon editing use-case is rather specific. I worry that this won't create much anticipation, or sales for that matter.

    On the other hand, I like how Synty Assets developed their "Build 2.0" modular asset set and I'm watching in dismay as users copy+paste their way to an entire level - with no way of making quick changes once the layout is done. But again: what are the game-specific use cases?

    I've seen users work around this by creating rooms and just snapping them together. For the most part this works rather well. So is there a need for an editing tool? And if so, at what level of detail? Walls+Floor+Ceiling or building construction or world design or prefab snapping or detail painting or ... ?

    So if you have a specific use-case in mind, particularly if you have a specific role-model game in mind that I can look at, please let me know! ;)

    ----------------------------------------------------------------------------------------------------------------

    In the meantime and while the crawler controller goes through review I want to exercise my interest in ludology and game system development. I've come to enjoy that very much while working on the crawler demo. My attention was recently brought to Vampire Survivors - and its countless clones. Specifically, and aptly named: Yet Another Zombie Survivors

    That's the only Survivors clone that got me hooked, probably because its "dressing" speaks to me the most and the fact that it's one of the few 3D survivors clones, and next to Soulstone Survivors the only one that looks the part. Speaking of which: Soulstone suffers from emphasizing visual effects way too much while it lacks this feeling of powerful impact.

    I want to spend some time making a Yet Another Zombie Survivors clone. There's a few things that I like about this idea, in no particular order:
    1. Survivors are popular. Ignoring the fact that some consider them dull and repetitive, they do provide that mindless entertainment to casually zone off for a quick play but also be able to go all nuts about it.
    2. It's simple enough to stream and tutorialize the creation of this sort of game.
    3. Since this sub-genre is relatively new, it's prone to evolve with only a few clones having made the logical jump to 3D visuals, and even fewer exploring other ludemes within the game's formula. How about taking the game's formula and use it to clean a world map from threats, undertaking quests with just enough of a story - like Puzzle Quest.
    4. It provides a rolemodel for tasks - meaning I don't have to come up with what to work on, I only need to extract tasks and prioritize them. I like that, because I need to move fast, and I'm not making a game, I want to make a modular game kit.
    5. YAZS has a single programmer (plus two dogs) - that confirms my notion that it's absolutely doable, especically considering I have the game design already set up and only a few things to work out myself (eg How does armor work? How do skills choose their target? How is evasion calculated?).
    6. They certainly won't mind seeing their game cloned seeing that they're cloning themselves. :) Okay, well, maybe not. But I feel it's ethically acceptable, and if anything, should be mutually beneficial as I am going to both dissect and promote their game just as I'm promoting Synty Assets.
    7. YAZS has so many systems used by so many other games. Creating those systems means they have a value beyond that game - stats, skills, upgrades, projectiles, combat, and so on.
    8. It's all systems I would enjoy working on, both technical and visual systems.
    Note to #7: I tend to think that this is where users struggle. Even though there are good systems already in the store => How to make several assets work nicely together? Or how to design an actual game within the huge framework of a "complete but generic" solution?

    Do you feel the same, or otherwise? Let me know ...