Feedback Blob Assets as a Standalone Package

I have been learning and developing using DOTS (both with and without the ECS part of it) since it was made publicly available. I wanted to give my impressions regarding BlobAssets as a standalone tool (not the Conversion Workflow that uses it).

Pros:

• Large amount of control over immutable data memory layout and usage.
• Push for separation of Runtime and Authoring Data which, for the most part, its a good thing.
• Allows easy-to-make custom conversion flows and hot-reloading to be put in giving us extremely fast iteration times (even in-build tools for "fully job-ified" games).
• Extremely easy to use with the JobSystem and sort-of-easy to use with Burst (which are -- to me -- the most useful parts of DOTS).
• BlobBuilder API could use some polishing, but is pretty solid and easy to use when converting some arbitrary data into Blob.

Cons:

• Is tied to the Entities Package (which means we can't use all other Job+Burst+Blobs without dragging all the ECS stuff into the project).
  • ECS is a great tool, but, after a lot of experimentation, I reached the conclusion that it may get more in the way of some types of problems than it helps to solve them. For those types of projects, I'd like to NOT have it in the project.
  • Burst and the JobSystem, however, are all around good tools and most projects would benefit greatly from using them. BlobAssets are also in this list of "all-around good tools". These I'd love to keep.
• Current way of integrating serialized blobs with Addressables is not intuitive and poorly documented (use TextAsset and get its bytes plus a MemoryStream deserialization to load from it).
  • We could probably use some editor support for a specific extension to make it easier to find serialized blobs (".blob" + a special icon for these extensions)
• No simple way of getting SIMD-optimized data from BlobArrays (Similar to ReinterpretLoad from NativeArray). Current work-around is to call BlobArray.GetUnsafePtr() to get a pointer but...
  • ...it can break if you attempt to nest BlobPtrs and/or BlobArrays into each BlobArray --- just have to know this and not do it.
  • ...it breaks immutability if you alter that memory -- so don't alter the memory using the pointer.
  • ...due to the 2 items above, it is very error prone and unecessarily risky -- the ReinterpretLoad would make it a lot easier to safely SIMD-ize algorithms that rely on BlobArrays.

Given the above, I'd like to propose that BlobAssets get moved to their own package. Here are the potential benefits and some suggestions on where to look for improvements. Keep in mind the suggestions below may have problems with them and are meant to start a discussion --- not saying these are the way to go:

• No stalling of BlobAsset improvements when ECS ties up (it'll tie up more often than the rest since it is a more complex problem to solve).
• Addition of other first-class Blob-based data structures (a hash-map would be great for certain use cases where you'd be randomly accessing the blob anyway and need some sort of mapping 1-1 mapping).
  • A couple of implementations are already floating around here in the forums
• Better support for ScriptableObjects as ways to generate one-or-more BlobAssets.
  • When calling BlobAssetReference.Write, allow passing in a ScriptableObject and the BlobAssetReference to serialize. Then serialize the Guid of the given SO in the blob.
  • Have overloads for BlobAssetReference.TryRead that yield the SO reference and/or Guid. This should allow for writing simple project-specific tools that use custom conversion tools.
• Better Blob Versioning Tools.
  • Maybe we can lean on whatever support is being considered for SOs to help with data conversions as per Unity's Roadmap. Although, to me, the real challenge is keeping systems backwards compatible without affecting perf and not necessarily migrating data between versions with different formats.
• Better integration with Addressables and AssetDatabase.
  • Editor UI BlobAsset Support.
  • Specific LoadAsync of BlobAssets that can optionally return the ScriptableObject or Prefab that is used to generate them aside from the deserialized blob. This would make it relatively simple for people to roll their own customized conversion workflows with hot-reloading that suit their specific project needs.

If you disagree, leave your reasoning here. Maybe you can help me see your point of view as well

 

Many excellent points. I noticed this problem when prototyping a library of utilities for native collections. I wanted to play around with support for BlobAssets, but that would mean bringing in all of ECS as a dependency.

While the ECS tooling and ecosystem matures, it seems more practical for many new and existing projects to skip entities and optimize via jobs or even just bursted functions.

 

Thanks To be honest, I'm enjoying the Jobs+Burst+Blobs(custom ScriptableObject/Prefabs/EditorWindows --- for authoring) workflows so much I don't think I'll be veering away from it even when the ECS tooling matures. Some projects just don't fit all that well to ECS (I'm currently working on a fighting game and, it turns out, the ECS overhead just adds unnecessary complexity to the codebase). For a fighting game, this feature here would be incredible as far as memory management is concerned -- and I'm still hoping it comes out soon.

I just set up a flexible, albeit a bit manual and project-specific, hot-reloading workflow in about a day's work and it was very simple and straight-forward to do so. I'll be setting up another thread with some feedback specifically regarding the JobSystem, but I really do believe ECS -- and its workflows -- are, by far, the least exciting part of DOTS. The rest of the ecosystem feels almost ready to go (and like coding in C/C++ except we get to dodge the horrible syntax and crappy dev environment --- which is awesome ). They are missing some quality of life stuff (like visual job graph) and some more learning/docs resources around Assembly to ease the transition into really using Burst.

 

I didn't really want to revive this thread... However, I get a feeling that if this doesn't happen before 1.0 gets released, it just isn't going to happen. I'd really hope to get some input from Unity on how likely something like this is to happen... If it isn't, then I can just roll my own "Relative Pointer" solution forking of of Unity's Blobs in its current state ---- but it'd be nice to know if I actually need to or not. Frankly, I'd be fine even just moving it into Collections if for whatever reason that's easier. I just feel like it's a huge part of the Burst+Jobs ecosystem that gets tied to Entities for reasons unknown (if there is a reason, I'd love to understand what it is)...

The points on the usefulness of blobs as a stand alone tool I mentioned are pretty much all still valid, so I hope this does happen --- as it'll greatly help games/genres that don't fit well to Unity's ECS but still want to live in DOD-land rather than the common OOP hellscape.

 

@mfuad It'd be nice to get an official position from Unity on this, whatever that may be. Any clarity you can provide on whether you believe this feedback is actionable for the 1.0 release of ECS (essentially, splitting out blobs into a separate DOTS but non-ECS package)?

 

@IsaacSeah Thanks for the response. It sounds like I'm better off splitting the package and maintaining it myself if I want it to exist separately. I'll get to that then

I'd love to hear the arguments for keeping it inside the Entities package, though. Other than workflow impact (which I don't dismiss, but in this case think it'd be worth it), I can't really imagine why not.

 

I've voted yes on this and think Blobs should exist in the Unity.Collections package. It's the closest thing. I'm not too keen for Blobs to be completely standalone but also not totally opposed to the idea.

Why I think it's not a standalone package comes down to blobs being unmanaged and having no safety around disposing. Entities handles that part but if you ever created a blob outside of entry points where Unity.Entities handles it for you, you'll have memory leaks when not adding the blob to the BlobAssetStore and not disposing said BlobAssetStore. None of this has any safety or leak checks.

 

I'd be more than ok with it inside Collections. I guess my title is a bit misleading hahaha Should've went with "Move BlobAssets out of Entities". The key thing I think has value is leveraging this type of data structure without incurring a dependency on all of ECS.