Feedback [Pre.47] My ECS Wishlist

Some of you might already be familiar with my mega review posts I do of the entire ECS stack every once in a while. You can find the last one in my signature.

Unfortunately, it is difficult for me to review the latest releases because there are some major breaking bugs that prevent me from experimenting. At the same time, the Latios Framework has gathered more and more attention from the Unity team, and I have come to realize the massive hurdle the team faces in trying to identify all the different tech issues I've encountered during development.

Rather than have them always reach out to me, I've created a living document which I plan to update as I remember issues, run into new issues, or issues get resolved by new releases. Let's shoot for more of the latter!

https://gist.github.com/Dreaming381/3aec938455f5e6f393a24e4febabd777

The document is grouped into three categories of severity. For convenience, I'll copy the high-severity items as follows as I would love feedback and discussion from everyone!

Physics and NetCode are not accounted for in this wishlist. Physics is architecturally incapable of fulfilling my use cases, and the below high-severity matters have stolen my time away from experimenting with NetCode.

Subscene Imports
Subscene import workflows have significant usability issues. Because they occur in a separate Unity process, they do not use Burst, cannot easily be debugged, have limited reporting capabilities of memory leaks and the like, and many engine features are not well tested when accessed in this mode (reading audio clips crashes 10-50% of imports for me).

The Latios Framework pushes the boundaries of what can be baked, with new and exciting high-level features. But that only works when baking itself works, which has been a constant pain point.

Currently I cannot use Pre.47 because of some weird serialization bug.

Transforms V2
Transforms V1 in 0.51 were heavy, but intuitive. There were clear rules as to when things were and were not updated. And at the end, everything funneled into LocalToWorld.

Now in Transforms V2, we have TransformAspect which tries to pseudo-synchronize local and world transforms, except sometimes things use or only update LocalToWorld instead, and sometimes they use both but only LocalToWorld is correct, but LocalToWorld isn’t synchronized as frequently. But then WorldTransform can only represent uniform scale, which is extremely limiting. So sometimes TransformAspect is correct, and sometimes it is completely wrong. And trying to replace it in order to extend transforms is impossible because all it takes is one asset to try using TransformAspect directly in order screw everything up.

Then there’s the performance aspects. Root entities have both local and world transforms, along with LocalToWorld, which requires 128 bytes total, enough to ensure that there will never be a full 128 entities in a chunk. LocalToWorld itself is problematic. It’s Rotation property gives incorrect values if there is non-uniform scale baked in the matrix. And why does the bottom row of the matrix even exist if Entities Graphics is going to ignore it?

Transforms V2 is a complete mess, and I ended up resorting to writing my own transform solution in Latios Framework just to have some sanity. But now users are concerned about compatibility with other assets. Please clean up this mess!

IAspect Feature Gaps
Aspect lookups are not only difficult to discover, but using them in IJobEntity requires a ton of boilerplate compared to ComponentLookup. There’s no SystemAPI methods for getting auto-cached handles or lookups.

For performance reasons, the Latios Framework is starting to have really complicated sets of components. A common example is that the Latios Framework triple-buffers animation data so that things like motion vectors and inertial blending can be easily evaluated. Yet rather than copy current to previous and previous to two-ago, these buffers rely on control components to rotate the roles. This behavior should be somewhat abstracted from the user, and IAspect solves this case beautifully. Unfortunately, users struggle to get such aspects from random entities in jobs.

SystemAPI Extensibility
The fact that we can’t use SystemAPI in static methods makes it extremely difficult to build extensions and common patterns. For example, I have a static method Physics.BuildCollisionLayer() that needs to schedule 5 jobs in sequence. While there are several variants, one variant requires the first job to perform chunk iteration. Securing such type handles is extremely problematic. The user has to manually cache and update a struct containing those handles, because this method can’t rely on SystemAPI. That’s a lot of unnecessary boilerplate burdened directly on the user.

Burst Generic Jobs Can’t Be Scheduled in Burst
Psyshock uses generic jobs in Physics.FindPairs() using a pattern that allows Burst to detect and compile the jobs both in the Editor and in builds without having to explicitly register the generic types with attributes. Unfortunately, the ILPP can’t pick up on it and patch these jobs to be Burst-schedulable. There should not be a discrepancy!

Generics and Codegen Accessibility
The previous three items could be potentially solved if the Latios Framework could easily inject itself into the codegen process to add support for these things. Unfortunately, codegen is very inaccessible to most people, and there is no certainty regarding how long such a solution would even last. At the same time, regressions surrounding generics keep popping up, and if they continue, they might prevent the Latios Framework from being able to leverage new versions of ECS, unless codegen became accessible. There are a ton of ways I would love to leverage codegen, so if you know things about this and would like to better understand the specific potential use cases or just want to help me start using them, please reach out to me via Unity forums PMs or Discord!

Skinned Mesh Rendering
The whole Skinned Mesh Rendering solution in Entities Graphics is problematic. It generates GC every frame, it doesn’t scale, and even the public API types of SkinMatrix and BlendShapeWeight fundamentally prevent more efficient algorithms like the ones Kinemation uses. I’ve been told numerous times that the skinned mesh rendering design is “experimental”. If that’s the case, please put that functionality behind an experimental scripting define so that you don’t have to support these flawed APIs long-term when you eventually get around to a better solution.

I’m only asking this because I have a sliver of hope that Entities.Graphics may adopt a design closer to Kinemation in which case I can delegate some features of Kinemation to the official package.

 

Thanks @DreamingImLatios, this is so complete and I don't hope to add more. But here is my experience:

Subscene is the most annoying thing that always get in my way when I try to bake some complex data. I've to change my way multiple times and in the same time try figuring out what's wrong with subscene. Now when subscene is finally working, my data becomes too complicated. I have to divide my runtime data into 2 parts: 1 for the ECS world and another for the GameObject world. All of this hurdle is because my project is 2D hybrid, and I want to synchronize data between ECS and GameObject in the most performant way: in batches. So I have to prepare my data in batches too. My data was neat before, but then Pre.44 suddenly hates managed types inside BlobAssets.

The second annoying thing is Generics. Now my codebase is too verbose. I feel that at the current state I can't hope to build any useful framework around Unity ECS because all have to be specialized.

 

You also mentioned Transforms V2. I thought I was going insane... These new transforms are a mess, unfortunately.

The more I use ECS, the more basic functionality/implementation issues I find. Hopefully all of these will get ironed out for the release candidate.

 

Spoiler

GC-Free UI in ECS?
It seems like every UI solution that involves text also involves GC allocations. We have FixedStrings. Why can’t we do UIs with them?

Yes this. I want official to make textmesh pro text able to directly accept FixedString. Currently it needs to convert string and make it GC every frame if u keep assigning to textmesh pro text.

 

Hybrid is ugly, to the point where anyone advertising support for it is false advertising. Although I'm surprised it is as big of a problem as it seems to be for people doing 2D. ISprites have Companion GO support. And there are several DMII-based solutions out in the wild. But honestly, I have no idea why the problem can't be solved with just a custom baker to fix things up for Entities Graphics and leverage that tech? So now I am curious, what are you using GOs for?

I'm surprised that ever worked. Managed data in blob assets don't make sense to me. The whole point of blob assets is that they are serializable and movable without anything needing to know what the underlying types are.

Yep. TMP allocates every frame. Even if it is only in the Editor, it is an awful workflow to have to make builds just to profile if there are GC allocs. Maybe UIToolkit is better in this regard, but I have yet to see anyone discuss GC with UIToolkit.

 

Spoiler

Skinned Mesh Rendering
The whole Skinned Mesh Rendering solution in Entities Graphics is problematic. It generates GC every frame, it doesn’t scale, and even the public API types of SkinMatrix and BlendShapeWeight fundamentally prevent more efficient algorithms like the ones Kinemation uses. I’ve been told numerous times that the skinned mesh rendering design is “experimental”. If that’s the case, please put that functionality behind an experimental scripting define so that you don’t have to support these flawed APIs long-term when you eventually get around to a better solution.

I’m only asking this because I have a sliver of hope that Entities.Graphics may adopt a design closer to Kinemation in which case I can delegate some features of Kinemation to the official package.

And also this. Really hope Entities Graphics team will improve this very soon instead of like other unity features that can't release anything after so many years or forever experimental. Currently I think Entities Graphics still have a lot of things need to improve for both CPU and GPU usage specially at mobile platform. It seems like currently no matter it's skinned mesh or static mesh, GPU usage is insanely high even camera is look at sky and not large amount of skinned mesh or static mesh.

 

While it is actually true it is false too
There is ZString from sysharp that can halp assign textmesh pro text without GC really just replacing symbols inside the same string instance. So it is easy to write extension method to accept FixedString.

Wish such logic or better to be Official API of TextMesh for sure

 

I'm surprised at how clunky Transform V2 is, tbh. Like, what does it even gain us?
I don't even think TransformAspect is a meaningful abstraction.

 

Looking at the source code of ZString, Neuecc is using the following API to set text without any allocations:
https://docs.unity3d.com/Packages/c...ml#TMPro_TMP_Text_SetCharArray_System_Char___

If you don't want to use ZString, you can do the same thing yourself by managing your own pool of

char[]

. There's even an overload with a

start

and a

length

in case you want to have a large buffer of

char[]

.

 

I don't understand why TransformAspect exists.

I understand why LocalToWorld exists. You kind of need something like it for physics interpolation, but it needs to be better explained to users. Maybe rename it.

 

Add support for adding chunk components in bakers please!

 

I've used a cached StringBuilder for this. Works good in builds. But the problem is that in the editor TextMeshPro will dump every string modification to an actual string (allocating GC) for the inspector textbox, even if said textbox isn't visible. It makes it much more difficult to profile GC in play mode in the editor.

It is already possible: https://github.com/Dreaming381/Lati...ation/Authoring/SkinnedMeshBaker.cs#L307-L310

 

At the beginning I was using NSprites of @Tony_Max, which has brilliant performance. But then the requirements changed and the game have to run on older Android devices which unfortunately have many issues with StructuredBuffer, which is the foundation of NSprites. The time is tight now so I can't afford to tinker more with NSprites and shader and hybrid becomes the most affordable solution at the moment. This might change when I have more room to breathe.

GOs are used to host SpriteRenderer. I've done a simple test to verify the performance on our Android 9 devices and while its performance is not as good as when using NSprites, the result is still fairly acceptable.

NSprites requires Materials and some custom data provided by ScriptableObject. Managed references in BlobAssets make it trivial for us to provide these data to NSprites rendering system. Otherwise it just becomes a lot more complicated and woefully verbose to prepare these data.

 

We can just allocate String with max chars we need (i.e.. for some score we need say 7 symbols max) and then just replace String content directly without any string builders or other methods and make TextMesh Dirty.

This is hacky but this way you can use String type whenever you need without reallocating and it works with DOTS

 

I still don't get how that would bypass TextMeshPro calling InternalTextBackingArrayToString() or how that would offer any advantages over reusing a StringBuilder. What am I missing?

 

TextMeshPro dont call InternalTextBackingArrayToString() if you use text property to set string so no GC

 

Thank you for mentioning that, I even didn't know that there always was

ComponentType.ChunkComponent<T>

. But I have a reason asking that. Every time I added chunk component it doesn't added with frist try, instead I have to press play or open / close subscene to get properly initialized entity with required chunk component.

For now my code looks like that:

Code (CSharp):

1. baker.AddComponent(new ComponentTypeSet
2. (
3.   ComponentType.ReadWrite<PropertyPointer>(),
4.   ComponentType.ChunkComponent<PropertyPointerChunk>()
5. ));

 

Unity just needs to bring Span API for many places already. We can use both managed and unmanaged memory. I've asked for TextMeshPro Span API a year ago. They still don't have a unmanaged DrawMeshInstanced method (there's a thread that asked for this in 2018). They're too slow with the changes.

In Entities I feel like they missed the point a while ago. ECS was about SoA representation and using tightly packed data and now look at how much data just the transform system brings, also back to AoS. I'm not fully sold to Aspects as well it looks like they're trying make it easy for OOP-minded people while bloating the ECS by adding too many ways to do things.

 

Simplest way to thinking about aspects is:
They add meaning for set of components that otherwise dont have any meaning

TransformAspect is not good sample of this

 

Wait, are you treating the string you assign as mutable using some pointer shenanigans?

 

Exactly.
This is what String builder do under the hood.
Allocate string with exact known length and then use it as char array.

But string builder work more like List and I work with string more like array.
And String builder make copy on ToString method but I pass string directly to TextMeshPro

 

Is there any way to pass FixedString into TMP_Text.SetCharArray(Char[]) or any other way without gc spike?

 

yeah, create a large

char[]

buffer, then set those chars when you want to fill it in.

Code (CSharp):

1. using System;
2. using TMPro;
3. using Unity.Collections;
4. using UnityEngine;
5.  
6. public class SetFixedStringToText : MonoBehaviour
7. {
8.     public TMP_Text text;
9.    
10.     private readonly char[] _charBuffer = new char[4096];
11.  
12.     public void SetText<TFixedString>(ref TFixedString fs)
13.         where TFixedString : unmanaged, IUTF8Bytes, INativeList<byte>
14.     {
15.         CopyToChars(ref fs, _charBuffer, out int length);
16.         text.SetCharArray(_charBuffer, 0, length);
17.     }
18.    
19.     public static void CopyToChars<TFixedString>(ref TFixedString fs, Span<char> charArray, out int length)
20.         where TFixedString : unmanaged, IUTF8Bytes, INativeList<byte>
21.     {
22.         int byteIndex = 0;
23.         int charIndex = -1;
24.         while (true)
25.         {
26.             var rune = fs.Read(ref byteIndex);
27.             if (rune.value == Unicode.BadRune.value)
28.             {
29.                 break;
30.             }
31.  
32.             charIndex++;
33.             charArray[charIndex] = (char)rune.value;
34.         }
35.  
36.         length = charIndex + 1;
37.     }
38. }

 

Full SoA can actually hurt performance more often than it helps. It is great when you have a small number of inputs that require a bit of math. But when your inputs and outputs widen (as is typical for most game code), then you run out of registers and prefetchers and suddenly your linear memory access starts to look more like random memory access. Small AoS structures significantly reduce register and prefetcher usage, such that if you use all the data in the structure together, you actually get better streaming. It does hurt ALU performance, but most people are bottlenecked more by bandwidth, so it is a good tradeoff. I experimented with both AoS and SoA implementations for pose sampling with AclUnity. AoS transforms were the clear winner, as algorithms could just work in-place.

I don't think it is so much about OOP as much as it is providing a way for systems designers to provide an API for their data while the data itself is complicated for performance reasons. I think TransformAspect would make more sense if things were more organized. I like the equivalent I wrote in Latios Transforms a lot actually. However, I think the OptimizedSkeletonAspect I wrote does a much better job demonstrating how important this is. Is it more OOP-like? Perhaps. But is it fast and easy to use? I'll let you be the judge of that. https://github.com/Dreaming381/Lati...emation/Components/OptimizedSkeletonAspect.cs

 

That might be the case if the systems are making use of all of the data (such as Transform Systems), however, I'd assume most game code doesn't care about Rotation and Scale.

 

I don't know what kind of game you are making, but for me the most common cases have been copying transforms, transforming between spaces (points, rays, bounding boxes, local vs world conversions, ect), and uploading to GPU for rendering. All of which require all three parts of the transform at once.