Most efficient way to load/create Raycast targets (e.g. Colliders) with DOTS?

This thread is the DOTS specific version of Most efficient way to load/create Raycast targets (e.g. Colliders)?

What is the most efficient way to load/create/serialize/deserialize a (Mesh)Collider with DOTS?

The goal is to be able to as fast as possible load a mesh from disk, raycast against it to determine:

• Vector3 hitpoint
• int triangle index

and then unload it again.

My current plan is to use MessagePack-CSharp as fast deserializer, and I am wondering if DOTS would be any help for quickly doing (many) raycasts?

 

Any hints on how I would do that if the meshes are not available during editor time, but will be loaded from file from a custom file format by the user? Currently I am creating the Mesh via new Mesh and set vertices, normals, indices, assign a MeshRenderer, Materials.

Haven't looked too much into the Subscene Workflow so far, but as far as I understood it it is designed more with converting .unity scenes to ECS - or maybe I am wrong on this on? Likely this would be more of a Pure ECS workflow correct?

 

Thank you very much Joachim. This is an important topic for me and I am super happy to get this answer

Hope Unite this year was as much fun for you as last year (or even more ^^)

 

@Joachim_Ante Sorry to be so bold to tag you, but this looks odd. Using the Unity.Physics code is an order of magnitude slower than using a MeshCollider:

The old way to create a MeshCollider in Unity took 162ms.

Code (CSharp):

1.  
2. //patchMpk is just a class holding the data. e.g. name, indexformat, vertices, indices,...
3. var sw = new Stopwatch();
4. sw.Restart();
5.  
6. var go       = new GameObject(patchMpk.Name);
7. var collider   = go.AddComponnent<MeshCollider>(go);
8. var mesh = new Mesh {
9.                 indexFormat = patchMpk.IndexFormat, // MUST be set before triangles!
10.                 vertices    = patchMpk.Vertices,
11.                 normals     = patchMpk.Normals,
12.                 uv          = patchMpk.Uv,
13.                 triangles   = patchMpk.Triangles, // MUST be set after indexFormat
14.             };
15. collider.sharedMesh = mesh;
16.  
17. sw.Stop(); // 162ms

Doing the same with Unity.Physics takes 3878ms (x23 as long)

Code (CSharp):

1. var sw = new Stopwatch();
2. sw.Restart();
3. var positionsData = patchMpk.Vertices.Select(v => new float3(v.x, v.y, v.z)).ToArray();
4. sw.Stop(); // 3ms and avoidable if I switch completly to float3
5.  
6.  
7. sw.Restart();
8. var positions = new NativeArray<float3>(positionsData, Allocator.Persistent);
9. var indices   = new NativeArray<int>(patchMpk.Triangles, Allocator.Persistent);
10. sw.Stop(); // 1ms and seems unavoidable
11.  
12. sw.Restart();
13. Unity.Physics.MeshCollider.Create(positions, indices);
14. sw.Stop(); // 3874ms and this seems off? Why does this take an order of magnitude (23x times as long) longer than using the old way?
15.  

Why does the Unity.Physics.MeshCollider.Create take so much longer than the MeshCollider way or where I am so extremly off?

 

This is way faster, but still 1.5 times as slow as the old way. What am I missing? The code examples look rather straight forward for such a simple test case

Code (CSharp):

1.  
2.     [BurstCompile(CompileSynchronously = true)]
3.     public struct CreateMeshColliderJob : IJob
4.     {
5.         [ReadOnly]
6.         private NativeArray<int> indices;
7.  
8.         [ReadOnly]
9.         private NativeArray<float3> vertices;
10.  
11.         public CreateMeshColliderJob(NativeArray<float3> vertices, NativeArray<int> indices)
12.         {
13.             this.indices  = indices;
14.             this.vertices = vertices;
15.         }
16.  
17.         public void Execute(int index)
18.         {
19.             Unity.Physics.MeshCollider.Create(vertices, indices);
20.         }
21.     }
22.  
23.  
24. var sw = new Stopwatch();
25. var job = new CreateMeshColliderJob(positions, indices);
26. job.Schedule().Complete();
27. swStop(); // 202ms
28.  

I've attached the CreateMeshAssembly copied from the Burst Inspector. Those measurements are from the editor playmode. However, it does not look better in a build either:

//Windows 64, build
//2279KB of mesh data (positions, indices, normals, uv and texture weights)
MeshCollider: 168ms
Unity.Physics.MeshCollider.Create (Bursted Job): 189ms

Which is now only 1.125 times as slow (so 12.5% slower) but I would have hoped that this will actually be faster, and to be very optimistic for what I would need to do, I wished it would be 50-100x times as fast (not saying that is realistic...) and with multiple runs, sometimes it get's as close as a few percent, but never faster.

I really wished I could build the MeshCollider once, persist the result disk, and then load the already calculated Spatial Acceleration Structure/Collider for this MeshCollider, s.t. I am really able to stream in large scenes (1Gb, 16GB, 32GB, 64GB or even 128GB and larger). Obviously not having them in memory at the same time but streaming them in out-of-core, but currently this performance maxes out at about 22MB/s from disk to Unity.Physics system (the loading time from disk is excluded here as the current bottleneck really is creating the MeshCollider)

 

Couple of things.

you can create the collider at design time and use the entities binary serialization and save it to disk. Then at runtime you just load in an already created collider. Really the only way to go for non procedural meshes where you have a lot or they are large.

https://gist.github.com/gamemachine/6eafec133e837249159743b61840b9c6

Their weld vertices code degrades severely as the size of the mesh increases. Not much you can do about this other then use the package source and comment it out until they fix it.

 

@snacktime Thank you from the bottom of my heart. I have been looking for something like this for ages. I am going to take a look into this right now.

 

That gist is a starting point. We have a bit more abstraction where we bake all colliders to a single file. If you look at entities serialization it has built in support for serializing NativeArray/NativeList.

It's a good in between approach generally for where sub scenes are overkill or don't fit the problem, but the performance still matters.

 

@snacktime I've tried to use your code (missing some extension methods e.g. ReadInt so I hope i changed it correctly) but I've encountered a few problems.

1. Regularly it crashes with an Overflow exception creating the StreamWriter ctor and a "random" large negative memory size value. I first thought it is related to job.Schedule().Complete() but it also occurs on the same thread with job.Run() so I believe it is because of unmanaged memory access?
2. After deserailization the MemorySize seems to be unintialized memory ("random" value)
3. ReadInt is not available, so I changed it to BlobAssetReference<int> x = reader.Read<int>(); which seems to be correct
4. Write(int) is not available either, so I changed it to
BlobAssetReference<int> x = BlobAssetReference<int>.Create(collider->MemorySize); writer.Write(x); which also seems to be working.

Create the (Mesh)Collider within a Burst job
Job

Code (CSharp):

1. [BurstCompile(CompileSynchronously = true)]
2.     public struct CreateMeshColliderJob : IJob
3.     {
4.         [ReadOnly]
5.         private NativeArray<int> indices;
6.         [ReadOnly]
7.         private NativeArray<float3> vertices;
8.  
9.         public NativeArray<Collider> Collider; // Return Value, Array of Size 1
10.  
11.         public CreateMeshColliderJob(NativeArray<float3>   vertices,
12.                                      NativeArray<int>      indices,
13.                                      NativeArray<Collider> collider)
14.         {
15.             this.indices  = indices;
16.             this.vertices = vertices;
17.             Collider      = collider;
18.         }
19.  
20.         public void Execute()
21.         {
22.             BlobAssetReference<Collider> blobAssetReferenceCollider = Unity.Physics.MeshCollider.Create(vertices, indices);
23.             Collider[0] = blobAssetReferenceCollider.Value;
24.         }
25.     }

Calling the Code

Code (CSharp):

1. var positions = new NativeArray<float3>(vertices, Allocator.Persistent);
2. var indices = new NativeArray<int>(triangles, Allocator.Persistent);
3. var jobResultCollider = new NativeArray<Unity.Physics.Collider>(1, Allocator.Persistent);
4. var job = new CreateMeshColliderJob(positions, indices, jobResultCollider);
5. job.Run(); // Seems to fill jobResultCollider[0]
6. //job.Schedule().Complete(); // Seems to not fill jobResultCollider[0]
7.  
8. // Serialize the Collider
9. int bytes=0; // bytes==36910
10. unsafe
11. {
12. Collider c = jobResultCollider[0]; //c.Type == Mesh, c.MemorySize==36906
13. bytes = UnityColliderSerialization.SerializeCollider(&c, serializedColliderPath); // Crashes regularly within StreamWriter ctor because of Overflow and negative sized MemorySize
14. }
15.  
16. // Deserailize the Collider
17.  
18. Collider deserializedCollider =
19. UnityColliderSerialization.DeserializeCollider(serializedColliderPath, bytes).Value;
20. // [I]deserializedCollider.Type == Mesh, [I]deserializedCollider.MemorySize is a random value <- Uninitialized memory?,[/I][/I]
21.  

Serialize and then deserialize it

Code (CSharp):

1. public class UnityColliderSerialization
2.     {
3.         public static unsafe BlobAssetReference<Unity.Physics.Collider> DeserializeCollider(string path, int bytes)
4.         {
5.             // bytes:36910
6.             using (StreamBinaryReader reader = new StreamBinaryReader(path, bytes))
7.             {
8.                 BlobAssetReference<int> x        = reader.Read<int>(); // readInt is not available? Changed to Read<int>
9.                 int                     length   = x.Value; // 36906
10.                 Collider*               collider = (Unity.Physics.Collider*) UnsafeUtility.Malloc(length, 16, Allocator.Temp);
11.                 reader.ReadBytes(collider, length);
12.                 var blob = BlobAssetReference<Collider>.Create(collider, length);
13.                 UnsafeUtility.Free(collider, Allocator.Temp);
14.                 return blob;
15.             }
16.         }
17.  
18.         public static unsafe int SerializeCollider(Unity.Physics.Collider* collider, string path)
19.         {
20.             int size  = collider->MemorySize;
21.             int intsize = sizeof(int);
22.             int bytes = size + intsize;//36910
23.  
24.             using (StreamBinaryWriter writer = new StreamBinaryWriter(path, bytes))
25.             {
26.                 BlobAssetReference<int> x = BlobAssetReference<int>.Create(collider->MemorySize); // no overload found, changed to BlobAssetReference<int>.Create(collider->MemorySize)
27.                 writer.Write(x);
28.                 writer.WriteBytes(collider, collider->MemorySize);
29.             }
30.             return bytes;
31.         }
32.     }

Any guidance/links to tutorials/examples are highly appreciated, as this is quite new territory for me.

 

I'd love to take a stab at this; can you upload a full project that showcases the performance issues you're seeing and any problems you are running into?

 

I am happy to do so. I just need some time to create a new project which does not contain any assets I am not allowed to share. I am going to start doing that right now.

 

@sschoener Here you are. On my machine it takes ~180ms for the MonoBehaviour workflow and 180-200ms for the Bursted IJob Workflow with Unity.Physics.

The Collider Serialization is not in this project - I was not able to get that running without crashing yet. As written above I am still struggling with this, as this is a rather new field for me and there is not much Documentation about that feature on the package documentation (or at least I have missed it)

For anyone interested this is an example project.

• It is a simple scene with two buttons and self-contained. Open the project and press play, then press the one button or the other.
• All code is to be found within ColliderCreationExampleScript.cs and CreateMeshColliderJob.cs
  
• It contains the actual mesh stripped off the texture and with a randomized y axis on all components, s.t. it is not reconstructible.
• You also won't see any mesh rendered as even if you would shift it to the camera there is no renderer created.
• The mesh data is positioned pretty far away because that is the original mesh position and in the actual streaming project it will use origin relocation to bring camera and mesh together.
  
• The first creation via IJob takes 10x-20x as long in the editor - I am not really sure why, but please run it at least twice to see the actual speed.
• The project contains memory leaks - I did not care about cleaning up the obvious permanently allocated native arrays as for this test the whole program will be killed anyway after each run.

 

Thanks; I've taken a quick look -- since you're essentially treating
Unity.Physics.MeshCollider.Create as a black box, there's little that you can do about its performance unless you are willing to customize it. Also, I see similar runtimes to what you have measured with the MeshCollider being slightly faster. However, updating Burst to 1.2-preview-6 makes the bursted job consistently faster (if only a few ms). So I generally see two options:

• Serialize the mesh collider (I understand that you have already tried this). So this is the actual problem you have, right?
• Run the raycast without building the collision mesh. How many rays are you casting against each model?

When you first hit the button in the editor, Burst starts to compile your job before running it. This is why you see the slowdown for the first run.

 

I think that you are almost there with the serialization. There is one issue with the memory you are allocating when deserializing. I've fixed it up, please also note the comments:

Code (CSharp):

1. using Unity.Collections;
2. using Unity.Collections.LowLevel.Unsafe;
3. using Unity.Entities;
4. using Unity.Entities.Serialization;
5. using Unity.Physics;
6.  
7. public static unsafe class UnityColliderSerialization
8. {
9.     public static BlobAssetReference<Collider> DeserializeCollider(string path)
10.     {
11.         // You do not have to pass a specific buffer-size to the reader. The buffer is
12.         // an optimiziation. You should avoid very small buffers, but the default size is
13.         // probably fine for your purpose.
14.         using (StreamBinaryReader reader = new StreamBinaryReader(path))
15.         {
16.             int length = reader.ReadInt();
17.             // We are deserializing the collider and want to keep it in memory, hence the
18.             // allocator should be persistent and we also definitely should *not* free it
19.             // immediately because we want to use the collider later on: If we free the
20.             // memory, we're essentially accessing random values in memory.
21.             void* collider = UnsafeUtility.Malloc(length, 16, Allocator.Persistent);
22.             reader.ReadBytes(collider, length);
23.             return BlobAssetReference<Collider>.Create(collider, length);
24.         }
25.     }
26.  
27.     public static void SerializeCollider(Collider* collider, string path)
28.     {
29.         // The same note about the buffer size applies here as well. you don't have
30.         // to set it.
31.         using (StreamBinaryWriter writer = new StreamBinaryWriter(path))
32.         {
33.             writer.Write(collider->MemorySize);
34.             writer.WriteBytes(collider, collider->MemorySize);
35.         }
36.     }
37.  
38.     // These extensions methods are taken from BinarySerialization.cs in the Entities package.
39.     public static int ReadInt(this BinaryReader reader) {
40.         int value;
41.         reader.ReadBytes(&value, sizeof(int));
42.         return value;
43.     }
44.  
45.     public static void Write(this BinaryWriter writer, int value) {
46.         writer.WriteBytes(&value, sizeof(int));
47.     }
48. }

This is how you would use it:

Code (CSharp):

1.         unsafe
2.         {
3.             UnityColliderSerialization.SerializeCollider((Collider*)job.Collider[0].GetUnsafePtr(), "D:\\test.collider");
4.  
5.             var newCollider = UnityColliderSerialization.DeserializeCollider("D:\\test.collider");
6.             UnityEngine.Debug.Assert(newCollider.Value.Equals(job.Collider[0].Value));
7.         }

 

@sschoener Exactly. This is my goal right now with the help of the gist provided by @snacktime.
I would estimate a few hundred thousands to a million rays (not per frame - per measurement/button click of a user)

I can run the the Unity.Physics Raycast without building the collision mesh? I'd expected building the collision mesh is actually registering the mesh in the physics world?

 

You can write the raycasting code yourself (doing ray/triangle intersection is actually pretty simple). Your initial post sounded like you wanted to load a mesh, run a single raycast against it, and unload it again. Depending on the circumstances, it could be faster to just do a brute force raycast against all triangles in the mesh than to build some sort of optimized collider first. But if you are doing millions of raycasts against a single mesh inbetween loading and unloading, then it is almost certainly a good idea to build some sort of acceleration structure like the mesh collider

 

Thank you snacktime. In hindsight the errors made seem obvious.

What still confuses me is why the generate Collider has a MemorySize of 0. Therefore I am pretty sure even the generated Collider is already invalid therefore the serialization later fails. The type is correctly identified as Mesh but this looks off.

Edit: Calculate AABB for the deserialized crashes because it fails with 0 vertices, so the collider indeed is invalid. And for the collider returned by the job it says center 0,0,0 with extends 0,0,0 so both are invalid (the vertices are for sure not all at 0,0,0). Arg....

Code (CSharp):

1. // After job - Before serialization
2. job.Collider.CopyTo(jobResultCollider); // from an example - I don't think this is necessary here as both should point to the same NativeArray
3. //job.Collider[0].Value: Type: Mesh expected Type: Mesh. MemorySize:0
4. //jobResultCollider[0].Value Type: Mesh expected Type: Mesh.MemorySize:-408135912
5.  
6. // After deserialization
7. // Deserialized Collider Type: Convex expected Type: Mesh. MemorySize:-1848526048, bytes: 4
8. // which is just the size of the int + MemorySize 0...

 

I can only speculate what is going wrong in your case, but this works flawlessly in the project you have posted using the serialization code I have posted above:

Code (CSharp):

1.     private void CreateWithJobs()
2.     {
3.         var job = new CreateMeshColliderJob(positions, indices, jobResultCollider);
4.         job.Schedule().Complete();
5.  
6.         ref var collider = ref job.Collider[0].Value;
7.         UnityEngine.Debug.Assert(collider.MemorySize > 0);
8.         var aabb = collider.CalculateAabb();
9.         UnityEngine.Debug.Assert(aabb.Extents.x > 0);
10.  
11.         unsafe
12.         {
13.             UnityColliderSerialization.SerializeCollider((Collider*)job.Collider[0].GetUnsafePtr(), "D:\\test.collider");
14.  
15.             var newCollider = UnityColliderSerialization.DeserializeCollider("D:\\test.collider");
16.             UnityEngine.Debug.Assert(newCollider.Value.CalculateAabb().Extents.x > 0);
17.         }
18.     }

Please note that according to the documentation of the Collider class, you should never pass a Collider by value. This line, for example, will be causing problems:

Code (CSharp):

1. var collider = job.Collider[0].Value;

You have to use them as references:

Code (CSharp):

1. ref var collider = ref job.Collider[0].Value;

 

@sschoener Thank you so much! Changing it to ref fixed the problem. Now I just need to test if the collider is actually working. But holy hell - this is EXACTLY what I needed so thank you two very much and if I by any chance can buy you a digital coffee/beer e.g. via paypal send me a pm to thank both of you.

This has troubled me over the last two years and at various points I have been close to giving up. The deserialization now takes 8ms which is 20x as fast as the MonoBehaviour workflow.

Where did you find the documentation for the Collider specifying we need to use ref? The only one I am aware of for Unity.Physics is this one - which is rather sparse.

Also: As I likely want to serialize/deserialize pretty much everything I need to load/unload along with the GameObject is there any documentation which states what I can serialize and what not. e.g. can I do the same trick for a Texture object, which for sure has a GPU counterpart and on CPU side is just the handle (I'm coming from OpenGL) and therefore I wonder if we have the relevant API for something like that.

 

You are very welcome No need to thank me, it's been educational for me as well. If you feel like it, consider taking public transportation more often, reducing meat consumption, or limiting the amount of plastic waste you create

Ha, now that's a nice speed-up

I mostly found it because I dug around in the source code for the package. All packages come with their source code and I like to understand things to their fullest extend when possible. It really should be made more obvious because this is a pretty sharp edge of the physics colliders. One place you can find this documentation is in Physics_MeshCollider.cs:

Code (CSharp):

1.     // A collider representing a mesh comprised of triangles and quads.
2.     // Warning: This is just the header, it is followed by variable sized data in memory.
3.     // Therefore this struct must always be passed by reference, never by value.
4.     public struct MeshCollider : ICompositeCollider

The reason for all of this is essentially that all colliders have a common header (the first few bytes) and that header is essentially what Collider is. Pointers to a specific collider type like MeshCollider can then be downcast to Collider, and value semantics for structs then treat it as if it was just the header. In the case of the MeshCollider it's yet a bit worse: Even MeshCollider itself is still just a header and it is followed by the data for the mesh collider in memory. It's like regular subclassing, but implemented manually and combined with a dynamic buffer trick you might be familiar from C . So yes, it should come with heaps of warning signs since it will be wildly surprising to C# programmers. I'm pretty sure that the documentation will be improved on this topic; the physics package is still in preview after all

Generally speaking, anything that you can use with DOTS/Burst should be serializable (in the sense that you can just write the bytes to disk and reload them from there). This does not apply to must "old" Unity objects.

 

It's more about how colliders itself using in DOTS land - Blobs, which have specific rules like ref's. (This is what I remember when picking physics)
Latest Unity session can help you with understanding it better + read source code + BlobificationTests.cs

 

In this case it is not about Blobs. For example, this is perfectly valid for, say, integers:

Code (CSharp):

1. void DoStuff(BlobAssetReference<int> blob) {
2.     int x = blob.Value;
3.     // Do stuff with x
4. }

With Collider on the other hand, this can lead to problems:

Code (CSharp):

1. void DoStuff(BlobAssetReference<Collider> blob) {
2.     // the actual collider in memory might be bigger than the Collider struct,
3.     // and this line will only copy the Collider struct into x, thus dropping all
4.     // additional data. Access to member functions of x might end up reading
5.     // data from the application's stack instead.
6.     Collider x = blob.Collider;
7. }

 

Yes agree, my memory about data structure in physics was a bit cleaned, cause I'm not using it in our game

 

Ya I should have said how it's called. It's the pointer you pass as shown here. There is also a built in property ColliderPtr on PhysicsCollider so you don't need to cast it yourself on every call.