Best way to load dynamic mesh data

Hello world

I have developed a component to load dynamic mesh data from a binary file. The data represent an animated object.
The file contains :

• Header
• 5 meshes (vertices, normals, UVs), representing the neutral state of the object (made of 5 parts)
• 1 diffuse map
• For each frame (could be 25, 250, 500...)
  • 5 morphed meshes (vetices, normales, UVs)
  • 1 normal map

Currently the script (in C#) can load the data by creating meshes and textures which are stored in arrays.
Then at some point, the script controls a MeshFilter and a Renderer components to display the animation.

Here is how I am doing it:

• Open the file.
• Read header (frame count, mesh count for one frame...)
• Allocate array corresponding to header
  • var frameToMesh = new Mesh[meshCount][frameCount]...
  • var frameToNormalMap = new Texture2D[frameCount]...
• Load meshes for neutral state.
• Create a Texture2D and load diffuse map.
• For f = 0 to frameCount
  • For m = 0 to meshCount
    • Load mesh and store it in frameToMesh[m][f]
  • Load normal map and store it in frameToNormalMap[f]
• Optimize meshes (Mesh.Optimize())

It works, but I have two problems:

1. The loading time are long. It's about 20 seconds for a 700 Mo file.
2. The memory footprint get huge and after one or two files of 700 Mo, Unity gets out of memory.

Problem 1.
The Profiler tells me that the "bold part" is the longest which is logic because this is where most of the data are read. To load a mesh, the script builds a vertices array, then it stores it in a newly created mesh's vertex field.
Is there a way to improve the loading ?
Is there a way to do the loading in another thread ? (I know Unity don't like you to modify things outside the main thread).

Problem 2.
I am not sure to correctly release the memory after the load work has finished. I presume that mesh and textures data should be uploaded to the GPU, thus freeing the RAM, but it do not seems to be the case.
Is there a way to "force" an upload of the data to GPU, then freeing the RAM ? (I have tried GC.Collect)

Thank you in advance for your help !!

Here is the script:

Code (CSharp):

1. public void Load()
2.     {
3.         print("[AnimMeshPlayer] Loading: " + sequenceFilePath);
4.  
5.         using (var reader = new BinaryReader(new FileStream(sequenceFilePath, FileMode.Open)))
6.         {
7.             // Read header
8.             reader.ReadChars(256); // File format
9.             _parameters.formatVersion = reader.ReadInt32(); //1...
10.             _parameters.frameCount = reader.ReadInt32();    //25
11.             _parameters.fps = fps;
12.             _parameters.MeshCount = reader.ReadInt32(); //5832
13.          
14.             Profiler.BeginSample("AnimMeshPlayer.Load.Init");
15.             // Init array
16.             _frame2partMeshes = new Mesh[_parameters.MeshCount, _parameters.frameCount];
17.             for (int i = 0; i < _parameters.MeshCount; ++i)
18.             {
19.                 for (int j = 0; j < _parameters.frameCount; ++j)
20.                 {
21.                     _frame2partMeshes[i, j] = new Mesh();
22.                 }
23.             }
24.  
25.             _frame2normalMaps = new Texture2D[_parameters.frameCount];
26.  
27.             _meshFilters = new MeshFilter[_parameters.MeshCount];
28.             Profiler.EndSample();
29.          
30.             #region Animated meshes
31.             // Extract animated meshes
32.             for (uint meshIdx = 0; meshIdx < _parameters.MeshCount; ++meshIdx)
33.             {
34.                 Profiler.BeginSample("AnimMeshPlayer.Load.MeshBuild");
35.                 Mesh currentMesh = _frame2partMeshes[meshIdx, 0];
36.  
37.                 // Create and config object
38.                 GameObject currentObject = (GameObject)Instantiate(_templateGameObject);
39.                 Transform coTrans = currentObject.transform;
40.                 coTrans.parent = _transform;
41.                 coTrans.localScale = Vector3.one;
42.                 coTrans.localPosition = Vector3.zero;
43.                 coTrans.localRotation = Quaternion.identity;
44.                 currentObject.name = meshIdx.ToString();
45.                 if (meshIdx < partSpecificMaterials.Length && partSpecificMaterials[meshIdx] != null)
46.                 {
47.                     currentObject.renderer.material = partSpecificMaterials[meshIdx];
48.                 }
49.                 if (meshIdx == 0)
50.                 {
51.                     _mainMaterial = currentObject.renderer.material;    // Saving material of mesh 0 (main) to allow normal map animation
52.                     _normalMapId = Shader.PropertyToID("_BumpMap");
53.                 }
54.                 _meshFilters[meshIdx] = currentObject.GetComponent<MeshFilter>();
55.  
56.                 // Vertex
57.                 int vertexCount = reader.ReadInt32();
58.                 //print(vertexCount);
59.                 // Storage variable to handle the multiple UV per vertex cases (vertex will be duplicated with new UV)
60.                 Vector3[] vertices = new Vector3[vertexCount];
61.                 Vector2[] uvCoord = new Vector2[vertexCount];
62.  
63.                 for (uint vertIdx = 0; vertIdx < vertexCount; ++vertIdx)
64.                 {
65.                     vertices[vertIdx] = new Vector3(reader.ReadSingle(), reader.ReadSingle(), reader.ReadSingle());
66.                     uvCoord[vertIdx] = new Vector2(reader.ReadSingle(), reader.ReadSingle());
67.                 }
68.  
69.                 int triangleCount = reader.ReadInt32();
70.                 int[] triangles = new int[triangleCount * 3];
71.  
72.                 #region Triangles
73.                 for (uint triIdx = 0; triIdx < triangleCount; ++triIdx)
74.                 {
75.                     uint idx = triIdx * 3;
76.                     // Tri A,B,C
77.                     triangles[idx] = reader.ReadInt32();
78.                     triangles[idx + 1] = reader.ReadInt32();
79.                     triangles[idx + 2] = reader.ReadInt32();
80.                 }
81.                 #endregion
82.  
83.                 //print(currentMesh.name + " : v=" + vertexCount + ", uv=" + vertexCount + ", t=" + triangleCount);
84.  
85.                 // Read diffuse texture
86.                 int texSize = reader.ReadInt32(); // 262913;
87.                 Texture2D diffuse = new Texture2D(256, 256);
88.                 diffuse.LoadImage(reader.ReadBytes(texSize));
89.                 if (compressDiffuseTextures)
90.                 {
91.                     diffuse.Compress(true);
92.                 }
93.                 currentObject.renderer.material.mainTexture = diffuse;
94.                 Profiler.EndSample();
95.  
96.                 Profiler.BeginSample("AnimMeshPlayer.Load.MeshApply");
97.                 // Apply to mesh
98.                 currentMesh.vertices = vertices;
99.                 currentMesh.uv = uvCoord;
100.                 currentMesh.triangles = triangles;
101.                 Profiler.EndSample();
102.             }   // Meshes read
103.             #endregion
104.  
105.             Profiler.BeginSample("AnimMeshPlayer.Load.AnimatedMeshRead");
106.             // Reading animated meshes
107.             for (int j = 0; j < _parameters.frameCount; ++j)
108.             {
109.                 for (int i = 0; i < _parameters.MeshCount; ++i)
110.                 {
111.                     int vertexCount = _frame2partMeshes[i, 0].vertexCount;
112.                     Mesh currentMesh = _frame2partMeshes[i, j];
113.                     currentMesh.name = i + " - " + j;
114.                     Vector3[] vertices = new Vector3[vertexCount];    // Set the same vertices as in reference frame (0)
115.                     Vector3[] normals = new Vector3[vertexCount];    // Set the same vertices as in reference frame (0)
116.                     Vector4[] tangents = new Vector4[vertexCount];
117.                     Vector2[] uv = new Vector2[vertexCount];
118.                     Array.Copy(_frame2partMeshes[i, 0].uv, 0, uv, 0, uv.Length);
119.                     int[] triangles = new int[_frame2partMeshes[i, 0].triangles.Length];
120.                     Buffer.BlockCopy(_frame2partMeshes[i, 0].triangles, 0, triangles, 0, triangles.Length * sizeof(int));
121.  
122.                     for (int k = 0; k < vertexCount; ++k)
123.                     {
124.                         vertices[k] = new Vector3(reader.ReadSingle(), reader.ReadSingle(), reader.ReadSingle());
125.                         normals[k] = new Vector3(reader.ReadSingle(), reader.ReadSingle(), reader.ReadSingle());
126.                         tangents[k] = new Vector4(reader.ReadSingle(), reader.ReadSingle(), reader.ReadSingle(), 1.0f);     // w is used to flip binormal, so 1 or -1. I do not know which value to assign... be carefull it's not causing a bug.
127.                     }
128.  
129.                     currentMesh.vertices = vertices;
130.                     currentMesh.normals = normals;
131.                     currentMesh.tangents = tangents;
132.                     currentMesh.uv = uv;
133.                     currentMesh.triangles = triangles;
134.                 }
135.                 // Read mesh 0 normal map
136.                 int texSize = reader.ReadInt32();
137.  
138.                 Texture2D bump = new Texture2D(256, 256, TextureFormat.ARGB32, false);
139.                 bump.name = this.name + "_" + j;
140.                 bump.LoadImage(reader.ReadBytes(texSize));
141.                 FormatToUnityNormalMapDirect(bump);
142.                 if (compressBumpTextures)
143.                 {
144.                     bump.Compress(true);
145.                 }
146.                 _frame2normalMaps[j] = bump;
147.             }
148.          
149.             Profiler.EndSample();
150.         }
151.  
152.         Profiler.BeginSample("AnimMeshPlayer.Load.Optimize");
153.         foreach (Mesh item in _frame2partMeshes)
154.         {
155.             //print(item.name + " _ " + item.vertexCount);
156.             // Finalize mesh
157.             //item.RecalculateNormals();
158.             item.RecalculateBounds();
159.             item.Optimize();
160.             item.UploadMeshData(true);
161.         }
162.         Profiler.EndSample();
163.  
164.         print("[AnimMeshPlayer] Report");
165.         print("[AnimMeshPlayer] " + _parameters);
166.     }

Using Unity 4.6

 

700 megabytes?!? That's enormous! No wonder it takes forever and crashes Unity.

However, yes, you can do most of this work in a thread. You can't access most Unity functions from outside the main thread, but things like building an array of Vector3's should be no problem.

I wish I could be of more help to you.

 

Ok, so I am thinking to populate the arrays from a thread, but can I set the mesh data from this thread also ?
Should I wait for the load thread to finish, then do all the "mesh.vertices = xxx" in the main thread.

I probably better to try it myself.

Thanks.

What about uploading to GPU ?

 

Yeah, I'd expect that you probably need to actually assign to the mesh on the main thread.

And I'm sorry, I haven't any idea about pushing to the GPU. My guess is that the data is copied to the GPU, at best, but never flushed from main memory. But I'm mostly making stuff up.

 

Ok, I managed to split the loading process with a thread. This thread is responsible to read the binary file and populate arrays (Vector3[] for vertices, normals and tangents; int[] for triangles; Vector2[] for UVs; and byte[] for diffuse and normal map). Then when this thread is done (without blocking the UI), the main thread starts to build the assets using the data (Mesh and Texture2D). This last process is blocking the UI...

Let the total time be T, then the loading part (LP) will use 50% of T without blocking the UI and the asset part (AP) will use the other 50% of T, while blocking the UI.

I have no other idea in order to decrease the time needed by AP. All it is doing looks like:

Code (CSharp):

1. _meshes = new Mesh[_parameters.MeshCount];
2. _mesh_diffuseMap = new Texture2D[_parameters.MeshCount];
3. for (int i = 0; i < _parameters.MeshCount; i++)
4. {
5.     _meshes[i] = new Mesh
6.     {
7.         name = "Mesh_" + i,
8.         vertices = _frame_mesh_Vertices[frame][i],
9.         normals = _frame_mesh_Normals[frame][i],
10.         triangles = _mesh_Triangle[i],
11.         /*... UVs, tangents ...*/
12.     };
13.  
14.     _mesh_diffuseMap[i] = new Texture2D(256, 256) { name = "Diffuse_" + i };
15.     _mesh_diffuseMap[i].LoadImage(_mesh_diffuseMapBytes[i]);
16. }

Question:
How to reduce the time for asset creation ? Is it possible ?

Thanks for your attention.

 

You should be able to build your mesh in a co-routine. (still executes on the main thread and is thread safe), it will not speed up the process but it will let you yield, allowing your UI to remain responsive.

you should also be able to speed the first half up by creating a basic parallel algorithm. For instance load the first half of your meshes in one thread and the second half in another. Let them run at the same time. You can extend this to 4 threads each loading a quarter of the data. More than 4 worker threads is probably not advisable depending on your target machines,but you can try and see if performance increases (essentially if you have more threads than cpu cores, the cpu will be switching between them and performance will be lost).

I am very curious as to what you are making there...

 

Hi Brominion

Thank you for this advice about co-routines!
I will try to use this, but even I am an *experienced* programmer, co-routines just make me
In my previous code sample, where should I put co-routine calls ? (I suppose WaitForEndOfFrame to split the for loop on multiple frame).

p.s. This script is used to load high-end facial animations. The animations are pre-calculated and saved in a file. The tricky part is that for each frame there is a morphing (vertices+normals+tangents modification) and a corresponding normal map.
So the current implementation, load the base mesh, then apply the vertices+normals+tangents+normalMap corresponding to the current frame.

Best regards.

 

I am not the best teacher, but i will try.

You can think of co-routines in terms of timeslicing. Each frame the main thread will execute all update functions, then all non-yielded coroutines, then the fixed updates.
Coroutines can be "paused" using the yield. You can pause execution practically anywhere in the code. Once the pause duration is over (it can be fractions of seconds, or until-end-of-frame, etc) , and its once again time to execute coroutines, the execution will continue after the yield statement. Otherwise they are just like normal functions.

This means you need to figure out how much work you can do in each loop of the coroutine and maintain acceptable framerate, this may differ depending on what is going on, a loading screen may not need to maintain 120fps, but if your loading in the background while gameplay is going on...


In your case you probably want to move all the code that generates the mesh into a coroutine function, launch it at some appropriate point (when the data is ready etc), and add a yield between line 15 and 16, this will mean one loop per frame. If that is not enough you can add more yields as needed between time consuming part of the code in order to split the execution over multiple frames.

In other cases you may want to do a time calculation and loop until some allotted time has passed before yielding, this is naturally more useful if the code in the loop is very quick, for example searching a graph, and you can to several "steps" in the same frame.

Make sure you only launch the coroutine once though so you don't have multiple copies of the function competing for the timeslice

hope that helps =)