Big byte array will cause memory leak

I find that If I create a big byte array, mono will NOT release the memory. E.g., the following codes will create a 300MB byte array, the memory cannot be freed, and I received out of memory exception in one minute. But If I only create a 30MB byte array, the memory will be freed successfully. This is very bad. Because we need to download a 300MB update file by using WWW class, and WWW class uses byte array. We are doomed if you cannot fix this bug.

Unity ver: 4.7.1f1

Code (CSharp):

1.  
2. public class ByteArrCheck : MonoBehaviour
3. {
4.     void Start()
5.     {
6.         StartCoroutine(CheckMemoryLeak());
7.     }
8.  
9.     IEnumerator CheckMemoryLeak()
10.     {
11.         int i = 0;
12.         while (true)
13.         {
14.             byte[] arr = new byte[314572800]; // 300MB
15.             arr = null;
16.             System.GC.Collect();
17.             System.GC.WaitForPendingFinalizers();
18.             yield return new WaitForSeconds(2f);
19.             Debug.Log(i++);
20.         }
21.     }
22. }
23.  

 

I see nothing in that code verifying that the garbage was not collected.

So I'm going to assume what you mean is that when you look in the profiler you are noticing that Unity does not release the memory back to the operating system.

Yeah, it doesn't... that's not the way Garbage Collection works...

See, the way memory works in the CLR (the runtime mono runs in) is that it reserves memory from the operating system, this is called the 'heap'. In this swath of memory it will stick your referenced objects and what not as your code runs. When GC runs (which it does on its own terms... you don't have to call it) it will go through the heap and clear out objects on the heap that aren't needed anymore, and reorganize the left over objects to defragment the memory.

The entire time this happens the heap's total size is allocated from the operating system. That does not change.

UNLESS you go and attempt to use an amount of memory that the heap is not large enough to support. When this occurs it has to request more memory from the operating system... this is a slow process. This is why it allocates a large portion up front instead of with every object you create.

So, when you go to load a 300MB file into memory, which is larger than the heap can handle. It has to go request 300 megs from the operating system. It uses it. You then GC that data. But the CLR maintains that memory for future use... you demonstrated to the CLR that you need this memory. Deallocating it back to the OS is expensive, and so is allocating it again if you need it again. So it'll maintain it as part of the heap for quite some time now...

This isn't a leak because the memory is still usable by your application. If you go and load yet another 300MB file after this one has been garbage collected, it won't ask the OS for another 300 megs... it has it already, and it will use it again.

A memory leak is run away memory... it's memory that gets filled up and forgotten about, never to be utilized again until the application haults.

 

I don't understand. There is no reference to that byte array. It should be collected by GC.

"So, when you go to load a 300MB file into memory, which is larger than the heap can handle. It has to go request 300 megs from the operating system. It uses it. You then GC that data. But the CLR maintains that memory for future use... you demonstrated to the CLR that you need this memory. Deallocating it back to the OS is expensive, and so is allocating it again if you need it again. So it'll maintain it as part of the heap for quite some time now..."

If you were right, the mono will take about 300MB memory, no matter how long the codes run. Because CLR has 300MB cached memory. But in fact, if you run the codes, you will receive out of memory exception in one minute.

 

Hrmm, let me run your specific code and see what's up.

Not on my unity dev machine right now, so it might be a little bit.

 

I encounter the same problem. How to solve it? Can anyone help me?

 

The array is most likely getting allocated to the Large Object Heap(Large Object Space in mono) and not being freed/re-used because the LOH is a pile of S***. There is also no guarantee that memory will be freed when calling GC.Collect()... Even if nothing is referencing the object. With large data you are better off writing a little unmanaged library that will allow you to free the memory immediately.

P.S. Why are your updates so insanely big? You should create a binary diff patcher so you only have to send the modifications. Assets can be compressed with LZ4 or LZMA to significantly reduce the size as well..

 

I have to create some AssetBundles from memory, which are large byte array. When I create those too many times, it will throw out of memory exception. I tried setting byte to null and GC.Collect(), but it did not work, the memory still not collected.
How to solve this kind of problem? Marshal.AllocHGlobal only creates unmanaged memory, but the AssetBundle.CreateFromMemory can only take byte[] as params.

 

I'm not sure how AssetBundles work in the engine. AssetBundle.CreateFromMemory could be keeping a reference to the byte array. The AssetBundle.CreateFromMemory method is also deprecated in 5.3.5. You should be using AssetBundle.LoadFromMemory now.

*Note* Unsafe code wont work in the webplayer.

 

Thanks for your reply.
It should have nothing to do with AssetBundle.CreateFromMemory or AssetBundle.LoadFromMemory.
I did a test: I only new a large byte array (hundreds of MB), then set it to null and GC.Collect(), the memory of byte array is not collected. However, if I call GC.Collect() every frame, it will collect memory.
How to collect memory immediately?

 

You should never directly call GC.Collect unless you know exactly what you're doing. There is a bigger underlying problem somewhere. The generational garbage collector wont free memory from the LOH as often. You can read more on mono's garbage collector here.

Is there something preventing you from creating multiple smaller AssetBundles? Smaller byte arrays will be collected much faster.

 

I'm facing this same issue. Running different Memory debugging tools it seems as though if a byte array is too big it is never recollected by Unity/Mono when Garbage Collecting.

 

I may be unexperienced, but to me, I see an infinite loop creating an infinite number of 300MB arrays that are never told they are no longer needed. Since the arrays are still 'needed', how can GC be run on them?

 

I believe this issue is related to this issue:
http://answers.unity3d.com/questions/225460/big-arrays-are-never-garbage-collected-why.html

 

That's not C++, that's C#. When you say

Code (CSharp):

1. arr = null;

you do say it's no longer needed.

 

Same problem here. Cannot avoid allocating big value of data - need to create AudioClip during runtime from Wav (converted from mp3). And still found no way to clean allocated memory.

 

5.6.6 f2 reproduced.
This problem has caused fatal "loss of sales" to our project.
use UnityWebRequest