UWP: Memory leak when downloading large files

The following code run on UWP will raise the memory by 100MB every iteration until the application crashes.

What am I missing here?

We're seeing behavior where we're downloading smaller files and saving them to disk, but never getting that memory back. This is the smallest reproducible example that looks to be the root cause of our memory issues.

This is running on a desktop with a ton of memory, but our application runs on a Hololens where we really only have 900MB, and where unity engine takes up 350MB standard leaving us with just 550MB for our app

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.Networking;
5. using System.IO;
6. using UnityEngine.Profiling;
7.  
8. public class KeepDownloading100MBFiles : MonoBehaviour
9. {
10.     private float downloadCompletion;
11.     private int fileWriteIter = 0;
12.  
13.     IEnumerator Start()
14.     {
15.         // This file is 100MB
16.         Debug.Log("downloading initial file ");
17.         UnityWebRequest request = UnityWebRequest.Get("http://speedtest.ftp.otenet.gr/files/test100Mb.db");
18.  
19.         request.SendWebRequest();
20.         while (!request.isDone)
21.         {
22.             yield return new WaitForSeconds(2);
23.             // Debug.Log("Download Completion: " + request.downloadProgress);
24.             downloadCompletion = request.downloadProgress;
25.         }
26.  
27.         Directory.CreateDirectory(Application.temporaryCachePath + "/test");
28.         string outPath = Application.temporaryCachePath + "/test/test_orig";
29.         File.WriteAllBytes(outPath, request.downloadHandler.data);
30.  
31.         // Memory usage increases by 100MB+ each loop
32.         while (fileWriteIter < 100)
33.         {
34.             yield return new WaitForSeconds(3f);
35.  
36.             request = UnityWebRequest.Get("file://" + outPath);
37.             yield return request.SendWebRequest();
38.  
39.             Debug.Log("Writing " + fileWriteIter);
40.             File.WriteAllBytes(Application.temporaryCachePath + "/test/test_orig_" + fileWriteIter, request.downloadHandler.data);
41.  
42.             fileWriteIter++;
43.         }
44.     }
45.  
46.     GUIStyle style = new GUIStyle();
47.     void OnGUI()
48.     {
49.         style.fontSize = 40;
50. #if UNITY_WSA && !UNITY_EDITOR
51.         GUILayout.Label("allocated RAM: " + (Windows.System.MemoryManager.AppMemoryUsage / 1048576f), style);
52.         GUILayout.Label("reserved RAM: " + (Windows.System.MemoryManager.AppMemoryUsageLimit / 1048576f), style);
53.         GUILayout.Label("mono RAM: " + (Profiler.GetMonoUsedSizeLong() / 1048576f), style);
54.         GUILayout.Label("download completion: " + downloadCompletion, style);
55.         GUILayout.Label("write iteration: " + fileWriteIter, style);
56. #else
57.         GUILayout.Label("allocated RAM: " + (Profiler.GetTotalAllocatedMemoryLong() / 1048576f), style);
58.         GUILayout.Label("reserved RAM: " + (Profiler.GetTotalReservedMemoryLong() / 1048576f), style);
59.         GUILayout.Label("mono RAM: " + (Profiler.GetMonoUsedSizeLong() / 1048576f), style);
60.         GUILayout.Label("download completion: " + downloadCompletion, style);
61.         GUILayout.Label("write iteration: " + fileWriteIter, style);
62. #endif
63.     }
64. }
65.  

Video of the app until it crashes: https://slack-files.com/T2VGY5GH5-FPB1EJVR9-1e0b56a5de

Screenshot of the crash: https://slack-files.com/T2VGY5GH5-FPG2D02P3-7872f7631c

Running Unity 2019.2.7f2

 

I suspected that maybe the infinite coroutine may be an issue, so I wrote this as a update loop with separate coroutines:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.Networking;
5. using System.IO;
6. using UnityEngine.Profiling;
7.  
8. public class KeepDownloading100MBFiles : MonoBehaviour
9. {
10.     private UnityWebRequest request;
11.     private float downloadCompletion;
12.     private int fileWriteIter = 0;
13.     private bool initialDownloadComplete;
14.     private bool downloadAndSaveComplete = true;
15.     private string initialDownloadPath;
16.  
17.     void Awake()
18.     {
19.         initialDownloadPath = Application.temporaryCachePath + "/test/test_orig";
20.     }
21.  
22.     void Start()
23.     {
24.         StartCoroutine(DownloadInitialFile());
25.     }
26.  
27.     void Update()
28.     {
29.         if (initialDownloadComplete && downloadAndSaveComplete)
30.         {
31.             downloadAndSaveComplete = false;
32.             StartCoroutine(DownloadAndSaveFile(initialDownloadPath));
33.         }
34.     }
35.  
36.     IEnumerator DownloadInitialFile()
37.     {
38.         // This file is 100MB
39.         Debug.Log("downloading initial file ");
40.         request = UnityWebRequest.Get("http://speedtest.ftp.otenet.gr/files/test100Mb.db");
41.  
42.         request.SendWebRequest();
43.         while (!request.isDone)
44.         {
45.             yield return new WaitForSeconds(2);
46.             Debug.Log("Initial Download Completion: " + request.downloadProgress);
47.             downloadCompletion = request.downloadProgress;
48.         }
49.  
50.         Directory.CreateDirectory(Application.temporaryCachePath + "/test");
51.         File.WriteAllBytes(initialDownloadPath, request.downloadHandler.data);
52.  
53.         initialDownloadComplete = true;
54.     }
55.  
56.     IEnumerator DownloadAndSaveFile(string path)
57.     {
58.         yield return new WaitForSeconds(3f);
59.  
60.         request = UnityWebRequest.Get("file://" + path);
61.         yield return request.SendWebRequest();
62.  
63.         Debug.Log("Writing " + fileWriteIter);
64.         File.WriteAllBytes(Application.temporaryCachePath + "/test/test_orig_" + fileWriteIter, request.downloadHandler.data);
65.  
66.         fileWriteIter++;
67.         downloadAndSaveComplete = true;
68.     }
69.  
70.     GUIStyle style = new GUIStyle();
71.     void OnGUI()
72.     {
73.         style.fontSize = 40;
74. #if UNITY_WSA && !UNITY_EDITOR
75.         GUILayout.Label("allocated RAM: " + (Windows.System.MemoryManager.AppMemoryUsage / 1048576f), style);
76.         GUILayout.Label("reserved RAM: " + (Windows.System.MemoryManager.AppMemoryUsageLimit / 1048576f), style);
77.         GUILayout.Label("mono RAM: " + (Profiler.GetMonoUsedSizeLong() / 1048576f), style);
78.         GUILayout.Label("download completion: " + downloadCompletion, style);
79.         GUILayout.Label("write iteration: " + fileWriteIter, style);
80. #else
81.         GUILayout.Label("allocated RAM: " + (Profiler.GetTotalAllocatedMemoryLong() / 1048576f), style);
82.         GUILayout.Label("reserved RAM: " + (Profiler.GetTotalReservedMemoryLong() / 1048576f), style);
83.         GUILayout.Label("mono RAM: " + (Profiler.GetMonoUsedSizeLong() / 1048576f), style);
84.         GUILayout.Label("download completion: " + downloadCompletion, style);
85.         GUILayout.Label("write iteration: " + fileWriteIter, style);
86. #endif
87.     }
88. }
89.  

Still crashes, see screenshot: https://slack-files.com/T2VGY5GH5-FPB3TN4HZ-47c3153eec

 

Thank you for your response @Aurimas-Cernius

Using the DownloadHandlerFile does fix this issue.

But it seems strange to me that we have no way to force clearing the memory when we know we're about to crash because we're running out of memory.

I've added

System.GC.Collect(100, System.GCCollectionMode.Forced, true);

after writing file to disk with File.WriteAllBytes, but that doesn't seem to do a lot.

The above code crashes after about 10 iteration (~1GB of app memory usage) on a PC that has 32GB of memory.
Running this on a Hololens 1 seems to crash the application after 20iterations at around ~1.3GB.

The wider issue is that we use a GLTF model loader from the MRTK codebase:
https://github.com/microsoft/MixedR...1.0/Assets/MixedRealityToolkit/Utilities/Gltf

Essentially, a GLTF is a large JSON file. The JSON file embeds textures and mesh data as a base64 string. When we load the model, the mesh data is read as a base64 string and extracted into a byte array (this has a large memory footprint). It seems to me that any time we allocate a large buffer (like a memorystream or a byte array), there's risk of never retrieving that memory back for later use.

Specific things I can point to in the GLTF plugin:

1. string gltfJson = File.ReadAllText(uri)

   https://github.com/microsoft/MixedRealityToolkit-Unity/blob/v2.1.0/Assets/MixedRealityToolkit/Utilities/Gltf/Serialization/GltfUtility.cs#L54-L6

2. glbData = new byte[buffer.Length]

   https://github.com/microsoft/MixedR...ies/Gltf/Serialization/GltfUtility.cs#L74-L88

What is the solution in this case?

 

Does this mean that unity just doesn't support loading in large files into memory?

 

I appreciate your feedback and help on this, and I agree with your assessment that in general we should stream instead of reading the file content into memory all at once.

In the above examples (where I wasn't yet using

DownloadFileHandler

), if I call Dispose on the web requests it should release all references to memory and should properly GC? I understand that this is bad practice

 

Ok, just to test this out I'm now disposing the request properly:

Code (CSharp):

1. using System.Collections;
2. using UnityEngine;
3. using UnityEngine.Networking;
4. using System.IO;
5. using UnityEngine.Profiling;
6.  
7. public class KeepDownloading100MBFiles : MonoBehaviour
8. {
9.     private int fileWriteIter = 0;
10.     private bool downloadAndSaveComplete = true;
11.     private bool paused = false;
12.  
13.     void Update()
14.     {
15.         if (downloadAndSaveComplete)
16.         {
17.             System.GC.Collect();
18.             downloadAndSaveComplete = false;
19.             StartCoroutine(DownloadAndSaveFile());
20.         }
21.  
22.         if (Input.GetKeyUp(KeyCode.P))
23.         {
24.             paused = !paused;
25.         }
26.     }
27.  
28.     IEnumerator DownloadAndSaveFile()
29.     {
30.         yield return new WaitForSeconds(1f);
31.  
32.         if (!paused)
33.         {
34.             string outWritePath = Application.temporaryCachePath + "/test/dump_" + fileWriteIter;
35.             string dirPath = System.IO.Path.GetDirectoryName(outWritePath);
36.             Directory.CreateDirectory(dirPath);
37.  
38.             using (UnityWebRequest request = UnityWebRequest.Get(Path.Combine(Application.streamingAssetsPath, "test100Mb.db")))
39.             {
40.                 yield return request.SendWebRequest();
41.  
42.                 File.WriteAllBytes(outWritePath, request.downloadHandler.data);
43.             }
44.  
45.             fileWriteIter++;
46.         }
47.  
48.         downloadAndSaveComplete = true;
49.     }
50.  
51.     GUIStyle style = new GUIStyle();
52.     void OnGUI()
53.     {
54.         style.fontSize = 40;
55. #if UNITY_WSA && !UNITY_EDITOR
56.         GUILayout.Label("allocated RAM: " + (Windows.System.MemoryManager.AppMemoryUsage / 1048576f), style);
57.         GUILayout.Label("reserved RAM: " + (Windows.System.MemoryManager.AppMemoryUsageLimit / 1048576f), style);
58.         GUILayout.Label("mono RAM: " + (Profiler.GetMonoUsedSizeLong() / 1048576f), style);
59. #else
60.         GUILayout.Label("allocated RAM: " + (Profiler.GetTotalAllocatedMemoryLong() / 1048576f), style);
61.         GUILayout.Label("reserved RAM: " + (Profiler.GetTotalReservedMemoryLong() / 1048576f), style);
62.         GUILayout.Label("mono RAM: " + (Profiler.GetMonoUsedSizeLong() / 1048576f), style);
63. #endif
64.         GUILayout.Label("write iteration: " + fileWriteIter, style);
65.         GUILayout.Label("paused: " + paused, style);
66.     }
67. }
68.  

It doesn't really behave much differently. Memory keeps going up until it reaches ~1.2GB. I then pause that ugly file download and dump (not application pause, just skip the logic) and I see that the GC does recover some memory, but leaving it running does not reduce memory back to what the application was at when it booted up.

Image: https://slack-files.com/T2VGY5GH5-FPS4D0RNU-345dbc8256

Is the behavior that I am expecting to see correct?

 

Hi, we're seeing the same behaviour in UWP (Hololens) using HttpClient. My current suspicions is that there is a large object heap issue in UWP/IL2CPP where some large buffers never get cleared. If I download objects of 0-200k and access them as a byte array everything appears to be fine (or stream them to disk), when we hit objects of 200k+ the memory climbs.

I could do this reliably with a test app on HoloLens using the following steps:

1. download a 1.6mb png
2. access the data as an byte[]
3. dispose of the stream and de-reference the array
4. wait a few seconds and start again.

Over a couple of minutes the memory use will push up towards 900mb.

If I convert the same image to a jpg (220kb) and perform the same process memory doesn't appear to grow (or nowhere not as fast).

Streaming the files to disk has no problems.