DownloadHandlerScript (to Stream) performance

I'm reading in some data from through http that either gets parsed as an XML or saved as binary data to a file.

In order to reduce the memory footprint, I figured I'd make an implementation of UnityEngine.Networking.DownloadHandlerScript that returns an implementation of System.IO.Stream. It works with blocks that are each an array of bytes. Once a full block has been read from the stream, I discard it, which is the memory reduction part.

The issue I have is the difference in download speed. The timing for a 10 MB file is roughly:

• DownloadHandlerBuffer: 0.8 to 1.0 seconds
• My own DownloadHandler: 2.7 to 2.9 seconds

I've already tried some optimizations:

• Using Array.Copy instead of copying in a loop in the script
• Recycling the blocks of bytes instead of letting the garbage collector handle it
• Removed the usage of Monitor.Wait / Monitor.Pulse to block and continue the reading thread

None of these have any noticeable effect on the time needed. I've even tried just not doing anything at all with the data, which also has no noticeable effect.

It really doesn't seem that it's my script that is spending too much time, but the download is simply slower with my own DownloadHandler. It takes roughly 2.7 to 2.9 seconds for the data to come in, while with DownloadHandlerBuffer everything is downloaded in 0.8 to 1.0 seconds.

The methods I'm overwriting of DownloadHandlerScript are:

• ReceiveContentLength
• ReceiveData
  
• CompleteContent
• GetProgress

Am I missing something here? Maybe a larger block size I can specify? Or is this just the overhead of DownloadHandlerScript?

 

Thanks for your quick reply Aurimas. I'm using a fairly old version of Unity (5.6.3f1), so no DownloadHandlerFile yet. (I would have used that.)

I understand DownloadHandlerBuffer is fast, but that doesn't seem to be the issue here. I made a dummy implementation of DownloadHandlerScript, which pretty much does nothing, but it still takes 2.7 to 2.9 seconds to receive the download:

Code (csharp):

1.  
2. public class KOGL4DownloadHandlerDummy : DownloadHandlerScript
3. {
4.     protected override void ReceiveContentLength(int contentLength)
5.     {
6.     }
7.  
8.     protected override bool ReceiveData(byte[] data, int dataLength)
9.     {
10.         return true;
11.     }
12.  
13.     protected override void CompleteContent()
14.     {
15.         Debug.Log("Done at " + Time.realtimeSinceStartup);
16.     }
17.  
18.     protected override float GetProgress()
19.     {
20.         return 0.0f;
21.     }
22. }
23.  

I can post the actual code, but this dummy shows the same time increase.

 

Yes, you are exactly right, 64K each call.

Passing a fixed 64K buffer in the constructor doesn't do much either.

I see a few options:

• Keep it like this, less memory use, but slower downloads
• Implement a Stream on top of DownloadHandlerBuffer with more memory use, but faster (and parsing can start while downloading)
• Migrate this project to a new version of Unity and implement a Stream on top of DownloadHandlerFile for both less memory use and speed
• Use .net WebRequest instead

I think I'll go for option 2.

Edit: Ok, option 2 doesn't work, because there is no way to get a partial download from DownloadHandlerBuffer. Back to the good old MemoryStream then...

 

Hi, question re: UnityWebRequest performance

We did a performance test of UnityWebRequest with DownloadHandleFile and equivalent in native code on iOS and the native implement is about 30-40% faster consistently.
Is there anything we can change on the Unity side to speed things up? I would of thought the WebRequest was as optimal as it could be but there must be a bunch of other things going we don't know about? Using 2018.2.21 at the moment.

 

Hi Aurimas, we used NSUrlSession for iOS - that said, our tests ended up being inconclusive
We had a local server and we tested a native version vs HttpClient async/await C# vs UnityWebRequest and UnityWebRequest was 2x slower

We then tried it via an S3 bucket and actually UnityWebRequest came in faster, so we're not sure what is going on.

We were trying to switch our download system from Coroutine based to async/await as we noticed our downloads were not downloading optimally - we weren't sure if it was the overhead of coroutines ( deeply nested ) and our network to the outside was hit and miss so using an internal local host, we got consistent numbers to say that using HttpClient async/await or just a straight iOS NSLUrlSession call was faster or so we thought.

Note: we performed all tests off the same device, point to the same servers and so on to keep as many parameters the same as possible.

Any ideas on the variation? I don't know how UnityWebRequest is constructed under the hood.

regards

 

We did some more tests, on Device with a downloadhandler things do seem to be quite a bit slower
We tested a 512MB file on S3
Android and iOS both exhibit slow performance so we may have to put time into a native downloader as we cannot explain the difference otherwise outside of the Update you mentioned.
Tried via safari and it came down on average in 30-40seconds on device. Our local test came down around 1min when done via UnityWebRequest and downloadfilehandler although we were using a small buffer ( will tweak this to see how to improve )

We also see this spewing in xcode though ( we are on 10.2.1 and Unity 2018.4.2 ) - apparently we can ignore?

[NetworkInfo] Signal strength query returned error: Error Domain=NSPOSIXErrorDomain Code=13 "Permission denied", descriptor: <CTServiceDescriptor 0x2810087c0, domain=1, instance=1>