Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Scripting Downloading data with UnityWebRequest

Discussion in '5.4 Beta' started by liortal, May 2, 2016.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,543
    Since a few releases ago, UnityWebRequest is available as an "experimental" feature.

    Although it's still experimental, i am looking at it as a replacement for the WWW class that we extensively use in our code.

    One point for us to consider moving to this new infrastructure is performance. I was wondering if transitioning to UnityWebRequest will benefit in any way with regards to performance ?

    Here are a few scenarios we're interesting in:
    1. Retrieving large JSON files from a remote server
    2. Retrieving textures at runtime from file/remote server
    3. Downloading and loading asset bundles from local file/remote server

    Things I'd like to see possible are:
    1. Memory: avoiding C# heap allocations (e.g: when accessing www.bytes or www.text). This could be done by reusing constant data buffers or even getting access to the raw native buffers somehow.
    2. Performance: Faster execution (faster downloads) by a better underlying implementation of the networking code.
    Are these supported? what are the current improvements of UnityWebRequest over the WWW class? (by current i mean with 5.4 or latest 5.3 versions)
     
    Last edited: May 2, 2016
    liortal, May 2, 2016
    #1
    JohnTube likes this.

  2. Agent_007

    Agent_007

    Joined:
    Dec 18, 2011
    Posts:
    899
    Agent_007, May 3, 2016
    #2

  3. Justinas

    Justinas

    Unity Technologies

    Joined:
    Sep 25, 2013
    Posts:
    12
    UnityWebRequest would give you optimized texture and asset bundle loads.

    However, at least today I am apt to look askance at giving access to raw buffers for www.bytes or www.text.

    It's not that I don't like to see memory being used conservatively, quite the contrary. The feature you want simply looks like a way to invite memory access disasters. For instance, consider a case when the lifetime of the download handler is shorter than the lifetime of the buffer you want to work on. You see, upon destruction the download handler would free the memory and you will be left with a buffer whose internal pointers are wild.

    For this reason www.text and www.buffer return managed copies of the internal buffer.
     
    Justinas, May 4, 2016
    #3

  4. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,543
    @Justinas thanks for your response ;)

    Regarding the case you present (data buffer lives longer than the download handler:

    First of all, this will probably always be the case (according to how i see the feature).

    Youwould allocate a managed array, big enough for your needs upfront, then give that to the DL handler and it should fill that up with data once its done. It can destroy the data once its done, but that means that the DLhandler was collected so nobody can access it anymore anyway.

    Imagine this usage scenario (saving downloaded data to disk):
    1. Allocate 4 mb byte[] in advance (this is enough for all current uses).
    2. Construct a DownloadHandler, pass the buffer to it.
    3. Once its done downloading, i can access the data, (save it to disk for example) without any GC alloc.
    You could define that as a new DownloadHandler actually to avoid any confusions or "disasters" as you call them :)
     
    Last edited: May 4, 2016
    liortal, May 4, 2016
    #4

  5. Justinas

    Justinas

    Unity Technologies

    Joined:
    Sep 25, 2013
    Posts:
    12
    The scenario that you describe does not need any exotic download handlers. There is a DownloadHandlerScript, from which you can derive and implement your own download handler.

    Your custom download handler may do these things ;-)
    1. In its c-tor allocate 4 Mb byte[] in advance.
    2. Once it's done downloading (during downloading, too) you can access the data, save the stream to disk or whatnot.
    The same technique could be applied for many other things, e.g., streamed decompression/decoding, save to file, etc., all with minimal memory and runtime overhead.
     
    Justinas, May 4, 2016
    #5

  6. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,543
    I see now that DownloadHandlerScript accepts a preallocatedBuffer parameter in its ctor.

    A few other questions pop to mind:

    1. Why isn't that an abstract class?
    2. Some methods on DownloadHandler are virtual, for example the 'data' property which internally calls GetData(), however GetData() is not overriden in DownloadHandlerScript to return the preallocated buffer. why is that ?
    3. If i have a preallocated buffer, how do i know what is the length of the content that was actually downloaded
     
    liortal, May 4, 2016
    #6

  7. Justinas

    Justinas

    Unity Technologies

    Joined:
    Sep 25, 2013
    Posts:
    12
    It is not an abstract class for technical reasons. Native code needs to be able to bind to the hooks in the DownloadHandlerScript.

    Also, I didn't override the GetData() because in order to override this method I would need to add preallocatedBuffer to the class definition, i.e., the users that don't use preallocatedBuffers would have to pay for something they don't use. The current interface is a compromise, it allows you to pass the preallocated buffer and you can override GetData() to return what you want.

    You need to track the length in ReceiveData. Also, before you receive your data a webserver may give you a hint what length you can expect via ReceiveContentLength, but I would advise to take this hint with a grain of salt. Webserver implementations are many, and UnityWebRequest cannot guarantee that this hint is sensible. It simply reports Content-Length back.
     
    Justinas, May 4, 2016
    #7
    landon912 likes this.

  8. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,543
    In that case how can i know how much of the buffer was actually used for the download?

    It would simply be the total length as received by ReceiveData then ?
     
    liortal, May 4, 2016
    #8

  9. Justinas

    Justinas

    Unity Technologies

    Joined:
    Sep 25, 2013
    Posts:
    12
    Yes.
     
    Justinas, May 4, 2016
    #9
    liortal likes this.

  10. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,543
    Cool i'll check it out.

    Does UnityWebRequest currently have any other advantages over WWW (when downloading textures or asset bundles) ?
     
    liortal, May 4, 2016
    #10

  11. Justinas

    Justinas

    Unity Technologies

    Joined:
    Sep 25, 2013
    Posts:
    12
    On transport level -- no. Data is data and it needs to be transferred. I will note, though, that WWW is going to be deprecated at some point, so getting acquainted with UnityWebRequest might be a good idea. Besides, I think it's nicer to work with and allows better opportunities for custom data stream handling.
     
    Justinas, May 4, 2016
    #11

  12. jkampitakis

    jkampitakis

    Joined:
    Dec 27, 2015
    Posts:
    56
    If someone tries to use the UnityWebRequest class to fetch a file from a url that has double slashes "//" like the following "https://s3.amazonaws.com/zoo.misc//misc/llgames_fourpics_data-el_GR.txt" he will see that the double slashes get transformed to single slash "/" so the GET request now tries to get the file from here "https://s3.amazonaws.com/zoo.misc/misc/llgames_fourpics_data-el_GR.txt"

    Why does this happen??
    Using the WWW class the GET request is correct (double slashes are preserved).
     
    jkampitakis, Jul 19, 2016
    #12
    Colin_MacLeod likes this.