Search Unity

Reading massive amounts of model data from the browser?

Discussion in 'Web' started by DeltaPiSystems, Nov 5, 2019.

  1. DeltaPiSystems

    DeltaPiSystems

    Joined:
    Jul 26, 2019
    Posts:
    30
    I have customer supplied model files that need to be loaded into Unity.

    I've been using SendMessage with JSON so far, and it's insufficient for the following reasons:
    1. There's a limit to the size of the strings parsed through SendMessage, I have to split the json into 1000000 character chunks to be viable.
    2. The Unity side has to recombine those chunks, currently using a StringBuilder with predetermined size.
    3. Unity has to parse the JSON, this is slow and memory hungry
    4. Large JSON files can't be parsed in a single go, parsing takes too much memory and WebGL crashes.
    5. Not only does the JSON need to be split in chunks, the data also needs to be split in segments and sent with some delays between each segment, for Unity to garbage collect the parsing data of the previously sent segment after a few frames.
    The data that has to be retrieved from the browser is relatively simple:
    strings, arrays of floats/ints, arrays of vector3s

    An idea I have but I'm unsure about:
    Unity can call external JavaScript functions, those can be used to return the data from the browser.
    Obviously, it's possible to serialize my data to some binary format, which would certainly be faster than JSON... But what if I can skip serialization completely?

    It's possible from the JavaScript side to allocate memory for strings and arrays in the Unity memory array, which is then filled and returned to Unity.
    I was wondering, is it possible to create structs using a JavaScript library (.jslib) and return that?

    The struct would be something like this:
    Code (CSharp):
    1. struct Data {
    2.   public string Id;
    3.   public int[] Numbers;
    4.   public Vector3 Points;
    5. }
    This way I wouldn't have to run any garbage collection, as there's no garbage to collect.

    It's not possible for Unity to read the model data directly from a server, the data is interacted with in JavaScript...

    I'm open for other ideas, obviously, as this is probably a rather complex situation.