How to prevent JSON parsing at each app launch?

Hi all,
I am asking for your advice. I have to parse a JSON file (approx. 6 MB), its content is used in the entire app, across multiple scenes. The parsing works fine and I can use it in every scene.
But, every time the app is launched, parsing is done again. How could I avoid this? Would you use SQLite to store the info of the parsed JSON locally and in a persistent way?

Thanks in advance for your comments!

 

It depends on a lot of things. First, is it actually a problem parsing the data every time the app starts? I assume so, or you wouldn't be asking about it. But how much of an impact does it have? 1 seconds? 10 seconds? More?

How necessary is all that data from the start of the app? Are you currently loading all of it merely because it's all in one file? Or do you only need a portion of it to start the game, and other portions later as you get to new sections of the game?

It's possible a database format is generally faster, but not necessarily. Depending on how you access the data, it could be faster to keep it in a single in-memory structure as you have it now, compared to converting to a database approach where you're issuing many queries against the data.

Before I tried changing the format significantly, I'd check whether its feasible to just break the data up into smaller chunks that load faster, especially if it's possible to isolate them to individual levels.

But it would be best if you could describe what kind of data it is, or how you're using it, to better inform you of whether a database approach seems better.

 

What JSON parser are you using? Are you deserialising into an object graph? I was going to suggest serialising your data out as an offline pre-process step using BinaryFormatter, but amazingly enough it looks like Json.NET is actually faster than BinaryFormatter: http://james.newtonking.com/archive/2010/01/01/net-serialization-performance-comparison (this is from 2010 so the situation may have changed).

 

Hi all, Many thanks for your thoughts. Yes, I am already using Json.NET for parsing the JSON. The JSON contains info I need everywhere to populate scroll views of a menu / navigation, similar to lists of an inventory.

The nesting is 6 levels deep. Once the user got to the last nested level (6th level) and chose a Child item from the scroll view list, I have to get the ID of that chosen Child6 and then look in "Persons" for a Person object that matches the passed ID. Currently, there are approx. 4.500 Person items. (I need to access the thumbs of Child and Person items quickly so that the cells of the scroll views can be quickly populated and that the scrolling runs smoothly. Currently, this works fine)

The structure looks like:

Root

       - Data

               - Child1

                       - Child 2

                       …

                               - Child 6



       - Persons

The parsing takes less than 8 seconds on my devices (iPad Air, Pixel 2, iPhoneX etc.) , but may take more time depending on the processors. I fear that this might bother users, if parsing happens each time the app is launched.
That is why I am looking for a solution so that the parsing has only to be done once on first app launch. I mean, if the user opens the app later, I do not want that the JSON parsing happens again. Ideally, the parsed info should be stored locally in a persistent way that "survives" app sessions.

 

I would try serialising your object graph to disk with BinaryFormatter and loading that instead to see if it makes a difference. That could even be done as a build-time process so it never runs on the final build.

 

Ok, thanks! I will try that.

 

That seems to imply that you have more than a kilobyte of data on each individual person; even with the substantial inefficiencies of JSON, that seems like a lot. Do you need all of that data on every person all at once? Maybe your main file could just include an index of the people, and then you could look up details on an individual person in separate files as-needed.

 

Well, the "Child" classes of the hierarchy also contain data and have several fields. I may not split or modify the JSON, because it is delivered and not created on my own. I have to use the JSON file as is.

 

I'm not sure why you believe that samizzo's suggestion of saving the data in a different format is workable but my suggestion of splitting up the file is not, but you'd know your circumstances better than me.

The simplest optimizations you could try are, of course:

• Load the file asynchronously in the background, so that other parts of your app will appear responsive even before it finishes loading
• Keep the data in memory when switching scenes so it does not have to be loaded again

 

Thanks @Antistone . After parsing the JSON, I store the data in a ScriptableObject and can access it after switching scenes. This seems to work.
However, I am loading and parsing in 2 different coroutines, and observe both to get notified as soon as the second coroutine is finished. Do you mean this by "Load the file asynchronously in the background"? I am asking, because the UI is not responsive as long as the coroutines are running. Do I have to start the Coroutines from a background thread?

I do not think that I may split the JSON file, because in a second stage, I would have to load the JSON from a server (RESTful API) and also check against updated data. So splitting the JSON would mean that the server has to deliver more than a single JSON file.

 

To load it asynchronously in the background in a coroutine you'd need to have a JSON parser which can run asynchronously and that you could yield from your coroutine. Otherwise as you noticed you'll see a stall when it comes time to deserialise. But if you could find one which has async support then you could use it. I'm not sure if Json.Net has an async API. Otherwise you could do it in another thread.

My reasoning for serialising the data out into a binary blob is that JSON parsing requires performing a lot of string operations (creating strings, comparing strings). I'm assuming (but don't know for sure) that if you have serialised the data out to a binary blob with a BinaryFormatter, it should be a lot quicker to re-create the object graph and populate it with data since BinaryFormatter writes out a lot of metadata about the classes involved and doesn't have to parse a lot of strings.

The other option which might be faster is to manually (de)serialise. So something like this:

Code (csharp):

1.  
2. class Persons
3. {
4.     public List<Person> People { get; private set; }
5.     public void Serialise(BinaryWriter writer)
6.     {
7.         writer.Write(People.Count);
8.         for (var i = 0; i < People.Count; i++)
9.             People[i].Serialise(writer);
10.     }
11.  
12.     public void Deserialise(BinaryReader reader)
13.     {
14.         People = new List<Person>();
15.         var numPeople = reader.ReadInt32();
16.         for (var i = 0; i < numPeople; i++)
17.         {
18.             var person = new Person();
19.             person.Deserialise(reader);
20.             People.Add(person);
21.         }
22.     }
23. }
24.  
25. class Person
26. {
27.     public string Name { get; private set; }
28.     public int Age { get; private set; }
29.  
30.     public void Serialise(BinaryWriter writer)
31.     {
32.         writer.Write(Name);
33.         writer.Write(Age);
34.     }
35.  
36.     public void Deserialise(BinaryReader reader)
37.     {
38.         Name = reader.ReadString();
39.         Age = reader.ReadInt32();
40.     }
41. }
42.  

This way you just blast through the data reading and writing exactly what you need. There'll be no field or type lookups which the other techniques would be doing. However note that there is no backwards compatibility so if you change the data that you store or you change the order of the data in (de)serialise you'll have to update your (de)serialise routines.

 

Depends how you're doing the actual work. Just putting something in a Unity coroutine doesn't make it asynchronous; control is only returned to the Unity engine while you yield.

Unity provides certain asynchronous operations that are specifically designed to be called from a coroutine and yielded; this allows the coroutine to wait without blocking while Unity finishes the other operation however it does it. For instance, a coroutine can yield a WWW object (deprecated) to wait for it to finish without blocking (the UI will continue to respond while you wait).

But if you just call a regular synchronous function, the fact that you are calling it from a coroutine doesn't make it asynchronous. Everything from one "yield" to the next "yield" is synchronous.

Starting your own thread is an option, but the separate thread should never start a coroutine. In fact, almost everything in the UnityEngine namespace is only safe to call from the main Unity thread; if you use your own thread, you generally have to rely on vanilla C# within that thread, and then write your results to some shared data structure and rely on some function in the main thread to periodically check for them until they're ready. (You can use a coroutine to do the periodic checking, if you like.)

 

Thank you, guys, for your advices! I will dig into