Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

How to read a json array

Discussion in 'Scripting' started by harpingseal, Apr 18, 2022.

  1. harpingseal

    harpingseal

    Joined:
    Sep 3, 2020
    Posts:
    57
    Hi so I have a josn table with around 40,000 entry which would look similar to this {{777,50,-4001},{777,50,-4001}}
    And I've saved it to a text in my resource folder, But now Im not sure how to go from there because most of them list for a dictionaries, Anyway I would like to be able to use the script to loop through and get the value sort of like this in lua
    Code (CSharp):
    1. for _,v in pairs(array) do
    2. print(v[1],v[2],v[3])
    3. end
     
  2. CortiWins

    CortiWins

    Joined:
    Sep 24, 2018
    Posts:
    150
    If you want to deserialize that json, you need to start with the question of which data structure looks like that as json. It looks like you have an array of arrays with 3 values each. Something like this

    Code (CSharp):
    1. var arrr = new int[3][]
    2. {
    3.     new int[]{1,2,3},
    4.     new int[]{11,2,3},
    5.     new int[]{1,222,333},
    6. };

    I just tried it with Unitys JsonUtility and it's not doing it well because JsonUtility doesn't deal well with those data structures. You may try json.net, thats more capable.
     
  3. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    38,520
    In general I highly suggest staying away from Unity's JSON "tiny lite" package. It's really not very capable at all and will silently fail on very common data structures, such as Dictionaries and Hashes and ALL properties.

    Instead grab Newtonsoft JSON .NET off the asset store for free, or else install it from the Unity Package Manager (Window -> Package Manager).

    https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347

    Also, always be sure to leverage sites like:

    https://jsonlint.com
    https://json2csharp.com
    https://csharp2json.io
     
  4. harpingseal

    harpingseal

    Joined:
    Sep 3, 2020
    Posts:
    57
    Hi the problem is, Im not sure how many entry I have and I couldn't possibly create 40,000 ish amount of variable, Is there a way that I could loop thorugh them? thanks, Heres the full file if you will https://pastebin.com/AbgwcrD9
     
  5. nijnstein

    nijnstein

    Joined:
    Feb 6, 2021
    Posts:
    78
    Seems to be just a vector3 array, something like this would work without any library if you only need this data as these libaries are usually not very performant with 40k entries:

    Code (CSharp):
    1. List<Vector3> list = new List<Vector3>();
    2. var records = json.Substring(2, json.Length - 4).Split("},{");
    3. foreach(var r in records)
    4. {
    5.    var data = r.Split(",");
    6.    Assert.IsTrue(data.Length == 3);
    7.    list.Add(new Vector3(int.Parse(data[0]), int.Parse(data[1]), int.Parse(data[2]));
    8. }
    although if you have more and different datafiles using a json library might be a lot more flexible
     
  6. Bunny83

    Bunny83

    Joined:
    Oct 18, 2010
    Posts:
    3,922
    This is not valid JSON. The outermost element is an object which has no keys. Instead the content inside looks like an array. However a json array has to use
    [ ]
    instead of
    { }
    . So no json framework would load your data properly.

    Json is extremely simply and has a clear and strict structure which you can see here.

    Your data is a valid LUA table, but that's only valid inside a lua script. This is not json.
     
  7. CortiWins

    CortiWins

    Joined:
    Sep 24, 2018
    Posts:
    150
    In that case, the solution is simple: Parse it manually. Shouldn't be hard as the data structure is simple with little variance.

    The text size is 255kb. The content is 40.000 x 3 values, assuming 32 bits per variable, thats below 500kb in memory. Just load it all, get the ressource as a string, and go through it bracket by bracket. string.IndexOf allows to search from a startIndex.
     
  8. Bunny83

    Bunny83

    Joined:
    Oct 18, 2010
    Posts:
    3,922
    Btw: I just had a quick look at the values in the table and there are a few strange values in that table which seems to be way off the range of all other values. Specifically

    Code (CSharp):
    1.  
    2. index: 8715 = [229866,308283,17595]
    3. index: 8876 = [-4059410,167519104,122707464]
    4. index: 9243 = [-2147483520,2147483520,2147483520]
    5. index: 9279 = [1014066708480,718790524928,-1384858517504]
    6. index: 9320 = [1175865,17334174,-6155725]
    7. index: 9322 = [38171308,556401792,-199228960]
    8. index: 9471 = [-186836896,194886896,-117432696]
    9. index: 9473 = [-857068928,894179264,-537436928]
    10. index: 13866 = [1071164,1048637,-1071164]
    11. index: 13870 = [1568929,1338573,-1568930]
    12. index: 15220 = [-2147225600,2147483648,2147483648]
    13. index: 15224 = [-700137,1247478,1299864]
    14.  
    I'm not sure what those values may represent, but you probably want to be careful with those. This may be some sort of heatmap data. Though it seems there's a lot of noise in those samples.
    PastebinPointCloud.gif

    So you probably need some sanity filtering and depending on the actual usecase some kind of clustering algorithm.
     
  9. harpingseal

    harpingseal

    Joined:
    Sep 3, 2020
    Posts:
    57
    Hi could you please show how you've done it, as it is a json array nested in another one, I have no idea how to construct the serializable class, Here for example

    Code (CSharp):
    1.  
    2. //Serializable Class
    3. using UnityEngine;
    4. using System.Collections;
    5. using System.Collections.Generic;
    6. using UnityEngine;
    7. using System;
    8. [Serializable]
    9. public class NewBehaviourScript<T>
    10. {
    11.     public T[] Items;
    12.     return Items;
    13. }
    14. //Code
    15.  
    16. using UnityEngine;
    17. using System;
    18. using UnityEditor;
    19. public class Test : MonoBehaviour
    20. {
    21.     static string str = "[[765,29,326],[735,29,-59],[778,98,-637]]";
    22.     static float[] a = JsonUtility.FromJson<NewBehaviourScript<float>>(str);
    23. }
    24.  
    Im positive It'll not work but perhaps you could guide me in the right direction? I based the class code from this https://stackoverflow.com/questions/36239705/serialize-and-deserialize-json-and-json-array-in-unity
     
  10. Bunny83

    Bunny83

    Joined:
    Oct 18, 2010
    Posts:
    3,922
    Well, since you now changed the data to actually be json, you can actually load it as json. However Unity's JsonUtility is quite limited. It only supports a single root object and it has to be an object and can't be an array. Second it does not support directly nested arrays, only nested arrays insided objects. Those could be inside another array, but direct nesting is not supported. This is because the JsonUtility class has the same limitations as Unity's normal serialization system.

    So you have to use a different json library. Most people use the Newtonsoft Json.NET library. Though I have my own SimpleJSON parser and made a Unity extension file to add direct conversion support for Vector3.

    I made the example above about one month ago and I wasn't sure if I still had it somewhere in one of my test projects. Though I've found it. It uses my SimpleJSON library. It still had the string "Replace" calls inside to exchange the curly braces with square ones. Though I should still do the same thing

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using SimpleJSON;
    5. using UnityEngine.Networking;
    6.  
    7. public class PastebinPointCloud : MonoBehaviour
    8. {
    9.     string url = "........";
    10.  
    11.     IEnumerator LoadData()
    12.     {
    13.         var request = UnityWebRequest.Get(url);
    14.         yield return request.SendWebRequest();
    15.         if (request.result != UnityWebRequest.Result.Success)
    16.         {
    17.             Debug.LogWarning("Error loading data from " + url + " due to " + request.result+ " / " + request.error);
    18.             yield break;
    19.         }
    20.         string json = request.downloadHandler.text;
    21.         json = json.Replace('{', '[').Replace('}',']');
    22.         var data = JSON.Parse(json);
    23.         foreach (JSONNode n in data)
    24.         {
    25.             Vector3 v = n;
    26.             v /= 200;
    27.             Draw3dCross(v, 0.01f, 100);
    28.         }
    29.         Debug.Log("count: " + data.Count);
    30.     }
    31.  
    32.     static void Draw3dCross(Vector3 p, float s, float time)
    33.     {
    34.         Debug.DrawLine(p + Vector3.forward * s, p - Vector3.forward * s, Color.blue, time);
    35.         Debug.DrawLine(p + Vector3.right * s, p - Vector3.right * s, Color.red, time);
    36.         Debug.DrawLine(p + Vector3.up * s, p - Vector3.up * s, Color.green, time);
    37.     }
    38.  
    39.  
    40.     void Start()
    41.     {
    42.         StartCoroutine(LoadData());
    43.     }
    44. }
    45.  

    For some reason the forum does not allow me to include your pastebin URL in my script. So just imagine your raw pastebin link is inside the url string.
     
  11. Bunny83

    Bunny83

    Joined:
    Oct 18, 2010
    Posts:
    3,922
    I never had issues like that here on the forum. It seems to be a strange new link policy? Anyways, line 9 would look like this, but without the space after the double slash.

    Code (CSharp):
    1. string url = "https:// pastebin.com/raw/AbgwcrD9";
    It seems this is allowed, really strange.