Looking for some guidance

I am building a financial data visualization app and I'm struggling with my data design. There is a significant amount of data to be moved around and used but the structure is the same for every object. I have a version running but I think there has to be a better way. First off, here's my model for the incoming data (this model represents one day for one stock.

Code (CSharp):

1. public struct IrisOption
2.     {
3.         [Serializable]
4.         public struct ExpireDates
5.         {
6.             public string date;
7.             public int strikeMin;
8.             public int strikeMax;
9.         }
10.    
11.         public string ticker;
12.         public string tradeDate;
13.         public float stockPrice;
14.         public float circumference;
15.         public ExpireDates[] expireDates;
16.         public int[] strike;
17.         public float[] yte;
18.         public float[] cVolu;
19.         public float[] cOi;
20.         public float[] pVolu;
21.         public float[] pOi;
22.         public float[] cBidPx;
23.         public float[] cValue;
24.         public float[] cAskPx;
25.         public float[] pBidPx;
26.         public float[] pValue;
27.         public float[] pAskPx;
28.         public float[] cBidIv;
29.         public float[] cMidIv;
30.         public float[] cAskIv;
31.         public float[] smoothSmvVol;
32.         public float[] pBidIv;
33.         public float[] pMidIv;
34.         public float[] pAskIv;
35.         public float[] divRate;
36.         public float[] delta;
37.         public float[] gamma;
38.         public float[] theta;
39.         public float[] vega;
40.         public float[] rho;
41.         public float[] phi;
42.         public float[] extVol;
43.         public float[] extCTheo;
44.         public float[] extPTheo;
45.     }

Using Apple as an example, there are approximately 1000 elements in each float array for each day (30,000 total data points). Here's how I'm managing the data right now. I download the CSV from my provider, parse it and create the JSON files that matches the model above. I then brotli zip them up and upload it to my CDN. These files are about 12 MB uncompressed and 3 MB compressed. (files are broken up into quarters ~60 trading days of data)

In unity, I download the file, decompress it and then deserialize it. This process takes between 800 ms and 2 seconds depending on the transfer rate. I then loop through the data to create NativeArray<float>, then I loop through the NativeArrays to calculate an xy position and a z rotation.

I have this working right now but this is a lot of overhead to get the data into a job.

I would like to bypass most of these steps by loading the data into a blobArray and then use either memcpy or better yet use pointers.

My questions - can I use blob arrays to not only represent the data in Unity but to store on my CDN. (Azure has a blob storage and transfer solution baked into Unity already - would this be faster than compressing the blobs and storing them on my CDN?)

-Where do I start? I don't have any experience working with memory like this. I understand the concepts and what I'm trying to do but understanding the concepts is different than actually writing code.

-What data structure should I use? blobArray<blobArray>? MultiHashMap? I'm open to keeping the non float[] data in JSON and loading it separately in order to keep the data structure as simple as possible for the blobs.

-Each data point is going to be represented by an entity. Should I just create the entities and load the raw data right into IComopenentData and then loop through the entities to create the xy position and z rotation? (I'm currently trying to run jobs to calculate these variables before I create the entities).

The system I have right now is fast enough for end of day data and I could move forward; however, phase 2 will be to incorporate live data and I would much rather build it fast the first time through than to come back in 9 months and refactor everything.

Thank you for your help.

 

By what you are saying, this is your major bottleneck. Right?
I am not sure how much you could. Improve here. Looks like lots of data to deal with.
Maybe need smaller chunk of data, different compression methods. Trying decompress chunks of data in parallel. Possibly using system threads, if not jobs. Checkout also for generated GC.

 

That is a bottleneck for sure, not much can be done about the data transfer part, the decompression is necessary to reduce the data charges. I would like to remove the deserialization if possible which is what I'm getting at by using blobs.

If I saved the data as blobs, wouldn't I be able to just load the blob and then point to it? ie no need to deserialize and loop through it to add the data to the NativeArray.

 

Blobs won't help you unless you are also creating those Blobs in a Unity application. Otherwise you have to mimic their data structure which can be confusing and may break with safety checks Enabled/Disabled due to the DisposeSentinel.

Is there a reason you can't deserialize directly to a NativeArray? The Unity.IO namespace has some useful utilities for such things.

 

@DreamingImLatios - I like that idea but after looking at the docs for Unity.IO, this would be over my ability.