Raw Data Export Example Script

Hi,

I have taken the time to get my game uploading all kinds of wonderful data and so far so good.

Now i would like to export that data and get it displaying so i can start drawing my own comparisons. The Unity Manual on Raw data exports seems to assume a lot of prior knowledge that im simply not understanding where to begin.

Im simply looking to
1. Export data (Using The REST API so i can perform the task from code).
2. Assign a field from that data to a variable.

Does anyone have an example of this they would be willing to share?

Thank you in advance.

 

Greetings Pixtopher,

My name is Skeledurr.

I have attached an example script that you can study to learn how to download export data from Unity. Please keep in mind the script requires a Json library and Unity.IO.Compress. Links below.


Unfortunately for this example I used JsonDotNet ($25 purchase from the asset store):
https://www.assetstore.unity3d.com/en/#!/content/11347

But I'm sure you can find any other free examples and use it instead.

Simple Json is a great alternative.
http://wiki.unity3d.com/index.php/SimpleJSON

Unity.IO.Compress. (This is a free download)
https://www.assetstore.unity3d.com/en/#!/content/31902


I have attached the example script to this post. Please let me know if you have any questions.

Many kind regards,
Skeledurr

p.s Can't wait for Dungeon League to be released.

Code (CSharp):

1.  
2. using Newtonsoft.Json;
3. using System;
4. using System.Collections;
5. using System.Collections.Generic;
6. using System.IO;
7. using Unity.IO.Compression;
8. using UnityEngine;
9.  
10.  
11. public class RawDataGetExample : MonoBehaviour
12. {
13.     private const string PROJECT_ID = ""; // Find this in your Analytics dash board / Configure / Project ID
14.     private const string API_KEY = ""; // Find this in your Analytics dash board / Configure / Raw Data Export API Key
15.     private const string BASE_URL = "https://analytics.cloud.unity3d.com";
16.  
17.  
18.     public List<string> curRequestIds = new List<string>();
19.     private float timeSinceLastCheck = 0;
20.     private float checkRate = 5f;
21.  
22.  
23.  
24.  
25.     private void Start()
26.     {
27.         CreateExport("2017-01-01", "2017-01-05", "appStart");
28.     }
29.  
30.     private void Update()
31.     {
32.         if(Time.time - timeSinceLastCheck > checkRate)
33.         {
34.             timeSinceLastCheck = Time.time;
35.             CheckRequests();
36.         }
37.     }
38.  
39.  
40.  
41.     // ************************
42.     //      CREATE EXPORT
43.     // ************************
44.  
45.     // Tell Unity to create us a file with all the data we want on the server.
46.     private void CreateExport(string startDate, string endDate, string dataSet)
47.     {
48.         Debug.Log("Start create export request.");
49.         string postUrl = BASE_URL + "/api/v2/projects/" + PROJECT_ID + "/rawdataexports";
50.        
51.         Dictionary<string, string> headers = new Dictionary<string, string>();
52.         headers.Add("Authorization", "Basic " +
53.             System.Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(PROJECT_ID+":"+ API_KEY)));
54.  
55.         headers.Add("Method", "POST");
56.         headers.Add("Content-Type", "application/json");
57.  
58.  
59.         Dictionary<string, object> dict = new Dictionary<string, object>();
60.         dict.Add("startDate", startDate);
61.         dict.Add("endDate", endDate);
62.         dict.Add("format", "json");
63.         dict.Add("dataset", dataSet);
64.         string data = JsonConvert.SerializeObject(dict);
65.         Debug.Log(data);
66.  
67.  
68.         WWW request = new WWW(postUrl, System.Text.Encoding.ASCII.GetBytes(data), headers);
69.        
70.         StartCoroutine(WaitForRequest(request));
71.     }
72.  
73.  
74.  
75.     // ************************
76.     //       PROCESS POST
77.     // ************************
78.  
79.     // When our request is complete process it.
80.     private void ProcessRequest(ReceivedData receivedData)
81.     {
82.         // If the file generation has completed on the server. Download it!
83.         if (receivedData.status == "completed")
84.         {
85.             if (curRequestIds.Contains(receivedData.id)) curRequestIds.Remove(receivedData.id);
86.  
87.             Download(receivedData);
88.         }
89.         // If the file generation is still running. Add the id to our request ids.
90.         // We will check on it again later.
91.         else if (!curRequestIds.Contains(receivedData.id))
92.         {
93.             curRequestIds.Add(receivedData.id);
94.         }
95.     }
96.  
97.  
98.  
99.     // ************************
100.     //      CHECK REQUESTS
101.     // ************************
102.  
103.     // Check current requests.
104.     private void CheckRequests()
105.     {
106.         for(int i = 0; i < curRequestIds.Count; i++)
107.         {
108.             GetExportStatus(curRequestIds[i]);
109.         }
110.     }
111.  
112.  
113.  
114.     // ************************
115.     //           GET
116.     // ************************
117.  
118.     // Check the status of a current export request.
119.     private void GetExportStatus(string requestId)
120.     {
121.         Debug.Log("Check status of request - id: " + requestId);
122.  
123.         string getUrl = BASE_URL + "/api/v2/projects/" + PROJECT_ID + "/rawdataexports/"+ requestId;
124.  
125.         Dictionary<string, string> headers = new Dictionary<string, string>();
126.         headers.Add("Authorization", "Basic " +
127.             System.Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(PROJECT_ID + ":" + API_KEY)));
128.  
129.  
130.         WWW request = new WWW(getUrl, null, headers);
131.  
132.         StartCoroutine(WaitForRequest(request));
133.     }
134.  
135.  
136.  
137.     // ************************
138.     //      REQUEST PROCESS
139.     // ************************
140.  
141.     IEnumerator WaitForRequest(WWW www)
142.     {
143.         yield return www;
144.  
145.         // check for errors
146.         if (www.error == null)
147.         {
148.             Debug.Log("WWW Ok!: " + www.data);
149.  
150.             ReceivedData receivedData = JsonConvert.DeserializeObject<ReceivedData>(www.data);
151.             ProcessRequest(receivedData);
152.         }
153.         else
154.         {
155.             Debug.Log("WWW Error: " + www.error);
156.         }
157.     }
158.  
159.  
160.     // ************************
161.     //      DOWNLOAD FILE
162.     // ************************
163.  
164.     // Download file from Unity.
165.     private void Download(ReceivedData receivedData)
166.     {
167.         Debug.Log("Download file - id: "+ receivedData.id+", fileList.Length: "+ receivedData.result.fileList.Length);
168.  
169.         StartCoroutine(DownloadProcess(receivedData.id, receivedData.result.fileList[0].url));
170.     }
171.    
172.     IEnumerator DownloadFileList(ReceivedData receivedData)
173.     {
174.         Debug.Log("Starting to download file list - file count: " + receivedData.result.fileList.Length);
175.  
176.         for(int i = 0; i < receivedData.result.fileList.Length; i++)
177.         {
178.             yield return StartCoroutine(DownloadProcess(receivedData.id, receivedData.result.fileList[i].url));
179.         }
180.  
181.         Debug.Log("Download file list complete.");
182.     }
183.  
184.     IEnumerator DownloadProcess(string dataId, string downloadUrl)
185.     {
186.         Debug.Log("Starting download - id: " + dataId);
187.  
188.         WWW www = new WWW(downloadUrl);
189.  
190.         while(!www.isDone)
191.         {
192.             yield return null;
193.         }
194.  
195.         Debug.Log("Download complete - id: " + dataId);
196.  
197.         string fullPath = Application.dataPath + "/"+ dataId+".gzip";
198.         File.WriteAllBytes(fullPath, www.bytes);
199.  
200.         Debug.Log("Decompressing file");
201.  
202.         byte[] file = File.ReadAllBytes(fullPath);
203.         byte[] decompress = Decompress(file);
204.         string text = System.Text.ASCIIEncoding.ASCII.GetString(decompress);
205.         HandleFileContents(text);
206.     }
207.  
208.  
209.     // ************************
210.     //      DECOMPRESS FILE
211.     // ************************
212.    
213.     static byte[] Decompress(byte[] gzip)
214.     {
215.         // Create a GZIP stream with decompression mode.
216.         // ... Then create a buffer and write into while reading from the GZIP stream.
217.         using (GZipStream stream = new GZipStream(new MemoryStream(gzip),
218.             CompressionMode.Decompress))
219.         {
220.             const int size = 4096;
221.             byte[] buffer = new byte[size];
222.             using (MemoryStream memory = new MemoryStream())
223.             {
224.                 int count = 0;
225.                 do
226.                 {
227.                     count = stream.Read(buffer, 0, size);
228.                     if (count > 0)
229.                     {
230.                         memory.Write(buffer, 0, count);
231.                     }
232.                 }
233.                 while (count > 0);
234.                 return memory.ToArray();
235.             }
236.         }
237.     }
238.  
239.  
240.     // ************************
241.     //   HANDLE FILE CONTENTS
242.     // ************************
243.  
244.     // Handle what ever you want to do with the data here.
245.     private void HandleFileContents(string jsonFromFile)
246.     {
247.         Debug.Log("file contents: " + jsonFromFile);
248.     }
249.  
250.  
251.  
252.     // *******************
253.     //      UTILITY
254.     // *******************
255.  
256.     // Lazy class for deserializing the json from the server.
257.     // Update the properties if you want to get different data.
258.     public class ReceivedData
259.     {
260.         public string id;
261.         public string status;
262.  
263.         public Results result;
264.  
265.         public class Results
266.         {
267.             public Files[] fileList;
268.         }
269.  
270.         public class Files
271.         {
272.             public string name;
273.             public string url;
274.         }
275.     }
276. }
277.  

 

Thank you so much Skeledurr!

Your response should totally get pinned as an example for future noobies.

 

Brilliant - thank you so much. Works like a charm so far.

 

You can also download the Raw Data Export files directly from the Analytics Dashboard. RDE does require a Pro license.

 

Hi @Skeledurr,

thanks for your script. It works in the Editor and if I create an .exe application.
I would like to get it running as WebGL version, so I can trigger it from anywhere. I get
"WWW Error: Unknown error"
when running the app on my server. It seems to be CORS related. I read a lot about it, e.g.

So the question is if Unity (https://analytics.cloud.unity3d.com) supports CORS, right? Anything else I could test or do?
Thanks

UPDATE: Tried again in browser and this is what Chrome reports:

For me this looks like there's no CORS support on the unity cloud subdomain. I'll create a separate thread and ask for it.

 

Instead of your approach, I might instead suggest a server-side scripting/web service api solution that does not require CORS. You could access the service from anywhere as a web service, and would not require CORS.

 

Hi,
I am using this script for post request but every time i got the Bad Request Error 400.
I want my Analytics data within unity.
any solutions.!!

 

It would be difficult to say what is wrong without knowing what the request looks like. I trust you have it working using curl as mentioned in the documentation? You don't want to call the API from within a game.

 

Question: how do you decipher the submit time string?

 

https://www.epochconverter.com/

 

is "ts" also a date?

Sorry if these are covered in an FAQ. If there is a page I can read please let me know.

 

Yes, ts = timestamp . This may help https://docs.unity3d.com/Manual/UnityAnalyticsRawDataExport.html

 

Do events get pushed in chunks? I am seeing submit times that are all the same.

 

Yes, they get sent in batches

 

ok great thanks for the info!