Search Unity

  1. Improved Prefab workflow (includes Nested Prefabs!), 2D isometric Tilemap and more! Get the 2018.3 Beta now.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. Let us know a bit about your interests, and if you'd like to become more directly involved. Take our survey!
    Dismiss Notice
  4. Don't miss out on the Unite LA Keynote for the latest information from Unity! Set a reminder!
    Dismiss Notice
  5. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  6. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Raw Data Export Example Script

Discussion in 'Unity Analytics' started by Pixtopher, Mar 14, 2017.

  1. Pixtopher

    Pixtopher

    Joined:
    Jan 16, 2014
    Posts:
    14
    Hi,

    I have taken the time to get my game uploading all kinds of wonderful data and so far so good.

    Now i would like to export that data and get it displaying so i can start drawing my own comparisons. The Unity Manual on Raw data exports seems to assume a lot of prior knowledge that im simply not understanding where to begin.

    Im simply looking to
    1. Export data (Using The REST API so i can perform the task from code).
    2. Assign a field from that data to a variable.

    Does anyone have an example of this they would be willing to share?

    Thank you in advance.
     
  2. Skeledurr

    Skeledurr

    Joined:
    Jan 5, 2015
    Posts:
    23
    Greetings Pixtopher,

    My name is Skeledurr.

    I have attached an example script that you can study to learn how to download export data from Unity. Please keep in mind the script requires a Json library and Unity.IO.Compress. Links below.


    Unfortunately for this example I used JsonDotNet ($25 purchase from the asset store):
    https://www.assetstore.unity3d.com/en/#!/content/11347

    But I'm sure you can find any other free examples and use it instead.

    Simple Json is a great alternative.
    http://wiki.unity3d.com/index.php/SimpleJSON

    Unity.IO.Compress. (This is a free download)
    https://www.assetstore.unity3d.com/en/#!/content/31902


    I have attached the example script to this post. Please let me know if you have any questions.

    Many kind regards,
    Skeledurr

    p.s Can't wait for Dungeon League to be released.


    Code (CSharp):
    1.  
    2. using Newtonsoft.Json;
    3. using System;
    4. using System.Collections;
    5. using System.Collections.Generic;
    6. using System.IO;
    7. using Unity.IO.Compression;
    8. using UnityEngine;
    9.  
    10.  
    11. public class RawDataGetExample : MonoBehaviour
    12. {
    13.     private const string PROJECT_ID = ""; // Find this in your Analytics dash board / Configure / Project ID
    14.     private const string API_KEY = ""; // Find this in your Analytics dash board / Configure / Raw Data Export API Key
    15.     private const string BASE_URL = "https://analytics.cloud.unity3d.com";
    16.  
    17.  
    18.     public List<string> curRequestIds = new List<string>();
    19.     private float timeSinceLastCheck = 0;
    20.     private float checkRate = 5f;
    21.  
    22.  
    23.  
    24.  
    25.     private void Start()
    26.     {
    27.         CreateExport("2017-01-01", "2017-01-05", "appStart");
    28.     }
    29.  
    30.     private void Update()
    31.     {
    32.         if(Time.time - timeSinceLastCheck > checkRate)
    33.         {
    34.             timeSinceLastCheck = Time.time;
    35.             CheckRequests();
    36.         }
    37.     }
    38.  
    39.  
    40.  
    41.     // ************************
    42.     //      CREATE EXPORT
    43.     // ************************
    44.  
    45.     // Tell Unity to create us a file with all the data we want on the server.
    46.     private void CreateExport(string startDate, string endDate, string dataSet)
    47.     {
    48.         Debug.Log("Start create export request.");
    49.         string postUrl = BASE_URL + "/api/v2/projects/" + PROJECT_ID + "/rawdataexports";
    50.        
    51.         Dictionary<string, string> headers = new Dictionary<string, string>();
    52.         headers.Add("Authorization", "Basic " +
    53.             System.Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(PROJECT_ID+":"+ API_KEY)));
    54.  
    55.         headers.Add("Method", "POST");
    56.         headers.Add("Content-Type", "application/json");
    57.  
    58.  
    59.         Dictionary<string, object> dict = new Dictionary<string, object>();
    60.         dict.Add("startDate", startDate);
    61.         dict.Add("endDate", endDate);
    62.         dict.Add("format", "json");
    63.         dict.Add("dataset", dataSet);
    64.         string data = JsonConvert.SerializeObject(dict);
    65.         Debug.Log(data);
    66.  
    67.  
    68.         WWW request = new WWW(postUrl, System.Text.Encoding.ASCII.GetBytes(data), headers);
    69.        
    70.         StartCoroutine(WaitForRequest(request));
    71.     }
    72.  
    73.  
    74.  
    75.     // ************************
    76.     //       PROCESS POST
    77.     // ************************
    78.  
    79.     // When our request is complete process it.
    80.     private void ProcessRequest(ReceivedData receivedData)
    81.     {
    82.         // If the file generation has completed on the server. Download it!
    83.         if (receivedData.status == "completed")
    84.         {
    85.             if (curRequestIds.Contains(receivedData.id)) curRequestIds.Remove(receivedData.id);
    86.  
    87.             Download(receivedData);
    88.         }
    89.         // If the file generation is still running. Add the id to our request ids.
    90.         // We will check on it again later.
    91.         else if (!curRequestIds.Contains(receivedData.id))
    92.         {
    93.             curRequestIds.Add(receivedData.id);
    94.         }
    95.     }
    96.  
    97.  
    98.  
    99.     // ************************
    100.     //      CHECK REQUESTS
    101.     // ************************
    102.  
    103.     // Check current requests.
    104.     private void CheckRequests()
    105.     {
    106.         for(int i = 0; i < curRequestIds.Count; i++)
    107.         {
    108.             GetExportStatus(curRequestIds[i]);
    109.         }
    110.     }
    111.  
    112.  
    113.  
    114.     // ************************
    115.     //           GET
    116.     // ************************
    117.  
    118.     // Check the status of a current export request.
    119.     private void GetExportStatus(string requestId)
    120.     {
    121.         Debug.Log("Check status of request - id: " + requestId);
    122.  
    123.         string getUrl = BASE_URL + "/api/v2/projects/" + PROJECT_ID + "/rawdataexports/"+ requestId;
    124.  
    125.         Dictionary<string, string> headers = new Dictionary<string, string>();
    126.         headers.Add("Authorization", "Basic " +
    127.             System.Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(PROJECT_ID + ":" + API_KEY)));
    128.  
    129.  
    130.         WWW request = new WWW(getUrl, null, headers);
    131.  
    132.         StartCoroutine(WaitForRequest(request));
    133.     }
    134.  
    135.  
    136.  
    137.     // ************************
    138.     //      REQUEST PROCESS
    139.     // ************************
    140.  
    141.     IEnumerator WaitForRequest(WWW www)
    142.     {
    143.         yield return www;
    144.  
    145.         // check for errors
    146.         if (www.error == null)
    147.         {
    148.             Debug.Log("WWW Ok!: " + www.data);
    149.  
    150.             ReceivedData receivedData = JsonConvert.DeserializeObject<ReceivedData>(www.data);
    151.             ProcessRequest(receivedData);
    152.         }
    153.         else
    154.         {
    155.             Debug.Log("WWW Error: " + www.error);
    156.         }
    157.     }
    158.  
    159.  
    160.     // ************************
    161.     //      DOWNLOAD FILE
    162.     // ************************
    163.  
    164.     // Download file from Unity.
    165.     private void Download(ReceivedData receivedData)
    166.     {
    167.         Debug.Log("Download file - id: "+ receivedData.id+", fileList.Length: "+ receivedData.result.fileList.Length);
    168.  
    169.         StartCoroutine(DownloadProcess(receivedData.id, receivedData.result.fileList[0].url));
    170.     }
    171.    
    172.     IEnumerator DownloadFileList(ReceivedData receivedData)
    173.     {
    174.         Debug.Log("Starting to download file list - file count: " + receivedData.result.fileList.Length);
    175.  
    176.         for(int i = 0; i < receivedData.result.fileList.Length; i++)
    177.         {
    178.             yield return StartCoroutine(DownloadProcess(receivedData.id, receivedData.result.fileList[i].url));
    179.         }
    180.  
    181.         Debug.Log("Download file list complete.");
    182.     }
    183.  
    184.     IEnumerator DownloadProcess(string dataId, string downloadUrl)
    185.     {
    186.         Debug.Log("Starting download - id: " + dataId);
    187.  
    188.         WWW www = new WWW(downloadUrl);
    189.  
    190.         while(!www.isDone)
    191.         {
    192.             yield return null;
    193.         }
    194.  
    195.         Debug.Log("Download complete - id: " + dataId);
    196.  
    197.         string fullPath = Application.dataPath + "/"+ dataId+".gzip";
    198.         File.WriteAllBytes(fullPath, www.bytes);
    199.  
    200.         Debug.Log("Decompressing file");
    201.  
    202.         byte[] file = File.ReadAllBytes(fullPath);
    203.         byte[] decompress = Decompress(file);
    204.         string text = System.Text.ASCIIEncoding.ASCII.GetString(decompress);
    205.         HandleFileContents(text);
    206.     }
    207.  
    208.  
    209.     // ************************
    210.     //      DECOMPRESS FILE
    211.     // ************************
    212.    
    213.     static byte[] Decompress(byte[] gzip)
    214.     {
    215.         // Create a GZIP stream with decompression mode.
    216.         // ... Then create a buffer and write into while reading from the GZIP stream.
    217.         using (GZipStream stream = new GZipStream(new MemoryStream(gzip),
    218.             CompressionMode.Decompress))
    219.         {
    220.             const int size = 4096;
    221.             byte[] buffer = new byte[size];
    222.             using (MemoryStream memory = new MemoryStream())
    223.             {
    224.                 int count = 0;
    225.                 do
    226.                 {
    227.                     count = stream.Read(buffer, 0, size);
    228.                     if (count > 0)
    229.                     {
    230.                         memory.Write(buffer, 0, count);
    231.                     }
    232.                 }
    233.                 while (count > 0);
    234.                 return memory.ToArray();
    235.             }
    236.         }
    237.     }
    238.  
    239.  
    240.     // ************************
    241.     //   HANDLE FILE CONTENTS
    242.     // ************************
    243.  
    244.     // Handle what ever you want to do with the data here.
    245.     private void HandleFileContents(string jsonFromFile)
    246.     {
    247.         Debug.Log("file contents: " + jsonFromFile);
    248.     }
    249.  
    250.  
    251.  
    252.     // *******************
    253.     //      UTILITY
    254.     // *******************
    255.  
    256.     // Lazy class for deserializing the json from the server.
    257.     // Update the properties if you want to get different data.
    258.     public class ReceivedData
    259.     {
    260.         public string id;
    261.         public string status;
    262.  
    263.         public Results result;
    264.  
    265.         public class Results
    266.         {
    267.             public Files[] fileList;
    268.         }
    269.  
    270.         public class Files
    271.         {
    272.             public string name;
    273.             public string url;
    274.         }
    275.     }
    276. }
    277.  
     

    Attached Files:

    Railon23, dishmop, cebollinos and 4 others like this.
  3. Pixtopher

    Pixtopher

    Joined:
    Jan 16, 2014
    Posts:
    14
    Thank you so much Skeledurr!

    Your response should totally get pinned as an example for future noobies.
     
  4. dishmop

    dishmop

    Joined:
    Sep 5, 2014
    Posts:
    4
    Brilliant - thank you so much. Works like a charm so far.
     
  5. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    2,421
    You can also download the Raw Data Export files directly from the Analytics Dashboard. RDE does require a Pro license.
     
  6. Railon23

    Railon23

    Joined:
    Jan 30, 2014
    Posts:
    21
    Hi @Skeledurr,

    thanks for your script. It works in the Editor and if I create an .exe application.
    I would like to get it running as WebGL version, so I can trigger it from anywhere. I get
    "WWW Error: Unknown error"
    when running the app on my server. It seems to be CORS related. I read a lot about it, e.g.

    So the question is if Unity (https://analytics.cloud.unity3d.com) supports CORS, right? Anything else I could test or do?
    Thanks

    UPDATE: Tried again in browser and this is what Chrome reports:
    For me this looks like there's no CORS support on the unity cloud subdomain. I'll create a separate thread and ask for it.
     
    Last edited: Jul 17, 2018