Search Unity

Raw Data Export best practices

Discussion in 'Unity Analytics' started by tcoles, Jan 31, 2018.

  1. tcoles

    tcoles

    Joined:
    Jan 30, 2018
    Posts:
    10
    I'm curious about running regular Raw Data Exports to support custom back end processes.
    1. How frequently can we (or should we) run an export? Daily? Hourly? (Less delay is preferable, of course.)
    2. What is the typical processing time for a single export (assuming the recommended frequency from #1)?
    3. I noticed the exports are being served out of S3: is there any possibility of your systems delivering data directly to an S3 bucket we own?
    Thanks!
     
    tcoles, Jan 31, 2018
    #1

  2. tcoles

    tcoles

    Joined:
    Jan 30, 2018
    Posts:
    10
    Also I haven't been able to find any reference in the documentation to a data retention period. Is Analytics data stored indefinitely? If not, how long?
     
    tcoles, Feb 13, 2018
    #2

  3. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    1,519
    @tcoles

    I'm going to meet with the team responsible for Raw Data Export in a couple of weeks to get more concrete answers for some of your questions.

    In the meantime, I can share what I know.

    For #2, I think the processing time will likely depend on the amount of data that is returned. If you are looking for every appRunning event from the last 30 days, that will be a lot of data. But if you are only looking for a specific custom event on a specific day, it should a bit easier to process. I will confirm with the team though.

    And the amount of data processed ties in with question #1. I think as long as you are being conscientious of your queries, it should be fine. I know there is a third-party tool that I have often recommended, which will pull your Raw Data into a PostrgreSQL database. The developer gave a talk at Unite LA 2016.

    I will have to check with the team on this request.

    We retain the raw data files generated for a request for 14 days. However, you can always make a new request to get those files again. The data itself is currently stored indefinitely. We are still working on a formal data retention policy. I will update you when we have more information.
     
    Last edited: Feb 27, 2018
    ap-unity, Feb 16, 2018
    #3
    tcoles likes this.

  4. JeffDUnity3D

    JeffDUnity3D

    Joined:
    May 2, 2017
    Posts:
    14,446
    Perhaps a better approach than PostrgreSQL for local processing is to use a NoSQL/JSON database since we return the Raw Data Export in JSON format. There are many such db's on the market, including MongoDB and Couchbase, among others. Couchbase has a free community edition, and the solution presented here requires no scripting and no database schema and provides a rich SQL-syntax query language https://support.unity3d.com/hc/en-u...ries-with-Unity-Analytics-and-Raw-Data-Export
     
    JeffDUnity3D, Feb 16, 2018
    #4
    MetaDOS and tcoles like this.