Search Unity

  1. We want you to join us at GDC this year! Take a peek at all of the events we will be hosting during the week of GDC.
    Dismiss Notice
  2. Tell us about your experience here and you’ll get early access to the 2018 Game Studios report + more goodies.
    Dismiss Notice
  3. Unity 2017.3 has arrived! Read about it here.
    Dismiss Notice
  4. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice
  5. We've closed the job boards. If you're looking for work, or looking to hire check out Unity Connect. You can see more information here.
    Dismiss Notice

Raw Data Export best practices

Discussion in 'Unity Analytics' started by tcoles, Jan 31, 2018.

  1. tcoles

    tcoles

    Joined:
    Jan 30, 2018
    Posts:
    2
    I'm curious about running regular Raw Data Exports to support custom back end processes.
    1. How frequently can we (or should we) run an export? Daily? Hourly? (Less delay is preferable, of course.)
    2. What is the typical processing time for a single export (assuming the recommended frequency from #1)?
    3. I noticed the exports are being served out of S3: is there any possibility of your systems delivering data directly to an S3 bucket we own?
    Thanks!
     
  2. tcoles

    tcoles

    Joined:
    Jan 30, 2018
    Posts:
    2
    Also I haven't been able to find any reference in the documentation to a data retention period. Is Analytics data stored indefinitely? If not, how long?
     
  3. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    808
    @tcoles

    I'm going to meet with the team responsible for Raw Data Export in a couple of weeks to get more concrete answers for some of your questions.

    In the meantime, I can share what I know.

    For #2, I think the processing time will likely depend on the amount of data that is returned. If you are looking for every appRunning event from the last 30 days, that will be a lot of data. But if you are only looking for a specific custom event on a specific day, it should a bit easier to process. I will confirm with the team though.

    And the amount of data processed ties in with question #1. I think as long as you are being conscientious of your queries, it should be fine. I know there is a third-party tool that I have often recommended, which will pull your Raw Data into a PostrgreSQL database. The developer gave a talk at Unite LA 2016.

    I will have to check with the team on this request.

    We retain the raw data files generated for a request for 14 days. However, you can always make a new request to get those files again. The data itself is currently stored indefinitely.
     
    tcoles likes this.
  4. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    983
    Perhaps a better approach than PostrgreSQL for local processing is to use a NoSQL/JSON database since we return the Raw Data Export in JSON format. There are many such db's on the market, including MongoDB and Couchbase, among others. Couchbase has a free community edition, and the solution presented here requires no scripting and no database schema and provides a rich SQL-syntax query language https://support.unity3d.com/hc/en-u...ries-with-Unity-Analytics-and-Raw-Data-Export
     
    tcoles likes this.