Processing Times Update

Hi all,

As many of you have noticed, our daily processing times has increased significantly in recent weeks and I wanted to give an explanation of why that is and what is in progress to improve it. To start, I would like to thank everyone for their patience, understanding and use of Unity Analytics. Making a product that provides useful insights to our developers is of the utmost importance to our team and we work every day towards that goal. To that end, we understand that the current processing times do not help with these goals and we are taking many steps to improve these times, which you can read about under Mitigation Steps.


Current State of Processing:

Right now, the dashboard is updated every 12 hours, and the time for processing to complete end-to-end is about 22 hours. This means that some new data will show up on the dashboard about every 12 hours, but a specific event that is sent (for example a brand new event) will take a full end-to-end cycle to appear. Additionally, our processing system works in batches, meaning we collect all the events received during the current processing cycle, and they all get processed the next time a cycle starts. If an event is sent at the beginning of the current processing cycle, for example, it would take two cycles to show up on the dashboard. Features like Livestream (Plus & Pro) and Raw Data Export (Pro) get their data before processing starts, and are therefore not affected by processing times.

This has been the way our processing system has worked the whole time, however the reason these times have increased so dramatically of late is due to the increasing amount of demand on our servers from all the lovely apps being created using Analytics, and challenges in scaling our system with the demand.


Mitigation Steps:

We understand the challenges this poses to our developers and know that many of you rely on quick turnover of your data to understand what is happening in, and to make informed decisions about, your game. I assure you we are working hard to fight this fire on multiple fronts.

- One way we are working to improve the current system is by being more intelligent about the way we manage our data. This includes upgrading hardware and refactoring code to reduce load on processing. For example we are currently looking into creating only the All Current Users segment for every project created, and adding our other 28 default segments once the app's dashboard has been accessed for the first time. Although many of these changes are small, each will allow a little more control and flexibility in the future and will also help us reduce our processing times.

- In addition to reducing processing times in our current system, we are also working on building out an entirely new data processing infrastructure. This new system will be faster and more scalable, allowing us to better serve the needs of our customers. This infrastructure will be rolled out in stages, and we will excitedly share many more details about it as soon as it's ready.

- Finally, we offer a Validator that shows all events sent from the editor and from development builds on devices in near-real time so that processing times of any length won't affect our users' ability to test whether events are being successfully sent. You can find this validator in the Integration tab of the dashboard, by selecting "Basic Integration for 5.2 Onwards" and then "Play to Validate" and scrolling down to the bottom of the page. The Validator works the same way sending events on devices do, meaning after the initial start, events are cached and flushed every 5 minutes, when a transaction event is sent, or when the app is restarted. We are also working on improvements to the Validator to make it easier to find and use.


Again, we are very grateful to our users and look forward to the time when we can announce reduced processing times!

 

Thanks for the update @erika_d. I'm gonna be 'that-guy' here:

When can we expect a return to something resembling a working system? How many days? Weeks?

It sounds like currently the processing backlog is bad (UA is ~3 days behind real-time) and getting worse. Can we expect it to keep getting worse for the immediate future?

 

Hi @Aurigan,

Unfortunately, I can't give a specific timeline of when these changes will roll out. The changes discussed in the first bullet are intended to be near-term changes to prevent things from getting worse, as you say. Because there are several small changes that we'll be implementing and rolling out over a period of time, its hard to predict specifics on when and the degree to which you'll see the effect. As far as the period of time over which they'll be rolling out, the best answer I can give is we want to roll them out as quickly as possible, but without forsaking proper reviewing and testing.

The changes in the second bullet are slightly longer term because they are more extensive, and therefore their testing must be even more rigorous.

As I'm sure you know, how long development and testing can take is unpredictable, and I don't want to lead you astray by hazarding a guess that turns out to be inaccurate. But know that making these changes are at the top of priority for us and we are working hard to get them to you as soon as we can.

 

fair enough, I'm reading this as 'will take at least several weeks to get to desired performance'.

Have/are you considering segmenting based on free/plus/pro users? Maybe you can get the people paying for Unity back to normality quicker by prioritizing processing their events.

 

I think building out the infrastructure functionality for that would probably take at least as long as the changes currently being made. But keep in mind Livestream (available to Plus and Pro subscribers and grandfathered Analytics projects) and Raw Data Export (available to Pro subscribers and grandfathered projects) are not affected by processing times.