Question Analytics Raw Data Export requests never completing

Over the last couple of years we have been trying to get the Analytics Raw Data Export API to work and we have repeatedly run in to issues with the service behaving irrationally and eventually becoming unresponsive, we have analytics data from a project since august 2017 we have never been able to export reliably, so we really want to get this working.

Here is an example of how we are creating a request and querying it's status to check if it completed:

Create export request

Response

Get request status

Response

Yesterday this worked perfectly with our .NET Core 6.0 based service, we could make the request and have it be fulfilled minutes later, this worked well. At some point around 12 hours ago the Raw Data Export API outright stopped completing our requests by never having them change status to "completed", they will always be reported as "running". Over 90 minutes later since making the request shown above, it has still not changed its status from "running".

Where this becomes extremely strange is that we have since then had an issue where the API every single minute will create three new requests that will end up completing in the time expected.


Now these requests do match what our own service would actually do in order to try to get the data we want from the API, but every single minute we can see that these requests are added to the API, yet these requests are being made without our own service actually running. We have tried completely shutting down every server running our analytics fetching code and even powered down development computers, the requests keeps being.

The Raw Data Export API severely needs to return the IP address that created the export request or allow us to regenerate a new secret key for the project to stop any lose service that might be spamming requests. We have no reason to believe the error is on our end as we have stopped every single possible computer that could be running our service and the Raw Export API itself has completely stopped completing our requests, this is an issue we have seen over and over for the last few years when we have tried using it.

We have experienced this before, in December 2020 we had the exact same issues when trying to get this to work when we looked at it last.

Our current conclusions are that we are being affected by one of the following:

Backlogged requests
It's possible our service by mistake might have generated too many export requests to the API and all the requests being made every minute are those waiting to be fulfilled. We have no reason to believe this is the case as the documentation does not mention this, listing all requests through the API does not show a bunch of queued up requests and the creation timestamp for the automatically created requests does not seem to be from any longer than a minute or two ago from when listing all requests is being done.

Export Raw Data API is generally unreliable
We have experienced many times in the past that the API will not fulfill requests for extended periods of time, it doesn't explain why we see a bunch of requests automatically appearing, but the API not completing new requests is a big problem.

Our service is running somewhere and is DDOS'ing our access to the Export API
We have no way to check where all our requests are coming from and we have done a lot of searching for where it could have been running, we manually upload the exported .Net Core 6.0 executable to the AWS Lightsail instance meant to run it, we don't have any continuous-deployment or other automated processes set up that might run it by accident.


We would love to have someone from Unity Technologies to weigh in on what might be going on, we simply cannot extract our data and every attempt we have made over the years eventually ends with the Export API becoming unresponsive for long periods of time.

 

@GMM That's a long message to ask if RDE is having issues! Which it is at the moment today, we are checking. FYI we will be replacing RDE with a new service via Unity Analytics 2.0 which is in beta now. This updated service will be available hopefully later this Q1

 

@JeffDUnity3D Thank you for the quick response, when do you expect RDE to be going offline and do you have a migration guide somewhere from RDE to the new system? Do you have an ETA on when we should expect it to be operational again?

We already checked https://status.cloud.unity3d.com/ to see if there were any outages with your services, why does RDE related issues not show up on there if you are aware of it?

We expected the issue could be on our side due to the continuously generating requests since we had no indication that the service had issues, in general we have had a lot of issues with RDE being unreliable every time we have tried to develop tooling for extracting customer analytics data on your platform.

 

I will update here when the service is ready again. Although the status page in theory is a good idea, as I've seen with other companies, the engineers are focusing on fixing the actual issue instead of updating the status page. You can join the UA2 beta now, but the data connector won't be available for likely a few months. https://support.unity.com/hc/en-us/articles/4408915768724-General-FAQ

 

Thank you for keeping track of it, I get why your engineers prioritize fixing the problem so everyone can get back to using the service, it’s just frustrating for us to now know about outages since it means we have once more spent an entire day trying to figure why our RDE tooling isn’t returning any data.

Thank you for linking to the FAQ for the new Unity Analytics system, we will be keeping an eye on it going forward