Upgrading DataPrivacy: Unity 2017.4 to Unity 2018.4

In our game we have other analytics systems such as AppsFlyer and Facebook Analytics. In Unity 2017.4, we added code to listen for Unity DataPrivacy status changes, for example when the user opted-out from the user's personal URL on Unity's website. When we detected the user opted out in Unity Analytics, we also disable data collection on our other systems.

However, now in Unity 2018, the DataPrivacy system has changed to where there's no longer a web query for the user's opt-out status in the DataPrivacy package code. After the user returns to our app from visiting the personal privacy URL, this code to force RemoteSettings to update in DataPrivacyButton kicks in:

Code (CSharp):

1.         void OnApplicationFocus(bool hasFocus)
2.         {
3.             if (hasFocus && urlOpened)
4.             {
5.                 urlOpened = false;
6.                 // Immediately refresh the remote config so new privacy settings can be enabled
7.                 // as soon as possible if they have changed.
8.                 RemoteSettings.ForceUpdate();
9.             }
10.         }
11.  

My question is, what's the best way for us to detect changes in privacy? Should we listen for the RemoteSettings.Updated event and then check the UnityAnalytics.limitUserTracking and UnityAnalytics.playerOptedOut or perhaps it's best to perform this check every frame in an Update() method?

 

What are you checking in the RemoteSettings update callback? Why do you believe RemoteSettings to be associated with the DataPrivacy settings? What web query were you using previously that is no longer working? Is your approach suggested above working for you? I would not recommend to do the check in Update, that can happen more than 60 times a second. But I don't believe you need to use RemoteSettings (which is being deprecated in favor of Remote Config https://unity.com/remote-config )

 

The code I posted above is directly from Unity's own DataPrivacyButton class, which is included in the latest Analytics Library package version 3.2.3. I'm aware that RemoteSettings will eventually be deprecated, but it clearly updates RemoteSettings to get the latest privacy settings (see the comment in the above Unity-owned package source code).

In the Asset Store version of DataPrivacy for Unity 2017.4, the DataPrivacy system queries (via WWW API) the user's opt-out status settings from Unity's servers. I assume from the above code that this is now happening via RemoteSettings in Unity 2018.3? Probably somewhere internally in the Unity Analytics library and no longer by the DataPrivacy code. The Analytics API added playerOptOutStatus since Unity 2017.4.

 

What is the code that you were previously using that you claim is no longer working? Does it return an error?

 

Previously we had modified the DataPrivacy code (Unity's Asset Store package code) so that whenever it received new opt-out status (via its own web request), we would also enable or disable data collection in our other analytics systems. We added a simple AddListener system to the DataPrivacy code to achieve this. In the newest 2018.4 DataPrivacy package (Analytics Library v3.2.3), the code to web-request opt-out status has been removed. My guess is that it was moved into the UnityEngine.Analytics library and now uses RemoteSettings. We use RemoteSettings for other things and there are no exposed properties for Unity Analytics that we can see, but that doesn't mean there aren't hidden properties for Unity-type stuff. I guess only @JeffDUnity3D can answer that.

Thanks!!

 

@tessellation Your approach looks correct, but as mentioned, don't check in Update

 

Sure, I suppose we could limit the check to once a second or something using a Coroutine. Is Analytics.playerOptedOut and Analytics.enabled known to be an expensive check?

 

Why don't you just check right after ForceUpdate? Or set a single shot co-routine after a second or two. Anything in Update is potentially expensive!

 

It's a good idea, however I'm inclined to check more regularly since even at app startup, Analytics will probably request updated settings asynchronously and I don't have any way of knowing when the opt-out settings might change. Analytics API doesn't provide a mechanism for listening to these changes.

 

They will most likely change during the opt out process, I would think. When would they be changing otherwise?

 

Like I said above, it could change at startup after the analytics remote settings are retrieved for the user's opt-out status.

 

How could it change? Only the user would change it. You might check once at startup, and once again when they attempt to opt-out. So I see only two times you might need to check it.

 

Correct, I could wait some arbitrary number of seconds before checking at startup and then never check again until the user returns from the game after the data privacy, at which point I would wait another arbitrary number of seconds, potentially missing the change due to internet lag. Seems easier just to check infrequently, all the time. Plus then we're future proof because it keeps the Analytics state in sync with our other systems.

 

Sorry, I don't quite follow. There are no arbitrary seconds after start up, what are you waiting on? I would not recommend polling, but it's up to you.

 

What I mean by arbitrary is that you can guess at how long it will take the user's device to contact Unity's server to get a response, but it's generally not predictable due to internet lag. Because there's no measurable impact from checking analytics state on Android devices (even every frame at 60fps), we're going to keep doing live checks every 1 second for the reasons I described above.

I've run into a related problem while in the process of testing on Android devices: Building with the Unity 2018.4.16f1 Analytics/DataPrivacy packages, after opting-out via the user's custom web site, Analytics never syncs up with the user's website opt-out state. Even after restarting the app, the state is incorrect:

Unity Analytics at Awake: enabled: True, playerOptedOut: False, limitUserTracking: False

Please advise if I'm doing something wrong or if I should post this as a Unity bug. As it is right now it breaks our GDPR/CCPA compliance.

Again Analytics Library (DataPrivacy) is v3.2.3

 

Yes your solution would work also. But there isn't a need to check every second if they haven't opted out yet in the game. Check at the beginning to see if they opted out in their LAST session, and wait for the opt-out in the current session and check (or start to check) only then. Granted not a callback, which would be ideal, but giving it two seconds (for example) would work correctly for 99.9% of all users (which you could confirm with Charles Proxy in your testing). Even those with the lag that you refer to would be identified on the next session during your start up check. Regarding your last concern, is this on an Android device? And the other Analytics packages that you refer to would be required to have their own opt-out process. Our process would only be supported for Unity components. But you have a point regarding the LimitUserTracking, it would affect ours if so, and would be considered an issue.

 

Yes, this is on Android and only affects Unity Analytics. The opt-out process isn't working in our testing because the client (user's Android device) itself doesn't know to stop sending data, regardless of whether the server stops storing the data or not.

 

Understood, I am checking.

 

Any updates? Should I submit this as a bug?

 

I will be looking at this today.

 

@tessellation I just tested this with the same versions that you are using, and I also see limitUserTracking = false when checking a few seconds after opting out and returning to the game. I will test again in 24 hours to see if there may be processing time involved, and will keep this thread updated.

 

Thanks for looking into it. With the old Asset Store version of DataPrivacy in Unity 2017.4, the opt-out would be queried immediately by the client via UnityWebRequest and OptOutStatus.optOut would be true as well as Analytics disabled and limitUserTracking = true. In theory we could disable the Analytics Library 3.2.3 package and use the DataPrivacy code from the Asset Store with Unity 2018.4, the only problem is that standard events are now moved into the same package as DataPrivacy and aren't part of the base Unity Analytics package.

 

Why would you need to disable the Analytics Library 3.2.3 in your scenario? Granted there looks to be a possible issue, I'm still seeing Analytics.limitUserTracking as false several hours later. I have let the engineering team know.

 

Because DataPrivacy class would be duplicated in the Analytics namespace (once in DataPrivacy Asset Store unitypackage and another in the Analytics Library 3.2.3 UPM package). To revert back to the Asset Store DataPrivacy system, you'd need to remove the UPM package, which would also remove Analytics Standard Events. I suppose I could hack it and extract those DLLs from the UPM package, but it's not a good long term solution.

I'm so glad you were able to reproduce it. I look forward to a quick fix.

 

We are hoping to address this within the next couple of days

 

Fantastic, thanks!

 

This has been addressed, please try again. After the user opts out, the Analytics settings are also updated (Analytics.playerOptedOut and Analytics.limitUserTracking are true)

 

I should add, it can take some time for the settings to show, up to 24 hours. In your case, you will likely want to check once, at the very start of the game, to see if they opted out in their last session. We are working to lessen the time frame to almost immediate.

 

I'm glad the problem has been fixed. I will test this shortly.

Thanks for the detailed information. Yes, I believe users will expect their data to not be sent by the client immediately after opting-out. I've had users complain about network connections to analytics servers, I think they used a packet sniffer to monitor the app's behavior.

 

It's batch processing time, not network delay

 

Yes and I'm glad Unity is working on having the server respond immediately to the app (client) with the current opt-out status instead of waiting to process (delete?) the user's existing data or whatever behavior it's doing now.