Bug Unity Analytics (New) in Offline Mode

Hello!

I've assigned this as a bug, but it could equally be argued as implementation.

So one of my test scenarios with the new Analytics was to run it in offline (airplane) mode.

When doing so I observed...

UnityServices.InitializeAsync() initialised after 1 second with a State of Initialised (which is great)

Events.CheckForRequiredConsents() failed after 10 seconds (fair enough, it needs a geo-lookup call)

However, when then calling Events.Flush() it throws the following exception (I'm assuming auto flush does the same)...

ConsentCheckException: The required consent flow cannot be determined. Make sure GeoIP was successfully called.

So this means all calls to Flush() need to be wrapped with exception handling, which is a pain. Could this not be built into the API please?

Or isn't there a case for supressing consent checks if required consents could not be determined? Or a fallback is device locale? Or a cached locale from a previous online session? Given it requires a UI and player input, anything other than on app init is not going to work well.

Basically, the demo code pattern showing await Events.CheckForRequiredConsents() causes knock on effects when in offline mode and I can see this catching a lot of people out.

 

@PeachPixels I just tested the following code with no Internet, and it displays the following. If this were the case, you would not want to send any events nor call Flush(). We may need to update the documentation.

Error reason = NoInternetConnection.

Code (CSharp):

1. async void Start()
2.     {
3.      
4.         string consentIdentifier = "";
5.         bool isOptInConsentRequired = false;
6.  
7.         try
8.         {
9.  
10.             var options = new InitializationOptions();
11.             options.SetAnalyticsUserId("my_test");
12.  
13.             await UnityServices.InitializeAsync();
14.             List<string> consentIdentifiers = await Events.CheckForRequiredConsents();
15.             Debug.Log("Count = " + consentIdentifiers.Count.ToString());
16.             if (consentIdentifiers.Count > 0)
17.             {
18.                 consentIdentifier = consentIdentifiers[0];
19.                 isOptInConsentRequired = consentIdentifier == "pipl";
20.             }
21.             if (isOptInConsentRequired)
22.             {
23.                 Events.ProvideOptInConsent(consentIdentifier, false);
24.             }
25.         }
26.         catch (ConsentCheckException e)
27.         {
28.             Debug.Log("Error reason = " + e.Reason.ToString());
29.         }
30.  
31.         Debug.Log("Done with Start!");
32.      }

 

In offline mode does the new Analytics not cache events locally? Am sure the documentation states it does.

UPDATE: From the docs...

There might be occasions where the SDK fails to send the Event, for example, when the Analytics service is down or the player is offline. The Events are stored in-memory (and to disc) until they can be uploaded.

If so, it's a fair assumption that if a developer can call Events.CustomData() in offline mode, the same should be for Events.Flush(). Even if it just silently exits and leaves the cached items for the next attempt.

Of course this can all be handled with a more complete pattern or handling in our own abstracted classes (in my case) but I can see a lot not doing this and getting caught out.

 

Good point! Again, we will follow up.

 

Hey Jeff!

With a fresher mind on this today, I've some more findings to report...

1.

UnityServices.InitializeAsync() does not need to be wrapped in the exception block. It will initialise successfully in offline mode and State will be set to Initialised

That's all good and for ease of use, the preferred approach imo.

2.

Events.CheckForRequiredConsents() does need to be wrapped in the exception block and will fail in offline mode (which it's clearly been designed to-do)

For consistency and ease of use, it might be better to follow the same approach as InitializeAsync. That is it should not fail (if offline) and its status determined by a property (much like State). Even better, why not wrap CheckForRequiredConsents() into InitializeAsync() and deal with it (in the app) via a callback?

In the event of a failed consent process then Events.CustomData() & Events.Flush() should still work (see point 4)

3.

As mentioned previously (and without knowing GDPR/CCPA/PIPL & App Store rules inside\out) having fallbacks for Events.CheckForRequiredConsents() offline mode is much needed. Even a fallback of 'assume no consent given' is preferable.

4.

I tested two Events.CheckForRequiredConsents() scenarios...

a) If the device was offline when this call is made, from that point on I could not see any events being cached to storage (Editor\Analytics\ArchivedEvents?) or sent if the device came back online. It seems that if this call fails, Analytics events will not be sent even with auto flush. The exception was that I did see a few standard events, but definitely no custom events.

b) If the device was online but the call was omitted (on game init) then events are not cached to storage (Editor\Analytics\ArchivedEvents?) but they are sent if using auto flush (every 60 seconds)

In both situations Events.Flush() fails. The only way it will work is if successful consent has been given.

Events were observed using Charles Proxy.

So basically..

The offline workflow when using Events.CheckForRequiredConsents() is breaking the new Analytics at present. It appears better to ignore the consent process (if offline) and manual flushes (if no consent given)

These are just my findings and may not be complete. I'm sure there are others scenarios that could be tested. On-top of that, you may see different behaviour.

But hopefully this all helps!

 

Hey Jeff!

I'm moving away from the new Analytics for the time being, but just observed the following...

If the device is offline at init then UnityServices.InitializeAsync() sets the state to Initialised but... I'm not seeing any events being sent if the device then comes back online.

The above call throws the following exception...

Curl error 28: Operation timed out after 10000 milliseconds with 0 bytes received

So assuming this is not being handled & re-thrown, then internally the API is setting State but has left itself in an undetermined state.

Basically if a device is offline during init then Analytics never appears to cache or send events, even if the device comes back online.

So...

Offline
UnityServices.InitializeAsync()
Events.CheckForRequiredConsents() <- Optional
Online
...
Events.CustomData() -> No event sent (or auto flush not being called)
...
Exit Game -> No event sent (or auto flush not being called)

FWIW legacy Analytics would handle and recover far more gracefully in offline situations, so definitely some changes needed in this area with new Analytics.

I hope this all helps!

 

Hey @Laurie-Unity

Thanks for the update.

I'm running v3.0.0-pre.4 here, installed on Tuesday.

Certainly for me, the offline issues are the biggest problem right now. That said, when online it's working well and the new dashboard is big step forward

Anything else I can help with, please just ask.

 

Thanks @Laurie-Unity

My game brings the various sub-bsystems online during game init in a linear fashion (using various coroutinues). Some (like analytics, ads etc) will try for N number of seconds then continue the init process if it the sub-system didn't initialise.

Those sub-systems will then keep trying in the background (every N seconds) until they do initialise. None of these sub-systems are mission critical, so they won't stop gameplay if not initialised.

The new analytics documentation states that events are cached even if the service is not initialised, then flushed when it eventually is. So I'm just testing the process now with various scenarios (offline before init and after etc) to see how well that works.

The trouble with the consent process as it stands is that ultimately it is very tied to having internet access, which is far from guaranteed with mobile devices. It also requires player input and obviously the consent ui can't popup mid-game (if that's when the device comes online). So for most this would mean adding the process to game init or at the latest, the home menu.

That was my thinking behind having (non-online) fall-back processes. It means consent can always be determined during game init.

Of course app store policies or legal may have other ideas on that

 

Hi @Laurie-Unity

So I've worked through a few offline scenarios with my new workflow (which initialises Analytics in the background only when the device has internet access) and the findings are below. It just shows one of my own custom events (FeatureVisited) and ignores standard events.

Scenario 1 (Device Offline on App Start, Device Online Mid-Game)

1) Device Offline
2) App Start
3) Analytics Init Fail (Background Retry Every N Seconds)
4) Home Menu
5) Visit FeatureA (Settings)
6) Level Start
7) Visit FeatureB (Pause)
8) Device Online
9) Analytics Init Success [FeatureA, FeatureB Events Posted - Seen in Charles Proxy]
10) Level End
11) App Exit

So as the documentation states, the events were cached (in memory?) and sent when the device came online. Which is good.

But obviously consent shouldn't be requested mid-game (stage 9) so in this scenario stage 3 or 4 are really the best options. Now that's not a difficult feature to build, but it's not made clear in the documentation and I can see others getting caught out by this.

Scenario 2 (Device Offline for Session 1, Online for Session 2)

1) Device Offline
2) App Start
3) Analytics Init Fail (Background Retry Every N Seconds)
4) Home Menu
5) Visit FeatureA (Settings)
6) Level Start
7) Visit FeatureB (Pause)
8) Level End
9) App Exit
10) Device Online
11) App Start
12) Analytics Init Success [FeatureA, FeatureB Events NOT Posted - Not Seen in Charles Proxy]

So in that scenario all events are lost. If this is a feature rather than bug, couldn't in memory events be flushed to storage on app exit for sending on the next successful init?

I've implemented Beta Analytics as best as possible for now, so will be moving onto something else until the next version.

Please reach out if you need any further info etc though.

Good luck!

 

Thanks for the detailed flows @PeachyPixels!
Do you know if the above is the current state of the offline use cases for the New Analytics or has any progress been made with later versions? @Laurie-Unity might also know more about this?
I am using 4.2.0 currently btw!

 

I've no idea to be honest. I converted my abstraction layer to the new Unity Analytics, tested as best as possible, then moved onto other things.

Current thinking (for my next game) is to actually not support Analytics at all. But if that changes I will re-test (with the current Verified version of Unity Analytics) and post here if there are any significant updates.

Good luck with your implementation.