Feedback Misc Feedback from FTUX of Unity Analytics Services

I recently watched the Unite talk on the new Unity Services and decided to try out Unity Analytics for our current in progress game. I'd like to preface this with some facts about our games and production, so that there is appropriate context for this feedback on our experience with setting up the package, and beginning recording of data.

About Us (and Me, the dev implementing the Analytics package) - We have worked on multiple shipped titles on Steam, Switch, iOS, and Android with small teams. These include well received titles such as QuiVr and Marble It Up! on Apple Arcade.

I mention this because some of the feedback I have is about the types of analytics we want to collect that don't seem to be a major focus of the package, but are still really important for games other than just indies making their first itch.io game (no offense intended)

Additionally, we are not strangers to analytics services, having used systems like Google Analytics, Amplitude, Firebase, etc. as well as the Unity Analytics that was available many years ago that we switched away from due to some of the same missing features that I'll be talking about here.

So without further ado, here are some friction points, issues, and good stuff I've found while investigating and implementing the most recent (4.3.0) Unity Analytics Service package and web portal.

 

Part 1 : Setting up the package

This is probably the smoothest experience I've encountered with Unity Services so far, so kudos!

The package is easy to install, and the Setup Guide is really helpful for getting you sending your first event, took about 5 minutes, and that includes me going through the extra steps of including a new dev environment for analytics events while using the Editor, and including a UserID since we already have player accounts being set up for multiplayer.

There was a bit of a delay with getting the first event to show up in the Event Browser, which caused me to relaunch the game a number of times with various logs to confirm I was actually successfully initializing the package, but I'll talk more on the delay later.

Overall, the process to get the package set up was straightforward and painless, great job!

 

Part 2 : Game Events
Here is where I started having some issues. Step 3 of the getting started guide is to create a custom event in the Event Manager, and boy does the complexity skyrocket with minimal info on how to create a useful custom gameplay event. There are a few major aspects that exacerbate this issue.

Default Events
There are so many default events and parameters, that it is immediately difficult to work with. Many of the default events (none of which can be disabled) only apply to mobile games, which is unfortunate. Things like push-notification, aquisitionSource, and adImpression, are totally meaningless on a lot of platforms, and should be opt-in instead of default and mandatory.

This issue is even worse with the parameters, there are 159 default parameters, that is insane and makes navigating parameters while creating new events cumbersome from the start. If we're working on a PC/Console game, we don't care about the 10 different acquisition parameters or the numerous Amazon params. A majority of the SDK API discusses classes that are not relevant to many games.

I think it would be advantageous to be able to opt into various analytics event groups when setting up analytics for a project. Being to able to go through a few groups like "User Acquisition", "Ads", "System", etc and just toggle them on or start fresh with no default events but the bare minimum 'gameX' ones.

It would also be nice to be able to modify how often (or if at all) gameRunning gets sent, when price is metered, it is nice to be in control how often events get sent, and it's totally reasonable to sacrifice some resolution about dropoff for less frequent events (seems to be 1s by default).

Creating Custom Events
The creation UI is tough, there is way too much empty space in the header and the default events have descriptions that take up too much space by default (should be collapsible) requiring a ton of scrolling while adding parameters, especially because the "Assign Parameter" is up at the top, so if you want to verify what params an event has before adding a new one, it requires scrolling down then back up.

I understand that custom events can't be deleted due to maintaining consistency for funnels, etc, but it should be possible to hide/archive them, so that they don't clutter up all search panels forever. The only place archived events should show up is when they are part of an existing funnel, or when searched for in the Event Manager with an "Include Archived" checkbox checked.

It also is really unfortunate that you seem to have to create custom events separately for each environment. At least, I didn't see a way to copy a custom event from dev to production once it was created, and that seems pretty silly, but at least custom parameters persist between environments! Which brings us to...

Creating Custom Parameters
The existing parameter types are not enough for game dev analytics and you all have to know that. String/Int/Float/Bool might cut it web analytics, but 3D games?

You absolutely need Vector2/3 types for any raw data usage, and it's totally fine if they won't be available in the web-based explorer, but heatmaps, anomaly detection, etc need positional data.

If other data types are available through API they aren't described anywhere when creating new events. Even the Analytics.CustomEvent docs have Vector3 in an example. But when setting up events, I want to be able to define those positional parameters.

Finally, it would be really nice to be able to have saved parameter-groups. For instance when I create custom events for gameplay-related tuning (for instance a roguelike choice being made, a player dying, etc), there are a number of game parameters I might always want to include like {Map, PlayerCount, DifficultyLevel, etc}. Having to add all these various parameters to each custom events is a pretty tedious process, but having a "GameplayEvent" parameter set that I could add would be great, and if we ever added a new feature we always wanted to track (eg. "AveragePlayerLevel"), we could just add that param to the GameplayEvent group and it would be added to all of the custom events that include that group.

Custom Event Engine Implementation
From the tiny amount of documentation about custom events, I have not been able to find a way to detect if an event was accepted/ignored from the editor side. You just fire off AnalyticsService.Instance.CustomData(name, data); and hope it was configured with the correct data. This wouldn't be an issue if there wasn't an hour delay from sending an event in editor to being able to see if it was configured correctly when it becomes visible in the dashboard going into the Valid/Invalid group.
It would be so nice, after having spent the time to set up a schema in the web dashboard, to have a "Copy Event Code" button that would create a bit of code like:

Code (CSharp):

1. AnalyticsService.Instance.CustomData("eventName", new Dictionary<string, object>
2.         {
3.             { "custom_param_1", INT },
4.             { "custom_param_2", STRING },
5.             { "custom_param_3", BOOL },
6.         });

that we could copy over to Unity and know that the event would be accepted, and we weren't missing any mandatory params or misspelling anything, as that is one major advantage of pre-defined event schema.

 

Part 3 : Data Analysis

Event Delay
The time delay for new events really hurts iteration time, something that Unity has always been a champion of even if there have been some bumps. There are plenty of Analytics services that offer immediate results, and if the issue is with user privacy, etc, just allow immediate data if it is coming from the Unity Editor. Waiting an hour to see if a custom event worked is nightmarish when setting up for the first time.

Data Exploration
Event filtering in the Data Explorer is pretty good, I wish it were easier to get out of a purely data-based approach to analysis. A lot of the time I just care about relative data across a timespan, not the specific data at each date. More non-line graph types (pie?) would be valuable.

With so many default events and params, finding the ones I care about is harder than I'd like, similar to creating the events mentioned above.

One awesome thing that some other analytics services provide for data exploration, is offering suggested values for string data. For example, when filtering for a specific version, or itemID, if there are only a few options, make it a dropdown instead of a pure string entry.

Overall I think the basic data exploration tools are nice and functional, and for more complex analysis the SQL Explorer is available, so nice job here!

Audiences
Similar to default events/params, there are too many Standard Audiences that are only useful for mobile games. Spenders are a useful group, but they should be opt in or able to be deleted, as a PC/Console game might not have any interest in these groups but they will clutter up the UI forever.

Other than that issue, there are some cool defaults in the list (unengaged, veteran), and it's cool that they can be used as filters in Data Exploration.

Funnels
These look great, but we don't have enough data to fully speak on how effective they are at showing actionable insights.

The one thing I imagine will become tedious is setting up the same funnel for multiple instances of the same type of content. For instance, the docs say you can use funnels to see if a dungeon is too hard. But if a game has 20 dungeons and wants to see the funnel for all of them, recreating the same funnel 20 times with only a single parameter change for each seems like frustration waiting to manifest. If there was a solution for this on the dashboard, I wasn't able to find it.

Dashboards
Let us create our own dashboards...

The Game Performance dashboard looks great, just let me create one with breakdowns of what modes are being played the most, the retention of specific features, etc.

Raw Data Export
I can't seem to find any way to get data into the engine so that we can use in-house editor tools to analyze the data using context. Whether this is heatmaps of places players have died, or the ability to create in-place data inspections on items (eg. a warning on an item is over 60% pick rate in games), having data available to query from the engine, or at least use in raw format, is necessary for many classes of analysis.

 

Conclusion
The Unity Analytics service looks promising, with strong tools for analysis. The biggest issues are that almost all the defaults feel like they're designed to improve monetization, not improve a game...

There are very few tools specific to game analytics, which makes it easier for other competing services to make reasonable claims about being superior, which is sad. Generic app analytics services shouldn't be able to compete so easily with Unity for game analytics.

I'm sure there will be marked improvements, but this isn't Unity's first attempt at creating a game analytics service, and currently it seems just as likely that Analytics will be rebooted in 4 years if this iteration doesn't get the adoption from the community needed to warrant continued investment.

I know that non-mobile games might not be the most likely ones to generate more than 25M events/month to start generating revenue, but these services aren't just supposed to be useful for the biggest spenders (I'm sure the biggest mobile games that use Unity like COD Mobile have their own internal analytics infrastructure), they should be one of many reasons why devs continue to choose Unity as their engine of choice, because we can make great games with it, and all the services you provide.

With the current state of the service, I will continue to keep my eye on updates and improvements, but currently I will also continue to look at other generic analytics services, as they are still very competitive in feature sets and ease of use.

 

From someone who has used GameAnalytics and DevToDev in the past, and spent the last week or so implementing the new Unity Analytics and Custom Events, I whole-heartedly agree with all your points.

Even though my game's are F2P/Mobile focused, you're 100% right on being forced into mobile specific/acquistion/spending related parameters which are pointless in a premium game.

It would be nice if you could create a new Analytics project using a Premium or Freemium template, or a completely blank one from scratch.

Not sure if I understand you correctly, but I would just track Valid Events and Invalid Events in the Event Browser, and if anything pops up in Invalid, usually you'll get a really descriptive error message of why the event was not accepted.
Once you don't see anything new coming into Invalid, you know all your events are coming through correctly.

 

Yep, I was able to find that, though I hadn't seen the full error description as I guess I got it right first try. The biggest issue then for me is the hour delay when testing new events.

I've updated my original post with more detail about this issue

 

Over 6 months later, and not a single one of these issues has been addressed.

Analytics 5.0 released with a whopping 12 items in its changelog from 4.4.2 - Isn't this the type of thing that Unity is putting money into instead of Gigia etc?