Announcing support for Multiple-Weighted Variants within Rules

We are adding support for “Multiple-Weighted Variants” within Rules that allows developers to define more than one variant per Setting in a Rule.

Remote Config uses Rules to target specific user groups and deliver different settings for each group. Rules are linked to Settings, which map to variables in the game code to override their default values when the audience criteria for a Rule is met.

For existing Remote Config users, you can use this new functionality to run a randomized experiment with two or more variants, commonly known as “A|B testing”.

You can now create a Rule where, instead of a single value mapped to a Setting, you can create multiple variants of that value for a particular Setting. Each variant also has an associated weight that defines the allocation of that particular variant across the population of players that the Rule applies to. You can either define custom weights for each variant – with the only condition being that the sum of weights across all variants must equal 100% – or if weights are all left undefined, Remote Config will assume uniform, random distribution for the variants.

Known Limitations
Analysis and Reporting
Remote Config is limited to the handling of the assignment and delivery of settings.

The system today does not do any of the following:

• Automatically track a target metric
• Perform any statistical hypothesis testing to calculate sample size or assess the significance of sample data
• Determine “winners”

Remote Config does create a unique variantId that is appended to the assignment event when serving assignments to the player at runtime. In this way, you can use the assignment summary and aggregate reports to describe how often each variant was selected (see Assignment Reports for more details).

If your game also sends user behavior events to Unity or your own analytics endpoint, you can cross reference this assignment summary report with a user behavior events report using the assignmentId, and then perform your own statistical analysis to determine the results of your test.

Variant Stickiness
Remote Config currently cannot track specific user assignments at runtime, so variant stickiness cannot be guaranteed. As it stands today, if the request attributes are identical, Remote Config will always serve the same assignment to the requesting user - including variant selection. However, if those attributes change, or the Rule configuration changes, there is the risk of serving up a different variant the next time that user requests an assignment.

A|B Testing Example
Step 1: Initial configuration
Go through the process of configuring your project for Unity Remote Config by following the steps here: Configuring your Project for Unity Remote Config.

After creating an Environment and Config (either in the dashboard or Editor GUI), create a new Rule with type “variant”, then link your Settings to this Rule.



Step 2: Add new variants to the Rule
Add a new variant to the Rule using either the Editor package GUI or in the Remote Config Dashboard.

Editor GUI



Dashboard
When using the dashboard, click on the “Settings” tab to add variants and edit variant-specific settings.




At the start, each Setting listed in the Rule will include one value by default. You can add variants by clicking on “Add Variants”.

Make adjustments as needed to the variant weights and setting values, then save your changes by selecting “Save” on the dashboard or “Push” in the Editor GUI.

Step 3: Enable Rule and Play Game
Finally, make sure your new Rule is set to “Enabled” to see your variant assignments at runtime.

 

Hi, I tried to configure the variant, but it doesn't work in my dashboard. The keys always marked as "undefined" even though I added 2 variants of keys.

This is the Parameters



And the Settings

 

Thanks for the heads up! We should be addressing this shortly

 

This should be working correctly now.

 

@markychoi , @JeffDUnity3D ,

Are there specific reasons why we should use Remote Config over the "A | B Testing" that is currently in the operate dashboard, or the other way around?

 

What is the conditions for "request attributes are identical"? If the user made request from another network, or another city - is this considered change of attributes, and possible another setting variant?

Also, can you, please, write a full example of running A/B test and measuring it's results? All i understood, is measurment fuctionality is not yet ready from the box, but can be somehow accomplished with third party analitycs and using assigmentID. That's what i am looking for, but i can find any info on how to do it exactly.
I will explain what i need. First i set up some experiment, rolling out some feature in two variants to 50% (group 1) /50% (group2). That's can be done with Remote Config. Second, i need to evaluate results. All i need for this, is for users to send to analytics an events like this "Exp1_Group1" and "Exp1_Group2". So i can find these users in the analytics and measure their metrics. But for this to work i need to somehow get this group info from Remote Config. But how i do this? Considering, that there can be multiple rules with variants, but assigmentID is just one field returned for the whole request?

 

So same user, same device, but from another city and/or network?

 

I just want to understand "attributes of request" which should be identical for the same variant to be assigned to the user, is it something developer can somehow ensure by following some rules, or it's somethig that changes based on external conditions like networks, locations and so on.

Also i updated post above with another question about A/B test.

 

Thanks! But is using customUserId is a must to achive this? I thought it's optional parameter. It's not hard to generate it, but to make it persisting across devices is a whole another story..

I still don't understand what exactly this "Assignment ID" means, and manuals don't help. Will it be unique for a combination "user+variant" or not? Or it will return a new value on each request and won't be persistant? This feature lacks a decent tutorial on how to do A/B measurement step-by-step.

Hmm.. This simple solution just didn't come to my mind. Thanks!

 

One of the things that attracted me to the Unity Analytics A|B solution was this line in the docs, because it suggested I could run an experiment on a remote setting that I had already implemented:

"You can use existing Remote Settings from the Release configuration as long as the settings are not used in any other active experiments."

I'm aware of the difference between Remote Config and Remote Settings, but it seems that following the methods above would require me to create a new release of my game to start sending the "Experiment Group" IDs, is that correct?

 

I'm not sure why, but if no userId is assigned - variants will always return second option in 50/50 weights.
Assigning user id (e.g. ConfigManager.SetCustomUserID(SystemInfo.deviceUniqueIdentifier); ) does the trick.