Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on Dec 8, 2022, between 7 am & 7 pm EST, in the DOTS Dev Blitz Day 2022 - Q&A forum, Discord, and Unity3D Subreddit to learn more about DOTS directly from the Unity Developers.
    Dismiss Notice
  3. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Announcing support for Multiple-Weighted Variants within Rules

Discussion in 'Unity Remote Config' started by markychoi, Jan 31, 2020.

  1. markychoi

    markychoi

    Unity Technologies

    Joined:
    Aug 4, 2014
    Posts:
    40
    We are adding support for “Multiple-Weighted Variants” within Rules that allows developers to define more than one variant per Setting in a Rule.

    Remote Config uses Rules to target specific user groups and deliver different settings for each group. Rules are linked to Settings, which map to variables in the game code to override their default values when the audience criteria for a Rule is met.

    For existing Remote Config users, you can use this new functionality to run a randomized experiment with two or more variants, commonly known as “A|B testing”.

    You can now create a Rule where, instead of a single value mapped to a Setting, you can create multiple variants of that value for a particular Setting. Each variant also has an associated weight that defines the allocation of that particular variant across the population of players that the Rule applies to. You can either define custom weights for each variant – with the only condition being that the sum of weights across all variants must equal 100% – or if weights are all left undefined, Remote Config will assume uniform, random distribution for the variants.

    Known Limitations
    Analysis and Reporting
    Remote Config is limited to the handling of the assignment and delivery of settings.

    The system today does not do any of the following:
    • Automatically track a target metric
    • Perform any statistical hypothesis testing to calculate sample size or assess the significance of sample data
    • Determine “winners”
    Remote Config does create a unique variantId that is appended to the assignment event when serving assignments to the player at runtime. In this way, you can use the assignment summary and aggregate reports to describe how often each variant was selected (see Assignment Reports for more details).

    If your game also sends user behavior events to Unity or your own analytics endpoint, you can cross reference this assignment summary report with a user behavior events report using the assignmentId, and then perform your own statistical analysis to determine the results of your test.

    Variant Stickiness
    Remote Config currently cannot track specific user assignments at runtime, so variant stickiness cannot be guaranteed. As it stands today, if the request attributes are identical, Remote Config will always serve the same assignment to the requesting user - including variant selection. However, if those attributes change, or the Rule configuration changes, there is the risk of serving up a different variant the next time that user requests an assignment.

    A|B Testing Example
    Step 1: Initial configuration
    Go through the process of configuring your project for Unity Remote Config by following the steps here: Configuring your Project for Unity Remote Config.

    After creating an Environment and Config (either in the dashboard or Editor GUI), create a new Rule with type “variant”, then link your Settings to this Rule.



    Step 2: Add new variants to the Rule
    Add a new variant to the Rule using either the Editor package GUI or in the Remote Config Dashboard.

    Editor GUI



    Dashboard
    When using the dashboard, click on the “Settings” tab to add variants and edit variant-specific settings.




    At the start, each Setting listed in the Rule will include one value by default. You can add variants by clicking on “Add Variants”.

    Make adjustments as needed to the variant weights and setting values, then save your changes by selecting “Save” on the dashboard or “Push” in the Editor GUI.

    Step 3: Enable Rule and Play Game
    Finally, make sure your new Rule is set to “Enabled” to see your variant assignments at runtime.

     
    jacobbev and vd_unity like this.
  2. bedivere_joyseed

    bedivere_joyseed

    Joined:
    Sep 1, 2018
    Posts:
    3
    Hi, I tried to configure the variant, but it doesn't work in my dashboard. The keys always marked as "undefined" even though I added 2 variants of keys.

    This is the Parameters

    upload_2020-5-6_12-6-36.png

    And the Settings
    upload_2020-5-6_12-7-34.png
     
  3. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    Thanks for the heads up! We should be addressing this shortly
     
  4. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    This should be working correctly now.
     
  5. JorisEertink

    JorisEertink

    Joined:
    Mar 15, 2017
    Posts:
    24
    @markychoi , @JeffDUnity3D ,

    Are there specific reasons why we should use Remote Config over the "A | B Testing" that is currently in the operate dashboard, or the other way around?
     
  6. Lisan

    Lisan

    Joined:
    Jun 17, 2009
    Posts:
    212
    What is the conditions for "request attributes are identical"? If the user made request from another network, or another city - is this considered change of attributes, and possible another setting variant?

    Also, can you, please, write a full example of running A/B test and measuring it's results? All i understood, is measurment fuctionality is not yet ready from the box, but can be somehow accomplished with third party analitycs and using assigmentID. That's what i am looking for, but i can find any info on how to do it exactly.
    I will explain what i need. First i set up some experiment, rolling out some feature in two variants to 50% (group 1) /50% (group2). That's can be done with Remote Config. Second, i need to evaluate results. All i need for this, is for users to send to analytics an events like this "Exp1_Group1" and "Exp1_Group2". So i can find these users in the analytics and measure their metrics. But for this to work i need to somehow get this group info from Remote Config. But how i do this? Considering, that there can be multiple rules with variants, but assigmentID is just one field returned for the whole request?
     
    Last edited: Jun 24, 2020
  7. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    So same user, same device, but from another city and/or network?
     
  8. Lisan

    Lisan

    Joined:
    Jun 17, 2009
    Posts:
    212
    I just want to understand "attributes of request" which should be identical for the same variant to be assigned to the user, is it something developer can somehow ensure by following some rules, or it's somethig that changes based on external conditions like networks, locations and so on.

    Also i updated post above with another question about A/B test.
     
  9. peterle_unity

    peterle_unity

    Unity Technologies

    Joined:
    Oct 3, 2017
    Posts:
    8
    Hi Lisan,

    Thanks for reaching out about Remote Config! For your "attributes of request" question... as long as the request contains the same "customUserId", that user will always get the same variant per rule (as long as the rule is still enabled). The other request attributes, such as networks and locations, will not have an effect on the variant selected for that user, as long as they still satisfy the JEXL condition for the rule itself. You could even have that user make a request from a different device, and they will see the same variant as long as they request using the same "customUserId".

    For measuring the results, you have a couple of options. The response you'll get from Remote Config will contain a unique "Assignment ID". You can generate a CSV report at any time on our web dashboard that will link up this assignment ID to the rules and variants specified per user.

    Alternatively, you could make this more explicit on the device itself for your own reporting by creating a string config key called (for example) "Experiment Group" and have the variant rule send back "Exp1_Group1" or "Exp1_Group2" to the device, so you can log that directly to Analytics to keep track.
     
  10. Lisan

    Lisan

    Joined:
    Jun 17, 2009
    Posts:
    212
    Thanks! But is using customUserId is a must to achive this? I thought it's optional parameter. It's not hard to generate it, but to make it persisting across devices is a whole another story..

    I still don't understand what exactly this "Assignment ID" means, and manuals don't help. Will it be unique for a combination "user+variant" or not? Or it will return a new value on each request and won't be persistant? This feature lacks a decent tutorial on how to do A/B measurement step-by-step.

    Hmm.. This simple solution just didn't come to my mind. Thanks!
     
  11. dri_richard

    dri_richard

    Joined:
    Mar 10, 2017
    Posts:
    153
    One of the things that attracted me to the Unity Analytics A|B solution was this line in the docs, because it suggested I could run an experiment on a remote setting that I had already implemented:

    "You can use existing Remote Settings from the Release configuration as long as the settings are not used in any other active experiments."

    I'm aware of the difference between Remote Config and Remote Settings, but it seems that following the methods above would require me to create a new release of my game to start sending the "Experiment Group" IDs, is that correct?
     
  12. peterle_unity

    peterle_unity

    Unity Technologies

    Joined:
    Oct 3, 2017
    Posts:
    8
    We use customUserId for assignments, if it exists in the request. If it doesn't, we fall back on a Unity generated userId that should be the same between user sessions for a game on a device, but we'd have no way of connecting users across games or devices.

    Each time a request is made for settings, we generate a unique Assignment ID. This Assignment ID can be used to find the user and any variants that were selected for that assignment, using our generated reports on our web dashboard. Each time that user requests settings, the variant that is assigned should stay persistent... although each time will generate a unique Assignment Id.
     
  13. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    2,926
    I'm not sure why, but if no userId is assigned - variants will always return second option in 50/50 weights.
    Assigning user id (e.g. ConfigManager.SetCustomUserID(SystemInfo.deviceUniqueIdentifier); ) does the trick.