Search Unity

  1. Unity 2018.1 has arrived! Read about it here
    Dismiss Notice
  2. Scriptable Render Pipeline improvements, Texture Mipmap Streaming, and more! Check out what we have in store for you in the 2018.2 Beta.
    Dismiss Notice
  3. If you couldn't join the live stream, take a peek at what you missed.
    Dismiss Notice
  4. Improve your Unity skills with a certified instructor in a private, interactive classroom. Learn more.
    Dismiss Notice
  5. ARCore is out of developer preview! Read about it here.
    Dismiss Notice
  6. Magic Leap’s Lumin SDK Technical Preview for Unity lets you get started creating content for Magic Leap One™. Find more information on our blog!
    Dismiss Notice
  7. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Questions about Unity Analytics A/B Testing

Discussion in 'Unity Analytics' started by gmcnaughton, Jun 4, 2018.

  1. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    Hi Unity!

    I have a couple questions about the beta A/B Testing service (we're using it in code via RemoteSettings):
    1. How do I reset remote settings in Unity Editor, to try to get into a different variant?
    2. Can I run A/B tests in development vs. production? Or when I click "Start Experiment", does that experiment begin immediately in both environments?
    3. Do control users get the setting value listed under the "Control" treatment, or the default value for the setting configured in Remote Settings? (In which case, if it's a new setting that does not exist in Remote Settings, what do control users receive?)
    I also have a couple "dev happiness" feature requests/questions:
    1. Is there any way to delete ended experiments?
    2. Is there any way to delete old settings used by ended experiments?
    3. Can I rename experiments? (I couldn't find a way to edit the name, even before starting it)
    4. Can I clone or modify an experiment (e.g., if I want to run it again with different values)?
    Thanks so much!
    Gordon
     
    jreed-curious and shochet like this.
  2. shochet

    shochet

    Joined:
    Dec 17, 2013
    Posts:
    14
    Excited to help test this - and I second the questions above, especially how to separate dev and production.
    Suggestion: Would be great to have more goals other than just retention. Monetization (IAP revenue) and other custom events would be really useful.
    Thanks-
     
    jreed-curious and gmcnaughton like this.
  3. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    962
    @gmcnaughton,

    Just a quick note about the AB Testing workflow.

    AB Testing will only use the Release configuration of Remote Settings. That means it won't affect the values sent to the Editor and Development builds (those platforms use the Development config.)

    So you could create a Remote Setting key in the Development Config and test that in the Editor. Once things are working correctly, you can transfer your Development settings to the Release config (via the Import/Export option).

    Now you can start an AB Test to see which values of the Remote Settings perform better with your users.

    However, the system is flexible enough that you don't need to go through the first steps of creating Remote Settings and testing them. If you know the exact settings you need, you can just create them when starting the AB Test. These settings will not be available in the standard Remote Settings page.

    Currently the Editor doesn't support variants.

    As mentioned, AB Testing is only available in production builds. The reason is that a valid AB Test requires a certain number of users in order to be statistically significant.

    Control users will get the setting in the Control treatment. The default value for that Remote Settings is used to set the default for that Setting, but if it is changed, then it will use the Control treatment value.

    Appreciate all the feedback. We definitely have plans to improve this feature, so I will make sure these ideas are passed on to the dev team.
     
    gmcnaughton likes this.
  4. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    @ap-unity, thanks for the detailed response, that was super helpful!

    Two more questions have come up in the interim:
    1. Is there a way to get a list of experiments/variants the user is currently in? We'd like to send these up to our backend as part of the user's metadata (for internal tracking and reporting)
      1. Our current workaround is to manually add an "ABTEST_<experiment>" treatment to each variant, which has the variant name as its value. Is that our best bet for the moment?
    2. The RemoteSettings docs mention a limit of "200 values per configuration. A setting with one segment, uses one value". Does this limit affect a/b tests? Are we in danger of hitting the 200 value cap by running multiple tests with multiple variants (especially since we can't delete finished ones?)
    Thanks again!
     
  5. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    @ap-unity bump - any thoughts on the second set of questions?
     
  6. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    962
    There isn't currently a way to do this. We have some ideas of how we might do this, but I can't say when that would be available.

    Yeah, that's probably your best bet.

    These quotas are separate.
     
    gmcnaughton likes this.
  7. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    Awesome - much appreciated! We'll keep sending feedback as we try this puppy out.
     
    ap-unity likes this.