Search Unity

AB Testing & release/development switch on remote config

Discussion in 'Unity Analytics' started by chrisendicott, Oct 11, 2018.

  1. chrisendicott

    chrisendicott

    Joined:
    Apr 19, 2017
    Posts:
    8
    Could someone explain how AB testing's use of remote config variables interacts with the release/development switch for the variables in remote config?

    If I create a variable in remote config in development mode, it doesn't show up as available as a treatment in AB testing which makes me assume you can only use AB testing on release variables? How does this interact with being able to test the AB test in the editor?

    Is there a reason the AB test page doesn't have the same release/development switch as the remote config page?
     
    chrisendicott, Oct 11, 2018
    #1

  2. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    1,519
    AB Testing will only use the Release configuration of Remote Settings. That means it won't affect the values sent to the Editor and Development builds (those platforms use the Development config.)

    You still can and should test your Remote Settings in the editor/dev builds first. How you do this is up to you, but since the Development config is completely separate from the Release config, you could give the variants different names (e.g. button_color_variant_1 and button_color_variant_2) and then have some way to switch between them in development. Or you could just keep the key the same and just make sure to test all values during development.

    The reason for only using the Release config is that AB Testing is only really useful if the sample size is large enough to deliver statistically significant results. This is more typical of a production environment than development.
     
    ap-unity, Oct 11, 2018
    #2

  3. chrisendicott

    chrisendicott

    Joined:
    Apr 19, 2017
    Posts:
    8
    Thanks - I appreciate the explanation.
     
    chrisendicott, Oct 12, 2018
    #3