Search Unity

  1. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  2. Improved Prefab workflow (includes Nested Prefabs!), 2D isometric Tilemap and more! Get the 2018.3 Beta now.
    Dismiss Notice
  3. Want more efficiency in your development work? Sign up to receive weekly tech and creative know-how from Unity experts.
    Dismiss Notice
  4. Participate with students all over the world and build projects to teach people. Join now!
    Dismiss Notice
  5. Build games and experiences that can load instantly and without install. Explore the Project Tiny Preview today!
    Dismiss Notice
  6. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  7. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Questions about Unity Analytics A/B Testing

Discussion in 'Unity Analytics' started by gmcnaughton, Jun 4, 2018.

  1. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    Hi Unity!

    I have a couple questions about the beta A/B Testing service (we're using it in code via RemoteSettings):
    1. How do I reset remote settings in Unity Editor, to try to get into a different variant?
    2. Can I run A/B tests in development vs. production? Or when I click "Start Experiment", does that experiment begin immediately in both environments?
    3. Do control users get the setting value listed under the "Control" treatment, or the default value for the setting configured in Remote Settings? (In which case, if it's a new setting that does not exist in Remote Settings, what do control users receive?)
    I also have a couple "dev happiness" feature requests/questions:
    1. Is there any way to delete ended experiments?
    2. Is there any way to delete old settings used by ended experiments?
    3. Can I rename experiments? (I couldn't find a way to edit the name, even before starting it)
    4. Can I clone or modify an experiment (e.g., if I want to run it again with different values)?
    Thanks so much!
    Gordon
     
    jreed-curious and shochet like this.
  2. shochet

    shochet

    Joined:
    Dec 17, 2013
    Posts:
    15
    Excited to help test this - and I second the questions above, especially how to separate dev and production.
    Suggestion: Would be great to have more goals other than just retention. Monetization (IAP revenue) and other custom events would be really useful.
    Thanks-
     
    jreed-curious and gmcnaughton like this.
  3. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    1,049
    @gmcnaughton,

    Just a quick note about the AB Testing workflow.

    AB Testing will only use the Release configuration of Remote Settings. That means it won't affect the values sent to the Editor and Development builds (those platforms use the Development config.)

    So you could create a Remote Setting key in the Development Config and test that in the Editor. Once things are working correctly, you can transfer your Development settings to the Release config (via the Import/Export option).

    Now you can start an AB Test to see which values of the Remote Settings perform better with your users.

    However, the system is flexible enough that you don't need to go through the first steps of creating Remote Settings and testing them. If you know the exact settings you need, you can just create them when starting the AB Test. These settings will not be available in the standard Remote Settings page.

    Currently the Editor doesn't support variants.

    As mentioned, AB Testing is only available in production builds. The reason is that a valid AB Test requires a certain number of users in order to be statistically significant.

    Control users will get the setting in the Control treatment. The default value for that Remote Settings is used to set the default for that Setting, but if it is changed, then it will use the Control treatment value.

    Appreciate all the feedback. We definitely have plans to improve this feature, so I will make sure these ideas are passed on to the dev team.
     
    gmcnaughton likes this.
  4. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    @ap-unity, thanks for the detailed response, that was super helpful!

    Two more questions have come up in the interim:
    1. Is there a way to get a list of experiments/variants the user is currently in? We'd like to send these up to our backend as part of the user's metadata (for internal tracking and reporting)
      1. Our current workaround is to manually add an "ABTEST_<experiment>" treatment to each variant, which has the variant name as its value. Is that our best bet for the moment?
    2. The RemoteSettings docs mention a limit of "200 values per configuration. A setting with one segment, uses one value". Does this limit affect a/b tests? Are we in danger of hitting the 200 value cap by running multiple tests with multiple variants (especially since we can't delete finished ones?)
    Thanks again!
     
  5. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    @ap-unity bump - any thoughts on the second set of questions?
     
  6. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    1,049
    There isn't currently a way to do this. We have some ideas of how we might do this, but I can't say when that would be available.

    Yeah, that's probably your best bet.

    These quotas are separate.
     
    gmcnaughton likes this.
  7. gmcnaughton

    gmcnaughton

    Joined:
    Sep 15, 2017
    Posts:
    5
    Awesome - much appreciated! We'll keep sending feedback as we try this puppy out.
     
    ap-unity likes this.
  8. jason_yak

    jason_yak

    Joined:
    Aug 25, 2016
    Posts:
    208
    Does anyone know how A/B testing will behave if a user has opted-out of the GDPR data collection if the test relies on country based segments?
     
  9. jason_yak

    jason_yak

    Joined:
    Aug 25, 2016
    Posts:
    208
    Or for that matter any remote settings that rely on segment data.
     
  10. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    2,758
    @jason_yak I am asking the Services team here for clarification regarding remote settings and GDPR. I would ask that you limit your question to a single thread however
     
  11. jason_yak

    jason_yak

    Joined:
    Aug 25, 2016
    Posts:
    208
    Thanks for the response, I’ll remove my previous questions in the other thread. Yea I just wondered if I’d posted in the wrong thread perhaps and thought to post here instead... thanks for looking into it.
     
  12. jason_yak

    jason_yak

    Joined:
    Aug 25, 2016
    Posts:
    208
    Ohh I’ll leave it, given that you responded there also. But point taken, sorry about that.
     
  13. Mogurito

    Mogurito

    Joined:
    Sep 21, 2015
    Posts:
    11
    Hi! I have a question. Is there a way to target the sample size to only users who have a specific app version? This is because we added a key in the last version of our game. Including old version in the test will corrupt the report because those old versions doesn't use that key.
     
  14. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    1,049
    @Mogurito

    The beta version of AB Testing does not support segments. But that is a feature we hope to include in V1 when it's released.
     
  15. Mogurito

    Mogurito

    Joined:
    Sep 21, 2015
    Posts:
    11
    That's great I'm looking forward to try that V1. By the way how is populated the sample size? It chooses from the current users or it's filled with new users only?
     
  16. ap-unity

    ap-unity

    Unity Technologies

    Joined:
    Aug 3, 2016
    Posts:
    1,049
    All current users will get the AB testing experience, though for retention outcomes, retention only refers to new users and whether they are retained.