Search Unity

  1. Unity 2019.1 beta is now available.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. We're looking for insight from anyone who has experience with game testing to help us better Unity. Take our survey here. If chosen to participate you'll be entered into a sweepstake to win an Amazon gift card.
    Dismiss Notice
  4. Want to provide direct feedback to the Unity team? Join the Unity Advisory Panel.
    Dismiss Notice
  5. Unity 2018.3 is now released.
    Dismiss Notice
  6. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice

Are PlayMode Test running in parallel on the same Scene?

Discussion in 'Scripting' started by Netbandit, Sep 27, 2018.

  1. Netbandit

    Netbandit

    Joined:
    Feb 16, 2014
    Posts:
    18
    Hi

    I'm a big fan of TDD, thus I have a lot of tests in my current project. Some of those tests are PlayMode Tests, which are executed by the player inside a scene.

    There are several of those tests, which create an GameObject with the name "MyObject" and start working on that object to test some things. Sometimes the object is deleted an recreated and so on.

    Now if every test-class in running alone, everything works perfectly. But if I start all my tests with "Run All" many tests which use the GameObject "MyObject" suddenly fail. For example the object is there, while it should not be there. Or the object is missing, when it should be there.

    I have the feeling, that the PlayMode tests are not well isolated from each other. It seems that every test-class in started in parallel on the same Scene, which would explain those race-conditions. But I was not able to find a detailed description how PlayMode tests are executed. That's why my questions:

    1.) Are the PlayMode Test classes executed in parallel?
    2.) Are they using all the same temporal scene (also in parallel)?
    3.) Is there any way to isolate the tests from each-other?
    4.) What happens when any of those tests change the scene? Will the scene be changed also for other tests?

    In fact I found out, that exactly this is the case, for tests in the same test-class. If any of those tests load a different scene and set it as active, every other test from the same test-class, which is executed later, will working in this scene instead of the temporal scene. But does this also apply for tests of different test-classes, which seems to be executed in parallel?
     
  2. chrisse27

    chrisse27

    Joined:
    Apr 20, 2016
    Posts:
    1
    Hi Netbandit! We're experiencing the same, but we also haven't found a way around this.
     
  3. Netbandit

    Netbandit

    Joined:
    Feb 16, 2014
    Posts:
    18
    Hi Chrisse27,

    my current workaround is to create an own Scene for every Test-Class, so that there is no Test-Class which is working in parallel on the same Scene. And inside the Test-Cases (methods of this Test-Class), I simply reload this scene in the beginning, so that all settings are reset to default for every new test-case.

    It is not nice and those tests take some time longer, but that was the only solution I found, which worked for me.
    I hope Unity will change this at any time.

    Best regards,
    Netbandit
     
  4. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    1,164
    I have a similar issue, which I ended up reporting as a bug: https://fogbugz.unity3d.com/default.asp?1103945_6gcb34pn4tqnmgbv

    The issue is that SetUp and TearDown methods definitely don't achieve the kind of isolation you'd expect. A very simple example occurs when scripts have some OnDestroy functionality. In those cases, if you run two tests in one batch, then the OnDestroy for objects loaded by the first test will occur after the first test's TearDown, and actually occur after the second test's SetUp is called.

    What this means: I have some tests that will 100% fail when run together, but which will 100% succeed if run one at a time.

    I also have no workaround for this, other than to tolerate the failure of a few tests, which I know will succeed when I Rerun Failed Tests. I tolerate this because A) I hope Unity fixes this at some point, and B) I don't have an automated build process to complain to me about build failures. If either of those changes, I'll look harder at a solution.

    Regarding your own case, I don't believe there's any parallelism about Play Mode tests. And with every test I run, I load the scene again. I've never had an issue where a gameobject created/changed in one test affects a gameobject in another test, except for DontDestroyOnLoad game objects, which I believe persist between tests when running tests in a batch.
     
  5. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    2,867
    Sorry for derailing the thread, but it's a bad idea to share fogbugz URLs publicly, you expose all of your submitted bugs to the public, including past and future ones which may or may not contain sensitive information. (of course if you're sure it's not a concern of yours then sorry for the intermission)
     
    Last edited: Dec 17, 2018
    dgoyette likes this.
  6. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    81
    Are you sure this still happen if you effectivelly load a clear new scene at startup AND teardown? This is what I had to do in order to isolate my tests