Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Question Setting up in other platforms

Discussion in 'Testing & Automation' started by Kalita2127, Jul 4, 2023.

  1. Kalita2127

    Kalita2127

    Joined:
    Dec 6, 2014
    Posts:
    279
    So finally I finished my unit test codes and I want to build to a specific platform. I'm pretty new in this unit test things. I found hat UnitySetUp is only called in the editor and that make me sad. That means my setup won't run in other platforms (let's say WebGL and android). I heard that you can make use of SetUp but it doesn't allow me to run couroutine. Is there any alternatives to run coroutine setup for a specific platform?
     
  2. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,909
    This should run in playmode. But keep in mind there are two types of test scripts: one for playmode, and one for editmode. The edit mode tests don't run in builds, the play mode tests do. What is a play or edit mode test is determined by the Assembly Definition file that the script(s) belong to.

    If you need a coroutine "setup" function you are definitely on the wrong track. There should never be any need to do any coroutine (or async) task during setup or shutdown. The test itself can run as a coroutine by using the [UnityTest] attribute.
     
  3. Kalita2127

    Kalita2127

    Joined:
    Dec 6, 2014
    Posts:
    279
    Seriously? So my setup and tear down is wrong? I'm testing on an app that requires waiting most of the time. Such as waiting player to connecting to the server and waiting for cleaning up session. I put the waiting functions on UnitySetup and UnityTearDown :')
     
  4. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,909
    That's perhaps something you should NOT test. ;)

    The same way one would not test a database (ignore exceptions). Simply because it takes too long to make a connection and clean it up. You want each test method to run within fractions of a second, and all tests within a few seconds. If you need to establish a network connection, you are by definition not perfoming unit tests (test the smallest possible unit of code) but integration tests (test that modules/frameworks work correctly together). The latter are complex, time-consuming tests you only run infrequently. Unit tests you should run while coding, just as you compile code you should be able to run tests alongside.

    If you were to connect and disconnect for each test method, this would take far too long overall. If you intend to keep the connection alive (sounds like it) then you risk introducing side effects into your tests because the tests now rely on the sequence of state changes as the tests run one after another. This is a "test smell"! The order of test execution cannot be taken for granted, nor should you write a "reset" method after every test because maintaining this is time consuming and will introduce errors of its own.

    Btw, the fact that you can connect and send data over the network needs no tests because this you can rely on when using a proven framework with tests of its own (eg Netcode). Whether a client state change is correctly reflected on the server and other clients needn't be tested - take this for granted!

    What you DO want to test is whether a client performs the right thing locally and DOES issue a RPC or variable change. Also separately test whether the server does the right thing receiving the RPC or state change. That's enough. You will have a ClientDoesX and a ServerDoesX test for each testable network state change instead.

    You may want to look into stub or mock classes, so your tests can hook into network state changes respectively simulate them. Basically you will want to set it up such that networked RPC calls and network variable changes are routed through a class that tests can hook into, either by subclassing or by writing against an interface so the tests can write a test class implementing that interface (but not actually sending the state, merely recording the method call).

    Example: Client presses a button => client should jump.
    • Client tests that it sends MakeMeJumpServerRPC() by providing a mock class that registers the call, and test asserts this: Assert.True(mockClass.DidCallJumpRPC)
    • Server tests that when receiving MakeMeJumpServerRPC it runs ConfirmJumpClientRPC() also with mock classes. It may additionally test that it does the right thing when receiving invalid data (ie negative or too high jump velocity) and that it does the right thing when the client is currently unable to jump (ie client in mid-air, client dead, etc)
     
    Last edited: Jul 4, 2023
  5. Kalita2127

    Kalita2127

    Joined:
    Dec 6, 2014
    Posts:
    279
    It's actually the set up method like this

    Code (CSharp):
    1.  
    2.         [UnitySetUp]
    3.         public IEnumerator LoadScene()
    4.         {
    5.             yield return null;
    6.  
    7.             UtilTestMethods.ForceGuestMode();
    8.  
    9.             yield return UtilTestMethods.Until(() => AppMain.State == EAppState.LOGIN && SceneManager.GetActiveScene().name == "Login");
    10.  
    11.             var loginView = GameObject.FindObjectOfType<LoginView>();
    12.  
    13.             var guestBtn = loginView.GetPrivateField<Button>("_skipButton");
    14.  
    15.             guestBtn.ExecuteUIClick(pointerData);
    16.  
    17.             yield return UtilTestMethods.Until(() => AppMain.State == EAppState.SELECT_AVATAR && SceneManager.GetActiveScene().name == "SelectAvatar");
    18.             yield return new WaitForSeconds(UtilTestMethods.NORMAL_TEST_DURATION);
    19.  
    20.             AppMain.DoneSelectAvatar();
    21.  
    22.             yield return UtilTestMethods.Until(() => AppMain.State == EAppState.IN_HUB && SceneManager.GetActiveScene().name == "Sky_City_Hub");
    23.             //yield return UtilTestMethods.Until(() => LoadingBuilder.Instance == null, 5f);
    24.             yield return new WaitForSeconds(UtilTestMethods.NORMAL_TEST_DURATION);
    25.         }
    26.  
    Basically the idea is waiting until the initialization is done. When the game is loaded then it runs the tests in order.
    While the tear down method is like this
    Code (CSharp):
    1.  
    2.         [UnityTearDown]
    3.         public IEnumerator UnloadScene()
    4.         {
    5.             UtilTestMethods.FixInvalidOperationExceptionInPlayerVoice();
    6.  
    7.             if (NetworkManager.CurrentSession != null)
    8.             {
    9.                 NetworkManager.Disconnect();
    10.             }
    11.  
    12.             yield return UtilTestMethods.Until(() => NetworkManager.Runner == null, 10);
    13.  
    14.             AppMain.State = EAppState.LOGIN;
    15.             //yield return UtilTestMethods.CleanInitTestScenes();
    16.         }
    17.  
    Pretty self explanatory I think.
    Aanndd yupp this is what I did! :') What should I do to do the set up correctly? :') Looks like I'm lost too far
    This is interesting.I've heard it before and not sure what's that for. Does unity have such mock things? Or can you recommend something to get it started?
    Thank you so much for your answer!
     
  6. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,909
    Okay so you definitely have an integration test there. You even rely on the GUI and specific scenes identified by string and if I get this right, you're actually moving the "player" through the whole flow of creating an avatar, connecting and lobbying and starting the game!

    You aren't going to maintain this for long because it's brittle and depends a lot on the whole scene hierarchy. It will break often, not because the test fails but because the project changes. That serves little purpose.

    You could make this one (!) integration test that you have now that does all that to confirm that the basics of connection and so on are working. But ... you will test this anyway with a manual test before an actual release thus there's really little to win by automating this flow.

    You can make this work however by calling the LoadScene and UnloadScene methods at the start / end of each test method, then you can use the yield statement and you would have a fully functioning integration test.

    But the gist of unit testing is to test the smallest possible "unit" of code. Like a method that computes a value. Given input X confirm that output is Y. That's a unit test. Or in terms of Unity: Given player is grounded and presses jump button confirm that player is no longer grounded in the next frame. As well as: given player is not grounded with downward velocity Y == 0 and presses jump button confirm that player's velocity Y remains equal or less than 0.

    Unity does not have a Mock framework but there are some that can be used. Here's a discussion about mocking. Personally, I would recommend to first make mocks manually so you understand what it takes to write them - it ain't much. And then you can consider whether it's actually worth installing a mock framework that may have a clunky API that may make tests less readable. Personally, I stay away from them as much as possible.
     
  7. Kalita2127

    Kalita2127

    Joined:
    Dec 6, 2014
    Posts:
    279
    First of all I'm not a native english speaker so I might not understand what you wanted to say to me. But if I understands you correctly, you suggested me to skipped the authentication flow and just go to the game scene then do the test there. Now I got confused because to be able to load the game scene I must login, go to avatar scene to customize, then go to the game scene. I also have a test if the avatar that we customized in avatar customization is also similar in the game scene(please tell me this is also unnecessary).
    Let's say "I want to test if the player can jump by using the space key"
    a.) Login first -> wait -> avatar creation -> wait -> then in the game scene simulate input test with InputTestFixture and assert if the player y position is greater than the origin position
    b.) Search straight for the jump function -> copy the code to our unit test function -> assert if the player position is greater than the origin position
    c.) Shouldn't be part of unit test
    I'm sorry but I don't understand what you meant by mocks manually. Can you give me some little code example if possible please?
     
  8. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,909
    You have a player (Avatar), right?
    That player has a character controller on it, right?
    If you drop that player (Avatar) in an empty scene, will it work without errors? Will it be controllable as usual? I hope so.

    If not => fix that! This is what unit testing forces you to do: write the code with the least amount of dependencies. I would absolutely expect that a character controller in any game will work in an empty scene.

    Now once you have the character controller working in ANY environment, you can skip the whole "go through entire menu and join network" part because you do not need it! You can just test whether the character controller can jump in an empty scene. A separate test can check if the jump command also issued the expected ServerRPC.

    The goal of unit testing is exactly to avoid a situation that you're currently in: that in order to test code, you have to load the ENTIRE system and even progress through various states until you get to the point where you can test the code. In that case, manual tests could be just as fast and effective.

    If you want to use unit testing, make sure you CAN actually test the smallest unit of code independently from everything else. To achieve that, you would normally begin writing the tests first, and then implement what the test is trying to test. This is called "test first" in test-driven development.

    Pointers:
    http://www.extremeprogramming.org/rules/testfirst.html
    https://medium.com/@tasdikrahman/f-i-r-s-t-principles-of-testing-1a497acda8d6

    One simple example would be to have an abstract base class ServerMessages that contains all ServerRPC messages as abstract methods. While in the game, a concrete subclass would implement these methods and perform the server methods. Whereas during testing you would have a "mock" implementation, also a subclass that implements all of these methods except that class merely increments an int field "MethodXyzCallCount" whenever the method gets called. The test assertions can then check if the call count is 1. How you "inject" this mock class into the player instead of the regular networked implementation is up to you. I would probably assign dependencies right after creating the object - in the game it would be the network implementation, in a test it would be the mock class.
     
    Kalita2127 likes this.
  9. Kalita2127

    Kalita2127

    Joined:
    Dec 6, 2014
    Posts:
    279
    All I can say is THANKS A LOT sir for enlighten me to the right path! Now I'm started to understand what's the unit test purposes is. It's not to test a feature, instead it used to test whether a single function would expect the desired result or not in an isolated environment. Am I right? So the one I wrote was an integration test not unit test?
    Does unit test in software development also divided into two test modes? Or in just unity only? Because from what you explained I started to see that maybe we rarely use play mode. And also why we need to build our unit tests in play mode?
    upload_2023-7-5_12-57-51.png
     
    CodeSmile likes this.
  10. Kalita2127

    Kalita2127

    Joined:
    Dec 6, 2014
    Posts:
    279
    For unity teams out there, if you see this thread please fix all of the broken links here. I feel that the sample provided there is very useful but I just couldn't opened it :(
     
  11. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,909
    That was totally an integration test, if not more. ;)

    Yes, that's what a unit test is: test a small unit of code to ensure it does the right thing. And that it does the right thing even when you feed it the wrong things, such as throwing an IndexOutOfBoundsException when accessing a negative index in a collection.

    Also, TDD as a methodology really helps develop code that is modular, decoupled. Adding tests after code was written is always way harder than writing tests first, then naturally writing the code so it is testable. Because code written without tests doesn't care that it needs to be testable, and thus programmers tend to add more dependencies and coupling their code to systems that may not or cannot exist during tests, such as (for example) external APIs like Steam or Firebase.

    No, this edit vs playmode testing is Unity-specific. Personally I test as much as possible using editor tests because it's faster to run them. Downside is: you cannot test them on a device. So for things like serialization that I need to ensure works correct on all platforms I write playmode tests.

    Playmode tests are also the ones that let you yield an IEnumerator. The editor tests also support that but yielding is not guaranteed to advance to the next frame in editor mode - you may have to yield hundreds of times in order for Time.frameCount to increment in the editor. Just something to be aware of when yielding in editor tests.
     
    Kalita2127 likes this.
  12. Kalita2127

    Kalita2127

    Joined:
    Dec 6, 2014
    Posts:
    279
    THANK YOU SO MUCH SIR! I feels like I'm nearly on the right path! Like really close! Now I got them in mind and not afraid to get lost for the second time. Again, thank you so much for your time sir! :D