Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug PlayMode Tests marked with [ValueSource] attribute fail to appear in Unity Cloud Build "Tests" tab.

Discussion in 'Unity Build Automation' started by Meatloaf4, Mar 15, 2023.

  1. Meatloaf4

    Meatloaf4

    Joined:
    Jul 30, 2013
    Posts:
    183
    Description of issue:
    PlayMode Tests marked with [ValueSource] attribute fail to appear in Unity Cloud Build "Tests" tab. Upon downloading the "Unit Tests (Play Mode)" file it appears they are in fact run, they just don't display under the "PlayMode Tests" section.

    Specifically in my case the tests that are not being displayed have the following signatures.
    Code (CSharp):
    1.    
    2. [UnityTest]
    3. public IEnumerator LevelDeterminismFreshLoadTest([ValueSource(nameof(LevelsToTest))] LevelData levelData,
    4.         [ValueSource(nameof(DefaultFrameTestBounds))] Vector2Int frameTestBounds,
    5.         [ValueSource(nameof(DefaultTimeToTests))] int timeToTest,
    6.         [ValueSource(nameof(PostLoadLevelWaitTimes))] float postLoadLevelWaitTimes)
    7.  
    8. [UnityTest]
    9. public IEnumerator LevelDeterminismRestartTest([ValueSource(nameof(LevelsToTest))] LevelData levelData,
    10.     [ValueSource(nameof(DefaultFrameTestBounds))] Vector2Int frameTestBounds,
    11.     [ValueSource(nameof(DefaultTimeToTests))] int timeToTest,
    12.     [ValueSource(nameof(PostLoadLevelWaitTimes))] float postLoadLevelWaitTimes)
    13.  
    14.  
    15. [UnityTest]
    16. public IEnumerator LevelConfigurationTest([ValueSource(nameof(LevelsToTest))] LevelData levelData)
    17.  
    Unity version:
    2020.3.12f1

    XCode version:
    Xcode 13.3.1

    Source control:
    git

    Local build platform:
    Windows? However the issue is specific to Unity Cloud Build dashboard.

    Build target:
    Android & iOS

    Relevant plugins being used (i.e. Google Play Game Services, other plugins that may perform automatic post-build behaviors)
    None

    Project name:
    Command Center Earth

    Build log (relevant chunk):
    Image below and a test-case from the play mode test file provided. I have excluded including test case file specifically out of caution for any sensitive data that could be contained in it.

    Code (csharp):
    1.  
    2. <test-case id="1071" name="LevelDeterminismFreshLoadTest(Simon (LevelData),(5, 60),4,0.0f)" fullname="LevelDeterminismPlayModeTests.LevelDeterminismFreshLoadTest(Simon (LevelData),(5, 60),4,0.0f)" methodname="LevelDeterminismFreshLoadTest" classname="LevelDeterminismPlayModeTests" runstate="Runnable" seed="1714533172" result="Passed" start-time="2023-03-15 00:49:19Z" end-time="2023-03-15 00:49:42Z" duration="23.296535" asserts="0">
    3.             <properties />
    4.             <output><![CDATA[<b>(Variable Log)</b>: Dynamic Parameter Variable '_levelStackCc' Encountered...
    5. <b>(Variable Log)</b>: Parameter '_levelStackCc' (of type 'StackCanvasController') promoted to a Variable in Blackboard 'Graph'.
    6. <b>(Variable Log)</b>: Dynamic Parameter Variable '_isPostLevelCCIntroTransitionAnimated' Encountered...
    7. <b>(Variable Log)</b>: Parameter '_isPostLevelCCIntroTransitionAnimated' (of type 'Boolean') promoted to a Variable in Blackboard 'Graph'.
    8. <b>(Variable Log)</b>: Dynamic Parameter Variable '_isPlayerWinner' Encountered...
    9. <b>(Variable Log)</b>: Parameter '_isPlayerWinner' (of type 'Boolean') promoted to a Variable in Blackboard 'Graph'.
    10. <b>(Variable Log)</b>: Dynamic Parameter Variable '_levelStackCc' Encountered...
    11. <b>(Variable Log)</b>: Parameter '_levelStackCc' (of type 'StackCanvasController') promoted to a Variable in Blackboard 'Graph'.
    12. <b>(Variable Log)</b>: Dynamic Parameter Variable '_isPostLevelCCIntroTransitionAnimated' Encountered...
    13. <b>(Variable Log)</b>: Parameter '_isPostLevelCCIntroTransitionAnimated' (of type 'Boolean') promoted to a Variable in Blackboard 'Graph'.
    14. <b>(Variable Log)</b>: Dynamic Parameter Variable '_isPlayerWinner' Encountered...
    15. <b>(Variable Log)</b>: Parameter '_isPlayerWinner' (of type 'Boolean') promoted to a Variable in Blackboard 'Graph'.
    16. ]]></output>
    17.           </test-case>
    18.  

    Does it build locally from a fresh checkout? (Instructions below):
    Yes
     

    Attached Files:

  2. Benjamin-Gooding

    Benjamin-Gooding

    Unity Technologies

    Joined:
    Apr 12, 2022
    Posts:
    290
    I took a look at your project and was able to see the issue. I've assigned a bug to one our web developers to resolve the issue with rendering the test results.
     
    Meatloaf4 likes this.
  3. Meatloaf4

    Meatloaf4

    Joined:
    Jul 30, 2013
    Posts:
    183
  4. Meatloaf4

    Meatloaf4

    Joined:
    Jul 30, 2013
    Posts:
    183
    Hey guys, I'm still seeing this is an issue on Cloud Build and it's kind of a big pain for us. Just curious if it's gained any traction.

    Thanks in advance!
     
  5. melinda_unity

    melinda_unity

    Unity Technologies

    Joined:
    Aug 9, 2022
    Posts:
    2
    Hi @Meatloaf4 ,

    Thanks for reaching out about this issue. We really appreciate your patience and for taking the time to bring it to our attention. We know it's been a pain for you, and we're sorry about that.

    The good news is that we've looked into the issue and fixed it. You should be able to see all your tests on the "Tests" tab on the Build History Details page now.

    Once again, thanks for letting us know about this. Your feedback is super important to us, and it helps us make our products better. If you have any other concerns, don't hesitate to give us a shout.

    Thanks,
    Melinda
     
    Meatloaf4 likes this.
  6. Meatloaf4

    Meatloaf4

    Joined:
    Jul 30, 2013
    Posts:
    183
    Ah this is great news! Thank you so much for fixing that issue.

    I also went ahead and checked that the fix works retroactively, and it does! Thanks so much!