Search Unity

Batch mode tests fail

Discussion in 'Testing & Automation' started by ST_ProductVIz, Jun 4, 2020.

  1. ST_ProductVIz

    ST_ProductVIz

    Joined:
    Nov 29, 2017
    Posts:
    23
    Hi,

    I'm running the command

    Unity.exe -batchmode -projectPath {pathToProj} -buildTarget standalone -logfile {pathToLog} -runTests -testResults {pathToXml} -testPlatform PlayMode -nographics

    from within an Azure pipeline's task and it always fails with exit code 2 if any of the tests fail. If all tests pass, the command terminates successfully with exit code 0. Unfortunately, no error or warning occurs in the log file. Untiy version is 2019.3.15f1

    Is it by design that running the tests in Unity batch mode should fail if any of the tests fail?
     
  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    Yes, the exit code behaviour is by design. Are you saying you are seeing the Editor exit with a nonzero exit code but none of the test results indicate any actual failure?
     
  3. ST_ProductVIz

    ST_ProductVIz

    Joined:
    Nov 29, 2017
    Posts:
    23
    Hi superpig,

    observed behaviour is:

    0 tests fail --> exit code 0
    1 or more tests fail --> exit code 2

    So if I understand it correctly, this is the expected behaviour?
     
  4. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    Yes, that's expected. You should get 0 for a successful run, 2 for a failed run, 3 for a run error (like if we weren't able to start the tests at all due to a compile failure) and 4 for 'platform not found' (e.g. asking to run Android tests when there's no Android platform support installed).

    You can see these return codes in the package source code, in com.unity.test-framework/UnityEditor.TestRunner/CommandLineTest/Executer.cs
     
  5. ST_ProductVIz

    ST_ProductVIz

    Joined:
    Nov 29, 2017
    Posts:
    23
    Got it, thanks!
     
unityunity