Question Test runner fails when using gitlab-ci

Hi!

I've set up a CI system for my project with gitlab-ci, using this repository as a template. I run the following batch script for the tests:

Code (Batch):

1. "Unity.exe" ^
2.   -projectPath "%cd%" ^
3.   -batchmode ^
4.   -runTests ^
5.   -testPlatform editmode ^
6.   -testResults "%cd%\editmode-results.xml" ^
7.   -logfile testlog.log ^
8.   || goto :error
9.  
10. echo "Tests successful!""
11. exit /b 0
12.  
13. :error
14. echo Failed with error #%errorlevel%.
15. exit /b %errorlevel%

The CI runner is running on my own computer on the same user, and when I manually go to the runner directory and run the scripts they are successful. However, when gitlab-ci does it, it fails with error code 255. The terminal shows nothing except for the command and the "Failed with error #255" message, and the result xml shows all the tests passed both when running local and on gitlab-ci. The log files (testlog.log) are almost identical except for the gitlab-ci one having two lines like this:

Code (log):

1. Symbol file LoadedFromMemory doesn't match image D:\Gitlab-Runner\builds\e4Y8wNDA\1\algoryx\external\unity-project\AGMSim\Library\PackageCache\com.unity.ext.nunit@1.0.0\net35\unity-custom\nunit.framework.dll

and the local one having these lines at the end:

Code (log):

1. Checking for leaked weakptr:
2.   Found no leaked weakptrs.
3. ##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1569912322617,"processId":29908,"allocatedMemory":1522783,"memoryLabels":[{"Default":1292},{"NewDelete":311155},{"Thread":-161},{"Manager":728},{"VertexData":480},{"GfxDevice":400},{"Audio":192},{"Font":1043876},{"Physics":177},{"Serialization":40},{"Terrain":112},{"String":51695},{"DynamicArray":28676},{"HashMap":14006},{"Curl":2192},{"PoolAlloc":8},{"ScriptManager":8971},{"Sprites":8},{"GI":400},{"VR":3956},{"Secure":8},{"Image":32768},{"EditorGui":40},{"EditorUtility":556},{"AssetImporter":32},{"RestService":256},{"UnityConnect":20224},{"Collab":696}]}

Anyone have any ideas what this could be about or how to investigate further? Another thing worth mentioning is that I have another job that builds the application in a similar way using the Unity CLI and it always works fine with no error messages.

I'm attaching the two log files (I had to change the file extensions to be able to upload them).

Thanks in advance,
Martin

 

I would also try to add -quit and -nographics to the args u are using to launch Unity with.

 

Thanks for the answer!

Adding the -nographics flag made no difference, and adding the -quit flag makes it so the application always exits with exit status 0, even when tests fail, and no xml file is generated.

I also tried using the -runEditorTests flag instead of -runTests, but that didn't change anything either.

I think I'll avoid checking the return status and just try parsing the xml file somehow instead...

 

When running Unity.exe like that, does the process block until Unity completes or does it exit immediately?
Also, do u have a chance to test the same on a MacOS based machine? (if that is even supported by gitlab-CI)

 

It blocks, both when I run it manually from a command prompt, and when the CI runs it. The command takes about a minute since it needs to regenerate the Library folder.

I haven't tried it on a Mac machine, but I could give it a try.

 

any updates? were u able to resolve this issue ?

 

Yes and no. I did the workaround I mentioned above, writing a Python script that parses the test log xml file, and it works fine. Since it worked I haven't researched the issue further.

For reference, here is the Python script I'm using:

Code (Python):

1. import sys
2. import xml.etree.ElementTree as ET
3.  
4. tree = ET.parse(sys.argv[1])
5. root = tree.getroot()
6.  
7. num_failed = root.attrib["failed"]
8. num_passed = root.attrib["passed"]
9. num_inconclusive = root.attrib["inconclusive"]
10. num_skipped = root.attrib["skipped"]
11. print("Number of tests passed: " + num_passed)
12. print("Number of tests failed: " + num_failed)
13. print("Number of inconclusive tests: " + num_inconclusive)
14. print("Number of skipped tests: " + num_skipped)
15. if root.attrib["result"] == "Passed":
16.     sys.exit(0)
17. elif int(num_failed) > 0:
18.     sys.exit(int(num_failed))
19. else:
20.     sys.exit(-1)

 

This is my .gitlab-ci.yml

Code (JavaScript):

1. stages:
2.   - editmodetests
3.   - playmodetests
4.   - build
5.   - deploy
6. unit-edit-test:
7.   stage: editmodetests
8.   script: "C:\\'Program Files'\\Unity\\Hub\\Editor\\2019.3.0b4\\Editor\\Unity.exe \
9.  -batchmode \
10.  -projectPath . \
11.  -runTests -testPlatform editmode \
12.  -logFile \
13.  -testResults ./TestResults/edit-tests.xml"
14.   tags:
15.     - unity
16. unit-play-test:
17.   stage: playmodetests
18.   script: "C:\\'Program Files'\\Unity\\Hub\\Editor\\2019.3.0b4\\Editor\\Unity.exe \
19.  -batchmode \
20.  -projectPath . \
21.  -runTests -testPlatform playmode \
22.  -logFile \
23.  -testResults ./TestResults/play-tests.xml"
24.   tags:
25.     - unity
26. unity-build:
27.   stage: build
28.   script: echo 'Building...'
29.   tags:
30.     - unity
31. playstore:
32.   stage: deploy
33.   script: echo 'Deploying...'
34.   tags:
35.     - unity

It works fine

 

What is your environment? I cant run tests from gitlab on mac

 

Have another issue when run gitlab runner from git lab. When execute this command locally everything works ok.

Code (CSharp):

1. /Applications/Unity/Hub/Editor/2019.2.19f1/Unity.app/Contents/MacOS/Unity -batchmode -projectPath .  -runTests -testPlatform editmode -testResults ./editmode-unit-tests.xml
2. /Users/yaroslav/builds/mpA4z3YE/0/root/mouseSim
3. 2020-02-13 01:25:35.521 Unity[2402:29587] Color LCD preferred device: Intel Iris Pro Graphics (low power)
4. 2020-02-13 01:25:35.521 Unity[2402:29587] Metal devices available: 2
5. 2020-02-13 01:25:35.521 Unity[2402:29587] 0: Intel Iris Pro Graphics (low power)
6. 2020-02-13 01:25:35.521 Unity[2402:29587] 1: NVIDIA GeForce GT 750M (high power)
7. 2020-02-13 01:25:35.522 Unity[2402:29587] Using device NVIDIA GeForce GT 750M (high power)
8.  
9. Aborting batchmode due to failure:
10. Scripts have compiler errors.
11.  
12.  
13. Native stacktrace:
14.  
15.     0   libmonobdwgc-2.0.dylib              0x0000000142d46a22 mono_handle_native_crash + 242
16.     1   libmonobdwgc-2.0.dylib              0x0000000142ca7ce1 mono_sigsegv_signal_handler + 220
17.     2   libsystem_platform.dylib            0x00007fff69ab542d _sigtramp + 29
18.     3   ???                                 0x0000000000000000 0x0 + 0
19.     4   Unity                               0x00000001054b7af8 _ZN12AudioManager14systemCallbackEP11FMOD_SYSTEM24FMOD_SYSTEM_CALLBACKTYPEPvS3_ + 424
20.     5   Unity                               0x000000010964ebaf _ZN4FMOD6Thread8callbackEPv + 175
21.     6   libsystem_pthread.dylib             0x00007fff69ac0e65 _pthread_start + 148
22.     7   libsystem_pthread.dylib             0x00007fff69abc83b thread_start + 15
23.  
24. Debug info from gdb:
25.  
26. (lldb) command source -s 0 '/tmp/mono-gdb-commands.aHzSb7'
27. Executing commands in '/tmp/mono-gdb-commands.aHzSb7'.
28. (lldb) process attach --pid 2402
29. error: attach failed: Error 1
30. ERROR: Job failed: exit status 1

 

Did you find a fix for this? I'm getting the exact same stacktrace when building for Android on GitLab with Unity 2019.2.21 (No tests running, and iOS builds from the same project does not have any issues).