Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only.

    Please, do not make any changes to your username or email addresses at id.unity.com during this transition time.

    It's still possible to reply to existing private message conversations during the migration, but any new replies you post will be missing after the main migration is complete. We'll do our best to migrate these messages in a follow-up step.

    On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live.


    Read our full announcement for more information and let us know if you have any questions.

Resolved How to use UnityEngine classes in command line test runners?

Discussion in 'Testing & Automation' started by eppz, Jul 10, 2017.

  1. eppz

    eppz

    Joined:
    Aug 2, 2014
    Posts:
    172
    I'm making C# libraries for Unity, doing TDD. Wrote some unit tests, currently running with NUnit directly (both in VS Code and Travis CI).

    I can hook up UnityEngine.dll, it compiles well. But when I run the tests, it can't find the methods during the test runtime.

    Simple methods like Debug.Log, or JsonUtility.FromJson all fails with MissingMethodException:
    Code (CSharp):
    1. System.MissingMethodException: Attempted to access a missing method.
    2.   at (wrapper managed-to-native) UnityEngine.JsonUtility:FromJson (string,System.Type)
    3.   at UnityEngine.JsonUtility.FromJson[T] (System.String json) [0x00001] in <ec28d45c6481407a9313eee207a323b3>:0
    As I looked into UnityEngine.dll, I found that most of the methods are falling back to an external implementation, like JsonUtility.FromJson:
    Code (CSharp):
    1. [GeneratedByOldBindingsGenerator, ThreadAndSerializationSafe]
    2. [MethodImpl(MethodImplOptions.InternalCall)]
    3. public static extern object FromJson(string json, Type type);
    (from https://github.com/MattRix/UnityDec...02/UnityEngine/UnityEngine/JsonUtility.cs#L25)

    But I'd like to know how to provide those external implementations in a test runner.

    Or am I just referencing the wrong dlls?
    I'm using the ones located in /Applications/Unity/Unity.app/Contents/Managed.
    But there are 7 other UnityEngine.dll files in the Unity.app bundle.

    ---

    As an example, here is a typical setup I use (this one without UnityEngine calls):
    Travis setup: https://github.com/eppz/Unity.Library.eppz.Extensions/blob/master/.travis.yml
    Project references: https://github.com/eppz/Unity.Library.eppz.Extensions/blob/master/Test.csproj
     
    Last edited: Jul 10, 2017
  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,670
    They're directly wired into the Mono runtime (hence 'InternalCall') via the Mono embedding API. You'd need to be running your tests under a Mono runtime that wire up those InternalCalls - or just use the Mono runtime launched by Unity itself, by having Travis etc launch Unity and ask it to run your tests.
     
    eppz likes this.
  3. eppz

    eppz

    Joined:
    Aug 2, 2014
    Posts:
    172
    Thanks!

    Currently I'm running NUnit using mono, like:
    Code (CSharp):
    1. mono /Users/eppz/Projects/Unity/Packages/NUnit/NUnit.ConsoleRunner.3.6.1/tools/nunit3-console.exe bin/EPPZ.Extensions.Test.dll
    Can I provide / link UnityEngine.dll somehow to this process?
     
    Last edited: Jul 10, 2017
  4. eppz

    eppz

    Joined:
    Aug 2, 2014
    Posts:
    172
    I tried to use countless NUnit shell scripts I found in the Unity.app package, like...

    Code (CSharp):
    1. /Applications/Unity/Unity.app/Contents/Mono/bin/nunit-console
    2. /Applications/Unity/Unity.app/Contents/Mono/bin/nunit-console2
    3. /Applications/Unity/Unity.app/Contents/MonoBleedingEdge/bin/nunit-console
    4. /Applications/Unity/Unity.app/Contents/MonoBleedingEdge/bin/nunit-console2
    ...but they are all throws the same MissingMethodException.
    What Unity uses to run tests in the Editor?

    ---

    Why I ask?

    I just want to avoid, like... ...switching back to Unity Editor every time I do some miiiiinnnor change in a library, then wait the entire project to be rebuilt, then run tests in Editor, then resolve a line number from the Inspector, then switch back to VS code, then lookup the error.

    Compared to: run an NUnit task on a small library dll I just built, right in VS Code, within seconds.
     
  5. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,670
    You would need to do your own build of Mono.

    Take a look at, for example, mono_marshal_init in marshal.c, line 195. That big sequence of 'register_icall` calls there is how the InternalCalls are registered with the Mono runtime.

    At engine/editor startup, Unity's C++ code makes calls similar to `register_icall` to wire up the InternalCalls you're seeing in UnityEngine.dll. So when we're executing managed code against our Mono instance, it has all the calls registered.

    To my knowledge, there is no way to dynamically register an icall from managed code at runtime.
     
    eppz likes this.
  6. eppz

    eppz

    Joined:
    Aug 2, 2014
    Posts:
    172
    I there any chance to access the running mono instance of Unity Editor from command line? Having that I could make a VS Code task, by which I could run the tests within.

    ---

    (on Travis CI speed is not a big concern, so I could go with using Unity.app command line directly there)
     
    Last edited: Jul 11, 2017
  7. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,670
    The Mono instance used by Unity is embedded inside the process - it is not a separate executable that can be launched independently. So yeah, you will need to launch Unity.app. We do have command-line parameters for activating the test runner, though.
     
    eppz likes this.
  8. eppz

    eppz

    Joined:
    Aug 2, 2014
    Posts:
    172
    Thanks, I came to the same conclusion.
    Made a shell script that runs tests in batch mode.

    Code (CSharp):
    1. #!/bin/sh
    2.  
    3. UNITY_EXECUTABLE="/Applications/Unity/Unity.app/Contents/MacOS/Unity"
    4. PROJECT_FOLDER="/Users/eppz/Projects/Unity/Prototype/eppz"
    5. RESULTS_FILENAME="eppz.Test.Result.xml"
    6.  
    7. echo "Run Unity.app (in batch mode) with editor test..."
    8.  
    9. time \
    10. "$UNITY_EXECUTABLE" \
    11. -batchmode \
    12. -projectPath "$PROJECT_FOLDER" \
    13. -runTests \
    14. -testResults "$PROJECT_FOLDER/$RESULTS_FILENAME" \
    15. -testPlatform editmode
    I had difficulties while trying in 5.5.0, but it works nicely with 5.6.2f1.
    It runs for ~18 sec even for a single test, but does the work (I can cut down 3 sec with -nographics).

    Code (CSharp):
    1. Run Unity.app (in batch mode) with editor test...
    2.  
    3. real    0m18.193s
    4. user    0m15.830s
    5. sys    0m3.390s
    Thanks again for pointing me to the right directions!
    (and sorry for somewhat fragment your vacation)
     
    Last edited: Jul 11, 2017
    superpig likes this.
  9. eppz

    eppz

    Joined:
    Aug 2, 2014
    Posts:
    172
    Aw, this one actually prevents me to use this with VS Code (while the project is open).

    "Opening a project in batch mode while the Editor has the same project open is not supported; only a single instance of Unity can run at a time." - from https://docs.unity3d.com/Manual/CommandLineArguments.html