Search Unity

  1. Calling all beginners! Join the FPS Beginners Mods Challenge until December 13.
    Dismiss Notice
  2. It's Cyber Week at the Asset Store!
    Dismiss Notice
FPS Beginners Mods Challenge Cyber Week Mega Sale

Are C# Expression Trees (or ILGenerator) allowed on iOS

Discussion in 'Experimental Scripting Previews' started by lluo, Aug 23, 2017.

  1. lluo

    lluo

    Joined:
    May 19, 2016
    Posts:
    20
    So after switching to the 2017.1 experimental .NET 4.6 profile, I could finally leverage the full power of C# expression trees in my serialization framework. However, a good question arises: are C# expression trees (and thus ILGenerator) considered dynamic code generation by iOS, and thus not really allowed for iOS publishing?

    Anyone has any insights into this?

    Cheers!
     
    lluo, Aug 23, 2017
    #1

  2. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    527
    as far as I know, everything that does dynamic code generation will either not compile or throw an exception on IL2CPP. you can test it on a build and see.

    I don't know if there is something that IL2CPP allows but is forbidden by Apple policies (short of writing your own bytecode interpreter)
     
    M_R, Aug 23, 2017
    #2

  3. DenisZykov

    DenisZykov

    Joined:
    Mar 31, 2016
    Posts:
    19
    You can parse, compile and execute expression trees on iOS with this asset using .CompileAot() extension on System.Linq.Expression.Expression<T> instance.
     
    Last edited: Aug 23, 2017
    DenisZykov, Aug 23, 2017
    #3

  4. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    The .NET 4.6 profile doesn't change the IL2CPP (and Apple restrictions). It is still an ahead-of-time compiler which does not support dynamic code generation. Basically, you cannot use anything in the System.Reflection.Emit namespace.
     
    JoshPeterson, Aug 23, 2017
    #4

  5. DenisZykov

    DenisZykov

    Joined:
    Mar 31, 2016
    Posts:
    19
    Actually Unity team could implement IL interpreter for AOT environment. It would be in 'gray' area of Apple policy.
     
    DenisZykov, Aug 23, 2017
    #5
    dCalle likes this.

  6. Arkade

    Arkade

    Joined:
    Oct 11, 2012
    Posts:
    596
    Hey, link just goes to assert store front page for me...?
     
    Arkade, Aug 23, 2017
    #6

  7. DenisZykov

    DenisZykov

    Joined:
    Mar 31, 2016
    Posts:
    19
    It is strange. I just copied it from browser address bar.
    http://u3d.as/oCH <- link in old Asset Store.
     
    DenisZykov, Aug 23, 2017
    #7
    Arkade likes this.

  8. lluo

    lluo

    Joined:
    May 19, 2016
    Posts:
    20
    How about things under System.Linq.Expressions only? I might be a little misleading, in that I have no explicit usage of anything under the System.Reflection.Emit namespace, I exclusively use the Expression Trees (System.Linq.Expressions); but I somehow got the impression that under the hood, Expression Trees actually generates IL code during the "Compile()" phase?

    So just to make the question exactly lean and mean: are C# Expression Trees (System.Linq.Expressions) allowed for iOS?
     
    lluo, Aug 23, 2017
    #8

  9. lluo

    lluo

    Joined:
    May 19, 2016
    Posts:
    20
    Hmm, I've actually got an idea: since my usage case doesn't really involve "dynamic code generation", even though the Expression Tree is used (with reflection) to inspect existing types and generate "logic" to process the data with the known types, it might actually be possible for me to take the compiled expression trees and create an assembly in a separate process (sort of a AOT process), and at the publishing or runtime, the generated assembly is shipped (or loaded at runtime), so there would be no runtime code generation whatsoever.

    The bits I'm looking now is: LambdaExpression.CompileToMethod based on this SO post: https://stackoverflow.com/questions...the-main-entry-point-to-a-new-executable-disk

    Please share your insights on this. Thanks again!
     
    lluo, Aug 23, 2017
    #9
    Arkade likes this.

  10. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    I don't think this will work. That example on Stack Overflow is using AssemblyBuilder, which is in System.Reflection.Emit. However to be sure, your best option is to try it in a small test project and see. We will do what we can to support dynamic code, as long as it does not violate an App Store restrictions.
     
    JoshPeterson, Aug 24, 2017
    #10
    VOTRUBEC and dCalle like this.

  11. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    527
    that can be done in the editor though. I have something similar for protobuf.net: I generate the DLL from my data classes via an editor MenuItem and put it into the project.
     
    M_R, Aug 24, 2017
    #11
    Ethan_VisualVocal and lluo like this.

  12. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    Indeed, the editor uses Mono JIT, so runtime code generation is fully supported.
     
    JoshPeterson, Aug 24, 2017
    #12

  13. lluo

    lluo

    Joined:
    May 19, 2016
    Posts:
    20
    Exactly as M_R suggested, the assembly generation would be done in the editor (a separate phase), and the generated assembly would be put in the Unity project, and gets referenced from there.
     
    lluo, Aug 24, 2017
    #13

  14. lluo

    lluo

    Joined:
    May 19, 2016
    Posts:
    20
    A follow up question: does IL2CPP automatically process an (Unity compatible) assembly that resides in the current Unity project (e.g. under "Assets/DLL" folder, so it looks like: "Assets/DLL/MyAssembly.dll")? Does IL2CPP work against "Assembly.Load(AssemblyName("MyAssembly"))"?

    Note that I'm aware that Assembly.LoadFrom(assemblyFilePath) would not work (see: https://forum.unity3d.com/threads/il2cpp-loading-c-assembly-at-runtime.417553/), but Assembly.Load(AssemblyName) seems only needing to find a "reference", and I'm assuming it should work with IL2CPP (or with an IL2CPP processed assembly)?
     
    lluo, Aug 25, 2017
    #14

  15. lluo

    lluo

    Joined:
    May 19, 2016
    Posts:
    20
    Unfortunately, Assembly.Load(AssemblyName) does not work against IL2CPP ecosystem.

    Based on my test (IL2CPP with the Xcode iOS simulator), if an assembly is referenced explicitly (such as invoking a function as defined inside the assembly in question), IL2CPP would pick the assembly up and does all the necessary transformation, and the generated C++ code is guaranteed to be referenced correctly, while if there are no explicit references to the assembly in question, the entire assembly would be stripped, and any runtime binding to the assembly would fail (even an in memory binding would fail, if implemented at all) - similar to the C++ linker that unreferenced symbols are stripped.
     
    lluo, Aug 25, 2017
    #15

  16. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    @lluo:

    Yes, the behavior you see is expected. IL2CPP does not convert assemblies that are not directly used, in order to keep code size down. I would recommend using something from this assembly elsewhere in the project to get it pulled into the build.

    If you are accessing parts of the code only via reflection, you will need to use a link.xml file to prevent the managed code stripper from removing that code as well.
     
    JoshPeterson, Aug 25, 2017
    #16

  17. lluo

    lluo

    Joined:
    May 19, 2016
    Posts:
    20
    lluo, Aug 31, 2017
    #17

  18. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    268
    I have heard that in C# recent version `Expression.Compile` has 2 implementation. It would fallback to use reflection interpreter in any platform that not support dynamic codegen

    https://stackoverflow.com/a/47476886/1513380

    If it's true then it would be useful in some case that not really need performance. Such as let user add math script to control object movement

    Would be anyone pleased to help test that is it really work on iOS and WebGL?
     
    Thaina, Nov 30, 2017
    #18

  19. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    We don't have support for an interpreter on iOS, WebGL, or any AOT platform in Unity. The Mono team has recently been working on an interpreter that might allow for something like this in the future, but it is not on the roadmap for Unity for the time being.
     
    JoshPeterson, Nov 30, 2017
    #19
    Ethan_VisualVocal likes this.

  20. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    268
    According to my investigation the reflection interpreter is natively supported in .NET framework since before 4.0. If unity was brought dotnet framework as is I think it would also contain the interpreter in the 4.6 engine

    Problem is I have no iOS device to test on my own. And the API for ensure interpreter usage is just added in 4.7.1
     
    Thaina, Dec 3, 2017
    #20

  21. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    I don't think there is a reflection interpreter in Mono now (although they are working on something like this).
     
    JoshPeterson, Dec 4, 2017
    #21

  22. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    268
    @JoshPeterson Doesn't dotnet 4.6 experimental use corefx?

    The implementation of Expression.Compile which also include reflection interpreter is in the corefx

    Could you please let me have information about what mono people trying to do about this case?
     
    Thaina, Dec 5, 2017
    #22

  23. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    No, the Unity .NET 4.6 code is from Mono, both class libraries and run time code.

    Here are some details about the Mono interpreter: http://www.mono-project.com/news/2017/11/13/mono-interpreter/
     
    JoshPeterson, Dec 5, 2017
    #23

  24. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    268
    Thaina, Dec 6, 2017
    #24

  25. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    268
    @JoshPeterson I was having conversation with mono people in gitter and this is what they answer me

     
    Thaina, Dec 8, 2017
    #25
    JoshPeterson likes this.

  26. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    Starting in Unity 2018.2, IL2CPP will use the Interpreter for Linq expressions, so some things have a chance of working. You will need to use a link.xml file, since the interpreter implementation uses reflection in internally, and the managed byte code stripper is a bit too aggressive with it.

    This link.xml should be enough:

    Code (CSharp):
    1. <linker>
    2.   <assembly fullname="System.Core">
    3.    <type fullname="System.Linq.Expressions.Interpreter.LightLambda" preserve="all" />
    4.   </assembly>
    5. </linker>
    In Unity 2018.2, you'll need to use the .NET Standard 2.0 Api Compatibility Level option in Unity. In Unity 2018.3 we're also supporting this with the .NET 4.x Api Compatibility Level. In the future Unity 2019.1 version, you won't need the link.xml file work around, the byte code stripper will be smart enough to keep this code around when it it used.

    With that said, you might still hit some limitations of the IL2CPP AOT engine. Specifically, these are often related to the use of value types. So a snippet like this, which uses only reference types (a string), will work:

    Code (CSharp):
    1. var body = Expression.Constant("Hey!");
    2. var lambda = Expression.Lambda(body);
    3. var function = lambda.Compile();
    4. var result = function.DynamicInvoke();
    But code which uses value types (like int or double) might run into AOT limitations. This occurs because IL2CPP shares implementations of generic type with reference type arguments, and so it can do a good bit of "runtime code generation" in these cases, since the only difference is the metadata, not the code.

    IL2CPP does not have the capability to share generics with value type arguments yet. This is an area of future development for IL2CPP, but it is not ready for production yet.
     
    JoshPeterson, Sep 11, 2018
    #26
    Thaina likes this.

  27. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    268
    @JoshPeterson Could we see a sample of code that would run into AoT limitation?
     
    Thaina, Sep 12, 2018
    #27

  28. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    Yes! This code won't work with IL2CPP currently

    Code (CSharp):
    1. var body = Expression.Add(Expression.Constant(42), Expression.Constant(1));
    2. var lambda = Expression.Lambda(body);
    3. var function = lambda.Compile();
    4. var result = function.DynamicInvoke();
    Those integers cause value type generics to be used, which IL2CPP cannot anticipate (since they are used via reflection), and so cannot generate code for them.
     
    JoshPeterson, Sep 12, 2018
    #28
    Thaina likes this.

  29. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    268
    Thank you. But this is really sad because math operation is what I really need for this feature
     
    Thaina, Sep 12, 2018
    #29

  30. juanfornos

    juanfornos

    Joined:
    Feb 17, 2018
    Posts:
    7
    @JoshPeterson if I understood correctly, an expression that returns a bool wouldn't work, would it?

    I'm "baking" a MethodInfo.Invoke (a function that returns a bool) with an Expression.Call to avoid allocs.

    In this scenario I'm getting:
    ExecutionEngineException: Attempting to call method 'System.Linq.Expressions.Interpreter.LightLambda::MakeRun0<System.Boolean>' for which no ahead of time (AOT) code was generated.
     
    juanfornos, Feb 18, 2019
    #30

  31. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    Yes, unfortunately this is correct. A method that returns bool will encounter the value type generics restrictions.
     
    JoshPeterson, Feb 19, 2019
    #31

  32. juanfornos

    juanfornos

    Joined:
    Feb 17, 2018
    Posts:
    7
    Thank you for your answer.

    I read in your previous posts that in the future IL2CPP will have the capability of handling value types. Is it possible to have an estimate of when it will be released?
     
    Last edited: Feb 19, 2019
    juanfornos, Feb 19, 2019
    #32

  33. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    That is correct. Unfortunately we don't have an estimate for that work to be completed.
     
    JoshPeterson, Feb 19, 2019
    #33

  34. Mgravitus

    Mgravitus

    Joined:
    Oct 8, 2015
    Posts:
    12
    Mgravitus, Jul 15, 2019
    #34

  35. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    No, I don't have an ETA or update on this feature, sorry!
     
    JoshPeterson, Jul 15, 2019
    #35
    Mgravitus likes this.

  36. Oruji

    Oruji

    Joined:
    Nov 5, 2012
    Posts:
    14
    Oruji, Aug 23, 2019
    #36

  37. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    4,141
    No, we don't have this on the public roadmap yet, sorry. I'm not sure when we will start work on it.
     
    JoshPeterson, Aug 23, 2019
    #37

  38. QFSW

    QFSW

    Joined:
    Mar 24, 2015
    Posts:
    2,481
    Is there any kind of work around to this such as perhaps boxing the values?

    Essentially I'm automatically supporting operators, but primitives don't have defined operators so I was using this to handle them (whereas non primitives I can just execute the operator method)
     
    QFSW, Oct 23, 2019
    #38