Search Unity

[Released] Roslyn C# - Runtime C# compiler

Discussion in 'Assets and Asset Store' started by scottyboy805, Mar 27, 2019.

  1. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Roslyn C# allows runtime loading of assemblies and C# scripts using the Roslyn compiler making it easy to add modding support or in-game programming to your project. In addition, Roslyn C# also includes code security verification that allows you to specify a number of security restrictions that loaded code must adhere to including illegal namespaces and types. This makes it much safer to load 3rd party code from unknown sources.

    Features
    • Compile and run C# scripts at runtime
    • Leverage the latest C# language features using the Roslyn C# compiler
    • Fast execution. Once compiled, external scripts will run as fast as game scripts
    • Allows modding support to be added easily
    • Code security validation means that unsafe code can be identified and discarded
    • Code security generates a detailed report upon failure containing information about illegal types used and every usage occurrence in the external code
    • Simple and easy to use API for assembly, type and instance reflection
    • Support for non-concrete communication using script proxies
    • Automatic type construction using correct method (AddComponent, CreateInstance, new)
    • Cached member tables for quick reflection
    • All scripts and example are organised into namespaces to avoid type name clashes
    • Fully commented partial C# source code included
    • Comprehensive .chm documentation of the API for quick and easy reference
    Limitations
    • Requires .Net 4.x API compatibility level
    • AOT platforms such as IOS are not supported
    • IL2CPP builds will fail at the moment.
    PC, Mac and Linux platforms are supported

    Demo

    Includes a small programming based game where the objective is to navigate a mouse out of a maze by writing code that performs the direction decision making. The code is written in game using the UI code editor and then compiled and executed to see how it performs.

    Screenshot1.png

    This example game demonstrates some of the compiler API's available as well as the script communication methods that Roslyn C# offers and it is also quite a fun challenge to try and create a simple AI that can reach the end of the maze consistently. All the code for this game is included in 2 fully commented C# scripts so you can clearly understand how it works.

    Need an code editor for your game?
    FeatureTrimmed.png
    Check out our other asset asset 'InGame Code Editor' which is an advanced text input field with line numbering and fully customizable syntax highlighting for C# and many other languages. Check out the asset store or forum page for more info.
     

    Attached Files:

    Last edited: May 7, 2019
    TeagansDad, Stardog, Artaani and 2 others like this.
  2. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Artaani likes this.
  3. pauliusuza

    pauliusuza

    Joined:
    Nov 1, 2016
    Posts:
    7
    Purchased! Looks great
     
  4. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Thanks for purchasing. Let me know if you do anything cool with it :)
     
  5. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    We are not giving up on Web GL and IL2CPP builds yet. We can see no theoretical reason (Apart from the build error we are receiving) that Roslyn C# cannot run under IL2CPP and not only that but WebGL as well. We have reported the IL2CPP build error we are receiving to the Unity team and will keep you posted with any updates.
     
    Artaani likes this.
  6. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Small update on IL2CPP support. The Unity QA team have reproduced the bug that is causing IL2CPP builds using Roslyn C# to fail so hopefully once fixed we can try once again to support IL2CPP and webGL. That does not mean we will not have any more issues building for IL2CPP after this bug is fixed but it will be a step in the right direction. Here is the bug causing us problems on the issue tracker.
     
    Artaani and OldYawn like this.
  7. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Roslyn C# version 1.0.1 has been submitted to the asset store for review and should be available for download within a couple of days. This version adds whitelist support to the security restrictions so you can now list references, namespace and types which are allowed instead of listing all restrictions.
     
    Artaani and OldYawn like this.
  8. lThanatosl

    lThanatosl

    Joined:
    Jan 25, 2013
    Posts:
    30
    Hey quick question, I bought your asset and came across something I'm too stupid to fix.

    Say I have an external class called A and another external class called B.

    If I try to have A contain a variable of type B, a bunch of exceptions will pop up, so I guess external classes don't know anything about each other. How would you tackle this? In my case, I have a class Actor and a class Attribute, and i just want the Actor to have a variable of type Attribute but that's when S*** gets crazy.

    Thanks!!
     
  9. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Hi,
    Are these external classes compiled separately? If so you should be receiving compiler errors about undefined types? Is that correct. If this is the case the you will need to add an assembly reference to the assembly that type B is defined in before compiling type A. Here is a shot example:

    Code (CSharp):
    1. using RoslynCSharp;
    2. using UnityEngine;
    3.  
    4. class Example : MonoBehaviour
    5. {
    6.     void Start()
    7.     {
    8.         ScriptDomain domain = ScriptDomain.CreateDomain("Example", true);
    9.      
    10.         // Compile source for type B
    11.         ScriptAssembly asm = domain.CompileAndLoadFile("insert type B source file");
    12.      
    13.         // Add an assembly reference
    14.         domain.RoslynCompilerService.ReferenceAssemblies.Add(asm.RawAssembly);
    15.      
    16.         // Compile source for type A which should now allow access to types defined in source B
    17.         domain.CompileAndLoadFile("insert type A source file");
    18.     }
    19. }
    This should allow you to use types defined in source B from source A.
     
    Last edited: Apr 27, 2019
  10. lThanatosl

    lThanatosl

    Joined:
    Jan 25, 2013
    Posts:
    30
    Oh I see, my design is basically the following:
    • Visual Studio Project: Contains classes that will be turned into the game's DLL. Basically has the absolute bare minimum the game is expected to have.
    • Unity Project (Mod creator): Uses the game's DLL and creates a folder of all the external scripts (the mod). Used to further enhance and create new functionality.
    • Unity Project (Main game): Uses the game's DLL, contains code such as System.IO, etc. Reads the external scripts from a mod by compiling them with your asset.

    The Main game project goes through all the scripts in a particular folder and compiles them like this (https://i.imgur.com/izX2dXc.png) Which is basically only good enough for the script to be placed in the scene attached to the given gameObject, and so its Start/Update/etc funcs are called. But yeah my issue is that doing this isn't enough for the script to know about other scripts.

    It seems like the solution you proposed will bring issues though, because the main game is what compiles the external code and the user will not be able to simply decide what gets compiled before what since they cant change that code.

    I was thinking maybe I need to compile all the external code into a DLL and have the main game load in 2 DLLs instead of 1, so 1 DLL done by me, and 1 DLL which contains the mod. The issue with this is that I wanted to keep mod code visible to others, but now it will be hidden behind the DLL unless a mod author decides to release not only the DLL but also the source, which just adds extra steps.

    Awesome asset btw!
     
  11. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Would it be possible for you to compile all of the mod scripts in the folder together. Roslyn C# supports compiling batches of source files which then act as though you are compiling a C# project in visual studio. Ie all the source files belong to the same assembly and as a result can communicate normally. If this is what you are looking for then you just need to use the overload of 'CompileAndLoadSources' or 'CompileAndLoadFiles' which accepts a string array. I may have had the wrong idea as I assumed that you wanted to compile the files individually.
     
  12. lThanatosl

    lThanatosl

    Joined:
    Jan 25, 2013
    Posts:
    30
    Oh that sounds sweet I'll look into that. No no I do all the loading of all the scripts at once, Basically I go through the mod's folder and all its subfolders and get all the scripts onto a list, and then i loaded each one the way i showed on the picture, but if the function that you mentioned allows them to know about each other then my problems would go away.

    I'll get started right away, thanks!!
     
  13. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Yest that would be the easiest and probably the least time consuming way of doing it. You will then need to change your loading code to something like this to process all types in the assembly instead of just the main type:

    Code (CSharp):
    1. void CompileFiles(ScriptDomain domain, GameObject go, string[] filePaths)
    2. {
    3.     ScriptAssembly asm = domain.CompileAndLoadFiles(filePaths);
    4.  
    5.     foreach(ScriptType type in asm.EnumerateAllMonoBehaviourTypes())
    6.         type.CreateInstance(go);
    7. }
     
  14. lThanatosl

    lThanatosl

    Joined:
    Jan 25, 2013
    Posts:
    30
    Good news! It worked like a charm!

    I decided to do something like this though (https://i.imgur.com/moAGnV4.png)

    Basically every external script can implement the static function OnScriptLoad, so that the user can decide what to do when the script is loaded. Maybe the mod creator wants the script to be attached to a gameObject, maybe not, so I'll let them decide.


    Thanks a lot for the help!
     
    scottyboy805 likes this.
  15. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Glad to hear you got it working. Yes you can choose whatever works best for you. I was only showing how you can assess all types in the assembly rather than the main type :). Let me know if you need anything else.
     
  16. albor63

    albor63

    Joined:
    May 1, 2019
    Posts:
    2
    Hi,

    I use Roslyn C# with Unity to run a script added as a Component to a GameObject.
    It works fine when running into the Editor.
    But it doesn't works when running outside the Editor as build.
    Is there a build setting to do or some dll to add ?

    Thanks.
     
  17. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Hi,
    No there is nothing special you need to do for a build. Can you try a debug build to see if you are getting any exceptions or anything. Also I assume you are running on Windows, Mac or Linux.
     
  18. albor63

    albor63

    Joined:
    May 1, 2019
    Posts:
    2
    Hi Sorry, it's an error on my part.
    The file containing the script was not found at runtime.
    All works fine. It's a real pleasure.
     
    scottyboy805 likes this.
  19. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Glad to hear that you got it working. Let me know if you have any more issues or questions.
     
  20. Player7

    Player7

    Joined:
    Oct 21, 2015
    Posts:
    1,390
    Isn't this just the same as Dynamic C# which I have..why is this not just an update?
     
  21. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Hi,
    Dynamic C# and Roslyn C# aim to achieve the same goal which is to compile and execute C# source code at runtime however there are some major differences:
    1. Roslyn C# as the name suggests makes use of the Roslyn Compiler service which means that the latest C# language features and frameworks are supported whereas Dynamic C# can only compile C# code equivalent to .Net 3.5 scripting API level.
    2. Roslyn C# includes a new and improved code security system which is able to provide much more detailed information about the location of illegal code making it easier to debug and fix security issues.
    3. Roslyn C# requires Unity 2018.1 or newer whereas Dynamic C# can support older Unity versions which is one of the main reasons for creating separate assets. We did not want to drop support for 2017 or even Unity 5 so Dynamic C# is still available to users who have projects running on older Unity versions.
    There are a few other differences but these are the main ones. We felt that these improvements justified creating a dedicated assets but both Dynamic C# and Roslyn C# will be maintained and updated. As you have purchased Dynamic C# you will also be eligible for a reduced price upgrade to Roslyn C# for $7.50 which we felt was fairest way to handle the update route.

    I hope this answers your question and that you understand why we took this approach.
     
    Artaani likes this.
  22. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    FeatureTrimmed.png

    We have now released a new asset which may be of interest if you have purchased Roslyn C#. InGame Code Editor is and advanced text input field with line numbering and full syntax highlighting for C# and many other languages as well as many more features. Check out the asset store or forum page.
     
    Artaani likes this.
  23. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Rolsyn C# version 1.0.2 has been submitted to the asset store for review. This version includes a minor bug fix for an issue where the compiler would receive incorrect data for an async file compilation request which causes the compile to fail.
     
  24. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Roslyn C# 1.0.3 has been submitted to the asset store for review. This version includes a minor bug fix where the compile would always load compiled assemblies into the current AppDomain even when another AppDomain was specified.
     
  25. Whitebrim

    Whitebrim

    Joined:
    Jul 11, 2017
    Posts:
    9
    [1143749] issue with IL2CPP was fixed in 2019.3
     
    Marcos-Elias likes this.
  26. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Thanks for letting me know. I just tried to build for IL2CPP using 2019.3.0a2 and still seem to be getting the same bug/error. Maybe they are adding the fix to the final release.
     
  27. Abenthum

    Abenthum

    Joined:
    Mar 2, 2016
    Posts:
    10
    Are you compiling native DLLs in case of IL2CPP?
     
  28. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    No, all assemblies are managed assemblies but due to a bug in IL2CPP we cannot support it at the moment. Unity have confirmed that the bug will be fixed so we hope to support it in the future.
     
  29. Abenthum

    Abenthum

    Joined:
    Mar 2, 2016
    Posts:
    10
    Oh -- so how are you executing code in managed assemblies from an IL2CPP player?
     
  30. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Well IL2CPP converts managed code to native at build time but as I say we cannot support it at the moment.
     
  31. Abenthum

    Abenthum

    Joined:
    Mar 2, 2016
    Posts:
    10
    :) Yeah I get that. But the point of your package is to compile and execute code during runtime, not build time. Even if you get Roslyn to run in an IL2CPP player build, all you get during runtime is compiled IL code, right? How are you then executing that in an IL2CPP player?

    More background: We're thinking about replacing our mono.csharp based development console with something that can work in IL2CPP builds as well. First thought was to build something reflection based or integrate some interpreted language, but that implicates a reduced feature set. Your package may be an alternative since you seem to have a plan to make IL2CPP with Roslyn work?!
    Assuming that bug gets fixed, naturally.
     
  32. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Sorry, it seems I misunderstood you. For runtime execution we have not yet tested but we have a few possibilities. The easiest and most straightforward would be to use reflection with JIT but do not know at this point whether IL2CPP supports this. Other possibilities are mono's IL interpreter 'mint' which could potentially be used for the loaded code or if all else fails we are currently working on a C# syntax tree interpreter using the roslyn compiler service as the parser so we are hopeful that we will be able to get something working. Obviously speed may play a factor at that point though so it is not an ideal solution.
     
    Abenthum likes this.
  33. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Roslyn C# version 1.0.4 has been submitted to the asset store for review and should be available for download within a day or two. This version fixes a minor bug in the settings window where adding define symbols to the compiler settings would actually add to the assembly references collection instead.
     
  34. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Roslyn C# version 1.1.4 has been submitted to the asset store for review. This version includes some major changes mostly relating to the code verification system:
    • Added new code verification system 'Trivial.CodeSecurity'
    • Improved the performance and memory usage of the code verification process
    • Added support for code verification on members. You can now mark individual members of a type as allowable or illegal
    • Added whitelists and blacklists for code verification to make it easier to setup restrictions
    • Added default rule for unlisted entries in code verification settings. If you have not specified the assembly reference, type name, namespace name or member name in the restrictions then the default rule will be used to determine whether it is allowable.
    • Fixed an issue where the compiler would generate many warnings relating to reference binding of higher assembly versions
    • Fixed a bug where the compiler would fail to load the assembly if 'Generate In Memory' was disabled.
    CodeVerification.png
     
    Yavvn likes this.
  35. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    480
    Hi, I ran into this Asset while looking for REPL solution for the interactive testing/debugging.
    The concept is similar to the C# Interactive Window from Visual Studio. Since VS' Interactive Window is based on Roslyn, I thought it is also possible for this Asset to support the Interactive Window.
    Could you please let me, know if it's possible to add the support?
    I believe REPL is a must for the interactive development but I'm surprised there isn't one exists yet(I find one based on Lua but it doesn't seem supported now) and it will make this asset very powerful.
    Thank you very much for your consideration.
    Cheers!
     
  36. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Hi,
    Thanks for your interest in our asset. This sounds like something we could potentially support although we would likley create something like this as an add on package as most users would likley not need these features when purchasing the main asset. What sort of features would you like to see from something like this? I assume the ability to run math expressions, evaluate branches, invoke method etc, anything else?. What about Unity API's? Would the ability to access the Unity api while the game is running be something you would like to see?

    Feedback would be much appreciated and we may be able to implement something in the future.
     
  37. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    480
    Hi,
    Yes, accessing both Unity API and my own API. Basically you will import assemblies and you can access anything within them

    Have you tried VS Interactive Window? They call it C# Interactive.

    I think there might be some framework let you write your own Interactive Window (to support intellisense, history and such)

    Thanks for your positive reply and I'm really looking forward to this.
     
  38. TheGabelle

    TheGabelle

    Joined:
    Aug 23, 2013
    Posts:
    94
    Have you tested this with Unity ECS? What are the limitations? It would be amazing if mods could create systems, jobs, components, etc.
     
  39. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Hi,
    We have not tested with ECS but I see no reason why it should not work as long as you add the necessary assembly references.
     
  40. mite51

    mite51

    Joined:
    Jun 25, 2015
    Posts:
    13
    My guess is that using ECS would require the burst compiler, which likely is not available at runtime.

    .....But the real reason I'm posting is because I wanted to try this asset with a project compiled in IL2CPP. I knew it would likely not work from reading here, but I wanted to know what the issues are, and I was hopeful there might be a way around it... or possibly it was already fixed in 2018.4.

    My first attempt was just to use the sample to see what the problem is, this resulted in the following runtime error:

    Code (CSharp):
    1. ArgumentException : Path is empty
    2. callstack
    3.     > System.IO.FileAccess
    4.     > System.IO.OpenRead
    5.     > Rosyln.Utilities.FileUtilites.OpenFileStream
    6.     ....
    Its failing doing some IO during compile, which I thought might be OK if I could avoid the IO somehow.

    Because my requirement is to send updated gameplay code via mission data, I thought maybe I could compile the code in editor, package the output assembly in binary then load that binary data in the project when compiled in IL2Cpp. It went well, the assembly compiled, I could grab is as a byte array and provide it to the built client. Then I got this :

    Code (CSharp):
    1. IL2CPP:AppDomain::LoadAssemblyRaw - "This icall is not supported by il2cpp"
    I was using System.Reflection.Assembly.Load, which I didn't think would work anyway... Then I tried domain.LoadAssembly ( the Roslyn version ), and that had exactly the same issue. This seems to be the main sticking point with any AOT platform ( IL2Cpp, iOS and android ) doing anything with dynamic code like this.

    Now I can see Unity fixing the IO problem, its possibly the assembly getting confused with colliding namespaces or something. However, I don't see them allowing LoadAssemblyRaw... but I could be wrong.

    Anyway, If someone has some insights or an idea, let me know. I would really like to have something like this work.
     
  41. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    503
    Hi,
    As you have discovered, IL2CPP support is not possible at the moment mainly due to the load assembly restriction. The IO problems could be fixed or even avoided all together by compiling in memory so there should be no issue there. The only potential issue I could see relating to IO is that WebGL which many users would like support for does not allow file access. At the moment Roslyn C# is only able to add assembly references via file paths which would need to change if we were to support WebGL.

    The dynamic loading issue we have now known about for some time and have started to implement a CIL interpreter which should be able to exectute externally loaded code. It is the early stages of development and we are only able to work on it in our spare time so it could be quite some time before we have something usable but I think this would be the only way to exectue externally loaded code on IL2CPP. This would also run the code slower than normal as there would be no Jit compiler and the runtime itself would be written in C#. We would also have to do alot of testing to ensure that it is stable and can indeed run on IL2CPP. At the moment we are only running tests on a development PC until we have implemented enough of the core.
     
  42. mite51

    mite51

    Joined:
    Jun 25, 2015
    Posts:
    13
    Good luck.. I hope you figure something out, I would love to use it in the future