Search Unity

  1. Click here to see what's on sale for the "Best of Super Sale" on the Asset Store
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

[Released] Roslyn C# - Runtime C# compiler

Discussion in 'Assets and Asset Store' started by scottyboy805, Mar 27, 2019.

  1. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Would you be able to try a new project and fresh import to rule out any issues with left over assets. Alos if you are able/willing to, you can send us the project to info(at)trivialinteractive.co.uk and we will take a look at what might be happening.
     
  2. alvaro_unity903

    alvaro_unity903

    Joined:
    Jan 20, 2020
    Posts:
    19
    I made a "Reset packages to default" because another problem and it seems fixed. Let me time, because I have 169 third errors now to deal with.
     
  3. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Ok great. Let me know you have any more issues.
     
  4. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    I've noticed that everytime I compile the exact same files in the exact same way the dll file I export is different, even a hash in the generated dll.... I was wondering is this to be expected? Is there any way I can prevent this? The reason I would like it to generate an identical DLL every time is because I'm sending this to clients and I don't want them to have to redownload it if they have a matching DLL already.

    Currently everytime the server is restarted all clients will have to redownload the DLL since it generates a different DLL everytime the server is started (and it is generating a different DLL every time). If this isn't possible I can live with it as well since the DLL file isn't too big and sends quickly, just curious.
     
    Last edited: Jun 1, 2020
  5. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Hi,
    When you say that the dll is different, I assume you are talking about the generated file name and assembly guid stored in the metadata? The file name is generated by our asset and is specifically designed to to overwrite any existing files so the guid file name will always be unique. If required, you can change the output file name before you call one of the compile methods to choose the output name:

    Code (CSharp):
    1. domain.RoslynCompilerService.OutputName = "MyAssembly";
    As for the metadata guid: I believe the compiler will regenerate this as part of the compile process so that each build is unique. Ie it is intended behaviour. There is a bit more information about that here.
     
  6. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    Hey, sorry I forgot to get back to you because this is not a priority problem for me, but no I mean when I write the dll file using
    Code (CSharp):
    1. stream.Write(result.assemblyImageData, 0, result.assemblyImageData.Length);
    I actually get a different file every time the same source is compiled. Everything works, but I was hoping it might be possible to make this always generate the exact same assemblyImageData given the same inputs.

    So yeah, in short the problem is result.assemblyImageData is different everytime the application runs. (It does produce the desired working assembly, just a different one every time.)

    I don't know enough about what's going on under the hood to know if this is the intended behaviour or not, or if there's a way around it... I figured if anyone would know it'd be you so that is why I am asking here.
     
  7. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Hi again,
    Thanks for clarifying.
    I believe this is caused by the compiler regenerating the assembly guid on evey build which it is likley doing because there is no specified guid value. Usually in visual studio or similar there would be a special source file located under the properties foldout called 'AssemblyInfo.cs' which contains metadata like assembly tille, version, guid, etc. You sould be able to add a similar source file to your compile request with a fixed guid and hopefully that will cause the same output assmebly every time although I am not completley sure it if would be the same or not. Here is an example assembly info source file:

    Code (CSharp):
    1. using System.Reflection;
    2. using System.Runtime.CompilerServices;
    3. using System.Runtime.InteropServices;
    4.  
    5. // General Information about an assembly is controlled through the following
    6. // set of attributes. Change these attribute values to modify the information
    7. // associated with an assembly.
    8. [assembly: AssemblyTitle("ExampleAssembly")]
    9. [assembly: AssemblyDescription("")]
    10. [assembly: AssemblyConfiguration("")]
    11. [assembly: AssemblyCompany("")]
    12. [assembly: AssemblyProduct("UltimateModdingToolkit")]
    13. [assembly: AssemblyCopyright("Copyright ©  2020")]
    14. [assembly: AssemblyTrademark("")]
    15. [assembly: AssemblyCulture("")]
    16.  
    17. // Setting ComVisible to false makes the types in this assembly not visible
    18. // to COM components.  If you need to access a type in this assembly from
    19. // COM, set the ComVisible attribute to true on that type.
    20. [assembly: ComVisible(false)]
    21. [assembly: Guid("12b8c257-89a2-472b-3e5b-6ca0a6e68d30")]
    22.  
    23. // Version information for an assembly consists of the following four values:
    24. //
    25. //      Major Version
    26. //      Minor Version
    27. //      Build Number
    28. //      Revision
    29. //
    30. // You can specify all the values or you can default the Build and Revision Numbers
    31. // by using the '*' as shown below:
    32. // [assembly: AssemblyVersion("1.0.*")]
    33. [assembly: AssemblyVersion("1.0.0.0")]
    34. [assembly: AssemblyFileVersion("1.0.0.0")]
    Line 21 is what you would want to keep consistent between compiles which will hopefully allow the output to be the same.
     
  8. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    Thank you for the suggestion, I will definitely give this a shot!
     
    scottyboy805 likes this.
  9. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    Do you know of a better way to do this as opposed to creating a file to add to the compilation request? Something like
    ScriptDomain.AddAttribute()
    or similar? My plan is to generate a MD5 hash of the concatenated hashes of the source files and convert that into a GUID to ensure it's the same every time... I was hoping their was a faster way to add the assembly than making a file every time.

    For now I will start implementing it with the file just to test and see if it works.
     
  10. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Hi again,
    There is no easy way to do this at the moment apart from manually creating the file and adding it to the compile request. We could possibly add an AssemblyInfo type or similar where you can set various assembly properties from code and then generate the script on our end if that could be something useful.
     
  11. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    That could definitely be very useful, but at least in my case only if this actually works in generating consistent output. I'm a bit busy with work right now so I haven't had a chance to test this but I will definitely do so eventually and let you know of the result. As always I appreciate the help.
     
  12. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Ok no problem. Let me know if it fixes your problem and then we can look at adding that feature in to the asset.
     
  13. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    So I managed to make it compile with the same GUID every time but it still produces a different file every time (although now the file contains the GUID I set every time too, there is just other differences). It seems like there's a ton of other GUIDs referenced in the generated DLL file (like 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.dll.pdb') so I'm assuming it'd be a real pain to try and make each compilation match.

    Either way this is a thing I can live with, it just means users who connect to servers will have to download a small clientside assembly everytime the server restarts. This is the price I pay for having the server compile the clientside code files instead of sending them to the client for compilation.

    It was also no hassle at all to just make a temporary file to add the compilation, I don't see it being worth your time adding it to the asset.
     
    Last edited: Jun 25, 2020
  14. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Thanks for the update. Sorry to hear it didn't work as expected and it does indeed sound like it would be too much effort to generate deterministic outputs. It seems like the compiler acutally attempts to prevent this, probably for a good reason so if you have a work around then that is probably best. Let me know if there is anything else I can help with.
     
    Crazycarpet likes this.
  15. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    Is it possible
    ScriptAssembly::LoadAssembly(string dllPath, ScriptSecurityMode securityMode);
    is leaving a file handle opened? I seem to be having an issue where once I call it I can no longer modify the file even after I unload the assembly. Also, should it even keep the file locked til the ScriptAssembly that is returned is deleted? If this is the case it's possible I'm not unloading the assembly properly and will look into it deeper.

    Getting the 1224 or whatever it is error that means 'file in use by another program'.
    What I am doing is: 1) Loading a DLL file and compiling it. | 2) Unloading the script assembly | 3) Edit the DLL file before reloading and compiling it.
    That's why I'm thinking I'm either messing up step #2 or 'LoadAssembly' is leaving a file handle opened.
     
  16. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Hi,
    This is actually intended behaviour of the CLR. when loading an assembly from file (which is what happens during that method call) the assembly file is locked until the application exits. There is also no way to unload assemblies once they have been loaded into the Unity app domain. You would have to destroy the app domain instead which ofcourse is not possible because all your game scripts would also be destroyed. There is a way around the file locking issue though by loading the assembly from memory:

    Code (CSharp):
    1. byte[] assemblyBytes = File.ReadAllBytes("Example/Assembly.dll");
    2.  
    3. scriptDomain.LoadAssembly(assemblyBytes, ScriptSecurityMode.UseSettings);
    Let me know if you have any more questions.
     
    Crazycarpet likes this.
  17. Crazycarpet

    Crazycarpet

    Joined:
    Dec 5, 2015
    Posts:
    17
    I think that's it, everything seems to be working and I think I'm finally done implementing Roslyn C# into my project. As always, thanks for the support.

    Great asset, great support, extremely easy to use. I appreciate your comments as they helped me improve the overall design of my implementation by helping me understand what's going on under the hood and hopefully anyone who runs into the same situations will be-able to find them helpful too.
     
    scottyboy805 likes this.
  18. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Glad to be of assistance. Let me know if there is anything else I can help you with.
     
  19. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Roslyn C# version 1.3.2 has been submitted to the asset store. This version adds further checks to the code security verification engine to prevent the usage of pinvoke. Using the blacklisting feature to disallow 'System.Runtime.InteropServices' or the 'DllImportAttribute' will not prevent pinvoke code from running as brought to our attentiion by a user. The reason for this is that the compiler will detect the DllImport attribute and emit pinvoke IL instructions and the attribute will be stripped allowing the code to slip though security verification. Further explicit pinvoke checks will now run to prevent external code from being called. There is also a new option in the settings window to reflect this 'Allow PInvoke' which we recommend remains disbaled.

    settings.png
    DLLImportUsage.png
    PInvokeError.png
     
    Last edited: Aug 31, 2020
  20. Saafris

    Saafris

    Joined:
    Nov 22, 2017
    Posts:
    4
    Hello

    Whenever I try to set the Roslyn settings, it resets to default any time I change the codebase. I do not want to have to select my preset every time I make a code change.

    I cannot find in the documentation how to add a blacklisted namespace through code, or how to load settings from a preset through code. How do I do these things?
     
  21. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Hi,
    Could you let me know which settings you are trying to change and the version of Unity you are using and we will look into the settings issue.

    All settings can be accessed at runtime though the 'RoslynCSharp' type:

    Code (CSharp):
    1. RoslynCSharp.Settings...
    To blacklist a namespace at runtime, you could use the following code:

    Code (CSharp):
    1. RoslynCSharp.Settings.SecurityRestrictions.NamespaceReferences.AddEntryName("System.IO.*", CodeSecurityRestrictions.CodeSecurityBehaviour.Deny);
     
  22. Saafris

    Saafris

    Joined:
    Nov 22, 2017
    Posts:
    4
    I'm not sure I understand.

    I'm using Roslyn through code with ScriptDomain, adding reference assemblies, (hopefully) adding security, and then compiling and loading files.

    Code (CSharp):
    1. var domain = ScriptDomain.CreateDomain("ExampleName", true);
    2.             domain.RoslynCompilerService.ReferenceAssemblies.Add( AssemblyReference.FromNameOrFile(". . ."));
    3.  
    4. ScriptAssembly assembly = domain.CompileAndLoadFiles(new[]
    5.             {
    6.                 ". . ."
    7.             });
    How do I add the Roslyn settings to this flow, or change the flow to be more correct? I wasn't able to use your above code, I had to adjust it to:

    Code (CSharp):
    1.  
    2. var roslyn = new RoslynCSharp.RoslynCSharp();
    3. roslyn.SecurityRestrictions.NamespaceReferences.AddEntryName("System.IO.*", CodeSecurityRestrictions.CodeSecurityBehaviour.Deny);
    4.  
    But I'm not sure how to make sure that applies.

    As for unity version, 2020.1.8f1
     
  23. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Hi,
    I am not exactly sure what you want to do.
    The settings that you modify via the settinw window are auto-loaded and applied to the compiler when creating your script domain via 'CreateDomain'. You can then use the 'RoslynCompilerService' property of the domain to access the compiler if you need to add additional references via code.

    If you want to access the settings that are assigned from the settings window then you can use the following code at runtime:

    Code (CSharp):
    1. RoslynCSharp.Settings
    That will load and return the active RoslynCSharp settings for the game.
     
  24. Saafris

    Saafris

    Joined:
    Nov 22, 2017
    Posts:
    4
    Ah, I see, it was unclear that editing those settings would affect my created domain. I assumed I was meant to pass it in, or modify the domain somehow.
     
  25. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    The settings are global to all script domains arnd are applied automatically when creating the domain as mentioned. If you need to change the settings after a domain has been created but still want to apply those changes, you can use the following method:

    Code (CSharp):
    1. class Example : MonoBehaviour
    2. {
    3.     void Start()
    4.     {
    5.         // Create a domain as usual
    6.         ScriptDomain domain = ScriptDomain.CreateDomain("Example", true);
    7.      
    8.         // Change some settings
    9.         RoslynCSharp.Settings.AllowUnsafeCode = true;
    10.         RoslynCSharp.Settings.AllowOptimizeCode = true;
    11.      
    12.         // Apply the modified settings to the compiler
    13.         domain.ApplyCompilerServiceSettings();
    14.     }
    15. }
     
  26. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    Roslyn C# version 1.3.3 has been submitted to the asset store. This version includes:
    • Fixed a bug in the code security engine where wildcard namespaces would not be handled correctly.
    If you have any feature requests or feedback for the asset then we would be happy to hear it :).
     
    Last edited: Nov 16, 2020
  27. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    856
    IL2CPP Update

    We have been working on IL2CPP support on and off for a little while now but I can finally share a working proof of concept that allows you to compile and execute C# source code at runtime on IL2CPP platforms, including WebGL.

    We have uploaded a WebGL demo of the included mouse maze game where you write code to control the movement direcition of a mouse in order to escape the maze. You can find the demo here.
    demo.png

    Important
    This is a proof of concept demo only at the moment and by no means complete, or even close to being production ready. Many aspects of executable code are partially implement or not implemented at all so you may enounter some errors when tyring to run different code. For example: Exception handlers are not implemented so there may be unexpected behaviour when throwing exceptions. We will continue to work on this to support as many featues as possible.

    How does it work?
    As you may already know, IL2CPP does not support loading of external assemblies such as the ones produced by the roslyn compiler which our asset uses. This is because all C# code is compiled ahead of time into C++ code and calls such as 'AppDomain.Load' can only work with assemblies that were compiled as part of this AOT process. The only feasable way to get around this limitation was to create an emulator for CIL bytecode which is exactly what we did. We created a CIL bytecode interpreter in pure C# code that fully supports the object model including types, members, generics ect. We use Mono.Cecil to read assembly definitions and extract the object metadata as well as method instruction sets to fully replicate the reflection system. IE. We have created our own implementation of System.Type, MethodInfo, FieldInfo etc. and you can use this replicated system as you would expect. For example, Invoking a method using the interpreter can be done like so:

    Code (CSharp):
    1. // Create an app domain where interpreted types/code will exist. Note this is not a 'System.AppDomain' instance
    2. AppDomain domain = new AppDomain();
    3.  
    4. // Replcement for 'Assembly.Load' or 'System.AppDomain.Load'
    5. CLRModule module = domain.LoadModuleStream(...);
    6.  
    7. // Find type with specified name
    8. Type t = module.GetRuntimeType("MyExampleType");
    9.  
    10. // Find method declared on type with specified name
    11. MethodBase method = t.GetMethod("MyExampleMethod", BindingFlags.Instance | BindingFlags.Public);
    12.  
    13. // Allocate a new instance of type 't'
    14. object instance = domain.CreateInstance(t);
    15. object someArgument = "ExampleString";
    16.  
    17. // Invoke an instance method
    18. method.Invoke(instance, new object[]{ someArgument});

    We will continue to work on this project in our spare time and update this thread with any news.
     
    Last edited: Nov 21, 2020
    Yavvn and ZhavShaw like this.
unityunity