Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only.

    Please, do not make any changes to your username or email addresses at id.unity.com during this transition time.

    It's still possible to reply to existing private message conversations during the migration, but any new replies you post will be missing after the main migration is complete. We'll do our best to migrate these messages in a follow-up step.

    On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live.


    Read our full announcement for more information and let us know if you have any questions.

How to force a Roslyn SourceGenerator to run?

Discussion in 'Experimental Scripting Previews' started by TJHeuvel-net, Apr 2, 2022.

  1. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    838
    I'm following the manual on how to get roslyn source generators to work, but i'd like to trigger the source generator to run based on a user action. How do i go about this? I've tried `CompilationPipeline.RequestScriptCompilation` to make a general recompile happen, but that doesnt trigger the source gen.

    Eventually i want the source generator to run when a user changes an asset.
     
  2. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    838
    I figured out a way, by calling AssetDatabase.ImportAsset on the current script file path. Works, but a bit icky.
     
    JesOb likes this.
  3. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    142
    Hi :)
    So theres a couple of things in play with Compilation and Source Generators:

    Unity tracks how much we need to compile. This is based on files changed, so we only invoke Roslyn on things we know needs to get recompiled. When you `CompilationPipeline.RequestScriptCompilation` that could result in nothing or not all getting recompiled, as input's to the compilation has not changed. (File hashes, arguments to Roslyn, other interesting things)

    Roslyn is run as shared, this allowed Roslyn to cache so next compilation will be faster, even when that compilation has changed as fx SyntaxTree's could be the same.

    Since Roslyn caches data, you should not Read files from it directly. Since the compilation might invoke Roslyn again, Roslyn could end up in reusing a Cached result as the input to that compilation for the source generator hasnt changed.

    Theres a thing called `additionalFiles`
    https://github.com/dotnet/roslyn/blob/main/docs/analyzers/Using Additional Files.md
    That fixes the Roslyn Cache, so what your SourceGenerator depends on needs to come from that.
    in 2022.2 we added support for AdditionalFiles (but not our yet).

    Since Unity does things in steps:
    - Compile
    - Domain Reload
    - Import Assets

    The Compilation Specific files has to be with knows extensions, additional files is postfixed with `.additionalfile` so we know to pick it up.

    You can use additionalfile today using `csc.rsp` file.
     
    TJHeuvel-net likes this.
  4. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    142
    Small update, the Additional files feature is going to be backported to 2021.2
     
    brunocoimbra and Neonlyte like this.
  5. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    838
    Awesome, thanks for the in depth response!
     
    HaraldNielsen likes this.
  6. cpatrascu

    cpatrascu

    Joined:
    Feb 18, 2019
    Posts:
    1
    Hi @HaraldNielsen,

    Has the Additional Files feature been backported to 2021 LTS? If not, would it be possible to get it done soon?
    We could really benefit from it in our project. Thanks!
     
  7. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    142
    @cpatrascu I just checked the code there, it should be part of 2021 LTS. Should be in since 2021.3.3f1
     
    rdjadu likes this.
  8. Kaligoth

    Kaligoth

    Joined:
    Mar 4, 2015
    Posts:
    8
    https://docs.unity3d.com/ScriptRefe...ompilerOptions.RoslynAdditionalFilePaths.html

    @HaraldNielsen It's more complex compared to what you wrote above if this document is to be trusted. And given that the documentation on Source Generators is lacking any information and there is some here, I am kind of inclined to think this is probably how it works.

    Let me phrase this as nicely as I can: It also matches the level of understanding of the tech/use cases and the unwieldiness one has come to expect from Unity in terms of supporting regular dotnet stuff. Quelle surprise!

    This is IMHO not the way to solve this. It excludes pretty much every source generator that uses additional files and is written for regular projects. One would normally filter by file extensions of additional files etc. and that is out of the window with having to use the .additionalfile extension. Not having "." in the file names excludes hierarchical and namespace-relevant stuff in generation - which again, is a common use case with those. Which people would know, if they had worked on source generators outside of the context of Unity. You also lose any editor support because guess what: they usually recognize which specific language editor to pick based on file extensions - but those all need to be the same.

    Having to specify all the files in the csc.rsp is unwieldy as f*** so please don't pretend like it's a viable workaround. It's probably what one should go for, given the situation, but it's not how this should have been implemented.

    Multiple ways to improve this, IMHO. Best one, given the limitations of the current project system: Add a project setting for specifying globs to match additional files. This could be just a text field. You can even have the whole thing where you can choose which generator gets which files, if you really want this (although the doc ALREADY specifies it doesn't have an impact). But it shouldn't be needed, because that's also not how it works in dotnet land. Every generator gets everything there and it's fine.

    Code (CSharp):
    1. MyAwesomeGenerator:**/*.lang;OtherGenerator:Assets/Sounds/**/*.wav
    Either way: I can't wait for 2025 or whenever the .NET project system work lands in Unity and these things just work out of the box.
     
    Last edited: Dec 16, 2022
    Anthiese likes this.
  9. Kaligoth

    Kaligoth

    Joined:
    Mar 4, 2015
    Posts:
    8
    Here's another workaround if someone else runs into this...


    Code (CSharp):
    1. public class AdditionalFilesPatcher : AssetPostprocessor
    2. {
    3.     public static string OnGeneratedCSProject(string path, string content)
    4.     {
    5.         if (!path.EndsWith("Assembly-CSharp.csproj")) return content;
    6.        
    7.         var myFiles = AssetDatabase.GetAllAssetPaths().Where(file => file.EndsWith(".some-extension"));
    8.        
    9.         var xDoc = XDocument.Parse(content);
    10.         var nsMsbuild = (XNamespace)"http://schemas.microsoft.com/developer/msbuild/2003";
    11.         var project = xDoc.Element(nsMsbuild + "Project");
    12.  
    13.         if (project == null) return content;
    14.        
    15.         var itemGroup = new XElement(nsMsbuild + "ItemGroup");
    16.         project.Add(itemGroup);
    17.        
    18.         foreach (var myFile in myFiles)
    19.         {
    20.             var include = new XAttribute("Include", myFile);
    21.             var item = new XElement(nsMsbuild + "AdditionalFiles", include);
    22.            
    23.             itemGroup.Add(item);
    24.         }
    25.        
    26.         content = xDoc.ToString();
    27.        
    28.         return content;
    29.     }
    30. }
    31.  
     
  10. Kaligoth

    Kaligoth

    Joined:
    Mar 4, 2015
    Posts:
    8
    Further addition: This is not enough because the editor doesn't build using the project file, it seems...
     
  11. Kaligoth

    Kaligoth

    Joined:
    Mar 4, 2015
    Posts:
    8
    2021.3.13f1 also doesn't seem to do _anything_ given my scenario, if I want to go with the route described in the docs.

    No matter if I put in the Assembly-Name (without extension) or the actual source generator class name. The additional files passed to my generator stay empty. Not that I should have to guess what is meant by [Analyzer Name]. But of course - it is not specified in the docs. Because that would actually make sense - and we can't have that now, can we?

    Honestly: This is dog s*** - and then you guys don't even reply here - great customer service!
     
  12. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    142
    @Kaligoth Its been vacation time, so that's the reason why ive been away.

    My time working in this area we have been pushing for a more standardized solutions, that takes time to do but we also want to enable you guys to use features like Source Generators, and not to wait years until we are back at where we should be. Myself as a C#/.NET developer would expect standardized solutions, so understand your frustration.

    The current solution is a solution that will be easy to transition to the csproj SDK with globbing of what Analyzers that the csproj is including. Adding that information in Project Settings will not, and will also by result not trigger our recompilation if those files are change/removed/added because of how the compilation pipeline are driven, and would delay work like the Dotnet modernization initiative we are working on, fx. move to csproj/msbuild compilation and move from Mono to CoreCLR.

    We cant parse all additional files to every compilation(.dll), as that would mean that all internal caches in Roslyn would be invalidated if just one of them has changed.

    The suggestion to use csc.rsp is a workaround, workarounds are not meant to be great but to unblock. csc.rsp will fx. not recompile if the additional file itself changes. That would enable you to add additional files that dont use the .add file. In Csproj's you also need to specify what AdditionalFiles to add.

    Im surprised you are saying `2021.3.13f1` do not work. Ill get someone to validate that, I only checked the code and release tags if it was included or not.

    To confirm, yes Unity today do not use the generated csproj's to compile with, its driven by the ADB, a custom pipeline to calculate the graph and in the end invoke Roslyn command line. Csproj's are only there for the IDE experience, and that's what we want to get away from, where you can maintain your own csproj's, so you are in control of fx. AdditionalFiles, or extra build steps.
     
    rdjadu and goncalo-vasconcelos like this.
  13. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    Did anyone have any luck trying to get this to work? I have the exact same problem, I tried guessing what [Analyzer Name] is supposed to be, but nothing what @Kaligoth tried works. I also tried to add this to the additional compiler arguments: /additionalfile:"Assets/sdaf.json" but this completely breaks unity. All materials turn pink and running a scene gives me these errors. upload_2023-2-28_21-33-40.png

    Could we get some more hints how to make this work or what the [AnalyzerName] is? @HaraldNielsen

    Also I tried to put the arguments into csc.rsp file in the Assets folder but that didn't do anything at all.
     
    Last edited: Feb 28, 2023
  14. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    142
    > Also I tried to put the arguments into csc.rsp file in the Assets folder but that didn't do anything at all.
    This sadly was a bug introduced in 2022.2, I fixed it and will be in from 2022.2.9f1 that will be out shortly, sorry for that!

    [Analyzer Name] is the analyzers filename without extensions.
    So if you have a analyzer named MyAswesomeAnalyzer.dll, `MyAswesomeAnalyzer` would be what you need to add there. I renotified the docs team about improving the docs around this here
     
  15. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    Thank you for the quick reply. I tried again to create a file in the format abc.MyAwesomeAnalyzer.additionalfile in the Assets folder, I still get no additional files in my source generator. Is there any other step that I missed?
     
  16. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    142
    @Ondrej98 what Unity version are you using?
     
  17. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    I updated to 2022.2.8f1 a couple of days ago in hopes it'd work there.
     
  18. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    142
    @Ondrej98 can you share a simple project with the setup?
    We do have automated tests for this, so it should work as described.
    As im going on vacation tomorrow, my coworker will try to help out
     
  19. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    Here is a simple project. It contains a GeneratorTest folder with the solution for the generator. The generated code is the same as in https://docs.unity3d.com/Manual/roslyn-analyzers.html. But it also contains a comment with the number of additional files at the top. The generated code shows the number of additional files is 0, eventhough there is a file named test.GeneratorTest.additionalfile in the Assets folder.
     

    Attached Files:

  20. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    982
    Hey!
    I'm using additional files in our project (Netcode for Entities) for long time. So I can assure that both GlobalConfig and additional files works.

    I tried you sample project and I modified the generator to output the file list. I used 2022.2.8f1. This is the result:

    Code (csharp):
    1.  
    2. foreach (var file in myFiles)
    3. {
    4.    context.ReportDiagnostic(Diagnostic.Create(
    5.       new DiagnosticDescriptor("test", "rrr", file.Path, "Debug", DiagnosticSeverity.Warning, true), Location.None));
    6. }
    7.  
    8. var sourceBuilder = new StringBuilder(
    9.    "//" + myFiles.Length + " -- " + " \n" +
    10.    @"
    11.         using System;
    12.         namespace ExampleSourceGenerated
    13.         {
    14.             public static class ExampleSourceGenerated
    15.             {
    16.                 public static string GetTestText()
    17.                 {
    18.                     return ""This is from source generator ");
    19.  
    20. sourceBuilder.Append(System.DateTime.Now.ToString());
    21.  
    22. sourceBuilder.Append(
    23.             @""";
    24.      }
    25.    }
    26. }");
    27.  
    This is the output in the Editor.Log
    Code (csharp):
    1.  
    2. warning test: /Users/cmarastoni/Downloads/CodeGenTest/Library/Bee/artifacts/200b0aEDbg.dag/Assembly-CSharp.AdditionalFile.txt
    3. warning test: /Users/cmarastoni/Downloads/CodeGenTest/Assets/test.GeneratorTest.additionalfile
    4.  
    I tried by both moving the dll in the Plugin and Assets folder. In both cases I got the same list
     
  21. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    982
    I'm trying on Mac but I don't think that would make any difference.
    If you are trying to use the analyzer from the IDE (i.e visual studio) it is not going to receive any additional file. Because the generate .csproj does not contain them
     
  22. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    That's interesting, I can also see the log in Editor.log. However, when I navigate to ExampleSourceGenerated in my IDE, the first line is still // 0 --, so when the file was generated the number of additional files was 0. How does your generated code look like? Is it the same?
     
  23. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    982
    It looks like this:
    Code (csharp):
    1.  
    2. //2 --  
    3.  
    4.             using System;
    5.             namespace ExampleSourceGenerated
    6.             {
    7.                 public static class ExampleSourceGenerated
    8.                 {
    9.                     public static string GetTestText()
    10.                     {
    11.                         return "This is from source generator 03/03/2023 17:15:41";
    12.                     }
    13.                }
    14.             }
    15.  
     
  24. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    982
    I will try again, removing all the library.
     
  25. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    982
    Ok, so I'm start guessing what going on.
    I was finally able to get this
    Code (csharp):
    1.  
    2. //0 --  
    3.  
    4.             using System;
    5.             namespace ExampleSourceGenerated
    6.             {
    7.                 public static class ExampleSourceGenerated
    8.                 {
    9.                     public static string GetTestText()
    10.                     {
    11.                         return "This is from source generator 03/03/2023 17:34:19";
    12.                }
    13.              }
    14.          }
    15.  
    And now I'm sorta guessing/understood why....

    First : look at this:
    Code (csharp):
    1.  
    2. Assets/Test.cs(7,43): warning CS0436: The type 'ExampleSourceGenerated' in 'GeneratorTest/GeneratorTest.GeneratorTest/ExampleSourceGenerator.g.cs' conflicts with the imported type 'ExampleSourceGenerated' in 'Unity.EditorCoroutines.Editor, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. Using the type defined in 'GeneratorTest/GeneratorTest.GeneratorTest/ExampleSourceGenerator.g.cs'.
    3.  
    The way your generator is written (zero check) embed in all assemblies the ExampleSourceGenerated.ExampleSourceGenerated class.

    I did this sliglty change to access the genered file:
    Code (csharp):
    1.  
    2. public class Test : MonoBehaviour
    3. {
    4.     static private ExampleSourceGenerated.ExampleSourceGenerated m_Generated; <-----
    5.     // Start is called before the first frame update
    6.     void Start()
    7.     {
    8.         Debug.Log("--");
    9.     }
    10. }
    11.  
    And I used a goto definition (I'm using Rider). I have two different outcome, depending on the circumstances:
    1- rider bring me to the correct source (the file that generate the dll comes from the dll)
    Screenshot 2023-03-03 at 18.11.50.png

    But in some cases, it bring me to another temp directory, where I presume the IDE generated the file (because it is running the generator). And because we don't pass the files in the .csproj that version emit 0

    But the one emitted when compiling from the editor (normal flow) is correct.
    Screenshot 2023-03-03 at 18.20.38.png

    So I'm guessing is mostly an IDE issue. And you can prove that it work by emitting the file in your own folder inside the generator (use full path)

    Code (csharp):
    1.  
    2. System.IO.File.WriteAllText("YOUR_HOME_FOLDER_PATH/test.txt", sourceBuilder.ToString());
    3.  
    if you do that, what happen?
     
  26. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    Yeah that works thank you. I am also using Rider. What do I need to do to be able access the correct generated file in the IDE? I tried restricting the generation with

    if (context.Compilation.AssemblyName != "Assembly-CSharp") return;


    I saw it suggested elsewhere on the forum, but the generated file inside "Assembly-CSharp" is still the incorrect one. However it still somehow runs the correct file. I am a bit confused by what is happening.
     
  27. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    Is there any solution to this @CMarastoni? If I try to use it for an actual project, the Rider will show a bunch of compile errors because it uses the wrong generated files, the ones without additional files but Unity will run it fine. It's a very inconvenient workflow. How are you dealing with it?
     
  28. Ondrej98

    Ondrej98

    Joined:
    Oct 18, 2016
    Posts:
    21
    Also I am using DOTS and ECS which you say uses additional files, but I don't have this problem with Unity's code. How is Unity's usage different to my example?
     
  29. AndyUnityDeveloper

    AndyUnityDeveloper

    Joined:
    Dec 7, 2013
    Posts:
    7
    FYI Both workarounds work for me as of Unity 2022.2.15

    A) Setting the additional file as a compiler argument in the Player Settings works. It applies to all projects it seems, as the Unity Editor shows the warnings generated by the analyzer for all files in the solution.
    upload_2023-4-24_20-45-34.png

    B) I am also using the AdditionalFilesPatcher AssetPostprocessor from @Kaligoth above so that I see the same warnings in Visual Studio when I open the corresponding files from Unity.