How does Unity do CodeGen and why can't I do it myself?

Hello!

Two things first:
Sorry if the title is confusing. I'll clear it up soon
And I'm also sorry if this doesn't really apply to the Entities stuff (and it kinda doesn't) but it's related to the process of building Entities, kinda. It seems like this would be the only place I could get help since Entities is the only thing using the stuff I mention below.

Now, I want to do some custom code-gen by myself. I've played around with Mono Cecil and it seems Unity is doing the same. The issue is triggering it. I've used the compilation finished hook but it's a bit unpredictable in my experience, and worst of all, my modified code doesn't survive the domain reload properly. So I can't disable the "reload domain" option that was introduced in 2019.3 and I LOVE that feature!
Looking at the entities source code, it seems Unity has streamlined the process a bit more and, from my understanding at least, you CAN disable the reload domain option and it will work fine. So naturally, I started copy & pasting to try and work it out. The most interesting thing seems to be in Unity.Entities.CodeGeun.EntitiesILPostProcessor which derives from ILPostProcessor. ILPostProcessor seems to be exactly what I want! But... I can't access it in any way. The Unity.CompilationPipeline namespace is just not accessible, for some reason. It almost seems like the Entities package has special access to it.
I looked around the entire entities package but can't find a way that Unity got a hold of it while I can't. Am I missing something really obvious here or are we not allowed to access it? And if not, why?

 

Unity use `InternalsVisibleToAttribute` to make one assembly visible to another.
And you can inject your own `InternalsVisiblesToAttribute` into `Entities.dll` by Cecil to access internal objects too.
https://stackoverflow.com/a/44329684
But this injected attribute will removed after you trigger some refresh process of package (usually while code compilation in my experience).

I've post a suggestion here wish Unity team could make a package to expose some useful internal method to users, but never have response.

 

Sure, you could inject it but I suspect it will just circle around to the original issue, being problems triggering the actual injection and keeping the injection properly. But it's just my theory.

 

For `ILPostProcessor`, you can just copy `Unity.CompilationPipeline.Common.dll` to your `Assets` or package directory, and refer it from your assembly.

 

For now, this workaround seems to work. I can't test a whole lot because Unity has decided it's going to fight against me at every opportunity it gets today.
I would still like a official way of doing this though and I hope they open up ILPostProcessor in the future. I can kinda see why they don't right now because I managed to lock up Unity several times when testing it.

 

Currently the API for ILPostProcessor is an internal feature and not publicly supported. We might make additional changes to the API and behavior in the future, so use at your own risk until we document it and actively start supporting it

That said, your ILPostProcessors asmdef name needs to be start with "Unity." and end with ".Codegen", ".Compiler", ".Compiler.Tests", ".Codegen.Tests" to get a reference to Unity.CompilationPipeline.Common.dll

See C# code for details: https://github.com/Unity-Technologi.../ScriptCompilation/UnityCodeGenHelpers.cs#L36

We are currently in the process of running the ILPostProcessors in a separate ILPostProcessorRunner.exe .NET Core application and they will therefore soon no longer run inside the Unity process. This has implications for how you write your ILPostProcessor and also makes setup of debugging a bit more involved.

Also note that Mono.Cecil is quite slow and we call every ILPostProcesor on every assembly, so I highly recommend that you perform checks that allow you to early out from IL post processing as soon as possible. So you do not spend a lot of time scanning assemblies without making any changes to them, as this will have a significant impact on iteration time.

An alternative to using the new ILPostProcessors API, is to use the existing CompilationPipline callbacks

https://docs.unity3d.com/ScriptRefe...tionPipeline-assemblyCompilationFinished.html

which you can setup in [InitializeOnLoad]

https://docs.unity3d.com/ScriptReference/InitializeOnLoadAttribute.html

There will be a bootstrap issue on startup because your code hasn't been compiled and loaded yet, so you can't post process the other assemblies that are compiled at the same time.

You can use SessionState to track this

https://docs.unity3d.com/ScriptReference/SessionState.html

and request an additional ScriptCompilation on startup to post process all assemblies

https://docs.unity3d.com/ScriptRefe...ilationPipeline.RequestScriptCompilation.html

 

Awesome, thanks for the detailed response!

I've used the CompilationFinished hook before to process all assemblies and managed to get all assemblies processed properly. Just that, as I described in my post, I can't use the 'fast enter play mode' settings because the domain doesn't reload the processed assemblies, it seems. I'm not sure if that's just a fact or if I'm doing something wrong. From my understanding, ILPostProcessor doesn't have that issue since Entities recommends you have 'reload domain' off, right?

 

There is no need to post assemblies when entering play mode, only when they are recompiled and then permanent changes are made to the assemblies. ILPostProcessors also only run after compilation and modify the assemblies and do not run when entering play mode.

Once the modifications are made to the assemblies on disk, they will always be used for domain reload during enter play mode. Unity does not overwrite the assemblies or modify them in any way when entering play mode, they are just loaded from disk.

 

@lukaszunity today I tested ILPostProcessor, and it works great in editor. Unfortunately it does not work in Tiny build. My custom ILPostProcessor is not fired, and 'Library\DotsRuntimeBuild\artifacts' contains unmodified dll. How can I force it to run in Tiny build?

edit: It looks like I fixed this issue. My asmdef had *.Codegen name instead of *.CodeGen. For some reason, Unity editor allows both names while bee takes only "CodeGen".