2018.3 Package Manager and ScriptAssemblies problems

Sorry about this long post. Summary at the bottom!

I am upgrading a project to 2018.3 and having various issues with package manager and the ScriptAssemblies folder. I think some of these issues are related to how we compile using dlls and import them to Unity. I am interested in hearing Unity's position on how to best handle external compilation with packman dependencies. With the addition of PackMan it is now necessary to import dlls from Unity's ScriptAssemblies folder in order to write code against it, and I'd like to know how Unity expects this to work.

Our workflow up until now has been to add ScriptAssemblies to our search path, and this has been very stable for a while. This is necessary for writing code against, say, TextMeshPro. But 2018.3 seems to be different.

It seems that in a recent-ish version the behaviour of ScriptAssemblies has been modified to wipe the directory. It seems to do this upon crash, but also upon TypeLoadException. For example if a nuget dependency accidentally gets added to both the output and Editor/ folders then it seems Unity wipes the ScriptAssemblies folder and it makes a bad situation worse.

PackMan doesn't seem to recover well from this. Sometimes it'll just not update, and I find that I have to pick a package and choose 'Reimport' to revive it.

Knowing when/what triggers a ScriptAssemblies wipe is kind of important if you are compiling against it. When it wipes I can't compile in Rider/VS because the deps disappear. I would like to know what does this, and also if any of these behaviours are considered to be bugs and issues.

There is an issue with the ScriptAssemblies getting wiped at startup, then loading in external dlls that depend on them. Unity will emit an error that looks like:

Code (CSharp):

1. Assembly 'Assets/lib/myLibrary.dll' will not be loaded due to errors:
2. Unable to resolve reference 'SomeOtherLibrary'. Is the assembly missing or incompatible with the current platform?

What seems to be happening is that my dlls are loaded, then they throw the exception because the given library it depends on doesn't exist. Then Unity compiles and the dlls exist in ScriptAssemblies, then I *think* everything is ok? It would be nice if unity could load the dlls *after* the ScriptAssemblies have been regenerated and not throw the error.

Lastly we have a real chicken and egg problem with package manager. We need it to compile its dlls so we can compile ours. We constantly get into weird states because of this. Ideally I would like to run package manager somehow to compile these dlls without necessarily running unity. This would be a huge boon. Packman really should be its own binary imho, which is controlled by unity.

So in Summary:
* What are the best practices for compiling code externally that relies on code in packages?
* What are the situations in which unity decides that it should wipe ScriptAssemblies?
* Are any of these situations bugs?
* Can we...not do this?
* Or put packages in their own folder?
* Feature Request: Make PackMan its own binary.

Apologies for the long post. Thanks!

 

1) you should not reference anything inside

Library/

. that folder is for Unity internal operations (importing, compiling, cache, etc...). it also gets wiped on 'Reimport All' (or manually by the user). also you should not version control it, therefore new clones will not have that folder in the first place
2) external DLL should not reference the result of source compilation inside Unity, as the source will reference every DLL and you end up with circular references (a very bad thing)
3) you can use Script Compilation API to compile scripts outside the Assets folder from inside Unity

 

Hi M_R, thanks for the reply.

What we're doing is compiling in VS, Rider or msbuild and creating our dlls outside of Unity and putting those dlls in unity. We have been doing our code this way for years due to the nature of our codebase. So alas your points 2&3 are not relevant to my situation as our code does not reside in the Unity Project in an uncompiled form. I do appreciate your reply tho.

This is why I'm asking Unity what we're supposed to do here. Unity is putting its PackMan-compiled code in ScriptAssemblies. When we need to link against it, its becoming more problematic in 2018.3 than it has been due to the problems i mentioned.

 

point 2 still applies, as you are putting your DLL in Unity. then Unity will add your DLL as a reference to everything that it compiles (including packages). therefore circular references (even if they are not used)

if you need to interact with package code, you can build a bridge:
- have a DLL with interface definitions. this will be referenced by your code. neither this nor your code should reference packages directly
- have scripts in Assets/ (or a custom upm package) implement your interfaces with (asmdef) references to target Unity Package
- initialize at runtime via DI

e.g:

Code (CSharp):

1. // Bridge DLL
2. public interface ITextMeshPro { void SomeAPI();}
3. public static class Factory {public static Func<ITextMeshPro> Create;}
4.  
5. // inside Unity
6. public class TextMeshProBridge : ITextMeshPro {
7.     /// implement API
8.     [RuntimeInitializeOnLoad] static void Init() {Factory.Create = () => new TextMeshProBridge();
9. }
10.  
11. // your other DLL
12. ITextMeshPro tmPro = Factory.Create();

or pull the bridge with reflection (i.e. scan all the assembly for any type implementing your interface and use that)


ps: we also had most stuff inside external DLL before the advent of asmdefs, but we didn't reference Unity from that (only pure logic was there, anything we needed was pushed at runtime into it)

 

The Library folder is not versioned and controlled by Unity, you should not add your own files to it.

You can setup a reference to the built package assemblies in Library/ScriptAssemblies and then emit your assembly into the Assets folder and then disable "Auto Referenced" (new 2018.3 feature) in the Plugin Inspector. This way your .dll will not be referenced automatically by all scripts, but only by .asmdef assemblies with explicit references to it.

If you look at the dependency graph on the Script compilation and assembly definition files page, you will see that you are inverting the dependencies according to how script compilation in Unity works. I can think of 2 issues that this will cause of the top of my head, there are probably more.

1) Precompiled assemblies are loaded before Unity compiles assemblies, so on clean startup without a Library/ScriptAssemblies folder, your precompiled assembly will be missing a reference and Unity will emit an error about a missing reference. I think there might be some cases where this doesn't happen, but it is not well defined behavior.

2) The script updater does not support the scenario.

We have seen people add references to Assembly-CSharp.dll in the past, before packages and .asmdefs were a thing and issues related to it. It can probably be made to work if you careful with how you handle .dlls and just be aware that it is not a setup we officially support currently.

 

An interesting idea! Definitely a bit heavy handed to have to wrap any PackMan library we want to use in our own interface. We'd also have to deal with the inability for interfaces to be accessible via the editor. Perhaps a non-interface solution would be more ideal. Thank you for this idea. While I'm not sure we will do something like this, it is always good to have options.

 

We are not adding files to it. I'm unsure as to what I may have said that might have alluded to this.

Thanks for pointing me to these new features! Quite interesting. But as I've mentioned, we have no code in the project itself, it is all in DLLs and so nothing would reference it.

I think there is some misunderstanding about our process so I will attempt to be more clear.
* All code is kept outside of all unity folders. It is divided into a large number of small dlls. Based on the project, the appropriate .csproj files are included into a .sln file.
* Our search paths are set up to include the correct version of Unity's dlls, and also the ScriptAssemblies folder, and of course our own folders. We are not referencing Assembly-CSharp.dll. The only reason we have recently added ScriptAssemblies to the search path is to handle Packman compiled Assemblies (aka TextMeshPro).
* We compile the code and put the resulting .dll files inside the Assets/ folder.

So we have no circular dependencies. We do reference unity dlls. But these dlls do not reference our code. Even if things referenced the code in these dlls in the assets folder, i do not see how this could become circular.

This is closer to our issue. These files get yanked out from under us for...some reason at some time? Not just on startup. Unity is putting files that need to be referenced by us in a volatile location. I'm asking for Unity to define this 'not well defined' behavior. If this is unsupported or poorly defined then is Unity basically saying 'don't compile your code externally'? Because as more code gets moved in to PackMan then what are we supposed to do?

We are not doing this, and wow, people would actually do that? That seems like a bad idea and understand why you wouldn't support it!

Thanks for your reply!

 

you can have your serialized fields be of type UnityEngine.Object, then cast/wrap when accessing them. or use abstract classes instead of interfaces:

Code (CSharp):

1. // DLL
2. public abstract class TextMeshProReference : MonoBehaviour {
3.     // apis you intend to use
4. }
5.  
6. // Bridge
7. public class ActualTMProReference : TextMeshProReference {
8.     // implementation and reference to TMPro stuff
9. }

or

Code (CSharp):

1. [Serializable] struct TMPReference {
2.     [SerializeField] Object ref; // <- write a custom inspector for this, so you only drag actual TMPro objects
3.     public ITMPro Value => Bridge.Create(ref);
4. }

where

Bridge.Create

will pull the correct wrapper from the implementation dll

as I said, packages would still reference DLLs in Assets folder. idk if it's intended or a bug

yes, you should consider Library contents as volatile. it gets wiped when you Reimport All, do a clean+build, or clean the various caches that exist in there. also the scripts get completely recompiled when you change platform or the defines, they may wipe the destination for safety reasons (e.g. if an asmdef has a define constraint, it must not exist if the symbol is not defined)

you can also grab the sources from the packages, compile them yourself and use the resulting DLL as one of your own, without having the actual package in the project

 

I have been attempting to do this myself with TextMesh Pro in the last few days, but have been unsuccessful because of all of the editor scripts and static files it uses. My end goal would be a Nuget package with a DLL of the compiled runtime and editor scripts, and all of the static content files that it relies on for custom inspectors, etc. All with minimum alterations to the original TextMeshPro package folder so that updates are relatively easy.

 

problem is that if you move the TMPro scripts out and use a DLL instead, the GUIDs will change and you need to reassign all the references.

I still think the bridge approach would be better. maybe you can use a tool to autogenerate code for it, something like this