Feedback (Case 1174117) Addressable dependency resolution is unreliable

So, we have been stuck on 0.5.3-preview for a long while now, which has been blocking upgrading to Unity 2019, and this is something that I have been trying to tackle. With every release after 0.5.3, our game just does not work. So, I have now rebuilt the Addressables setup and catalogue from scratch in 1.1.7, fixing all asset duplication issues we had at the same time. To my great surprise, the game still does not work.

After much investigation, I have concluded that there is something very wrong with dependency resolution. We have very complicated dependency graph, assets depend on other assets, which depend on other assets, etc. I have made sure that anywhere where an asset depends on an asset in another asset bundle, the latter is addressable so that it is explicitly referenced by that bundle.

There are no duplicated dependencies anywhere at this point, both by design and by analysis rule.

Now, when we load, say, the main character, we get *a lot* of spam, which did not occur in our old setup in 0.5.3:

The referenced script (Unknown) on this Behaviour is missing!
(Filename: C:\buildslave\unity\build\Runtime/Scripting/ManagedReference/SerializableManagedRef.cpp Line: 197)
The referenced script on this Behaviour (Game Object '<null>') is missing!
(Filename: C:\buildslave\unity\build\Runtime/Mono/ManagedMonoBehaviourRef.cpp Line: 333)

According to other threads, this is caused when you have dependencies multiple levels deep with duplication. This is not the case here. Moreover, what follows is even weirder.

The system uses a very bespoke version of UMA. UMA serializes character recipes as text, which can live in assets, and then references to other assets use a soft-reference system that is similar to addressables. We scrapped the latter, when recipes are serialized any references to unmanaged assets are put in an array of UnityEngine.Object so that Unity can detect dependencies and are only stored in asset files. Loading arbitrary text recipes is not supported. At deserialization time, managed references are restored using the managed reference array. This has worked for a long time now without problems... until the missing script spam started.

Now, all of a sudden, I'm getting invalid cast exceptions. What is occurring is that the native objects are deserialized correctly but their managed wrappers are instantiated as UnityEngine.Object! If I debug that code, the name of the object is correct, setting Selection.activeObject to it I can see in the inspector that all the data is there, but object.GetType() returns UnityEngine.Object!

What is even worse is that this seems to be entirely non-deterministic. Which objects fail to load varies.

I have managed to make a repro on a test project using nothing but ScriptableObjects and I am utterly stumped as to where to go from here.

A, B, C and D are separate ScriptableObject-derived classes. A, B and C have a UnityEngine.Object reference. I created the following asset structure:

All four chains (i.e. objects A1, A2, A3 and A4) are referenced by a prefab. The prefab and all objects in all chains are addressable and all in a Packed Separately group.

Chains 3 and 4 always deserialize correctly.

Chains 1 and 2 *occasionally* deserialize correctly. Most of the time, the reference to object B is dropped to an instance of UnityEngine.Object. Inspecting the new object with SerializedObject/SerializedProperty I can see that the native object is deserialized correctly, but I cannot do anything with it, since it is the wrong class in the managed domain.

If I change the types of the references to be more specific, the references are outright dropped to null instead.

... ideas?

Edit: I have now submitted this as a bug. @unity_bill @PaulBurslem it would be great to get your input on this, I am utterly stumped after spending an extraordinary amount of time in the guts of the system trying to debug this, even making asset dependency diagrams to see what on earth is going on.

 

You wouldn't happen to have any ideas, @Ryanc_unity?

 

Ah, cool. That makes sense, I did manage to repro it with both the version of addressables that worked for us and the latest, which would mean that the original wasteful packing we had just didn't trigger this. I guess I need to park my task (and hence the 2019.x upgrade for our project) until the fix is released.

 

So, QA have reproduced this issue and have also been unable to reproduce it in 2019.3a12, which is the version @unity_bill has already said this fix should land in. Now, this blocks us upgrading to 2019.x, so I hope it'll be backported at least to 2019.2. It'd be good if it was backported to LTS and 2019.1 too, although I suspect that the latter is not going to happen, as the stable TECH stream release is 2019.2 now.

 

I can confirm that the issue appears to be fixed. Just updated a fresh clone of the project to 2019.3.0a12 and for the first time since 0.5.x everything works, with our new zero-dupe packing strategy and everything. Fingers crossed for the first non-alpha version with the fix on, the game is live so we cannot have the rest of the team on the alpha.

One issue, the fix did not make it to the changelog. Neither my bug number nor the original issue in the issue tracker are anywhere in it, or anything that sounds like it for that matter. That'll make it difficult to track what version the fix lands in on the non-alpha releases.

 

Looks like this didn't make it into 2019.2.1.

 

So I have just been informed that it is going to be *at least* 2019.2.5 before this fix lands into TECH and 2018.4.9 before it lands into LTS. This is very disappointing, to be perfectly blunt.

It also raises a lot of questions as to the viability of the interrelation of Unity Package releases and core engine releases. We spend time evaluating packages, suggesting improvements and so on, only to end up in a vicious circle of a new Unity release requiring a new version of a package, which triggers an engine bug, which is fixed by a newer version of the engine, which is in alpha, which might at some point, fingers crossed, be back ported to the versions we can actually release with.

This is complicated even further by the fact that this is happening across multiple packages, not just Addressables. We are forced to update to the 19.x cycle because of issues in LWRP that are fixed by newer releases that require that version of the core engine.

 

Beautifully said

 

so given that latest of 2019.2 is .1. We can expect this to land in 2019.2 in about 4-5 weeks?

 

Thank you for the update. Basically, I got an email from QA that it the fix is approved for release but not added to the queue to be backported yet and that with the current release schedule that means "at least" 2019.2.5, which would be quite a while. It was specifically the fact that it looked like it was not even in the queue yet that worried me, as I could see .5 becoming .6, becoming .7 and so on.

To be honest, the main thing that I found frustrating is that the bug was fixed while 2019.2.0 was out. It was understandable that it was not going to hit .1 or .2, there is no way it could have been merged and QA'ed on time, but five point releases later with a qualifier of “at least” was a bit much, considering how much time both myself and everyone else who has posted here with this problem have lost on it. .4 is a much more reasonable prospect.

Still, my point stands, there is a big problem with the way package and Unity versions interrelate, especially with preview packages. For example, the way Unity alphas and betas are run at the moment, submitting bugs and then posting on the forum tends to result in rapid resolution (thanks @LeonhardP! ). When, however, a preview package is being tested and it unearths issues in the core engine, the same priority is not given. Seeing how packages lose support for older (even LTS) versions of the engine rather quickly, that is not ideal.

 

Thank you for the explanation, and that’s really good, it allows us to transition to 2019.2 a fair amount of time before the next release and the workflow improvements I’ve achieved with the latest addressables, the new catalogue and my performance fixes are tremendous. Bundle building time was reduced from anything between 1:30hr and 4hr down to minutes when a lot of content is cached, which is most of the time.

It would be beneficial to be upfront about this stuff, as we can then be more confident when testing preview packages. The more we can test them, the closer they will be to our needs and expectations as developers upon release.

I do have to admit, it’s spoiling me that I can go in, investigate and fix issues in our build process without hitting the black box that is the native side of the engine for people that don’t have a source licence quite as often, this was absolutely not the case before the SBP landed.

Understood, I expect that part of the engine to be a bit of a minefield, considering how much would break if it goes wrong.

 

This is excellent news! Thanks Ryan!

 

So, the fix has landed on 2019.2.3f1! One slight bit of weirdness, getting a lot of missing script warnings that were happening when the error was occurring. These warnings do not happen in 2019.3.0b1.

 

Yep, cleared the build cache and rebuilt several times since then. The dependencies are now loaded correctly, the only hint of the issue having being there is the warning.

Interestingly enough, the warning does not happen on my repro project, only on the real project, and so far it *appears* to be harmless.

 

Hm, wouldn't that be Editor-only? I'm getting this on builds as well.

 

I'll see if I can make a repro, I cannot attach the real project itself, for contractual reasons.