Search Unity

Feedback (Case 1174117) Addressable dependency resolution is unreliable

Discussion in 'Addressables' started by AlkisFortuneFish, Aug 3, 2019.

  1. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    So, we have been stuck on 0.5.3-preview for a long while now, which has been blocking upgrading to Unity 2019, and this is something that I have been trying to tackle. With every release after 0.5.3, our game just does not work. So, I have now rebuilt the Addressables setup and catalogue from scratch in 1.1.7, fixing all asset duplication issues we had at the same time. To my great surprise, the game still does not work.

    After much investigation, I have concluded that there is something very wrong with dependency resolution. We have very complicated dependency graph, assets depend on other assets, which depend on other assets, etc. I have made sure that anywhere where an asset depends on an asset in another asset bundle, the latter is addressable so that it is explicitly referenced by that bundle.

    There are no duplicated dependencies anywhere at this point, both by design and by analysis rule.

    Now, when we load, say, the main character, we get *a lot* of spam, which did not occur in our old setup in 0.5.3:

    The referenced script (Unknown) on this Behaviour is missing!
    (Filename: C:\buildslave\unity\build\Runtime/Scripting/ManagedReference/SerializableManagedRef.cpp Line: 197)
    The referenced script on this Behaviour (Game Object '<null>') is missing!
    (Filename: C:\buildslave\unity\build\Runtime/Mono/ManagedMonoBehaviourRef.cpp Line: 333)

    According to other threads, this is caused when you have dependencies multiple levels deep with duplication. This is not the case here. Moreover, what follows is even weirder.

    The system uses a very bespoke version of UMA. UMA serializes character recipes as text, which can live in assets, and then references to other assets use a soft-reference system that is similar to addressables. We scrapped the latter, when recipes are serialized any references to unmanaged assets are put in an array of UnityEngine.Object so that Unity can detect dependencies and are only stored in asset files. Loading arbitrary text recipes is not supported. At deserialization time, managed references are restored using the managed reference array. This has worked for a long time now without problems... until the missing script spam started.

    Now, all of a sudden, I'm getting invalid cast exceptions. What is occurring is that the native objects are deserialized correctly but their managed wrappers are instantiated as UnityEngine.Object! If I debug that code, the name of the object is correct, setting Selection.activeObject to it I can see in the inspector that all the data is there, but object.GetType() returns UnityEngine.Object!

    What is even worse is that this seems to be entirely non-deterministic. Which objects fail to load varies.

    I have managed to make a repro on a test project using nothing but ScriptableObjects and I am utterly stumped as to where to go from here.

    A, B, C and D are separate ScriptableObject-derived classes. A, B and C have a UnityEngine.Object reference. I created the following asset structure:
    graphviz (1).png
    All four chains (i.e. objects A1, A2, A3 and A4) are referenced by a prefab. The prefab and all objects in all chains are addressable and all in a Packed Separately group.

    Chains 3 and 4 always deserialize correctly.

    Chains 1 and 2 *occasionally* deserialize correctly. Most of the time, the reference to object B is dropped to an instance of UnityEngine.Object. Inspecting the new object with SerializedObject/SerializedProperty I can see that the native object is deserialized correctly, but I cannot do anything with it, since it is the wrong class in the managed domain.

    If I change the types of the references to be more specific, the references are outright dropped to null instead.

    ... ideas?

    Edit: I have now submitted this as a bug. @unity_bill @PaulBurslem it would be great to get your input on this, I am utterly stumped after spending an extraordinary amount of time in the guts of the system trying to debug this, even making asset dependency diagrams to see what on earth is going on.
     
    Last edited: Aug 3, 2019
  2. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
  3. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    221
    Ya, this is due to an engine bug that has a fix currently going through the approval and merge process now. Will let you know when it is fixed and back ported.
     
    unity_bill likes this.
  4. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    Ah, cool. That makes sense, I did manage to repro it with both the version of addressables that worked for us and the latest, which would mean that the original wasteful packing we had just didn't trigger this. I guess I need to park my task (and hence the 2019.x upgrade for our project) until the fix is released.
     
    unity_bill likes this.
  5. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    So, QA have reproduced this issue and have also been unable to reproduce it in 2019.3a12, which is the version @unity_bill has already said this fix should land in. Now, this blocks us upgrading to 2019.x, so I hope it'll be backported at least to 2019.2. It'd be good if it was backported to LTS and 2019.1 too, although I suspect that the latter is not going to happen, as the stable TECH stream release is 2019.2 now.
     
  6. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    221
    Was just about to post in here that the fix landed in 2019.3a12 just the other day. Backports have been opened for 2019.2, and 2018.4. As you mentioned a backport to 2019.1 won't happen as the tech stream is now 2019.2. Sorry I missed that cutoff window.
     
    unity_bill and AlkisFortuneFish like this.
  7. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    I can confirm that the issue appears to be fixed. Just updated a fresh clone of the project to 2019.3.0a12 and for the first time since 0.5.x everything works, with our new zero-dupe packing strategy and everything. Fingers crossed for the first non-alpha version with the fix on, the game is live so we cannot have the rest of the team on the alpha.

    One issue, the fix did not make it to the changelog. Neither my bug number nor the original issue in the issue tracker are anywhere in it, or anything that sounds like it for that matter. That'll make it difficult to track what version the fix lands in on the non-alpha releases.
     
  8. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    Looks like this didn't make it into 2019.2.1.
     
  9. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    So I have just been informed that it is going to be *at least* 2019.2.5 before this fix lands into TECH and 2018.4.9 before it lands into LTS. This is very disappointing, to be perfectly blunt.

    It also raises a lot of questions as to the viability of the interrelation of Unity Package releases and core engine releases. We spend time evaluating packages, suggesting improvements and so on, only to end up in a vicious circle of a new Unity release requiring a new version of a package, which triggers an engine bug, which is fixed by a newer version of the engine, which is in alpha, which might at some point, fingers crossed, be back ported to the versions we can actually release with.

    This is complicated even further by the fact that this is happening across multiple packages, not just Addressables. We are forced to update to the 19.x cycle because of issues in LWRP that are fixed by newer releases that require that version of the core engine.
     
    RecursiveFrog and zornor90 like this.
  10. zornor90

    zornor90

    Joined:
    Sep 16, 2015
    Posts:
    156
    Beautifully said
     
  11. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    221
    First a small status update, it looks like the fix is on track for landing in 2019.2.4f1 unless something critical occurs such as a critical security flaw requiring an update. Which happened last time I gave exact dates on a fix on the forums just like this. The LTS release is still looking like 2018.4.9f1 as we require backports to land in the higher versions first to avoid situations where a user updates from latest 2018 to latest 2019 and functionality has reverted just because a fix has not landed.

    I realize it's really frustrating waiting for fixes to land, especially since I was the engineer that worked on this particular issue and really try to make it so fixes can be landed in backports as quickly and smoothly as possible. I'm really sorry I could not get this fix out to you all sooner, but we do have a process in place that ensures quality of backporting fixes just so we don't introduce more bugs than we fixed. This process, as well as the process of verifying the release before we ship it out takes time.
     
  12. CDF

    CDF

    Joined:
    Sep 14, 2013
    Posts:
    740
    so given that latest of 2019.2 is .1. We can expect this to land in 2019.2 in about 4-5 weeks?
     
  13. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    Thank you for the update. Basically, I got an email from QA that it the fix is approved for release but not added to the queue to be backported yet and that with the current release schedule that means "at least" 2019.2.5, which would be quite a while. It was specifically the fact that it looked like it was not even in the queue yet that worried me, as I could see .5 becoming .6, becoming .7 and so on.

    To be honest, the main thing that I found frustrating is that the bug was fixed while 2019.2.0 was out. It was understandable that it was not going to hit .1 or .2, there is no way it could have been merged and QA'ed on time, but five point releases later with a qualifier of “at least” was a bit much, considering how much time both myself and everyone else who has posted here with this problem have lost on it. .4 is a much more reasonable prospect.

    Still, my point stands, there is a big problem with the way package and Unity versions interrelate, especially with preview packages. For example, the way Unity alphas and betas are run at the moment, submitting bugs and then posting on the forum tends to result in rapid resolution (thanks @LeonhardP! :) ). When, however, a preview package is being tested and it unearths issues in the core engine, the same priority is not given. Seeing how packages lose support for older (even LTS) versions of the engine rather quickly, that is not ideal.
     
  14. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    221
    Tech releases usually are 1 release a week. 19.2.1 was released the 14th so 19.2.4 should be the 4th. Again, this is if everything goes smoothly with 19.2.2 and 19.2.3.

    Monday when I was providing that information to QA, it had not been added to the queue, it was just added yesterday. QA was also trying to cover his bases with the possibility of it getting pushed back due to unforeseen release issues this the "at least" note. Also I read the release calendar incorrectly which didn't help the estimate I provided him. I'm fairly certain I read the release calendar correctly this time.

    I fully understand that frustration, and that is definitely something I feel we can do better on for communicating expectations of packages and expected version support while it is in preview. For us, Addressables & SBP have very little integrations into the editor, and because of this, we have not had to make breaking changes to require changing the minimum supported Unity version. However other packages do not have as small as an editor footprint which is where situations like what you are describing with the LWRP have come from.

    One of the biggest reasons for the slow turn around on this fix was that I almost had to make a breaking editor change that would not have been able to be backported and struggled with finding a way to fix this without needing to break compatibility for close to 3 weeks.
     
    CDF likes this.
  15. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    572
    Thank you for the explanation, and that’s really good, it allows us to transition to 2019.2 a fair amount of time before the next release and the workflow improvements I’ve achieved with the latest addressables, the new catalogue and my performance fixes are tremendous. Bundle building time was reduced from anything between 1:30hr and 4hr down to minutes when a lot of content is cached, which is most of the time.

    It would be beneficial to be upfront about this stuff, as we can then be more confident when testing preview packages. The more we can test them, the closer they will be to our needs and expectations as developers upon release.

    I do have to admit, it’s spoiling me that I can go in, investigate and fix issues in our build process without hitting the black box that is the native side of the engine for people that don’t have a source licence quite as often, this was absolutely not the case before the SBP landed.

    Understood, I expect that part of the engine to be a bit of a minefield, considering how much would break if it goes wrong.