Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug Hidden Dependencies AssetDatabase.GetDependencies & Addressables

Discussion in 'Addressables' started by daxiongmao, Nov 3, 2021.

  1. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    I am currently trying to clean up my projects dependencies for helping to build addressable bundles.

    I have a tool that makes it easier to se the results of AssetDatabase.GetDependencies.

    I have identified several cases where dependencies show in this list and probably should not.
    I need to still verify if these issues translate into a final build.
    I have tested addressable systems where setting some of these cause new items to show up in the final build list analysis rule.

    1. Particle Systems: particle systems store values even when a setting has changed not using that property anymore. The one I found is Shape Emitter, when setting to mesh, the mesh is still assigned even when switching to another shape type.

    2. Materials: any texture slot ever stored in a material stays set, even when changing the Shader. These show as dependencies.

    3. Changing code and removing a property that has references to other items. The editor interfaces show the property gone but the actual prefab still has the data saved. (In this case I did not checkout the prefab first, but from just having used unity a lot, I don't believe is causing it to not update). To fix it you have to change something save and it removes the data.

    4. Scenes: I had a prefab that had some of the issues about. I went through and cleaned up everything. But it was still showing up. Deleting the object. Removed it. Undoing that brought it back. Editing the scene manually shows the GUID still that was removed. Making changes, reordering and anything i tried would not cause that entry to be removed when saved. The only way I found to fix it was delete the old prefab and then drop in the prefab again. Then saving the scene the data was removed and the dependency gone.

    Maybe when doing a build some more smarts are being employed to eliminate these issues but like I mentioned before it doesn't seem addressable is doing it.

    Hopefully something could be done to make the dependency checking a little smarter to remove these.
     
    LaurieAnnis likes this.
  2. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    5: Animation controllers. Removing a state does not remove the reference to the animation. Still haven't figured out how to remove it without editing the file. But not seeming to see in the addressable output buildlayout.txt. There is a hidden Layer in the controller that is not anywhere in the UI.

    So maybe the addressable has some better logic in it. Would be nice if you could be moved into AssetDatabase so easy to correlate things.
     
    Last edited: Nov 3, 2021
    LaurieAnnis likes this.
  3. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    6: Another one is FBX models, if they have had their textures extracted at some point then later set to material import = None, they hold the references to the external materials still.
     
    LaurieAnnis likes this.
  4. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,816
    Flagging for the team to review!
     
  5. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    Thanks.
    I have not verified how many of these false positives make it into the builds or addressable.
    I have seen some get added in and others not.

    It would be nice to have this function match what the build would produce.


    Unity 2021.1.25
    Addressables: 1.18.19
     
    Last edited: Nov 6, 2021
  6. pillakirsten

    pillakirsten

    Unity Technologies

    Joined:
    May 22, 2019
    Posts:
    346
    Hi @daxiongmao thanks for the feedback! I would suggest to use the ContentBuildInterface methods instead. GetPlayerDependenciesForObjects or CalculatePlayerDependenciesForScene in particular might be useful.

    AssetDatabase.GetDependencies contains editor-only dependencies, and excludes dependencies that are determined dynamically at build time.

    Otherwise using the buildlayout.txt file is the best way to determine what dependencies exist.
     
  7. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    Thanks,

    I will look at those.
    I was hoping to avoid making builds to find dependencies.

    Any chance of getting buildlayout.txt in a format that would be easily parsed?
    Any chance of dumping a buildlayout.json too?

    I have been doing some work to parse it already. It would be nice as well that the total size listed on the group could be of the total contents and the final build.
    Currently I am trying to parse the assets in it to calculate some total memory sizes they seem to be uncompressed while the groups total size is the final bundle compressed size.

    I am building tools to estimate the total memory usage of a character in a our game by finding references, textures sizes and mesh sizes etc. And trying to make it so the addressable groups contain no indirect dependencies. In a more automated way.

    Having to pull data together from Profiler.GetRuntimeMemorySizeLong, build layout to verify and the profiler on device to confirm.

    Profiler.GetRuntimeMemorySizeLong seems to report 2x for texture memory. I think its mentioned elsewhere on the forums in the editor.
     
  8. pillakirsten

    pillakirsten

    Unity Technologies

    Joined:
    May 22, 2019
    Posts:
    346
    There are some plans to display the buildlayout.txt data in a new UI tool/window, but sadly it's not high priority at the moment.

    Good to know about the group size in the buildlayout.txt. I'll create a ticket to see if we can add that data.

    Have you tried using the "Bundle Layout Preview" rule in the Addressables Analyze window (Window Asset Management > Addressables > Analyze)? It will list the direct and indirect dependencies for all your addressable groups.

    Here's a small example of a prefab (Assets/Cube.prefab) that uses a custom material (Assets/New Material.mat):
    upload_2021-11-23_10-41-55.png

    Note that the "Bundle Layout Preview" rule runs some build tasks to calculate the dependency data, so it can still consume quite a bit of time.

    Calling Profiler.GetRuntimeMemorySizeLong in the Editor is an estimate at best since the value returned is platform dependent. Profiling on the device should give the most accurate information.
     
    Last edited: Nov 23, 2021
  9. LilMako17

    LilMako17

    Joined:
    Apr 10, 2019
    Posts:
    43
  10. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    @pillakirsten Thanks.

    The texture memory seems to match up with iOS at least when you divide by 2. And then convert to MB using 1024^2 and rounding to one decimal. Lol

    I have seen the analysis build rule and have used it. I just can’t do anything with it. My tools now let me select the dependencies and add them to groups. If there could be more functionality it would be more useful. Now its output is unusable from automation standpoint.

    I tried making my own analyzer but its not real clear what is going on there exactly and i didn't want to invest time trying to figure it out. And that analyzer makes use of a lot of internal classes that are not available outside the package. I am using addressable as an internal package right now because of other issues so I could do it but it seemed too much effort.

    Which is where if there was a more usable format for the outputs it would make it a lot easier to create tools around.

    @LilMako17 Thanks, I will check that out.

    Reading the post it seems he is dealing with same issues trying to parse that file.
    The output format is not good for parsing. Seems mainly for human reading. Which doesn't help a lot.
     
  11. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    @pillakirsten I did try the ContentBuildInterface I got it working somewhat, but there isn't much documentation or examples on it. I found some other packages making some use of it. But It seems easy to get stuck in infinite loops due to it being unclear to use and getting circular references.
    Even the code i found using it seems to have to add a lot of extra stuff to make use of it.

    Which is why the AssetDatabase.GetDependencies is so much easier to use.
    Would really like to have an API that is that easy to use but give build time results.
    Seems it is something that unity has already built and uses in a few places if it could be exposed with the simpler interface.
     
    Petr777 likes this.
  12. Petr777

    Petr777

    Joined:
    Nov 8, 2017
    Posts:
    49
    @pillakirsten I agree with @daxiongmao , it would be cool if Unity can provide simple API for determining dependencies "as is in the build"
    Are there any plans to create an API for that?
     
  13. pillakirsten

    pillakirsten

    Unity Technologies

    Joined:
    May 22, 2019
    Posts:
    346
    Last edited: Feb 18, 2022
  14. pillakirsten

    pillakirsten

    Unity Technologies

    Joined:
    May 22, 2019
    Posts:
    346
    @daxiongmao Yeah the way the Analyze window was implemented, the results printed out are just read-only/non-interactable. The immediate solution we can do is expose some of the internal API or maybe add some new API that would return useful info (i.e. return a list of addressable groups objects instead of just the group names).

    What specific APIs did you want to be exposed?
     
  15. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    412
    I would take the buildlayout.txt in CSV at a minimum. Or the addressable build analyzer output in JSON.
    Something that is easier to parse. JSON would probably be a lot better.

    upload_2022-2-19_2-25-7.png
    Could this just be written to a file as well? I don't think you can even copy paste this at the moment.
    Better would be to include the sizes of those assets that are listed in the Buildlayout.txt.

    You obviously have the code somewhere to figure it all out. Can that be put up as an example?
    Or wrapped in a internal API that has a callback for every dependency.
    As i mentioned in the suggested features, I bet there is already an in memory representation of this data before dumping to the logs. How about just serializing that to JSON somewhere.

    Really like I mentioned previously having an API that works like AssetDatabase.GetDependencies but excludes assets not in the build would be really nice.

    I think exposing something like this lets up build tools to help where addressable is missing.
    My main tool now make it much easier to work with groups. And not having vs just saying find everything missing or duplicated and throw it into some large group and then having to sort it out later.

    Its one of the things unity is usually good about, having access for us to build tools. So Addressable or Build teams can focus on core stuff.

    I submitted the ideas.
    Thanks[/user]
     
  16. pillakirsten

    pillakirsten

    Unity Technologies

    Joined:
    May 22, 2019
    Posts:
    346
    @daxiongmao Awesome thanks for the feedback! I've created a ticket on the Addressables board to improve the usability of the "Bundle Layout Preview" analyze rule. And a second ticket to add the ability to export buildlayout as a json.

    The requested API (having something similar to AssetDatabase.GetDependencies) will be handled by the engine build pipeline team.
     
    Last edited: Feb 22, 2022