Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice
Dismiss Notice
Join us on Dec 8, 2022, between 7 am & 7 pm EST, in the DOTS Dev Blitz Day 2022 - Q&A forum, Discord, and Unity3D Subreddit to learn more about DOTS directly from the Unity Developers.

Official Import Activity Window (2021.2.0a19 and up)

Discussion in '2021.2 Beta' started by Unity_Javier, May 26, 2021.

  1. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Import Activity Window

    The Asset Database holds a lot of information about the state of your project which has not been available to users in the past. Being able to answer questions such as “why did my asset re-import”, or “how long did this asset take to import” are all questions that come up too often and a good answer is hard to get without investing some time into each individual situation. In order to improve this, we’ve created the Import Activity Window!


    Using this window, inspecting what the AssetDatabase knows about a particular asset in detail is now possible, and the possibility for a further self-diagnosis will be made possible.

    Accessing the Import Activity Window
    New window and context menus have been added to Unity that will allow you to use the Import Activity Window.

    Window -> Analysis -> Import Activity Window

    This will show the Import Activity window with the Overview for a project selected by default.




    Right Click on an Asset -> View in Import Activity Window




    Inspector Menu -> View in Import Activity Window



    Below are some of the main features of the Import Activity Window:

    Overview Panel



    The overview is meant to provide high level information on which assets either took the longest to import, or have the most dependencies. This will help to surface the most expensive items both for import time, but also as a way to verify which assets are susceptible to being reimported as their number of dependencies increases.

    Asset List
    The following information is available, in sortable columns:
    • See when assets were imported
    • Show how long it took to import each asset and sort by import duration
    • See which Importer was used to import a particular asset
    • Clicking an asset will then show the Asset Details panel, where you can see:
      • Its GUID
      • When it was imported
      • How long it took to import (and comparison to the previous import, if applicable)
      • Reasons for import
      • Produced files in the Library folder
      • A searchable & sortable list of dependencies



    There are many more features available within the window, and we look forward to hearing your feedback on it, and we're also happy to answer any questions you may have.
     
    Last edited: Jun 4, 2021
  2. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Bump!
     
    oxysofts, makaka-org and Aras like this.
  3. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,964
    It looks super interesting, but we're not in a position to try to update a big project that would need it right now.
     
    AcidArrow, Unity_Javier and Jes28 like this.
  4. Zarkend

    Zarkend

    Joined:
    May 11, 2016
    Posts:
    20
    Looks amazing to be honest :D
     
    Unity_Javier and Jes28 like this.
  5. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    732
    Would it possible to have this as a package, and use it in older (LTS) releases? Because this looks like a life saver.
     
  6. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    @TJHeuvel-net good question!
    We didn't make this a package on purpose because the internal data structures that it depends on are tied to the AssetDatabase and that's built into the Editor, so they're quite tightly coupled. The upside is that the window should "always" be compatible with whatever editor you're on (from 2021.2.0a19 onwards).

    However, if the demand is high enough we could do something with the Databases themselves, such that the Databases could be exported to an intermediary format which this window (or other tools) could then load, but there's no plans for that just yet.
     
    oxysofts, NotaNaN and TJHeuvel-net like this.
  7. IAndrewNovak

    IAndrewNovak

    Joined:
    Nov 29, 2013
    Posts:
    76
    Hello. I have some feedback.

    Would be nice to copy asset info fields (like GUID, Path, etc).
    Also, need more info about Duration 23 ms (-50%). It is not clear where this -50 % came from.
    I didn't find the option to hide Editor Assets from the Import Activity assets list.
     
  8. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Hey @IAndrewNovak!
    Thanks for the feedback.

    Sure thing, we can make those fields copyable :)

    The -50% comes from comparing the import duration to the previous import duration of that asset. I didn’t mention it in the original post because it was getting a bit long, but in the options dropdown you can enable “Show Previous Imports” and that will add a new list of imports to the window that you can browse :)

    Currently all Assets show up, but I guess you want to only see assets under the “Assets/“ folder? Might be good to have a toggle for showing package assets, so we don’t clutter the view.
     
    ccjay and mahdi_jeddi like this.
  9. IAndrewNovak

    IAndrewNovak

    Joined:
    Nov 29, 2013
    Posts:
    76
    Thanks for the reply.
    Yes. I want to only see assets under the “Assets/“ folder.
     
  10. mahdi_jeddi

    mahdi_jeddi

    Joined:
    Jul 18, 2016
    Posts:
    216
    The reason is the most interesting part to me. Once the project reimported everything after we updated TextMeshPro package. Would've been interesting to see why Editor decided to do that.
     
  11. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,770
    My understanding is that whenever a new C# based asset postprocessor appears (or disappears), all assets are reimported. They kinda have to if you're going for 100% consistency (since asset postprocessor could potentially alter the imported result of any asset).

    Now, TextMeshPro does contain an asset postprocessor, I think (why, I don't know).

    That probably would explain it.
     
    Unity_Javier likes this.
  12. mahdi_jeddi

    mahdi_jeddi

    Joined:
    Jul 18, 2016
    Posts:
    216
    Makes sense, Possibly either had one and got removed or added one. It shouldn't have reimported everything, but I guess that's a limitation of the meta-data kept for imported assets to avoid reimporting them.
     
    XiWenbo likes this.
  13. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,313
    Is this information accessible via editor scripting API too?
     
  14. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Hi @Peter77,

    The APIs & the datastructures are marked as internal currently, so you could access them for testing with the right .asmdef, but we still kept them that way so that they can be changed, if necessary, before we make them public.
     
    Ruchir and mahdi_jeddi like this.
  15. Xelnath

    Xelnath

    Joined:
    Jan 31, 2015
    Posts:
    349
    Hi Javier, no offense to you directly, but Unity has along history of putting out cool tools that don't scale well and are super non performant.

    Can you set this up in a way that it doesn't take forever to load the screen, handles 120,000 assets gracefully (NO MASSIVE SCROLL BARS), doesn't memory alloc the entire application to death and repaints efficiently?
     
    Ramobo and nilserik like this.
  16. Ramobo

    Ramobo

    Joined:
    Dec 26, 2018
    Posts:
    212
    I'd organize it by folder, at least when ordering by name.

    Using UI Toolkit for everything (not just *some* things) and virtualizing the asset list should do it. Hell, it seems that most leaf nodes in this window are `IMGUIContainer`s. There should be misleading advertisement penalties for companies that don't use their own products.
     
    Ruchir likes this.
  17. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,752
    IMGUIContainer
    is a unity "product"...
     
  18. Ramobo

    Ramobo

    Joined:
    Dec 26, 2018
    Posts:
    212
    Is that a joke that I don't get? The quotes make it seem so.
     
  19. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,752
    No joke. There is no meaning behind what you said. They are using a unity made tool on a way they intended it. IMGUIContainer is a tool in a big, complex set of tools which allows us (and them) to use code made a while ago play along with the new system while everyone rewrite their systems to use the new one. It's software engineering 101. You don't migrate everything at once, especially not in a complex software like unity. But you already know all of this, why do I have to spell this out?
     
    Last edited: Jun 5, 2021
  20. Ramobo

    Ramobo

    Joined:
    Dec 26, 2018
    Posts:
    212
    UI Elements was added as experimental in 2017.1 and released in 2019.1, it seems. If they've updated their Mono fork in a few months, they can rewrite the Console window with UI Toolkit in 2 years. Different teams, but you get the idea. Console, Project, Hierarchy, Game, Lighting... all still pure IMGUI. The longer they keep piling changes up on the old codebase, the worse an eventual rewrite gets.

    I'm sure that you get the sentiment: Acme releases this product that's gonna fix everything and it's utterly broken because they haven't felt the pain of using it for any serious project.
     
    Last edited: Jun 5, 2021
  21. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Hi @Xelnath ,
    Thanks for the feedback.

    I had originally profiled the opening of the window using a 200,000 file project but it appears that something has regressed. I'll improve the loading time, and I'm currently exploring some approaches:
    1. Return a smaller list (i.e. a % of total assets initially or first 100 items, for example), then have a button for "load remaining assets" or something to that effect.
    2. Make the sorting faster when opening the window. I've verified this to be the biggest bottleneck on a project with 900,000 files. With #1 this might be less of an issue, but if all assets do get loaded, sorting by path, import time, duration, etc. would still be slow so there's something to explore there.

    doesn't memory alloc the entire application to death
    Will also have a look at what can be done there. I've read some of the replies to the thread and there's some ideas there, so I'll keep my eyes open for it :)
     
    stuksgens, ccjay, mariandev and 2 others like this.
  22. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Hi @Ramobo,

    So, the main issue is that parts of UIToolkit are in a package, and other parts are not. This means that for certain features I can rely on UIToolkit, while for others I have to use IMGUI as you've pointed out (the search field being another problematic area). Its a sort of catch-22, because UIToolkit benefits from a "faster" release cycle by being in a package, but we don't have access to it all in the editor, though this is incrementally getting addressed.

    Earlier in the thread its been noted that a package could be made for this, but that introduces a number of other dependencies and potential for incompatibility with different releases and versions of the editor.

    That being said, I'll double check IMGUI to see if something can be done about the tree views (caching of elements, improved code path, etc.) to see if I can decrease the repaint cost of the visible list.

    Hope that helps to shed some light into some of the decisions we made along the way!
    Thanks for the feedback so far, it helps to figure out if the priorities are in the right place. :)
     
    Wattosan, ccjay, mariandev and 6 others like this.
  23. dlorddd

    dlorddd

    Joined:
    Aug 13, 2016
    Posts:
    6
    Hi @Unity_Javier ,
    this new feature looks great. I could not try it out yet but I am curious to ask the following:
    Does the Import Activity Window highlight any activity that is related to Unity Accelerator (upload to, download from, etc.)? It would be a great benefit once we upgrade our project to stable 2021.2 in the future.
     
    codestage likes this.
  24. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    799
    Hi Javier, some more feedback after trying this out:

    - some filenames (e.g. all package.json imports) are truncated in the middle, and only show the path on hover. There's more than enough space.

    upload_2021-6-22_23-18-21.png

    - especially the dependencies list could benefit from another splitter, there's tons of space and a tiny window to scroll through them right now

    upload_2021-6-22_23-19-50.png

    - would be great if "SourceAsset/MetaFileHash/xxx", "SourceAsset/HashOfSourceAssetByGUID/xxx" and the actual source asset would be grouped (or these typical things in a foldout) - hard to see what belongs to what otherwise

    - right now it seems the _actual_ import time isn't visible, e.g. when you have URP in the project there's
    - "Decal.shadergraph" which takes 130ms and has 195 dependencies
    - "SpeedTree8_PBRLit.shadergraph" which takes 7120 (!) ms and has 222 dependencies
    I can only assume the bulk of that import time is the shader compilation but I'm not sure.
    A similar issue for e.g. "Mono.Cecil.Pdb.dll" (1885ms) and "Newtonsoft.Json.dll" (1033ms) where I can't actually see the reason why these take so long – if it's possible to show more there that would be great.

    - for Custom Dependencies it would be great if their current value and the value on import would be visible
    upload_2021-6-22_23-25-15.png

    Otherwise I agree with other voices here that performance would probably benefit from VisualElements, and that some hierarchical view would be great as well. Thanks for a great tool, please keep improving it :)
     
  25. SideSwipe9th

    SideSwipe9th

    Joined:
    Jan 10, 2019
    Posts:
    46
    Hey @Unity_Javier. This is looking awesome. I remember at the start of this year debugging some frequent asset reimports with you, and this looks like it'll be amazing for it. Can't wait until the 2021 LTS release comes and we can upgrade our projects for this!
     
    Unity_Javier likes this.
  26. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Thanks @SideSwipe9th !
    I’ve landed a couple optimizations based on feedback from this forum thread and they’ve been backported to 2021.2 as well :)
     
  27. SideSwipe9th

    SideSwipe9th

    Joined:
    Jan 10, 2019
    Posts:
    46
    Hey @Unity_Javier , with the 2021 LTS release earlier today, we've started the process of upgrading our projects. I'm definitely looking forward to digging deep into the remaining few assets that get frequent reimports at seemingly unrelated times.

    One quick question I have, with the dependency list, is there an easy way to convert the various hashes into a human readable format? I'm aware of AssetDatabase.GUIDToAssetPath. In the screenshot below there is what appears to be three GUIDs, one in Dependency Name, two in Dependency Value. The one in Dependency Name matches the GUID of the asset in the field above the dependencies list. If I was to input the two dependency values into an editor script that invokes GUIDToAssetPath and outputs the value to the debug console, would I then get the asset paths of the assets that depend on the asset that was just imported?

    If the answer to this is yes, a follow up question is will this functionality for easy lookup be included in a future release?

    Thanks!

    upload_2022-4-13_1-4-20.png
     
  28. Nyphur

    Nyphur

    Joined:
    Jan 29, 2016
    Posts:
    45
    Does the Import Activity Window only cover the first iteration? Often there are issues where an asset gets stuck in "importing (iteration 2)" for minutes at a time but this doesn't seem to be reflected in the Import Activity Window so we can't debug it.
     
    SideSwipe9th likes this.
  29. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Hi @SideSwipe9th , long time :)

    This is a usability issue that we have discussed, and its pretty annoying tbh, since you're right, its not totally clear if we're talking about a Hash or a GUID since they're both the same. I'll add this usability fix to me todo list, since I think it should be generally straightforward to do!

    Here's some snippets from my internal notes:

    SourceAsset/MetaFileHash/[guid] , [Hash128]
    This means that the guid here will give you the path to the .meta file, and the Dependency Value is a Hash128 (i.e. the hash of the contents of the .meta file)

    SourceAsset/HashOfSourceAssetByGUID/[guid] , [Hash128]
    The GUID here will point to an asset that an artifact depends on (it not just itself, since, for example, Prefabs depend on the Hash of their Nested Prefabs).

    Artifact/HashOfContent/ArtifactKey([guid]) , [Hash128]
    This dependency is registered when an asset depends on the import result of another asset, and the Hash128 value is the ArtifactID (i.e. a unique ID that states which revision of another artifact/import result a particular asset depends on).

    ImporterRegistry/PostProcessorVersion/[postprocessorType] , [Hash128]
    This is a hash of all the Postprocessors in a project which affect a particular type (i.e. ModelPostprocessor, TexturePostprocessor, etc.) and the Hash128 is a combined hash of their versions (defined in the AssetPostprocessor.GetVersion function)

    I think that should cover "most" of the illegible situations, but I can provide more info if some more come up.
     
    SideSwipe9th likes this.
  30. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Hey,
    You can see previous revisions for an asset if you enable the "Show Previous Imports" option, in the Options dropdown:


    Then you can click on the current artifact, and the difference reporting should tell you why a particular asset was reimported. We subtly highlight the current revision in bold in the window, so the latest version is the bold one, and you can use that and its "Reason for import" to track down why something reimported multiple times.

    We have a manual page for the Import Activity Window now too :)
    https://docs.unity3d.com/2022.1/Documentation/Manual/ImportActivityWindow.html
     
  31. SideSwipe9th

    SideSwipe9th

    Joined:
    Jan 10, 2019
    Posts:
    46
    @Unity_Javier yeah, been a while. Just devving away.

    Awesome. I should be able to make an editor script from this info that'll output in human readable form what the dependencies are, so that we can analyse if they're correct (eg, Prefab A depends on Mesh A) or incorrect (eg Prefab B depends on Mesh A due to it previously containing Mesh A, but doesn't any longer). As well as figure out if we have any objects that have interdependencies that maybe can be broken, preventing reimports when one of the dependencies changes.

    Also Nyphur is a colleague of mine. The question he's asking is not so much, why is this asset being reimported multiple times (which is what I was asking), but why is an asset taking what feels like unnecessarily long to import. The duration tab is fantastic for figuring out which assets take the longest to import at a glance, but it still doesn't give much insight yet into why it's taking an FBX file say 244ms to import (eg Chicken_1.FBX in your image)

    It's also unclear right now if that 244ms is the full time it takes that asset to pass through the pipeline (eg if Chicken_1.FBX is the only asset being reimported, then from when AssetDatabase.ImportAsset or AssetDatabase.RefreshAsset is invoked, to when AssetPostprocessor.OnPostprocessAllAssets is invoked), or if its only representative of what the blocking Import progress bar calls "iteration 1". As for example, in the image below this particular material has at least two import iterations.

    upload_2022-4-20_17-28-16.png
     
    Unity_Javier likes this.
  32. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Ok, I misread that :p

    So, that is *possible* but it would be a large undertaking, as we'd be adding some sort of import diagnostics to all importers, but also to various areas of the AssetDatabase. We would also need to serialize all this information, since we would most likely want to persist it. We might be able to prototype something to get a feel for how large this would actually be.

    In the meantime, using Superluminal (https://superluminal.eu/) could help give insight into what is going on, at least from a high level. I can't recommend that tool enough :D
     
    SideSwipe9th likes this.
  33. Nyphur

    Nyphur

    Joined:
    Jan 29, 2016
    Posts:
    45
    Yeah, specifically I have two main issues:
    • The import activity window shows me a series of materials that have been imported, and we happen to know there's a bug causing each of them to be imported 24 times in a row. When I go into the import activity window, it shows only the last import and there are no previous versions listed and no reason for import.
    • As SideSwipe9th pointed out, the import duration numbers in the window don't seem to match the real time spent importing. Each one of them shows import times from 2ms to 5ms, but in reality I can see that it's only importing about 2 per second. It looks to me as if it's not counting the time spent in "Iteration 2" when importing, which is by far most of the import duration and the part we would want to diagnose.
    If there are many circumstances under which the window discards previous import data, and if it can only account for iteration 1 when importing, that does limit the tool's usefulness. Are the further import iterations usually waiting on third party assets doing something, or is this still the base Unity engine?
     
    SideSwipe9th likes this.
  34. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    130
    Hey!
    So, the intermediate results should not be discarded. The only way that would happen is during startup, where we run Artifact Garbage Collection before we even do the first project scan.

    Ok, so if there are no Artifacts present, I would guess that the same asset with everything being the same keeps getting queued for import.
    One way to check that in the Editor log is by looking for a line that looks like:

    Code (CSharp):
    1. Start importing ... using Guid(adf7bea9401c1834380d55601add6cfb) Importer(-1,00000000000000000000000000000000) -> (artifact id: 'cc5e7fa392c836bafaee4dc3c65a8516') in 0.004119 seconds
    Look for the artifact id value, and see if its exactly the same each time.

    Currently there's no support for detecting if the exact same asset is force reimported multiple times, but I think its good feedback since that would help make the tool even more complete.

    Does that help?
     
  35. SideSwipe9th

    SideSwipe9th

    Joined:
    Jan 10, 2019
    Posts:
    46
    It does!

    The bugs Nyphur was referring to in his last post were in Digger and MicroSplat. Digger has an integration with MicroSplat that hooks into its terrain sync events. Here's a brief rundown of the bugs, both are now fixed in their respective assets.

    The bug in MicroSplat was that on scene save, each MicroSplat terrain was forcing every other MicroSplat terrain to sync. This resulted in a the number of sync events being the square value of the number of Microsplat terrains in your scene; eg 4 terrains = 16 sync events.

    The bug on Digger's side was that, every time MicroSplat invoked a sync event, Digger would resync itself. Because of the bug in MicroSplat, Digger would receive a number of sync events equal to the square value of the number of terrains plus the number of terrains. Eg, 4 terrains = 20 Digger syncs. As part of Digger's sync, after doing some cleanup it would subscribe a function to EditorApplication.delayCalls. That function forced the Asset Database to re-import (using ImportAssetOptions.ForceUpdate) the material variant, of the MicroSplat material, that Digger uses for each terrain it is attached to. It would do this regardless of whether there were any underlying changes to the MicroSplat material. And because EditorApplication.delayCall seems to be executed sequentially, as the number of terrains with MicroSplat and Digger in your scene increased, there would be an exponential increase in the number of assets being sequentially reimported.

    When it came to the ImportActivityWindow, you've already identified that because the same asset was being reimported multiple times, and the window doesn't detect when this occurs, it would only show the results of the last of those imports. And because of this, I now understand why we'd see the same material being imported 24 times, as we had 24 terrains in our scene, each with MicroSplat and Digger, but only see a single result in the activity window.

    This was a particularly frustrating bug for us and our sanity. Because of the high number of terrains, we'd see total save times (ie, time from when save is pressed to when we can interact with the editor again) in excess of five minutes, on machines that are fairly powerful; in my case a Ryzen 5900X with RTX3090. Thankfully fixed now though :)