Search Unity

  1. Calling all beginners! Join the FPS Beginners Mods Challenge until December 13.
    Dismiss Notice
  2. It's Cyber Week at the Asset Store!
    Dismiss Notice

Building Asset Bundles "Compiling Scripts..." step repeats for each bundle?

Discussion in 'Editor & General Support' started by BurningToad, Mar 16, 2016.

  1. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    Hello,

    I'm trying to convert a very large project (100gb project folder, 250+ scenes) to use the latest Asset Bundle system, in hopes of saving build time for builds with small changes between them.

    Currently we use the old asset bundle system for a few types of assets, but the scenes are all built with the standard Unity build process. I am trying to convert these scenes (and their required assets) to asset bundles. I've ended up with a process that I think is very similar to how Unity builds scenes by default, with each scene in it's own file (now asset bundle), and shared assets files to go with each scene (separate asset bundle from the scene.)

    Now, building these all the first time seems to take 4-5 hours (a regular build for us is about 2.5 hours.) An incremental asset bundle rebuild with no, or very small changes, goes quickly (minutes.) However, during the initial build, and any other build with a decent number of changes, I've noticed while it is building the bundles, it seems to repeat a "Compiling Scripts..." step for every bundle (or at least every scene bundle.) This step in our project takes about 40-60 seconds unfortunately, and because we have so many scenes, that really adds up. The other parts of the asset bundle build steps seem to go very quickly, but then it get stucks on "Compiling Scripts..." again, etc. So in effect, it looks as if the build process is compiling the scripts ~250+ times when we do a full asset bundle rebuild. Is this really necessary? Is it actually compiling scripts in this step, or doing something else? I don't see why they would need to be recompiled for every asset bundle.

    We are on 5.3.3p3. I have tried various asset bundle build options (such as DisableWriteTypeTree and IgnoreTypeTreeChanges) and they all seem to behave the same in this respect.

    Has anyone else noticed this, and found a way to avoid it, or is there perhaps something Unity could do to improve this?

    Thanks
     
  2. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    I double checked, and it is indeed only asset bundles with scenes inside them that cause this "Compiling Scripts...." step. So, I can speed things up a bit by putting more scenes per asset bundle, but for dependency and other reasons, I would much prefer to have one scene per asset bundle.
     
  3. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    I've also confirmed that this happens in a sample project, such as the Asset Bundle Manager sample from Unity. Just add extra scenes, and put them in their own bundle, and build asset bundles. You can see the Compiling Scripts step flash by... it goes quick in this case because script compilation is fast, but in a project like ours with 40+ second compile times... it is a real pain.
     
  4. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    I submitted a bug report for this, 780886
     
  5. SteenLund

    SteenLund

    Unity Technologies

    Joined:
    Jan 20, 2011
    Posts:
    452
    Hi,

    Thank you for the report. I was actually just made aware of this issue by an other customer 2 days ago. It is a bug script compilation should of course only happen once.
     
  6. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    Great, glad you guys are aware now! A fix for this would really help out!
     
  7. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
  8. Dizzy-Dalvin

    Dizzy-Dalvin

    Joined:
    Jul 4, 2013
    Posts:
    46
    I don't know if it's a separate issue, but not only does it recompile all the scripts for each of the bundled scenes, it also rebuilds all the associated assets and that takes a huge amount of time. Any minor change to one asset, and it will rebuild all of the scenes (in our case it takes about 20 mins). It even does so for empty scenes.
     
  9. Vincent-Zhang

    Vincent-Zhang

    Unity Technologies

    Joined:
    Jan 23, 2013
    Posts:
    106
    It should be a separate issue, seems asset bundle incremental build totally breaks for you. Could you please open a bug for it with your project? Then we can have a look.
     
  10. Dizzy-Dalvin

    Dizzy-Dalvin

    Joined:
    Jul 4, 2013
    Posts:
    46
    Yeah, you're absolutely right. After further investigation, we have found out that it happens somewhat randomly and is not limited to scenes. It won't rebuild the bundles when run immediately after the previous build, but sometimes just 15 minutes later Unity decides a bunch of bundles need to be rebuilt, often even empty scenes and small single object bundles that reference only a model, texture, material and a shader. Looking at Editor.log we can see the list of files included in each bundle and we're sure none of them changed in any way (git also conifrms this).

    I would file a bug, but our assets folder is currently 24 Gb, so I think attaching it is going to be problematic. We've tested on a clean project and everything worked fine, so the problem seems to be with our code. Now, is there a way to debug how Unity decides if the assets were changed? Even if there may not be an API for this, maybe we could reimplement the check ourselves (calculating hashes?) to try to catch the moment when this happens and for which files to further detect what causes this.
     
  11. lishuai

    lishuai

    Joined:
    Feb 18, 2012
    Posts:
    7
    me too
     
  12. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    197
    Same here, this was ok a while ago but now we get assetbundles forcing a complete script recompile per assetbundle..very slow.
     
  13. jeremyfp

    jeremyfp

    Joined:
    Oct 14, 2013
    Posts:
    6
    Has anyone found a workaround for this? Incremental building for asset bundles seemed to be working for me in 5.3.3p2 but after upgrading to 5.3.4p6 (and then 5.3.5) Unity is re-building untouched asset bundles. I can't seem to find any rhyme or reason to it.
     
  14. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    Any chance you are using the unity cache server? In some recent patches, I have noticed that having the cache server enabled during the build process causes Unity to reimport many things that it doesn't need to. If you are using it, try disabling it.

    The specific issue this thread is about is when a scene does actually need updating, and it needs to build a bundle with a scene in it, it re-compiles scripts for each such bundle.
     
  15. jeremyfp

    jeremyfp

    Joined:
    Oct 14, 2013
    Posts:
    6
    Not using the Unity cache server.

    Seems to be something I'm doing on my end. I wanted to detect asset bundle changes (when rebuilding) and then automatically increment an asset bundle version number. The approach I'm taking:

    - Compare the main platform manifest (in this case, iOS.manifest) checksum before and after calling BuildPipeline.BuildAssetBundles.
    - If the checksum changes, increment the version number that's stored in an xml file (that's also part of an "xml" bundle) and then call BuildPipeline.BuildAssetBundles again.

    This seems to work well enough at first. If I rebuild the asset bundles again Unity reports no changes, as expected. However, if I switch to another app and then back to Unity (I'm running OSX) and then rebuild bundles, it rebuilds the main iOS bundle and the CRC value in iOS.manifest changes. I'm guessing that some sort of asset refresh occurs when the app is brought back into the foreground.

    I've tried forcing a refresh of the two bundles/manifests (iOS, xml) before second call to BuildPipeline.BuildAssetBundles, but that doesn't seem to help. I'm at a loss and honestly a bit frustrated that we can't just easily make use of the seemingly unused "ManifestFileVersion" value. I'm sure there's another/easier way to go about this so if anyone has suggestions I'm all ears.
     
  16. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
  17. SteenLund

    SteenLund

    Unity Technologies

    Joined:
    Jan 20, 2011
    Posts:
    452
    @BurningToad

    The fix is unfortunately not a simple fix. Large refactoring of our player building had to be done.
    But it getting closer to completion, but is probably still going to take some time to get ready for shipping.
     
    BurningToad likes this.
  18. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    Alright, thanks very much for the status update!
     
  19. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    Looks like the issue is marked as resolved for a future release, great! Any idea if the fix will make it to a 5.3 patch, or only 5.4? Thanks!
     
  20. SteenLund

    SteenLund

    Unity Technologies

    Joined:
    Jan 20, 2011
    Posts:
    452
    The fix will be in 5.5,

    Change is too high risk to be backed ported and we need to make sure it is proper tested before a release.
     
  21. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    54
    Alright, thanks for the info again. Luckily our regular build times have come down quite a lot with a recent Unity patch that reduced our build times by 2 hours (we suspect shader compilation fixes during scene building?) and also another 2 hours when we turned off "Optimize Mesh Data" as a build option (wish we found that sooner...) So we've gone from roughly 6 hour builds to 2 hour builds, making the transition to Asset Bundles less of a priority (though still something we want to do when this compilation fix makes it in!)
     
  22. GXMark

    GXMark

    Joined:
    Oct 13, 2012
    Posts:
    402
    I just wanted to get an update on this bug ? I'm running 5.5p3 and still getting this recompilation occurring each asset bundle build. Very annoying little bug as i have many small asset bundles each taking on this extra overhead. Any time frames for when this will be completed ?
     
  23. SteenLund

    SteenLund

    Unity Technologies

    Joined:
    Jan 20, 2011
    Posts:
    452
    @GXMark
    How are you building your bundles?

    This fix only applies to building asset bundles with scenes and only when you build multiple bundles at the same time.
     
  24. DarkWindfury

    DarkWindfury

    Joined:
    Mar 12, 2015
    Posts:
    1
    @SteenLund
    Hello, in our project we have the similar issue that occurs when we iteratively in one method call build a lot of small bundles (prefabs). After each bundle in task manager you can see that separate mono.exe process starts and hangs on for a 10+ seconds, after which terminates. What does it stand for? With older unity versions 5.2.x it took 30 minutes, but now more than half and hour =(