Search Unity

  1. Unity 2019.1 beta is now available.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. We're looking for insight from anyone who has experience with game testing to help us better Unity. Take our survey here. If chosen to participate you'll be entered into a sweepstake to win an Amazon gift card.
    Dismiss Notice
  4. On February 28th the Feedback website will shut down and be redirected to the Unity forums. See the full post for more information.
    Dismiss Notice
  5. Want to provide direct feedback to the Unity team? Join the Unity Advisory Panel.
    Dismiss Notice
  6. Unity 2018.3 is now released.
    Dismiss Notice
  7. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice

The assetbundle build is different each time if I remove the Library folder

Discussion in 'Asset Bundles' started by watsonsong, Sep 12, 2017.

  1. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    322
    I am building the assetbundles to support the "hot" update the game asset. So I want to keep my asset bundle binarary consistency so thay the user will not download the whole assets.
    But if I delete my Library folder without any change to my Assets folder, the build result is not the same. I have to commit the Library folder to the version control to avoid this, it really mess up the project.
    I known there have a lot of discussion about this issue, but I never find a right way.
    Is there any way to reslove this, or I am missing something.
     
  2. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    165
    We've done quite a bit to fix binary stability of asset bundle builds. What version of Unity are you using, and by chance do you have any tools or packages that do anything with assets on import?
     
  3. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    322
    I am using Unity 5.6 and the svn as version control. I submit my assetbundle to the svn, but each time I delete Library folder and rebuild assetbundle, I found a lot of bundle is changed.
    I write a script to deal with the import setting for each folder, make sure the import setting in the right way. Different setting for different folder.
     
  4. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    165
    Ok, so the svn stands out as a possible source of the problem:
    - Make sure your SVN does not modify the line endings of any text files on checkout. This will definitely cause the asset hash to be different in certain cases. Unity uses unix line endings when it writes out any text files, regardless of platform. So if your SVN is modifying the line endings to match the platform, that will mess up the hash calculation and make it think the asset has changed and thus need a rebuild.
    - Timestamps are also a big deal with SVN as from what I remember SVN tends to mess up the timestamps of when the asset was last modified. If that is still the case, then that would make the asset hash calculation system mess up again and make it think the asset has changed.

    Finally, I'd also test to see if the bundles themselves are actually different run to run. Use a diff application like DiffMerge to check the 2 bundle files and see if they are the same or different. If they are different, you can use WebExtract on the bundles to pull out the internal content, then run binary2text on the extracted internal content to get a more human readable version of the bundle data. Then run DiffMerge again on the text files and you can then see exactly what changed. If you post or direct message that information I can definitely look over it (note: on vacation next week) and see if it is something we have fixed, or still need to fix. WebExtract & binary2text are found in the Unity install location under the Data/Tools directory.
     
  5. onur-gg

    onur-gg

    Joined:
    Dec 14, 2015
    Posts:
    3
    We also have same issue with GIT and any version of Unity 5.2+. I'm sure generated asset bundles are same, because they have same CRC but different hash in their manifest. We reduce cases with defining all unity files as "-text" in gitattributes, so git not changes the line endings. But as far as I know it's impossible to preserve actual time stamps in GIT. It's actually quite annoying because we almost never achieve to NOT generate asset bundles. There is always one developer/artist who imported the assets which differs from rest of team :(

    Edit: The problem may not be caused from different timestamps. What we observes in our team is that we usually ends up with same binary and meta file, however the importer generates one hash, and rest of team generated another hash for same asset bundle.
     
    Last edited: Sep 22, 2017
  6. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    3,267
    This can also happen if even a single meta file is missing in the directory chain. IE: If you have assets/foo/bar/myfolder/mything, and the meta file for foo is not checked in, it will get generated on each machine with a different GUID and cause mything.bundle to get a different hash, even though it isn't actually different.
     
  7. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    165
    Ya, so that would definitely cause the hash to be calculated differently one machine to another. Unity Cloud Build actually had the same issue with their service, and they used a hook (https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) to modify the "Last Modified" timestamp of all the files to match the last commit time to change that file based on the script found here: https://git.wiki.kernel.org/index.p...mestamp_of_the_commit_which_last_touched_them
     
  8. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    322
    I think the build system for assetbundle should find a way to get rid of the line-end or timestamp. If the two file is the same, the assetbundle must the same.
    Use other tool or script to make sure the file line-end or timestamp is so easy to broken.
     
  9. onur-gg

    onur-gg

    Joined:
    Dec 14, 2015
    Posts:
    3
    I do some playground, I think timestamps are irrelevant. My steps are in below. Is there any possible cause you can suggest?

    Unity Version 2017.2f3 - Windows
    [Master] create a repo
    [Master] create unity project (switch Android)
    [Master] use gitignore and gitattribute files (attached to post)
    [Master] initial commit
    [Master] import 2 sprites assign a bundle name (sprites.bundle)
    [Master] commit
    [Master] update timestamp (access, and modified. attached to post)
    [Master] build bundle
    [Master] commit
    [Master] update timestamp
    [Master] build bundle (no diff)
    [Master] push
    [Slave-1] clone repo (open from command line with -buildTarget android)
    [Slave-1] update timestamp
    [Slave-1] check timestamps (identical with master)
    [Slave-1] build bundles (sprites.bundle is same, Bundles and Bundles.manifest generated with CRC_NEW old CRC_OLD)
    [Slave-1] commit diff
    [Slave-1] build bundles (no diff)
    [Master] pull
    [Master] build bundles (sprites.bundle is same, Bundles and Bundles.manifest generated again with CRC_OLD)
    [Slave-2] clone repo (do not sync timestamps, open from UI, switch to android from Build Settings)
    [Slave-2] build bundles (sprites.bundle is same, Bundles and Bundles.manifest generated with CRC_NEW, identical with Slave-1)
    [Slave-2] commit diff
    [Slave-2] build bundles (no diff)
    [Slave-1] pull
    [Slave-1] build bundles (no diff)
    [Master] pull
    [Master] build bundles (sprites.bundle is same, Bundles and Bundles.manifest generated again with CRC_OLD)

    In my previous experiences the problem will grow in time. For example if project 100 bundles. At first only the "Bundles" is different and then rest of the bundles become different machine to machine in time.
     

    Attached Files:

  10. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    165
    @onur-gg
    I double checked the hash calculation code since it's been a while since I last looked at it. You are correct, timestamps are not part of the hash calculation.

    Ok, based on what you are saying, I take it you are building your asset bundles to a folder called "Bundles" thus you end up with a Bundles and Bundles.manifest file correct? Also just to understand, sprite.bundle is the same in all your test cases, this includes the hash and crc contained in the sprite.bundle.manifest file, but Bundles & Bundles.manifest is not the same, correct? Can you send me a direct msg with the bundles built from Master & Slave-1 so I can look at the internals?
     
  11. onur-gg

    onur-gg

    Joined:
    Dec 14, 2015
    Posts:
    3
    + Ok, based on what you are saying, I take it you are building your asset bundles to a folder called "Bundles" thus you end up with a Bundles and Bundles.manifest file correct?
    - Yes, It's located Assets/Bundles. My build code, activeBuildTarget is "android"

    Code (CSharp):
    1. BuildPipeline.BuildAssetBundles("Assets/Bundles", BuildAssetBundleOptions.None,
    2.             EditorUserBuildSettings.activeBuildTarget);
    + Also just to understand, sprite.bundle is the same in all your test cases, this includes the hash and crc contained in the sprite.bundle.manifest file, but Bundles & Bundles.manifest is not the same, correct?
    - Yes. sprite.bundle and sprite.bundle.manifest is same for all git clones

    + Can you send me a direct msg with the bundles built from Master & Slave-1 so I can look at the internals?
    - Sent you DM
     
  12. qujianbiao

    qujianbiao

    Joined:
    Jun 26, 2017
    Posts:
    1
    I also encountered the same problem, I use [Unity Assets Bundle Extractor] tool found that when packing material in prefab to AssetsBundle, leading to the two pack AssetsBundle is different, but the two AB files, unity hash value is the same, think the file has not changed, but the actual file MD5 has been different.(unity5.6.1p4)

    ab file1:

    Code (CSharp):
    1. 0 Material Base
    2. 1 string m_Name = "mat_128"
    3. 0 PPtr<Shader> m_Shader
    4.   0 int m_FileID = 5
    5.   0 SInt64 m_PathID = 1902212346509020161
    6. 1 string m_ShaderKeywords = ""
    7. 0 unsigned int m_LightmapFlags = 5
    8. 1 bool m_EnableInstancingVariants = false
    9. 0 int m_CustomRenderQueue = -1
    10. 0 map stringTagMap
    11.   0 Array Array (0 items)
    12.    0 int size = 0
    13. 0 vector disabledShaderPasses
    14.   0 Array Array (0 items)
    15.    0 int size = 0
    16. 0 UnityPropertySheet m_SavedProperties
    17.   0 map m_TexEnvs
    18.    0 Array Array (1 items)
    19.     0 int size = 1
    20.     [0]
    21.      0 pair data
    22.       1 string first = "_MainTex"
    23.       0 UnityTexEnv second
    24.        0 PPtr<Texture> m_Texture
    25.         0 int m_FileID = 0
    26.         0 SInt64 m_PathID = -325076255573081539
    27.        0 Vector2f m_Scale
    28.         0 float x = 1.000000
    29.         0 float y = 1.000000
    30.        0 Vector2f m_Offset
    31.         0 float x = 0.000000
    32.         0 float y = 0.000000
    ab file2:
    Code (CSharp):
    1. 0 Material Base
    2. 1 string m_Name = "mat_128"
    3. 0 PPtr<Shader> m_Shader
    4.   0 int m_FileID = 5
    5.   0 SInt64 m_PathID = 1902212346509020161
    6. 1 string m_ShaderKeywords = ""
    7. 0 unsigned int m_LightmapFlags = 5
    8. 1 bool m_EnableInstancingVariants = false
    9. 0 int m_CustomRenderQueue = -1
    10. 0 map stringTagMap
    11.   0 Array Array (0 items)
    12.    0 int size = 0
    13. 0 vector disabledShaderPasses
    14.   0 Array Array (0 items)
    15.    0 int size = 0
    16. 0 UnityPropertySheet m_SavedProperties
    17.   0 map m_TexEnvs
    18.    0 Array Array (3 items)
    19.     0 int size = 3
    20.     [0]
    21.      0 pair data
    22.       1 string first = "_MainTex"
    23.       0 UnityTexEnv second
    24.        0 PPtr<Texture> m_Texture
    25.         0 int m_FileID = 0
    26.         0 SInt64 m_PathID = -325076255573081539
    27.        0 Vector2f m_Scale
    28.         0 float x = 1.000000
    29.         0 float y = 1.000000
    30.        0 Vector2f m_Offset
    31.         0 float x = 0.000000
    32.         0 float y = 0.000000
    33.     [1]
    34.      0 pair data
    35.       1 string first = "_MaskTex"
    36.       0 UnityTexEnv second
    37.        0 PPtr<Texture> m_Texture
    38.         0 int m_FileID = 0
    39.         0 SInt64 m_PathID = 0
    40.        0 Vector2f m_Scale
    41.         0 float x = 1.000000
    42.         0 float y = 1.000000
    43.        0 Vector2f m_Offset
    44.         0 float x = 0.000000
    45.         0 float y = 0.000000
    46.     [2]
    47.      0 pair data
    48.       1 string first = "_NoiseTex"
    49.       0 UnityTexEnv second
    50.        0 PPtr<Texture> m_Texture
    51.         0 int m_FileID = 0
    52.         0 SInt64 m_PathID = 0
    53.        0 Vector2f m_Scale
    54.         0 float x = 1.000000
    55.         0 float y = 1.000000
    56.        0 Vector2f m_Offset
    57.         0 float x = 0.000000
    58.         0 float y = 0.000000
    _MaskTex, _NoiseTex is what I later modified shader to join
     
    Last edited: Mar 29, 2018
  13. BetaDwarf

    BetaDwarf

    Joined:
    Mar 4, 2015
    Posts:
    10
    @Ryanc_unity @onur-gg did you resolve this issue? We are seeing a similar issue. We are using PlasticSCM with a local server setup and have been able to replicate the issue with both Unity 5.6 and 2017.4.

    The issue is that some of the asset ID's has changed and that it orders dependencies differently:

    m_Dependencies (vector)
    size 3 (int)
    data "shaders" (string)
    data "ui" (string)
    data "vfx" (string)
    vs

    m_Dependencies (vector)
    size 3 (int)
    data "ui" (string)
    data "shaders" (string)
    data "vfx" (string)
     
  14. leuconoe

    leuconoe

    Joined:
    Oct 14, 2014
    Posts:
    1
    bump, Because of this problem, you can not configure a new build machine.
    Please let me know how this problem is solved.
     
  15. JohnSonLi

    JohnSonLi

    Joined:
    Apr 15, 2012
    Posts:
    583
    different hash different machine, this is not what we had expected,do you find solutions??