Bug Nondeterministic build of a bundle containing shaders(EditorDataHash changed)

We have a project, which put all shaders used in a single asset bundle. I made a test about whether the build is deterministic by cleaning cache and rebuild. I found the hash of the bundle containing shaders changed in Android build.(We have over 8000 asset bundles, only the asset bundle containing shaders changed. And this issue is not reproducible consistently, I tried multiple times later, but only appeared once again)

I made some investigation about this issue:

1. The size of these two different asset bundles is identical
2. Use VBinDiff to compare uncompressed asset bundles(Extracted by WebExtract), they are different at only 1 place, which is 32 bytes
   
3. Use AssetStudio to export shaders from asset bundles, and the results are identical
4. Use AssetStudio to export raw data of shaders from asset bundles, and found one of the shaders is different
5. Use AssetStudio source code to debug the deserialization, and found the different data is 'EditorDataHash' of one shader. Since we support GLES2 and GLES3x in Android build, so there are two EditorDataHash(Hash128), and the size is 32 bytes, matched the finding by VBinDiff.
6. The problem shader(Effect/Effect_ChuChang_Xue) contains two passes, ForwardBase and ShadowCaster. Only the EditorDataHash of ForwardBase pass is different.

The following is the screenshot of VBinDiff:


The following is part of the log of two Android builds (Left is the first build):


It seems that when doing the second build, Unity considered some variants of the FORWARD pass of the shader(Effect/Effect_ChuChang_Xue) need to be recompiled. By the way, Unity considered all other shader variants cached, and didn't recompile them.

Between the builds, I didn't change the project assets or settings manually. I only opened the project, cleaned the cache of Addressables, closed it, and then built from command line using our build script. Also, I confirmed the last modification time of the problem shader is five months ago.

We are using Unity 2020.3.30 and Addressables 1.19.19.

Anyone has some ideas about this issue?

 

@Joachim_Ante

 

@aleksandrk
@richardkettlewell

 

Hi,
The issue reproduced twice on our build machine. At the first time, the shader is created by Shader Forge(The different EditorDataHash belongs to ForwardBase pass). At the second time, the shader is Hidden/PostProcessing/Lut2DBaker, coming from PostProcessingStack package(The different EditorDataHash belongs to the third pass).

The asset bundle contains 96 shaders, and other shaders keep unchanged between rebuilding.

 

I have some new information about this issue, maybe it's helpful for you to find the cause.

Since we have Unity source code license, I added a log in detail::ComputeShaderSnippetKey to record the information about snippet.m_AssetPath and editorDataHash[5].

The part of the log of the first build:

Code (CSharp):

1. Compiling shader "MODX/MaskFlash" pass "Pass 0" (vp)
2.     2 / 2 variants left after stripping, processed in 0.00 seconds
3.     starting compilation...
4. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 63c3f4dd53840566fb7a577e072bbdf8, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
5. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 63c3f4dd53840566fb7a577e072bbdf8, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
6.     finished in 0.00 seconds. Local cache hits 2 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 0 variants (0.00s CPU time), skipped 0 variants
7. Serialized binary data for shader MODX/MaskFlash in 0.00s
8.     gles (total internal programs: 1, unique: 1)
9.     gles3 (total internal programs: 1, unique: 1)
10. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 63c3f4dd53840566fb7a577e072bbdf8, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
11. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 63c3f4dd53840566fb7a577e072bbdf8, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
12. Compiling shader "ModX/UI/FocusFont" pass "Pass 0" (vp)
13.     2 / 2 variants left after stripping, processed in 0.00 seconds
14.     starting compilation...
15. ComputeShaderSnippetKey: Assets/shaders, 4008358f0c64e488e5930e175fc0fa5a, 63c3f4dd53840566fb7a577e072bbdf8, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
16. ComputeShaderSnippetKey: Assets/shaders, 4008358f0c64e488e5930e175fc0fa5a, 63c3f4dd53840566fb7a577e072bbdf8, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
17.     finished in 0.00 seconds. Local cache hits 2 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 0 variants (0.00s CPU time), skipped 0 variants
18.  

The part of the log of the second build:

Code (CSharp):

1. Compiling shader "MODX/MaskFlash" pass "Pass 0" (vp)
2.     2 / 2 variants left after stripping, processed in 0.00 seconds
3.     starting compilation...
4. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 2f67e6f90fb753b52c4d90024fe201c0, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
5. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 2f67e6f90fb753b52c4d90024fe201c0, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
6.     finished in 0.00 seconds. Local cache hits 2 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 0 variants (0.00s CPU time), skipped 0 variants
7. Serialized binary data for shader MODX/MaskFlash in 0.00s
8.     gles (total internal programs: 1, unique: 1)
9.     gles3 (total internal programs: 1, unique: 1)
10. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 2f67e6f90fb753b52c4d90024fe201c0, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
11. ComputeShaderSnippetKey: Assets/shaders, 0afb63eada6fe75cfa11c64ccf486d2c, 2f67e6f90fb753b52c4d90024fe201c0, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
12. Compiling shader "ModX/UI/FocusFont" pass "Pass 0" (vp)
13.     2 / 2 variants left after stripping, processed in 0.00 seconds
14.     starting compilation...
15. ComputeShaderSnippetKey: Assets/shaders, 4008358f0c64e488e5930e175fc0fa5a, 2f67e6f90fb753b52c4d90024fe201c0, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
16. ComputeShaderSnippetKey: Assets/shaders, 4008358f0c64e488e5930e175fc0fa5a, 2f67e6f90fb753b52c4d90024fe201c0, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
17.     finished in 0.02 seconds. Local cache hits 0 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 2 variants (0.05s CPU time), skipped 0 variants
18. Serialized binary data for shader ModX/UI/FocusFont in 0.00s
19.     gles (total internal programs: 1, unique: 1)
20.     gles3 (total internal programs: 1, unique: 1)
21. ComputeShaderSnippetKey: Assets/shaders, 4008358f0c64e488e5930e175fc0fa5a, 2f67e6f90fb753b52c4d90024fe201c0, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
22. ComputeShaderSnippetKey: Assets/shaders, 4008358f0c64e488e5930e175fc0fa5a, 2f67e6f90fb753b52c4d90024fe201c0, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
23.  

Before the second build, I only reimported a folder containing some shaders and cleaned the cache of Addressables and SBP.

It seems that the EditorDataHash[1](which is snippet.m_IncludeHash) of many shaders became 2f67e6f90fb753b52c4d90024fe201c0 from 63c3f4dd53840566fb7a577e072bbdf8. MaskFlash.shader is one of them and it only includes UnityCG.cginc.

 

It dosen't happen every time, but I reproduced twice due to reimporting shaders recently. At the first time, I only reimported one shader. After that, I tried to reimport the shader multiple times that day, but the issue didnt appear again(Until I reimported the folder today).

 

Hi, @aleksandrk

I think I found the cause.

Firstly, the change of 'snippet.m_IncludeHash' is due to the change of Unity version. We used Unity 2020.3.26 before, and upgraded to Unity 2020.3.30 recently. The 'UnityCG.cginc' file is changed between these two versions. So it's not a bug, but I think it's not expected: the change of 'UnityCG.cginc' between 2020.3.26 and 2020.3.30 is in the comment, and dosen't cause any change of the compiled result, but the change of EditorDataHash causes the generated bundle becomes different.

The real cause of the change of EditorDataHash is 'snippet.GetCodeHash'. This method is not threadsafe, so the generated hash may become some unexpected value. I added some logs in detail::ComputeShaderSnippetKey to confirm it:

Code (CSharp):

1. Compiling shader "Effect/Noise_Mask_Pro2" pass "Unlit" (vp)
2.     4 / 4 variants left after stripping, processed in 0.00 seconds
3.     starting compilation...
4. ComputeShaderSnippetKey: Assets/Plugins/3rd/EffectNoisePro/shader, 32e14f291b5b46231607026671945570, 2f67e6f90fb753b52c4d90024fe201c0, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
5. CodeHash is wrong: 32e14f291b5b46231607026671945570, expected: 1c03bf4473e48264e60ddcf200a6ac92
6. ComputeShaderSnippetKey: Assets/Plugins/3rd/EffectNoisePro/shader, 1c03bf4473e48264e60ddcf200a6ac92, 2f67e6f90fb753b52c4d90024fe201c0, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
7. ComputeShaderSnippetKey: Assets/Plugins/3rd/EffectNoisePro/shader, 32e14f291b5b46231607026671945570, 2f67e6f90fb753b52c4d90024fe201c0, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
8. CodeHash is wrong: 32e14f291b5b46231607026671945570, expected: 1c03bf4473e48264e60ddcf200a6ac92
9. ComputeShaderSnippetKey: Assets/Plugins/3rd/EffectNoisePro/shader, 1c03bf4473e48264e60ddcf200a6ac92, 2f67e6f90fb753b52c4d90024fe201c0, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
10.     finished in 0.03 seconds. Local cache hits 2 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 2 variants (0.05s CPU time), skipped 0 variants
11. Serialized binary data for shader Effect/Noise_Mask_Pro2 in 0.00s
12.     gles (total internal programs: 2, unique: 2)
13.     gles3 (total internal programs: 2, unique: 2)
14. ComputeShaderSnippetKey: Assets/Plugins/3rd/EffectNoisePro/shader, 32e14f291b5b46231607026671945570, 2f67e6f90fb753b52c4d90024fe201c0, 800500000000000091f80b0c9ae40b0c, 19093d926519dad5c93fc42486cf5a44, 80674e4b8db2c9e94b0d2257466895cd
15. CodeHash is wrong: 32e14f291b5b46231607026671945570, expected: 1c03bf4473e48264e60ddcf200a6ac92
16. ComputeShaderSnippetKey: Assets/Plugins/3rd/EffectNoisePro/shader, 32e14f291b5b46231607026671945570, 2f67e6f90fb753b52c4d90024fe201c0, 800900000000000091f80b0c9ae40b0c, 6ade5ebe42640c786109bbe5569251a5, 80674e4b8db2c9e94b0d2257466895cd
17. CodeHash is wrong: 32e14f291b5b46231607026671945570, expected: 1c03bf4473e48264e60ddcf200a6ac92

The expected hash above is calculated in the method detail::ComputeShaderSnippetKey:

Code (CSharp):

1. Hash128 expectedHash;
2. ComputeHash128(snippet.GetCode().c_str(), snippet.GetCode().size(), expectedHash);

 

Based on what I found yesterday, I created a project to reproduce this issue, Case 1428947.

 

Hi, will this be handled and also fix the similar issue in Addressables (since it relies on AssetBundle)? Any ETA?
Thanks

 

Hi, is there a ticket number to track this?

 

Hi, it seems that Unity didn't create a public link for this issue. The current status is

EDIT (2022-08-20):
I confirmed this issue has been fixed in 2023.1.0a6

 

For 2021.3.x and 2022.2b which exact version fix the issue?

 

Alright. I hope you can prioritize the fix to 2022.2b since I will upgrade project to 2022.2b soon.

 

When updating Unity 2022.03.9 -> 2022.03.10, many bundles have changed. Upon analysis, i found that the 'm_EditorDataHash' was modified in the builtin shaders 'Sprites/Default', 'Standart' and other.
I checked the builtin shaders, they did not change. In attach output `unitydatatool dump` from bundle and screen of bundle GO.
This is normal or bug?