Accelerator is caching an empty/corrupt asset.

Hi,

We have just experienced a problem where a broken asset (0 bytes long) got uploaded to the accelerator and now all builds fail, with this error in the log:

Error loading the file 'Library/Artifacts/32/326e47e820bc46df02a2958b7dd90697'. File is either empty or corrupted, please verify the file contents.

From the accelarator logs I can see that one of our build slaves uploaded the 0 sized asset (utc timestamps):

{"level":"debug","ts":"2020-10-23T22:18:19.874Z","msg":"handled getRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11481,"remote":"192.168.16.140:64501","segment_id":24841,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","status":"ok","status_code":0,"size":3818424}

{"level":"debug","ts":"2020-10-23T22:18:27.919Z","msg":"handled getRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11482,"remote":"192.168.16.140:64500","segment_id":24838,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","status":"ok","status_code":0,"size":3818424}

[B]{"level":"debug","ts":"2020-10-23T22:19:32.878Z","msg":"handled putRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11480,"remote":"192.168.16.140:64499","segment_id":26128,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","size":0,"status":"ok","status_code":0} << BAD UPLOAD[/B]

{"level":"debug","ts":"2020-10-23T22:34:18.813Z","msg":"handled getRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11488,"remote":"192.168.16.141:63744","segment_id":7224,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","status":"ok","status_code":0,"size":0}

{"level":"debug","ts":"2020-10-23T22:44:45.715Z","msg":"handled getRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11490,"remote":"192.168.16.140:52390","segment_id":7224,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","status":"ok","status_code":0,"size":0}

This is the log from the Unity that uploaded the corrupted asset (utc+2h timestamps):

[00:19:19] :        [Post Processing Catalog Entries] Start importing Assets/Sprites/Characters/Atlases/Low/Otho_Armored.png using Guid(e3f006035ddb0de4e82479b8ed477cdf) Importer(-1,00000000000000000000000000000000)

[00:19:19] :        [Post Processing Catalog Entries]

[00:19:19] :        [Post Processing Catalog Entries] TPImporter: OnPreprocessTexture(Assets/Sprites/Characters/Atlases/Low/Otho_Armored.png)

[00:19:19] :        [Post Processing Catalog Entries] TPImporter: sheet exists in db, no changes

[00:19:19] :        [Post Processing Catalog Entries] TPImporter: Successfully loaded tpsheet file containing 370 sprite definitions

[00:19:19] :        [Post Processing Catalog Entries] TPImporter: set texture type + mesh type

[00:19:19] :        [Post Processing Catalog Entries] TPImporter: meta data hasn't changed

[00:19:19] :        [Post Processing Catalog Entries]

[00:19:23] :        [Post Processing Catalog Entries] TPImporter: Texture scale factor: 1

[00:19:23] :        [Post Processing Catalog Entries] TPImporter: Updating geometry of 370 sprites

[00:19:23] :        [Post Processing Catalog Entries]

[00:19:33] :        [Post Processing Catalog Entries] Done importing asset: 'Assets/Sprites/Characters/Atlases/Low/Otho_Armored.png' (target hash: '812098c51feb6381628b111786045e53') in 13.586947 seconds

[00:19:33] :        [Post Processing Catalog Entries] RemoteAssetCache::AddArtifactToCacheServer - artifactKey='Guid(e3f006035ddb0de4e82479b8ed477cdf) Importer(-1,00000000000000000000000000000000)' Target hash='812098c51feb6381628b111786045e53' Key (and library path)='Library/Artifacts/32/326e47e820bc46df02a2958b7dd90697'

[00:19:33] :        [Post Processing Catalog Entries] RemoteAssetCache - Upload - Artifact - success:true, namespace:81e94844d19a16919208533e08183531, key:326e47e820bc46df02a2958b7dd90697

[00:19:33] :        [Post Processing Catalog Entries] RemoteAssetCache::AddArtifactToCacheServer - artifactKey='Guid(e3f006035ddb0de4e82479b8ed477cdf) Importer(-1,00000000000000000000000000000000)' Target hash='812098c51feb6381628b111786045e53' Key (and library path)='Library/Artifacts/c4/c48e12a7e1eb5bf520e3f4b4b7fa076b'

This is the error we get when trying to build:

[23:40:50] :        [Unity Package Manager] Error loading the file 'Library/Artifacts/32/326e47e820bc46df02a2958b7dd90697'. File is either empty or corrupted, please verify the file contents.

[23:40:50] :        [Unity Package Manager] UnityEditor.AssetDatabase:GetMainAssetTypeAtPath(String)

[23:40:50] :        [Unity Package Manager] UnityEditor.Localization.LocalizationAssetPostProcessor:OnPostprocessAllAssets(String[], String[], String[], String[]) (at Assets\com.unity.localization@0.8.0-preview\Editor\Asset Pipeline\LocalizationAssetPostProcessor.cs:35)

[23:40:50] :        [Unity Package Manager] System.Reflection.MonoMethod:InternalInvoke(Object, Object[], Exception&)

[23:40:50] :        [Unity Package Manager] System.Reflection.MonoMethod:Invoke(Object, BindingFlags, Binder, Object[], CultureInfo)

[23:40:50] :        [Unity Package Manager] System.Reflection.MethodBase:Invoke(Object, Object[])

[23:40:50] :        [Unity Package Manager] UnityEditor.AssetPostprocessingInternal:InvokeMethod(MethodInfo, Object[])

[23:40:50] :        [Unity Package Manager] UnityEditor.AssetPostprocessingInternal:PostprocessAllAssets(String[], String[], String[], String[], String[])

We don't know how to fix this problem, if we delete the broken assets from the cache, a different Unity instance will upload the broken version again. We also tried deleting the entry, and do a force reimport in the hopes that a fixed version would get uploaded, but it appeared like a broken version got uploaded.
./bin/unity-accelerator cache delete 127.0.0.1:10080 78b2fdc254b0a4f8a4693a81bb96598a 326e47e820bc46df02a2958b7dd90697


This time around we could fix it by just changing the guild of the asset (as there were only a few references to the broken asset), but that might not be possible next time, if the asset is referencved from all over.
Please advice on how this should handled.

Unity: 2019.4.8f1
Accelerator: v1.0.604+g695e221

 

I can't tell if the imported asset was also 0 bytes long on the build machine, that build have been cleaned since then.
I found a few other errors in the log of build taht pushed the 0 bytes long asset to the accelerator:


A few (around 10) of these:

[00:21:46] :        [Post Processing Catalog Entries] Failed to load 'C:/b/1/work/79d8696630371996/Library/Artifacts/c8/c87624ddefb095d21b7874a9a632447e'. File may be corrupted or was serialized with a newer version of Unity.

[00:21:46] :        [Post Processing Catalog Entries] UnityEditor.AssetImporter:GetAtPath(String)

[00:21:46] :        [Post Processing Catalog Entries] UnityEditor.ModelImporterPostProcessor:OnPostprocessAllAssets(String[], String[], String[], String[])

[00:21:46] :        [Post Processing Catalog Entries] System.Reflection.MonoMethod:InternalInvoke(Object, Object[], Exception&)

[00:21:46] :        [Post Processing Catalog Entries] System.Reflection.MonoMethod:Invoke(Object, BindingFlags, Binder, Object[], CultureInfo)

[00:21:46] :        [Post Processing Catalog Entries] System.Reflection.MethodBase:Invoke(Object, Object[])

[00:21:46] :        [Post Processing Catalog Entries] UnityEditor.AssetPostprocessingInternal:InvokeMethod(MethodInfo, Object[])

[00:21:46] :        [Post Processing Catalog Entries] UnityEditor.AssetPostprocessingInternal:PostprocessAllAssets(String[], String[], String[], String[], String[])

[00:21:46] :        [Post Processing Catalog Entries] UnityEditor.AssetDatabase:Refresh(ImportAssetOptions)

[00:21:46] :        [Post Processing Catalog Entries] UnityEditor.AssetDatabase:Refresh()

[00:21:46] :        [Post Processing Catalog Entries] GooglePlayGames.Editor.GPGSAndroidSetupUI:PerformSetup(String, String, String) (at Assets\GooglePlayGames\Editor\GPGSAndroidSetupUI.cs:200)

[00:21:46] :        [Post Processing Catalog Entries] Build:BuildAndroid(Boolean, Boolean, Boolean, Boolean, Boolean, Boolean) (at Assets\Editor\Build\BuildAndroid.cs:148)

[00:21:46] :        [Post Processing Catalog Entries] Build:BuildAndroidDev(Boolean, Boolean, Boolean, Boolean) (at Assets\Editor\Build\BuildAndroid.cs:80)

[00:21:46] :        [Post Processing Catalog Entries] Build:BuildAndroidDev() (at Assets\Editor\Build\BuildAndroid.cs:41)

[00:21:46] :        [Post Processing Catalog Entries]

[00:21:46] :        [Post Processing Catalog Entries] [ line 0]

[00:21:46] :        [Post Processing Catalog Entries] (Filename: Assets/GooglePlayGames/Editor/GPGSAndroidSetupUI.cs Line: 200)

This was a clean build, so the Library folder did not exist at the start of the build, so it's more likely that the file was coprrupted than it was from a different version of Unity.


A few assertions of and a abort:

[00:24:57] :        [Building Player] Assertion failed on expression: 'success && actual == (UInt64)size'

[00:24:57] :        [Building Player] (Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 453)

[00:24:57] :        [Building Player]

[00:24:57] :        [Building Player] Assertion failed on expression: 'success && actual == size'

[00:24:57] :        [Building Player] (Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 462)

[00:24:57] :        [Building Player]

[00:24:57] :        [Building Player]

[00:24:57] :        [Building Player]

[00:24:57] :        [Building Player] Aborting batchmode due to failure:

[00:24:57] :        [Building Player] Fatal Error! dataOffset != file.GetPosition()

I can't find any messages in the log about running out of disk space, but the errors kinda leads me to believe that it could have happend, and Unity did not react to failed writes?

 

Did you find a fix for this case?

This error still appears rarely but it does.