Search Unity

ScriptableBuildPipeline - Known issues, roadmap, ...

Discussion in 'Asset Bundles' started by pavelkouril, Oct 30, 2018.

  1. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Hello,

    I've been testing SBP to build asset bundles (jncluding integrating the new cache server), and I've been hitting issues when building asset bundles (which will get built correctly using the legacy built-in pipeline).

    Therefore, I would like to ask if there is a list of known issues (e.g., some scenes/prefabs references to scripts will get broken, empty paths when using the cache server integration, etc.) with the current pipeline, which could save me hours/days of time by trying to hunt the issue in a big project.

    Also do you have some approximate roadmap? Especially, when the package is expected to go out of preview?

    Anything will be helpful, really. :)
     
  2. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    @pavelkouril The documentation included with SBP details what differences to expect (such as variants not being supported). Everything else should work, and if not, it's a bug we need to fix (and no aware of) so please open a bug report for the issue. We did just push out a new version: 1.1.1-preview that fixes issues with asset bundle's that contain scenes.

    As for roadmaps, we are not ready to share anything just yet in this area, but going out of preview should be soon.
     
  3. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Well,

    for one we are hitting an issue I submitted as case 1096354; unfortunately without repro case, because I don't have any idea how to reproduce this. :(

    The other issue was getting errors when using cache server with the SBP and prefabs/GOs in scene referenced non-existing Mono Behavior (but the component was still there); we fixed this on our side by removing the references, and I will try to prepare a repro case once I'll have an hour or two of free time and will submit it using the reporter. :)
     
  4. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    Are you using the same CacheServer for SBP and your Project?

    The SBP docs should call out that you need to use a separate CacheServer for Build Artifacts from Project Artifacts otherwise a collision will occur as the CacheServer was originally not designed to handle two different sets of artifacts for the same asset.
     
  5. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    I was using a separate instance, but I tried to upload the editor cache to the cache server.

    However, I still can't get reliably over the issue with UnityEditor.Build.Pipeline.Tasks.CalculateSceneDependencyData.Run resulting in an ArgumentException because of an empty path (BuildCache.GetCacheEntry); it seems that some asset has dependency, which doesn't exist already, and the AssetDatabase.GUIDToAssetPath then returns empty path?

    Is there a way to determine (ideally outside of the build process) which assets can have invalid dependencies, so that we can fix them? I tried checking for non-existing MonoBehaviours, but those are fixed already.
     
  6. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    Hmm, honestly sounds like a edge case I should just fix as Assets (or references in this case) can disappear, but the backing data still can be valid as you could restore the asset and the reference will connect back up without any changes. In this case though it sounds like something might be automatically creating objects during the build process incorrectly? I'm not should though as I haven't run into this locally and been able to debug. PM sent with more info.

    On SBP / CS side, how are you uploading the editor cache to the cache server? I take it you wrote your own upload script?
     
    Last edited: Nov 20, 2018
  7. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    We uploaded the data using the dialog that appeared in Unity after installing SBP, "Assets/Cache Server/Upload All Assets"

    Is it also possible that e.g. some Image component has missing texture as a reference (that was just a wrongly deleted file), and the GUID is still in the meta of the asset, but it doesn't exist anymore? We also create some files during the build process, but I don't think any of them are added as dependencies of scenes in any fashion.
     
  8. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    Ahh, "Assets/Cache Server/Upload All Assets" is for Asset artifacts, not Build artifacts.

    Hmm, it's possible. I don't remember specifically testing that case, and if it is returning that object as a dependency, that's wrong and I need to fix the "ContentBuildInterface.PrepareScene" API to not do that (also ContentBuildInterface.GetPlayerDependenciesForObjects).
     
  9. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    I'll could try to filter the results of ContentBuildInterface.GetPlayerDependenciesForObjects on my side in SBP tomorrow to those where AssetDatabase.GUIDToAssetPath actually returns existing non-empty path and write the result if it helped with this issue?
     
  10. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Will the CS get the build artifacts from the build machine while I build with it for the first time? Or without doing the custom upload script, the server won't contain the artifacts at all, even during subsequent run?
     
  11. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    SBP uploads artifacts when it generates them as some artifacts need to (currently) follow a Least Recently Used cache pattern while others don't. So first run it will store a local copy of the artifacts, and upload a copy to the cache server (This is if you set the CacheServerHost and CacheServerPort values on the passed in BundleBuildParameters (or BuildParameters) object.
     
  12. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    Ya, filtering those results should work. You would only want to filter the ObjectIdentifier out if guid.Empty() is false and if the path returned is null or empty. As that should indicate missing reference, so save to skip skip.
     
  13. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    I added a filter condition

    Where(x => x.guid.Empty() || (!x.guid.Empty() && AssetDatabase.GUIDToAssetPath(x.guid.ToString()) != string.Empty && AssetDatabase.GUIDToAssetPath(x.guid.ToString()) != null))
    to the results of GetPlayerDependenciesForObjects in CalculateAssetDependencyData and PreviewSceneDependencyData, and I'm still getting the same error unfortunately.

    So either I'm doing the filtering wrong, or there is still some other source for dependencies that have empty path. :/
     
  14. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    Try this:
    BuildCache.cs Line 92, add Debug.Log(entry); to the beginning of that function and run it again with the console open, cleared, and collapse turned off. Then look at the entry log right before the error and paste that Log(entry) output and the full stack trace of the error to me.
     
  15. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Code (CSharp):
    1. {51f772310d7ea1e4d9e26e9cca2497af, c6d57e38e20f24411f5b94918cea3dd8}
    2. UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
    3. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    4. UnityEngine.Logger:Log(LogType, Object)
    5. UnityEngine.Debug:Log(Object)
    6. UnityEditor.Build.Pipeline.Utilities.BuildCache:GetUpdatedCacheEntry(CacheEntry) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:101)
    7. UnityEditor.Build.Pipeline.Utilities.BuildCache:NeedsRebuild(CachedInfo) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:118)
    8. UnityEditor.Build.Pipeline.Utilities.BuildCache:LoadCachedData(IList`1, IList`1&) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:260)
    9. UnityEditor.Build.Pipeline.Tasks.CalculateSceneDependencyData:Run() (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Tasks\CalculateSceneDependencyData.cs:57)
    10. UnityEditor.Build.Pipeline.BuildTasksRunner:Run(IList`1, IBuildContext) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Shared\BuildTasksRunner.cs:50)
    11. UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&, IList`1, IContextObject[]) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:108)
    12. UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:35)
    13. UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles_Internal(String, IBundleBuildContent, BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:88)
    14. UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles(String, AssetBundleBuild[], BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:54)
    15. foo.editor.<ExecuteBundleBuild>c__AnonStorey2:<>m__0() (at Assets\Editor\Packaging\BuildBundles.cs:804)
    16. BuildPipelineEx.BuildContext:ExecuteWithBuildPlayerSettings(Func`1) (at Assets\Plugins\Editor\BuildPipelineEx\BuildContext.cs:603)
    17. foo.editor.BuildBundles:ExecuteBundleBuild(BuildContext, String, AssetBundleBuild[]) (at Assets\Editor\Packaging\BuildBundles.cs:804)
    18. foo.editor.BuildBundles:DoExecuteBuild(BuildContext, AssetBundleSetBuildHelper, String, String, String) (at Assets\Editor\Packaging\BuildBundles.cs:248)
    19. foo.editor.FinalizeBuildStep:Execute(BuildContext) (at Assets\Editor\Packaging\BuildBundles.cs:118)
    20. BuildPipelineEx.CommandLineBuildBridge:ExecuteCustomBuildSteps(BuildContext, BuildPhase) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:118)
    21. BuildPipelineEx.CommandLineBuildBridge:PostBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:635)
    22. BuildPipelineEx.CommandLineBuildBridge:DoRunBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:597)
    23. BuildPipelineEx.CommandLineBuildBridge:RunBuild() (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:510)
    24. (Filename: Assets/Editor/Packaging/BuildBundles.cs Line: 804)
    25.  
    26. ArgumentException: Path is empty
    27.   at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x00027] in <f2e6809acb14476a81f399aeb800f8f2>:0
    28.   at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean anonymous) [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
    29.   at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access) [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
    30.   at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess)
    31.   at UnityEditor.Build.Pipeline.Utilities.HashingMethods.CalculateFile (System.String filePath) [0x00004] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\HashingMethods.cs:294
    32.   at UnityEditor.Build.Pipeline.Utilities.BuildCache.GetCacheEntry (UnityEditor.GUID asset) [0x000bc] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:68
    33.   at UnityEditor.Build.Pipeline.Utilities.BuildCache.GetUpdatedCacheEntry (UnityEditor.Build.Pipeline.Utilities.CacheEntry entry) [0x00040] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:106
    34.   at UnityEditor.Build.Pipeline.Utilities.BuildCache.NeedsRebuild (UnityEditor.Build.Pipeline.Utilities.CachedInfo info) [0x0006b] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:118
    35.   at UnityEditor.Build.Pipeline.Utilities.BuildCache.LoadCachedData (System.Collections.Generic.IList`1[T] entries, System.Collections.Generic.IList`1[UnityEditor.Build.Pipeline.Utilities.CachedInfo]& cachedInfos) [0x00175] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:260
    36.   at UnityEditor.Build.Pipeline.Tasks.CalculateSceneDependencyData.Run () [0x0006e] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Tasks\CalculateSceneDependencyData.cs:57
    37.   at UnityEditor.Build.Pipeline.BuildTasksRunner.Run (System.Collections.Generic.IList`1[T] pipeline, UnityEditor.Build.Pipeline.Interfaces.IBuildContext context) [0x00097] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Shared\BuildTasksRunner.cs:50
    38. UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
    39. UnityEngine.DebugLogHandler:LogException(Exception, Object)
    40. UnityEngine.Logger:LogException(Exception, Object)
    41. UnityEngine.Debug:LogException(Exception)
    42. UnityEditor.Build.Pipeline.Utilities.BuildLogger:LogException(Exception) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildLogger.cs:77)
    43. UnityEditor.Build.Pipeline.BuildTasksRunner:Run(IList`1, IBuildContext) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Shared\BuildTasksRunner.cs:57)
    44. UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&, IList`1, IContextObject[]) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:108)
    45. UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:35)
    46. UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles_Internal(String, IBundleBuildContent, BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:88)
    47. UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles(String, AssetBundleBuild[], BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:54)
    48. foo.editor.<ExecuteBundleBuild>c__AnonStorey2:<>m__0() (at Assets\Editor\Packaging\BuildBundles.cs:804)
    49. BuildPipelineEx.BuildContext:ExecuteWithBuildPlayerSettings(Func`1) (at Assets\Plugins\Editor\BuildPipelineEx\BuildContext.cs:603)
    50. foo.editor.BuildBundles:ExecuteBundleBuild(BuildContext, String, AssetBundleBuild[]) (at Assets\Editor\Packaging\BuildBundles.cs:804)
    51. foo.editor.BuildBundles:DoExecuteBuild(BuildContext, AssetBundleSetBuildHelper, String, String, String) (at Assets\Editor\Packaging\BuildBundles.cs:248)
    52. foo.editor.FinalizeBuildStep:Execute(BuildContext) (at Assets\Editor\Packaging\BuildBundles.cs:118)
    53. BuildPipelineEx.CommandLineBuildBridge:ExecuteCustomBuildSteps(BuildContext, BuildPhase) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:118)
    54. BuildPipelineEx.CommandLineBuildBridge:PostBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:635)
    55. BuildPipelineEx.CommandLineBuildBridge:DoRunBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:597)
    56. BuildPipelineEx.CommandLineBuildBridge:RunBuild() (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:510)
    57. (Filename: <f2e6809acb14476a81f399aeb800f8f2> Line: 0)
    I tried searching for the
    51f772310d7ea1e4d9e26e9cca2497af and
    c6d57e38e20f24411f5b94918cea3dd8. The 51f772310d7ea1e4d9e26e9cca2497af was a reference to a Missing (Audio Clip) on a prefab, but even when I removed these references (can't find a single file having this string in the Assets folder), I got the same error. Or are these asset dependencies stored somehow cached and I will need to evict that cache too?
     
  16. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    Oh! That error makes a lot of sense. It's trying to load data from the cache, because the entry exists, but the file in the cache is not found. So something failed to get added to the cache on the first build.

    Purge the build cache, run the build again and watch for errors. It's likely I missed cleanup in BuildCache.cs on move failures in an edge case.
     
  17. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    I'll try it tomorrow them; by purging the build cache, you mean something in Library folder, or just the contents of the cache server (if there are any).
     
  18. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    "Library/BuildCache" is the local copy of the build cache. Purging, is just deleting that folder. There is also a menu item for this: "Window > Asset Management > Purge Build Cache".
     
  19. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Ok, thanks!

    We were finally able to build using SBP. Shame there was no speedup for builds when using SBP+Cache Server for us, so we will probably wait until the package goes out of preview (hopefully there will be less edge cases by then?).
     
  20. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    332
    SBP still gives you local speed ups on subsequent builds. Cache Server is just if you wish to share the build cache from one machine with other machines. Since you were able to get it working using just SBP, I would purge the Cache Server you were using for this and reconnect SBP to the Cache server and try again. Don't use "Assets/Cache Server/Upload All Assets" and make sure this cache server is ONLY connected to SBP via the BundleBuildParameters input class and not the Editor via Editor Preferences.
     
  21. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    @ryanc-unity could you please explain the workflow of working with the build cache and SBP?
    In the documentation, i've only seen a code example of how to manually upload assets to a cache server.

    Which cache server is that? Is it the same as the Unity cache server used by the editor? does it work with the Accelerator ? (which is the newer version of the cache server ?)

    How doe ? should i use it "manually" like the code sample in the docs show ? in what ways does it optimize build times? what kind of artifacts does it cache?

    Would really love some clarifications :)
     
    PD_Boudreaux likes this.
  22. PD_Boudreaux

    PD_Boudreaux

    Joined:
    Feb 3, 2021
    Posts:
    8
    @liortal According to this post https://forum.unity.com/threads/sbp...-does-not-work-with-unity-accelerator.955632/ the current latest version of SBP (1.19 at time of writing) does not support the Unity Accelerator.

    I too would like to know answers to your other questions. Once SBP 2.0 is released, it could really use some documentation updates for how to use it with the Unity Accelerator and general documentation improvements to answer other questions that aren't documented like "Is SBP for use with AssetBundles only?"