Search Unity

Bug Burst corruption

Discussion in 'Burst' started by tertle, Dec 5, 2022.

  1. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    I've been getting a lot of burst cache corruption again. Current versions
    Unity 2022.2.0f1
    Burst 1.8.2

    They manifest like this
    Code (CSharp):
    1.  
    2. NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object
    3. This Exception was thrown from a function compiled with Burst, which has limited exception support.
    4. 0x00007ff7026706fd (Unity) burst_signal_handler
    5. 0x00007ff809428b4c (ntdll) RtlDeleteAce
    6. 0x00007ff8094012c6 (ntdll) RtlRaiseException
    7. 0x00007ff809450f4e (ntdll) KiUserExceptionDispatcher
    8. 0x00007fff921e84cf (9c33427477f2fcf9f52aa87bcd77068) Unity.Physics.Systems.CreateJacobiansSystem::Unity.Physics.Systems.CreateJacobiansSystem.OnUpdate (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.physics@1.0.0-pre.15/Unity.Physics/ECS/Base/Systems/UnityPhysicsSimulationSystems.cs:271)
    But they are popping up everywhere, including code gen files

    Code (CSharp):
    1. NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object
    2. This Exception was thrown from a function compiled with Burst, which has limited exception support.
    3. 0x00007ff7026706fd (Unity) burst_signal_handler
    4. 0x00007ff809428b4c (ntdll) RtlDeleteAce
    5. 0x00007ff8094012c6 (ntdll) RtlRaiseException
    6. 0x00007ff809450f4e (ntdll) KiUserExceptionDispatcher
    7. 0x00007fff92157e4b (bc02d18ffb1d97782999fc13c0444f5) BovineLabs.Game.SubScenes.LoadWithBoundingVolumeSystem::BovineLabs.Game.SubScenes.LoadWithBoundingVolumeSystem.OnUpdate (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Temp/GeneratedCode/BovineLabs.Game/LoadWithBoundingVolumeSystem__System_502371246.g.cs:33)
    This eventually leads to an editor crash after a lot of spam

    I can fix it by
    - turning off burst (obviously not ideal)
    - closing editor, deleting burst cache, opening editor
    - turning safety off (and optionally on again) to force a burst recompile
     
    tertle, Dec 5, 2022
    #1
    Shinyclef likes this.

  2. MarcoPersson

    MarcoPersson

    Unity Technologies

    Joined:
    Jul 21, 2021
    Posts:
    53
    Hi @tertle, do you have a consistent way of reproducing this? Or have noticed any specific circumstances that trigger this?
     
    MarcoPersson, Dec 8, 2022
    #2

  3. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    I seem to reproduce it quite frequently in one of my projects and I think it's somewhat related to codegen (maybe even netcode codegen specifically) and recompiling during burst compiling this.

    Sorry I was meant to follow this up more since this post but I have a massive milestone due tomorrow so I haven't had time. I'll try find a somewhat consistent repo this weekend in project.
     
    tertle, Dec 8, 2022
    #3

  4. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    No repo yet, but I took note of exactly what happened on the most recent one

    upload_2022-12-9_7-39-36.png

    On a recompile I got an infinite wait on entering play mode, force breaking inside unity shows it's waiting on an native AssetDatabase operation so I can't investigate further.

    Force closing unity at this point and restarting produces a corrupted burst cache and requires a second unity restart to fix.
     
    tertle, Dec 8, 2022
    #4

  5. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    @MarcoPersson ok so the most common way I get this is

    - after a script change, hit play while unity is still recompiling
    - sometimes (reasonably often for me) it causes an editor lockup during subscene loading/baking
    - force close editor via task manager
    - start project again

    Burst is now corrupted

    Hooking up a debugging during the lockup it seems stuck on
    Code (CSharp):
    1. AssetDatabasExperimental.ProduceArtifact()


    So yeah, not sure how much of this is on burst team vs entity team vs asset team since I think a lockup in assetdatabase / baking
     
    tertle, Dec 16, 2022
    #5

  6. scottjdaley

    scottjdaley

    Joined:
    Aug 1, 2013
    Posts:
    163
    I'm encountering a similar problem. Editor locks up. Cursor is invisible, no loading bar taking focus. Attaching a debugger and I see a thread with the same callstack as this (AssetDatabaseExperimental.ProduceArtifacts).

    I'm not seeing any burst errors after an editor restart. But I am seeing this one, although it doesn't seem correlated with the editor freezing.

     
    scottjdaley, Dec 16, 2022
    #6

  7. MarcoPersson

    MarcoPersson

    Unity Technologies

    Joined:
    Jul 21, 2021
    Posts:
    53
    From the stack-trace you posted, that part on its own at least doesn't seem to be directly related to Burst.
    I'd recommend reporting the freeze bug so my AssetDatabase colleagues can have a look at it.
    If there's still a Burst bug hiding there I'd gladly have a look at it if I can repro it

    Do note that people are starting to go on holidays now, so you might not see much movement until after new-years though
    :)
     
    MarcoPersson, Dec 16, 2022
    #7
    Anthiese likes this.

  8. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    So while the error I posted above is definitely an issue, I think there is more going on here.
    I'm finding my function pointers being scrambled on domain reload

    Code (CSharp):
    1.  
    2. DivideByZeroException: System.DivideByZeroException: Attempted to divide by zero
    3. This Exception was thrown from a function compiled with Burst, which has limited exception support.
    4. ...
    5. Shattered.Game.States.OptionsStateSystem.__codegen__OnCreate(System.IntPtr self, System.IntPtr state) -> void_aef4d46f9ec6d18a03913cfa3cd76414 from Shattered.Game, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.burst@1.8.2/.Runtime/unknown/unknown:0)
    6. object:wrapper_native_000001B790D8A8C0 (intptr,intptr)
    7. Unity.Entities.SystemBaseRegistry:ForwardToManaged (intptr,Unity.Entities.SystemState*,void*) (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/SystemBaseRegistry.cs:363)
    8. Unity.Entities.SystemBaseRegistry:CallForwardingFunction (Unity.Entities.SystemState*,int) (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/SystemBaseRegistry.cs:311)
    9. Unity.Entities.SystemBaseRegistry:CallOnDestroy (Unity.Entities.SystemState*) (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/SystemBaseRegistry.cs:381)
    10. Unity.Entities.World:DestroyAllSystemsAndLogException () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/World.cs:1011)
    11. Unity.Entities.World:Dispose () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/World.cs:306)
    12. Unity.Entities.World:DisposeAllWorlds () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/World.cs:332)
    13. Unity.Entities.DefaultWorldInitialization:DomainUnloadOrPlayModeChangeShutdown () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/DefaultWorldInitialization.cs:101)
    14.  
    OnCreate in a system somehow got called during on a domain reload destroying the world.
    It should have been calling OnDestroy but invoked OnCreate - it's like all my burst function pointers are getting muddled on domain reload.

    -edit-

    even weirder, is this gets called even if I've never entered play mode to create the system and I've verified it isn't in the editor world. Thoroughly confused what is going on.
     
    Last edited: Dec 19, 2022
    tertle, Dec 19, 2022
    #8

  9. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    287
    > OnCreate in a system somehow got called during on a domain reload destroying the world.
    It should have been calling OnDestroy but invoked OnCreate - it's like all my burst function pointers are getting muddled on domain reload.

    Yes, this is indeed caused by a bug in Burst - it took us a while to track down - quite a "fun" one :) It was exactly what you said - function pointers got mixed up after a domain reload. The fix will be in the next version of Burst.
     
    tim_jones, Jan 11, 2023
    #9
    FlightFight, MarcoPersson, Shinyclef and 2 others like this.

  10. Shinyclef

    Shinyclef

    Joined:
    Nov 20, 2013
    Posts:
    505
    Thank you for finding this. It has been a thorn in my side, completely demotivating me. It's not really feasible for me to leave burst enabled at the moment due to the frequency of crashes in my project, so this fix can't come soon enough.
     
    Shinyclef, Jan 11, 2023
    #10

  11. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    So I believe there is still a (very rare) corruption that is still possible

    What you're seeing here is a bursted OnUpdate in a system that only exists at runtime

    BovineLabs.Netcode.States.Client.ClientJoinGameStateSystem.__codegen__OnUpdate (System.IntPtr self, System.IntPtr state) <0x24e762387d0 + 0x0005a> in <53584b9e4565462ea4644f8b6cf28928>:0

    Being called from a baker

    Unity.Entities.BakingUtility:postprocessBake(World, BakingSettings, BakingSystem) (at ./Library/PackageCache/com.unity.entities@e98ed4d1fd/Unity.Entities.Hybrid/Baking/BakingUtility.cs:141)

    These are the filters for the ClientJoinGameStateSystem, clearly no baking flag
    Code (CSharp):
    1.  
    2. [UpdateInGroup(typeof(SimulationSystemGroup), OrderFirst = true)]
    3. [WorldSystemFilter(
    4. WorldSystemFilterFlags.ClientSimulation | WorldSystemFilterFlags.ThinClientSimulation,
    5. WorldSystemFilterFlags.ClientSimulation | WorldSystemFilterFlags.ThinClientSimulation)]

    Unfortunately I have no repo

    Potentially related to

    Which I see slightly more often
     
    Last edited: May 5, 2023
    tertle, May 5, 2023
    #11

  12. creativelefty

    creativelefty

    Joined:
    Jan 28, 2017
    Posts:
    10
    Unity 2022.3.15f1 - seems still not resolved the burst bug?
    Note: I don't use any jobs nor Burst Compile in my code.

    I have the following settings:
    Project Settings>Burst AOT Settings>Enable Burst Compilation DESELECTED
    Project Settings>Editor>Enter Play Mode Settings>Enter Play Mode Options ON
    Project Settings>Editor>Enter Play Mode Settings>Reload Domain ON
    Jobs>Burst>Safety Checks ON
    Jobs>Burst>Enable Compilation DESELECTED

    If Project Settings>Burst AOT Settings>Enable Burst Compilation is SELECTED don't see the bugs anymore (CORRECTION UPDATE: Still occurs regardless of whether Selected or not), so can I assume that even if not using any Burst Compile or Jobs in my code I still need to have Enable Burst Compilation SELECTED?

    (Below,errors when Enable Burst Compilation DESELECTED)

    InvalidCastException: Specified cast is not valid.
    (wrapper castclass) System.Object.__castclass_with_cache(object,intptr,intptr)
    System.Runtime.InteropServices.Marshal.GetDelegateForFunctionPointer[TDelegate] (System.IntPtr ptr) (at <7ec8e29954a6455daa48484a381ec418>:0)
    Unity.Burst.FunctionPointer`1[T].get_Invoke () (at ./Library/PackageCache/com.unity.burst@1.8.11/Runtime/FunctionPointer.cs:68)
    Unity.Entities.Content.RuntimeContentManager.ProcessQueuedCommands$BurstManaged () (at ./Library/PackageCache/com.unity.entities@1.0.11/Unity.Entities/Content/RuntimeContentManager.cs:706)
    Unity.Entities.Content.RuntimeContentManager+ProcessQueuedCommands_00001AC3$BurstDirectCall.Invoke () (at <464ba5a980eb4cf189a68ccd0ba6ad41>:0)
    Unity.Entities.Content.RuntimeContentManager.ProcessQueuedCommands () (at ./Library/PackageCache/com.unity.entities@1.0.11/Unity.Entities/Content/RuntimeContentManager.cs:675)
    Unity.Entities.Content.RuntimeContentSystem.Update () (at ./Library/PackageCache/com.unity.entities@1.0.11/Unity.Entities/Content/RuntimeContentSystem.cs:94)
    UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at <64861216782042de93237b5c061bff1b>:0)
     
    Last edited: Dec 18, 2023
    creativelefty, Dec 17, 2023
    #12

  13. Miro_Brodlova

    Miro_Brodlova

    Unity Technologies

    Joined:
    Apr 28, 2023
    Posts:
    23
    Hi @creativelefty ,Thanks for sharing this is happening to you.

    That looks like odd behavior indeed. Is it possible for you to make an official bug report regarding this behavior, and include your project that recreates it? (Just in case you haven't made official Unity bug reports before here's how one goes about it)
    That way we can take a proper look at it!
     
    Miro_Brodlova, Jan 3, 2024
    #13
    chadfranklin47 likes this.