Question Any way to get readable stack traces from burst in release builds?

My game is announcing soon and playtesting is underway. Unfortunately, most crashes occur in burst code. Users have to tell me what they were doing and send me a save file so I can debug it in the editor. Is there any way to get readable burst errors from release builds? Currently they look something like this:

This really hampers my ability to address crashes. If I can't easily reproduce, I don't even have any clues for what part of the code is behaving badly. This will be especially problematic when the game releases later this year and I start getting crash reports from users that aren't explicitly testing the game for me.

 

Is often much easier to debug burst crashes in build than errors in editor. You need to get access to the dmp file though, but then you should be able up see the exact line it crashes on.

Services like backtrace $$ can upload this to you automatically instead of having to ask your users to upload it.

TLDR: learn how to debug dmp files

 

I was actually just researching this and was going to start using backtrace. Debugging the dmp file was the missing element for me. I'll look into this.

 

Alright so I got WinDbg going, but the analyzer is pointing at a different line than I'd expect.

In my burst job, I intentionally added code that crashes my app:

nonUniformScaleLookup[entity] = new NonUniformScale();

In this case, the entity in question does not have a uniform scale, so it crashes. If I watch it in the Unity editor without burst on, the stack trace points me right at this line. However, windbg (and visual studio), both give me this result instead for the release build crash dmp:

And the stack trace is:

Notably my burst job is not present. I'm not even sure why the ReleaseChunk method would throw in this case anyway.

I'm using the pdb file found in the build directory

E:\Nevergames\final-factory\Library\BurstCache\Windows-Intel

154566d2150fce0f7a2fefd2be0610a9.pdb

Any ideas why it wouldn't point to my burst job at all?

 

Hmm sorry note sure, I've only ever used visual studio for debugging pdbs and never had an issue with that proper link.

-edit-

Actually one thing to note though is that there is no safety in builds therefore often your 'illegal' operations won't cause an issue on the spot. It'll just write really bad data to something and eventually crash elsewhere. Maybe that's what is going on here.

 

This must be true. In component lookup, the setter does this:

Code (CSharp):

1.                 ecs->AssertEntityHasComponent(entity, m_TypeIndex);
2.  
3.                 if (m_IsZeroSized != 0)
4.                     return;
5.  
6.                 void* ptr = ecs->GetComponentDataWithTypeRW(entity, m_TypeIndex, m_GlobalSystemVersion, ref m_Cache);
7.                 UnsafeUtility.CopyStructureToPtr(ref value, ptr);

ecs->AssertEntityHasComponent(entity, m_TypeIndex);

This line does nothing, it only fires when checks are on. Then it grabs the pointer for the data with that entity, and that method's comment says this:

// This method will return an invalid pointer if the entity does not have the provided type. It is the caller's
// responsibility to ensure that the type exists on the entity.

So yeah, it's just copying some garbage data into the lookup and failing later.

I wonder if a good idiom would be to always manually check yourself if, when using ComponentLookup, the entity has the component, and throwing an exception if not. Forcing the lookup would be slower, but in most jobs this performance cost wouldn't matter. You could leave it out of critical hotpaths.

 

Just to confirm the above observations: if I add in my own "safety checks" before accessing the ComponentLookup in my own job and throw an exception in this case, the stack trace does indeed point at exactly the right place in windbg (or visual studio).

The perils of native programming...