Mono.pdb file, needed to debug an infinite loop

I felt very good about this page on the Unity Documentation:

http://docs.unity3d.com/Manual/WindowsDebugging.html

but in order to be able to have a minimal WinDBG support, I need the embedded mono.pdb file, which is not distributed with Unity 5.2 or Unity 5.3 (It's instead present in Unity 5.4 beta).

Can someone at Unity be so kind to provide this file?

 

ops, I forgot to mention that I tried that, but it looks like 5.3 is not available either. I can see that the last pdb available is on May 2015. https://symbolserver.unity3d.com/mono.pdb/ are you sure it's there? I can tell you which signature Windbg is looking for (it will always mismatch)

P.S.: thanks a lot for the help, it's very appreciated!

 

Also let me add this, I found out that our infinite loop issue happens inside the Globalgamemanager::Transfer function...any clue what GlobalGameManager can be? Never heard of it.

 

I am testing both with Unity 5.2 and Unity 5.3. Since you say you don't support 5.2, let's focus on 5.3:

First let's talk about the .PDB. I have built a standalone windows 32 bit development client, with script debugging option on (probably not needed, but I added it anyway). I then run it and attach WinDBG. WinDBG uses this Symbol path:

cache*c:\symbols-cache;srv*https://msdl.microsoft.com/download/symbols;srv*http://symbolserver.unity3d.com/

then I use whatever command, for example .analyze -v

It tries to load all the symbols and then this is what I get:

SYMSRV: http://symbolserver.unity3d.com//mono.pdb/F2C8DA40ABC24B4FB462F7FB70D949571/mono.pdb not found

I am not sure how the symbol file hashing works, but F2C8DA40ABC24B4FB462F7FB70D949571 is not in your server.

----------------

About the GlobalGameManager::Transfer. We can reproduce the freezing both with 5.2 and 5.3. I used process explorer to analyze the threads. The thread running GlobalGameManager::Transfer never ends and takes 25% of the CPU constantly (while the other threads are not doing anything). I need to find out what's wrong, any help is very appreciated.

 

OK that's probably it then, I am using the patch version 5.3.2p2. Can you please confirm that 5.3.2f1 is available on your servers?

About the stack, unluckily I lost it, but it appears that the thread is waiting for a signal, that never comes, as result of an Animation Event. At this stage I have not enough information yet, but I will get back once I know more.

question: why don't you ship the pdb file with Unity?

 

ah I got the stack...I am not 100% sure if this is the freezing I am after...I still need more investigation:

ntdll.dll!ZwWaitForAlertByThreadId+0xa
ntdll.dll!RtlReleasePath+0x5fa
ntdll.dll!RtlReleasePath+0x4fa
ntdll.dll!RtlReleasePath+0x39f
ntdll.dll!RtlEnterCriticalSection+0x129
ntdll.dll!RtlEnterCriticalSection+0x40
Unity.exe!GlobalGameManager::Transfer<StreamedBinaryWrite<1> >+0x3c6eb7
Unity.exe!GlobalGameManager::Transfer<StreamedBinaryWrite<1> >+0x3aa870
Unity.exe!GlobalGameManager::Transfer<StreamedBinaryWrite<1> >+0x3b80c5
Unity.exe!GlobalGameManager::Transfer<StreamedBinaryWrite<1> >+0x3b9102
Unity.exe!GlobalGameManager::Transfer<StreamedBinaryWrite<1> >+0x3b92cd
Unity.exe!GlobalGameManager::Transfer<StreamedBinaryWrite<1> >+0x3ba1e6
Unity.exe!GlobalGameManager::Transfer<StreamedBinaryWrite<1> >+0x3a17bc
Unity.exe!AnimationEvent::Transfer<YAMLRead>+0x3668e3
Unity.exe!AnimationEvent::Transfer<YAMLRead>+0x3669b8
Unity.exe!Renderer::Transfer<YAMLWrite>+0x230e7
Unity.exe!LevelGameManager::Transfer<StreamedBinaryWrite<0> >+0x2d74
Unity.exe!LevelGameManager::Transfer<StreamedBinaryWrite<0> >+0x2e5f
Unity.exe!LevelGameManager::Transfer<StreamedBinaryWrite<0> >+0x315f
Unity.exe!LevelGameManager::Transfer<StreamedBinaryWrite<0> >+0x31d7
Unity.exe!AnimationEvent::Transfer<YAMLRead>+0x138421

 

Great thanks! Further investigation leads me to think now that the problem (I guess a deadlock) is happening in FMOD native code.

This time I used windbg (the stack you say is broken was generated by Process Explorer, probably didn't have the symbols loaded) and the stack is of the thread consuming the most CPU is

00 0d97f8c0 00d1177b RobocraftClient!FMOD_OS_SupportsSSE+0xf
01 0d97f8e0 00d0d19b RobocraftClient!FMOD_Resampler_Linear+0xb
02 0d97f950 00cc81a9 RobocraftClient!FMOD:SPWaveTable::read+0x50b
03 0d97f99c 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
04 0d97f9e8 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
05 0d97fa34 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
06 0d97fa80 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
07 0d97facc 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
08 0d97fb18 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
09 0d97fb64 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
0a 0d97fbb0 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
0b 0d97fbfc 00cc81a9 RobocraftClient!FMOD:SPFilter::read+0xd9
0c 0d97fc48 00d0d718 RobocraftClient!FMOD:SPFilter::read+0xd9
0d 0d97fc84 00cc70b0 RobocraftClient!FMOD:SPSoundCard::read+0x38
0e 0d97fcf8 00ccabb9 RobocraftClient!FMOD::Output::mix+0x2e0
0f 0d97fd3c 00cc0d8d RobocraftClient!FMOD::OutputPolled::threadFunc+0x319
10 0d97fd48 00e5772a RobocraftClient!FMOD::Thread::callback+0x6d
11 0d97fd80 00e577b4 RobocraftClient!_endthreadex+0x3a
12 0d97fd8c 76c33744 RobocraftClient!_endthreadex+0xc4
13 0d97fda0 77489cd4 KERNEL32!BaseThreadInitThunk+0x24
14 0d97fde8 77489c9f ntdll!__RtlUserThreadStart+0x2f
15 0d97fdf8 00000000 ntdll!_RtlUserThreadStart+0x1b

I think it's due to something inside Fabric.

Thanks a lot again for the mono.pdb...should I bother you again if I update the editor?

 

yes and I don't think it's the first time we are having this issue. Do you have an advice to be sure I am right? So far I know that the thread is taking 25% of the CPU, I used the same thread ID in windbg, push K and that was the call stack.

 

Sorry to take advantage of your kindness, but could you also generate the 5.2.2f1 for me? Our live client is running on that version and I am debugging that one too with the help of our QA department.

 

I'll do it, thanks for the suggestion

 

Thanks a lot! This is the result of the windows performance. Please note it's the first time I heard of it, I am not 100% sure I used it right, but the result seems the same:

 

I just realized that I can't find a way to call mono_pmip((void*)0x1b45558c) from windbg. the c++ evaluator ?? seems to not be able to evaluate functions in this way. Can you confirm? Is visual studio the only option available then?

 

oh I didn't because all the other cycles where showing 1...I will do it tomorrow, the file should be still saved. I appreciated a lot your help!

 

about the blog. I read that already. It's new stuff to me, but as far as I understood .call actually calls the function from the application from the current thread. I verified it because the main thread call stack was showing mono_pmip as last call. Also it matches the logic:

Thread is set up for call, 'g' will execute.
WARNING: This can have serious side-effects,
including deadlocks and corruption of the debuggee.

 

No I don't think .call is what we want (the evaluation of a function). That's probably why in the Unity3D documentation there is written to use Visual Studio to run mono_pmip. I'll try that next time it freezes.

OK roger that! Tomorrow first think I will send you the image (if you have windows performance analyzer I can even send you the report file)

 

Good Morning. I am back with more information. First the file you were looking for:

https://drive.google.com/file/d/0BynSzm3EhYu5RnlDRjQ0YTE4V0FrRTBVNG9zaW1KV3dYZ3ZF/view?usp=sharing

than the reply from Fabric author, which seems to confirm my fears:

"Yep this looks like a classic FMOD deadlock, their threading implementation is not that great!!.... usually these are caused because other threads are stalling or crashing so in order to get a better picture you really need to check the state of all the game related threads... for example it's possible that FMOD's streaming thread is waiting to open/read a file (for example streaming music) but Unity's filesystem is also streaming a scene/asset bundle which then might wait for the game thread which waits for FMOD to release the DSP connections etc.... or it's possible that the mixing thread has encountered an error or again it's stalling on reading a file (yes I have seen it doing that in there as well).... unfortunately it's very hard to track these issues especially without having a complete picture of the threads or access to the source code.... Your best bet will be to submit this issue into Unity ideally with a project that they can reproduce.

With regards to what you can do to improve the situation... perhaps you can reduce/remove the number of DSP effects (if you have any active) or reduce the number of audio sources you are trying to play or stream (Fabric can help in there). Also if you are using the Fabric's external group components or dynamic audio clip loading perhaps or audio source pool you can disable them temporarily and see if that makes a difference.

Ultimately it's only Unity that can fix this issue properly. "

My priority is to understand what causes this and how I can prevent this from happening in future. This is the third time we release to the public a client that randomly freezes (for the same reason) and of course it causes an uproar among the community.
So far we fixed it almost randomly changing the audio setup, which, of course, is not sustainable as strategy. I hope I can find out the solution with your help. So thank you again for following up.

 

new findings from this morning investigation:

it seems that the deadlock happens when a sound is disabled (entering in a critical section) and a resampling of an audio is happening. We shouldn't have any resampling, it's a mistake, so we are removing all them. I'll keep you updated.

 

Damn, sorry, my bad, I chose the wrong file!

new link: https://drive.google.com/file/d/0BynSzm3EhYu5R0plMk5ONktXMXNFeG1mLVNJd0RBVzlqOGVB/view?usp=sharing

also I am quite sure I found out the issue (but not the cause yet)

The main thread is locked in this critical section (ignore the mono_pmip call, apparently the VS immediate window DOES the same thing of the windbg .call function)



and the hanging thread is:


I found it out through the !lock command:

 

Images fixed, I am quite frustrated because I can't get mono_pimp to work. It always raises an exception when it's called. I need to know what 0x6a9c5df is

 

Could you please explain to me if the AudioSource RemoveFromManager has been called from a gameobject destroyed or simply deactivated?

 

We are not using OnAudioFilterRead explicitly, but it's possible that Fabric (our audio engine) does. I sent this thread to the author, so I hope he will be able to reply.

 

I disassembled Fabric and OnAudioFilterRead is used in 3 places. I can't add much more information, because without being able to use mono_pimp (it crashes during evaluation), I can't tell you what is calling what.

 

More info. It's used in 3 different Monobehaviour.

AudioCapture should be for recording, therefore we are not surely using it.
AudioPanner is for the 5.1 audio, that we don't support
Finally StereoSpreader, which we don't use either, is to recreate a stereo image from a mono source. We don't use it either.

So I don't think it's involved. Isn't my WPR helping at all?

 

we don't use DSP filters, so I would not take in consideration us explicitly using it. However we cannot understand why the Resample_Linear function is called. What's the reason? Why is there? All our sounds are sampled at 44khz and our audio settings sample rate is 44khz. Why is this function called then? Also is it normal that DSPFilter::read is used even if we don't use DSP filters at all (neither through Fabric nor through Unity audio mixer)?

About the unloading..Fabric uses a pool of AudioSources. I guess, but only guess, that what it does is to disable the AudioSource, without destroying it, when it's recycled. This is actually what you can see from the stack. A GameObject is destroyed/deactivated and the AudioSource is recycled (I guess in order to do it, it must remove it from the table of active audio sources)

 

thanks,this makes things clearer. We have a limited level of control, because like I said, we use Fabric which automates a lot of stuff. It's also not simple to know what exactly is going on. We think it's due to sounds related to projectile of new weapons we introduced (or explosions of course) and those sounds are played continuously during a multi player match. The client doesn't crash only because the player uses a specific weapon, but it could be just because sits next to another player using that weapon. Also it's pretty random due to the high number of possible combinations (our game is a building game with physic)
In the past we solved similar cases just changing the audio setup, but this process is too empiric to be used as a strategy (in fact this problem keeps on happening over the time).
You told me to ignore the DSP stack, but what about the main thread? There it's obvious that a gameobject is disabled/destroyed...shouldn't we focus on that to pin down the case?

 

the frequency is also very random. Knowing that it happens with specific weapons, we used those weapons until we meet a crash. Some times it happens on the first match, other times we need to wait 4-5 matches. Overall it usually happens between the first 30 minutes of playing...I'd say most of the times around 15-20 minutes of gameplay.

I guess a dmp file or a WPR file would not be useful for you to debug the case?

Can you please explain to me why the stack shows a gameobject setactive? Can you please confirm that it means that the object has been destroyed and or disabled?

BTW the stacks shown are a constant...it always shows these stacks.

 

Ok the only information I can add right now is that Fabric uses AudioSource only inside a single MonoBehaviour (Called AudioComponent). This MonoBehaviour doesn't have an OnDisable, but it does have a OnDestroy.

I really appreciate your help and I hope you can find a fix. I will be here to help if you need more information, but we have a big issue: we need to release a new client this week and I need to find a solution for the freezing. What are the options left? Shall I ask to our audio designer to try to change some configuration?

 

Tomorrow I will ask to reproduce the freeze and I will create a new DMP file.

 

Hello,

DMP file is available here: https://drive.google.com/file/d/0BynSzm3EhYu5TFFnMUhybnBrZFk/view?usp=sharing

On our side, we are still investigating to understand what we changed that introduced the problem. It would be VERY useful if you can teach us some trick to identify the sound from the DMP or frozen executable. Any other further idea to let us pin down the issue would be useful as well.

it would be quite useful actually to know what these functions

0x685311a
0x685307b
0x74dc294
0x15f7120a
0x678c5df

Edit: nvm the pointers, I don't think they are useful if the application is not actually running. Meanwhile I finally figured out how to use mono_pmip from windbg. .Call actually works, but I think the result cannot be evaluated if the thread is locked.

 

We finally pinned down the cause of the issue. It was due to a looped, not virtualized, sound attached to a GameObject which is disabled after a while. I really cannot understand how something in nature so random as a deadlock can be caused by a specific sound only.

I tried to recreate the same scenario in an empty scene, but on its own the sound doesn't cause the freezing.

Since those are Fabric features, I doubt you can help further, but if you have any information, I am here for future reference.