General stability of Beta? (Crashes per day)

Hi,

we use Unity 5 Beta since a couple of weeks and have around 3-10 crash per working day, depending on the typical use of the editor. (In Unity 4.6, this was a lot worse - due to the 32-bit limit, so we are kind of used to frequent crashes.)

The editor usually crashes when switching to play mode for the X'th time. Recently the crashes seem to occur much more frequently (most probably not because of Unity, but because we keep adding assets). I can reliable reproduce errors now (with Beta 19) by loading my test scene here and switching to play mode two times, so I thought I give it a shot and ask whether you are interested in a repro-case, or whether memory-related crashes in huge projects are a work in progress anyway?

The crash seems to be memory-related, but the exact message varies. Sometimes there is a console log with a totally strange stack trace. Or some generic "TypeInitializationError" in standard .NET classes or just things like "SceneView"..). Sometimes, the Unity bug reporter tool opens after the crash, sometimes not.

We didn't file a standard bug report with the report tool yet since the problem seem to occur only in our big project repository, which is around 50 GB to upload. If you are interested, we could arrange an GIT access or a snapshot download to our repo (compressed "only" a couple of GB).


------------------
A typical crash stack trace in some "strange" place:

Code (csharp):

1. NullReferenceException: Object reference not set to an instance of an object
2.   at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00007] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/NumberFormatter.cs:553
3.   at System.NumberFormatter.FormatGeneral (Int32 precision, System.Globalization.NumberFormatInfo nfi) [0x00077] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/NumberFormatter.cs:1322
4.   at System.NumberFormatter.NumberToString (System.String format, System.Globalization.NumberFormatInfo nfi) [0x00070] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/NumberFormatter.cs:1065
5.   at System.NumberFormatter.NumberToString (System.String format, Single value, IFormatProvider fp) [0x00078] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/NumberFormatter.cs:887
6.   at System.Single.ToString (System.String format, IFormatProvider provider) [0x00000] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/Single.cs:256
7.   at System.String.FormatHelper (System.Text.StringBuilder result, IFormatProvider provider, System.String format, System.Object[] args) [0x00152] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/String.cs:1941
8.   at System.String.Format (IFormatProvider provider, System.String format, System.Object[] args) [0x00000] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/String.cs:1873
9.   at UnityEditor.PrefColor.ToUniqueString () [0x00042] in C:\buildslave\unity\build\Editor\Mono\Animation\ZoomableArea.cs:255
10.   at UnityEditor.Settings.Set[PrefColor] (System.String name, UnityEditor.PrefColor value) [0x00000] in C:\buildslave\unity\build\Editor\Mono\Animation\ZoomableArea.cs:422
11.   at UnityEditor.Settings.Get[PrefColor] (System.String name, UnityEditor.PrefColor defaultValue) [0x00021] in C:\buildslave\unity\build\Editor\Mono\Animation\ZoomableArea.cs:419
12.   at UnityEditor.PrefColor..ctor (System.String name, Single defaultRed, Single defaultGreen, Single defaultBlue, Single defaultAlpha) [0x0000d] in C:\buildslave\unity\build\Editor\Mono\Animation\ZoomableArea.cs:217
13.   at UnityEditor.SceneView..cctor () [0x00014] in C:\buildslave\unity\build\Editor\Mono\Sprites\SpriteUtilityWindow.cs:66
14. Rethrow as TypeInitializationException: An exception was thrown by the type initializer for UnityEditor.SceneView

Attached is another typical message we see often (CheckDisalow.png)

 

Oh, and btw.. I am a co-worker of Korbinian who posted a similar problem here, although his post is from stand-alone builds, where these problems here are from in-editor "play" mode changes. I am not 100% sure, but I think I saw some "System.MonoCustomAttrs" in the editor-stack traces too, so this might be as well the exact same error..

 

My "reliable" test case here, where it crashes after exactly play/stop/play in my current test scene stops being reliable when I remove a couple of prefab instances and place them again. (I need more than 2 play-mode changes for the crash).

So this might not be only an out-of-memory problem. Another co-worker suspects that bug #653352 he reported is not in Beta 19 yet, although its marked as fixed as "in next beta release"?

 

bump? Anybody from Unity likes to comment?

 

Awesome. I'll zip some archive together and ask our admin to put it on some downloadable location and then file a bug report with the link today.

@KEngelsoft: Could you check whether the fix for 653352 is already included in Beta 19, please?

 

The problem seems to be a really sneaky ninja bug. After some necessary DLL replacements to be able to launch the project outside our environment here, the problem isn't easily reproducible anymore (it still crashes occasionally, but not predictable anymore).

I will work in this "DLL replaced" environment in the next couple of days and hope to get another test case where the bug occurs reliable and report back..

 

I know your post was about your specific issue your seeing; but from the title I thought/assumed you were polling people about 5's general stability and people might come here wondering (if they arent in the beta)

So my experience after using Beta 5 is that most nights I do not crash at all. About once per week I go through an evening where the editor will crash after I make a scene change, then a code change, and then tab back to the editor.

Oddly enough the next evening it goes away. So its possible a system restart might fix it when it comes up but of course I'm reluctant to do that because I'm trying to get my work done for the evening.

So I give it a 4 out of 5 stars for stability right now.

 

Seems like you guys are getting useful debug messages at least. I crash about 1/4th the time I hit play in the editor and its always just an Access Violation crash so nothing useful to report.

 

Yes. That's the most common crash here too. The editor.log in your Local AppData directory usually only contains a line "CRASH!!!" somewhere in the middle but no usefull stack trace thingie or anything.

Although... 1/4th of the time.. that's hard.. I feel you, bro..

 

@KEngelstoft : I have send a bug report under Number 668506 together with a download link for an striped down snapshot of our project (I removed all audio and textures).

It seems not to be memory related. After deleting all textures and wav files, the problem still occurs. The memory usage of the editor is merely 1.3GB after startup.

I would be glad for any kind of feedback what could cause the frequent crashes.

Best regards, Imi.

 

Does the stack trace output to editor.log, or output_data.txt for standalone builds, ever output any beyond "OUPUTTING STACK TRACE" with Windows 5b14 32-bit or 5b18 64-bit? My team is seeing frequent crashes, and we'd really like a lead from a log file, or some kind of debugging capability.

 

RC1 still contains the frequent crashes for us. Is there any progress in this issue? Could you reproduce the crash?

The bug issue (668506) has one reply from Ieva from QA team asking whether the lots of console logging errors after loading the project in the editor are normal. I replied that they are. No other reply so far.

Can I be of help to get this resolved?

 

It seems like our bug reporter or something is not invoking?

Is there a dialog or something before the process closes? If so, you then have a chance of taking a crash minidump or otherwise to get a crash stack. If you could get that, and add it to the bug report, it would be most helpful.

 

There is usually no unity bug reporter opening at the end. Sometimes there is, but as I said - the actual errors are varying.

I was hoping that you can reproduce the case to be able to better debug the cause(s). (After the initial library import, it takes me about two minutes of play/stop/assembly reload/play/stop/assembly reload ... until it crashes.)

But I will try to get a .dmp file via Visual Studio, if that is of any help..?

 

Is there any news here? The frequent crashes really bogging us down. I estimate, that we are loosing at least 25% of our work time waiting for Unity to crash and restart. The stability seems to be worse than ever before, (agian: this is probably related to us pumping more and more assets into the project, not because Unity versions getting more instable).

Some developer here even don't bother anymore trying to press "Stop" in the editor but just directly killing the Unity.exe process from task manager when they pressed played once, as the probability to get it playing a second time is near to zero.

The fact that unity is gone "Release Candidate" with these stability issues makes me wonder whether we are just an exception here and whether we use some "unusual" code part of the editor that we could avoid to not have it crash so often... Any hints from the crash dumps so far?

 

No, there is still nothing usefull. It seems to be the same problem as I wrote before: Usually the line "Crash!!!" is the first hint of something gone wrong.

After that, there is a list of loaded DLL's and then

Code (csharp):

1.  
2. ========== OUTPUTING STACK TRACE ==================
3.  
4. 00000000084E8933 (Mono JIT Code) [/Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/Enum.cs:186] System.MonoEnumInfo:GetInfo (System.Type,System.MonoEnumInfo&)
5.  

and that's it. Before the Crash!!! - line there isn't anything suspicious either (i.e. lots of "this message is harmless" lines)

But sometimes, there are gazillions of strange exception like in my very first post before the crash. Here is an example of the first exception of another crash I had just now:

Code (csharp):

1.  
2. Platform assembly: C:\Program Files\Unity\Editor\Data\Managed\Unity.DataContract.dll (this message is harmless)
3. Platform assembly: C:\Program Files\Unity\Editor\Data\Mono\lib\mono\2.0\System.Core.dll (this message is harmless)
4. Platform assembly: C:\Program Files\Unity\Editor\Data\Managed\Unity.IvyParser.dll (this message is harmless)
5. Error enumerating assemblies looking for package manager. System.NullReferenceException: Object reference not set to an instance of an object
6.  
7.   at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00007] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/NumberFormatter.cs:553
8.  
9.   at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00032] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/NumberFormatter.cs:1013
10.  
11.   at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00025] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/NumberFormatter.cs:939
12.  
13.   at System.Int32.ToString () [0x00000] in /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/Int32.cs:672
14.  
15.   at System.Text.RegularExpressions.Syntax.Parser.ResolveReferences () [0x00000] in <filename unknown>:0
16.  
17.   at System.Text.RegularExpressions.Syntax.Parser.ParseRegularExpression (System.String pattern, RegexOptions options) [0x00000] in <filename unknown>:0
18.  
19.   at System.Text.RegularExpressions.Regex.CreateMachineFactory (System.String pattern, RegexOptions options) [0x00000] in <filename unknown>:0
20.  
21.   at System.Text.RegularExpressions.Regex.InitNewRegex () [0x00000] in <filename unknown>:0
22.  
23.   at System.Text.RegularExpressions.Regex.Init () [0x00000] in <filename unknown>:0
24.  
25.   at System.Text.RegularExpressions.Regex..ctor (System.String pattern, RegexOptions options) [0x00000] in <filename unknown>:0
26.  
27.   at System.Text.RegularExpressions.Regex.Match (System.String input, System.String pattern, RegexOptions options) [0x00000] in <filename unknown>:0
28.  
29.   at System.Text.RegularExpressions.Regex.Match (System.String input, System.String pattern) [0x00000] in <filename unknown>:0
30.  
31.   at Unity.DataContract.PackageVersion..ctor (System.String version) [0x00000] in <filename unknown>:0
32.  
33.   at UnityEditor.Modules.ModuleManager.InitializePackageManager (System.Reflection.Assembly assembly, Unity.DataContract.PackageInfo package) [0x0006e] in C:\buildslave\unity\build\Editor\Mono\ParticleSystemEditor\ParticleSystemModules\UVModuleUI.cs:55
34.  
35.   at UnityEditor.Modules.ModuleManager.RegisterPackageManager () [0x0003e] in C:\buildslave\unity\build\Editor\Mono\ParticleSystemEditor\ParticleSystemModules\SubModuleUI.cs:43
36.  

I don't think that these are the places where things go wrong. To me, this screams like a memory corruption bug within either the C++ part of unity or even a bug in Mono.exe. That's why I was hoping to give you a reproducible test case. (In my experience, crash dumps of memory corruption bugs are not super helpful as they show you the state when things already went wrong for a while. When I have to trace one of these bugs down, I always try to get my hands on reliable repro case to be able to debug while the stuff is happening).


Anyway.. I've emailed the last two editor logs (where the examples above are coming from) to the bug issue 668506. Maybe you can have a look and find some anomalies or identify some known problems?

Thanks for your help,
Immanuel Scholz

 

We are around 20 people and have probably installed over 100 unity installations so far since Unity 5 Beta. Since the beta always installs into a new folder, probably less than 10% of the time, it _might_ have happened that someone installed unity over an existing folder. In basically all the installations, these crashes happen. So.. no: I can definetely rule out an corrupted unity installation as the source of the problem.

The fact that the "package manager" is mentioned on the top of the stack trace is distracting at best. My initial post shows a very similar stack trace but with "SceneView" at the top. I think the errors are not in the package manager nor in the SceneView.

I don't quite understand what you mean with this. NullReferenceExceptions are runtime exceptions, not compile errors. How can they be "in our scripts"?. Yes, of course we are trying to fix exceptions when we encounter bugs that cause them in our game.

Or are you referring to the fact that the first exception in the log is a NullReferenceException in System.NumberFormatter.ResetCharBuf? Then, please observe that there is not a single line outside the Unity editor in this stack trace where I could do any kind of fixing.

Beside, if my intuition is right and this is a memory corruption, then remember, that for memory corruption bugs the crash stack trace does not show the problem! The problem already happened. The stack trace just shows the first piece of code that was unlucky enough to access the corrupted memory.


Regards,
Immanuel Scholz

 

Thanks a lot Tomas. . Reading from your response you think it may be a bug in Mono.exe, right? We use some rare features like IL code emitting, custom attributes (everywhere), thread static variables, FormatterServices.GetUninitializedObject and tons of lambdas, linq and yield functions... anything particular that caused lots of trouble in Unity & mono in the past? (Some are used editor-only and others are used in its own server process later but run inside Unity while the editor is in playmode)

(With enough time, these features could be replaced with more "normal" things if they are known to cause hazard?


Here is my stack trace I got just now:

ntdll.dll!00007ffd456d84be() Unknown
kernel32.dll!00007ffd42d1f587() Unknown
Unity.exe!0000000140dc133f() Unknown
Unity.exe!000000014018a9c2() Unknown
KernelBase.dll!00007ffd42cc1c72() Unknown
ntdll.dll!00007ffd45721368() Unknown
ntdll.dll!00007ffd457024a6() Unknown
ntdll.dll!00007ffd45713b1d() Unknown
ntdll.dll!00007ffd456d8407() Unknown
ntdll.dll!00007ffd45712caa() Unknown
000000003f400000() Unknown
000000003630c1ed() Unknown


Given this stack trace, you suspect 0000000140dc133f and 0000000140dc133f might be the culprit, right?
Both addresses returned "0x0000000000000000 <NULL>" for me. I'll try the 32-bit approach tomorrow or this week (we have some mean deadlines upcoming days ).

Greetings,
Immanuel Scholz

 

Up to now, I could not reproduce the error when running the 32-bit version. (Sadly, switching to 32-bit editor is not an option for us due to the 3gb memory limit.)

I inspected a dozend more crash dumps so far and mostly got no function name at all from the unnamed call stack entries. Once, the function was inside the SyntaxTree UnityVS dll, once there was no unnamed call stack entry at all and a couple of times, the only function line that was translated was the "System.MonoEnumInfo:GetInfo" that I posted above from the editor.log.

If there is anything we can do to help resolve the crash issues, please feel free to contact me. Until then, we hope that things just turn out to be better in the future . (TBH, most of my co-worker are so used to the crashes by now, they don't even bother)

Regards,
Immanuel Scholz.

 

Hi Tomas,
I'm a coworker of Immanual and I was able to get some stack information using your method:

[native part]
ntdll.dll!RtlVirtualUnwind() Unknown
kernel32.dll!RtlVirtualUnwindStub() Unknown
Unity.exe!StackWalker::ShowCallstack(void *,struct _CONTEXT const *,int (*)(void *,unsigned __int64,void *,unsigned long,unsigned long *,void *),void *,int) Unknown
Unity.exe!winutils:rocessInternalCrash(struct _EXCEPTION_POINTERS *,bool) Unknown
KernelBase.dll!00007ffb9471ce8a() Unknown
ntdll.dll!RtlUserThreadStart$filt$0() Unknown
ntdll.dll!__C_specific_handler() Unknown
ntdll.dll!RtlpExecuteHandlerForException() Unknown
ntdll.dll!RtlDispatchException() Unknown
ntdll.dll!KiUserExceptionDispatch() Unknown

[Mono stack - unrolled using mono_pmip]
0x0000000000000000 <NULL>
0x0000000026564ab0 " System.MonoEnumInfo:GetInfo (System.Type,System.MonoEnumInfo&) + 0xd3 (00000000054C87C0 00000000054C98E2) [00000000021A67F8 - Unity Child Domain]"
0x0000000000000000 <NULL>
0x0000000000000000 <NULL>
0x0000000000000000 <NULL>
0x0000000000000000 <NULL>
0x0000000000000000 <NULL>
0x000000002d986c20 " (wrapper runtime-invoke) object:runtime_invoke_void (object,intptr,intptr,intptr) + 0x0 (00000000054010D0 000000000540130D) [00000000021A67F8 - Unity Child Domain]"
0x000000009c485990 " UnityEditor.SyncVS:GetInstalledVisualStudios () + 0x0 (0000000008178F00 00000000081798B9) [00000000021A67F8 - Unity Child Domain]"
0x0000000000000000 <NULL>
0x0000000000000000 <NULL>

Seems to be originating from GetInstalledVisualStudios() in the UnityEditor.dll. Might be related to another enum related error we had awhile ago:
http://forum.unity3d.com/threads/cr...rs_from_class-g_free-access-violation.286431/

Hope this helps!