Mono Upgrade Unity editor crash when changing scene only with .NET 4.x (100% reproducible)

I'm not sure it's the right place for that, so don't hesitate to tell me where to go if it's not.

Some background:
When going for Unity 2018.3.0f2, I saw that .NET 3.5 was depreciated, so I went to .NET 4.x.
After a few fixes here and there, it finally compiled, but Unity kept on crashing with no error message, no crash log, and nothing in the log file.

It took me a few days to locate the origin of the problem, it's linked to the memory allocation. In some situation Unity crashes (the editor, but it also crashes in release).
It is not the quantity of the allocation which is the culprit, it seem to be a mix of quantity and size of memory blocks allocated.
And it's not the allocation itself which is the problem. In fact, the allocation works perfectly well, it's only when going for another scene that Unity crashes.

I've created a very tiny project which reproduce the problem 100% of the time.
It has a few files, but each file has no more than a couple of lines of code, so in the end it's very small.
It's probably possible to cut it down even more, but I stripped down the 20k lines of code of my current project down to the 19 lines of code of this test project, and I haven't bothered to try to simplify it further as I think it's small enough to debug (especially since about 2/3 of the code is used to display the bug and only 1/3 to actually crash Unity).


Small explanation about what the test project does:
In the first scene, you will see 3 input fields and one 'Next scene' button.
When the 'Next scene' button is clicked, it will allocate 'Nb events' * 'Nb records' * 'Pool capacity' events using some lists and arrays, and after that it will go for the second scene.
It is possible to change these values in order to find out the threshold that crashes the program (it's not the same when in Unity and when in release, and I suspect that it may change depending on the OS/computer).
But for now, do not change the values, they crash perfectly well as they are on all the computers I could try them on.
The second scene is the same, except that it has no input fields, the button reads 'Exit' and after a few seconds it displays 'No crash'.


How to reproduce the crash (works 100% of the time):
The test project is attached to this post.
Steps to reproduce:
- unzip the AllocCrash.zip file
- open the project with Unity 2018.3.02f2 (I've tried a couple of older Unity version, same problem)
- open the 'InitialScene' scene
- start the project
- press the 'Next scene' button
- observe that the input fields disappear
- observe that the button now reads 'Exit'
- observe a Unity crash with no crash log and nothing helpful (at least I think so) in the Editor.log file


Steps to demonstrate that the problem doesn't occur with .NET 3.5:
- relaunch Unity
- re-open the same project
- go to the Player settings and set the project to use .NET 3.5
- launch the project again
- click on the 'Next scene' button
- observe that the input fields disappear
- observe that the button now reads 'Exit'
- wait a few seconds
- observe that the test 'No crash' is displayed in the center of the screen
- observe that there is no crash

I insist that the crash occurs when changing scene.
The allocations are working perfectly, my full project can work for hours without changing scene with all the allocation already done.
And anyway, in the test project the fact that the input fields disappear indicates that we are in the second scene, and the allocations are done before changing scene, which is proof enough.

 

Done again.
I already did it about a month ago, but I got no e-mail confirmation, I guess that it's been lost somewhere.
I used another email this time, just in case.

Is there a way for me to ensure that you received the bug report?

 

You're right, this time I received a confirmation e-mail.

I double-checked and I am sure I haven't received a confirmation mail for my first bug report.
I clearly remember seeing the same window explaining me that I should receive one along with a few information about bugs reporting.
Maybe it was a bad idea to report a bug December 23rd.

Anyway, now I got the bug report confirmation (1117663) so it's OK, thanks for the help.


Note that I am also interested if other people have come across such a problem, because I can't imagine that I'm the first to have that problem, especially with .NET 4.x being around for more that one year now (or I am a lot more unlucky that what I thought...).

 

Yup, that instantly crashes. You're doing some pretty funky things in the repro case with referencing a very specific static field in a very generic non-static class, and mixing delegates into the mix, so I assume that
a) that's what's exposing the 4.5 bug
b) that's why nobody else has discovered it.

Impressed by the repro case, though, it's got quite a lot of code.

I tried to dig into it some more for fun, then my computer bluescreened. May or may not have been related.

 

Thanks for confirming that it crashes on a different environment than mine.
I never had a blue screen related to that bug, and I reproduced it at least a hundred times (probably more...) when I used dichotomy on my repository in order to find out THE commit which made it happen (which was in December 2017...) and afterwards, when I extracted the relevant parts in order to create a small repro project.
So I think your blue screen is probably unrelated to that specific bug.

As for the repro case, it's only 19 lines of code according to the Visual Studio metrics. It's mostly declarations in fact, there's nearly no actual instructions involved.
I mostly played with collections and generics, the 'static' is only there to ensure that the references shouldn't be garbage collected, I just tried without it and it still crash anyway.
It's probably possible to reduce the code a bit further, but I doubt it would help Unity to fix it as it's tiny enough.


From the way it crashes, it may well be a stack overflow, and knowing that Unity clean-up a few things when going to another scene, I suspect that Unity is using some recursion algorithm in order to clean some garbage, going through allocated blocks of memory, blowing the stack if there are too many of them.
If that's that, they should catch it very easily with the dev environment, after that it would just be a matter of refactoring a recursive algorithm in an iterative one, which is usually quite easy, but I suspect that the tests would take some time, as a mistake in that area has the potential of being really nasty for a lot of people...

 

I also am having issues with the .NET 4.x runtime version. I left a comment on your issue case but I want to bring attention to this problem so might as well post here too.

In my case, it's a recursive dungeon generation algorithm causing the crash. The algorithm only manages to run for about 30 iterations before crashing the entire editor. To be fair, that algorithm contains a lot of stuff. I pass a struct containing some data as a parameter to the function, the data inside that struct is edited as the function goes through multiple RNG rolls and finally the updated struct is passed to this same function again to "step" to the next tile in my map according to the direction set inside the struct.

But the thing is I never had issues with .NET 3.5. Hell, I wrote the original version of this algorithm using Java and even with the annoyance that is the JVM I didn't get any issues. So given that things works perfectly using .NET 3.5 but not using .NET 4.X, I'm assuming this has to be an issue with the runtime and hopefully not an issue with my code.

Sadly it seems that there are only a handful of bug reports about this particular issue so I'm a bit spooked by the upcoming removal of NET 3.5 from Unity 2019.

 

Okay well that's an odd one. I'm just here throwing in the information I managed to find about this issue out there so that the next poor soul who ends up down the same rabbit hole I did has a bit more info on the subject.

Basically I managed to fix my issue by shoving the ref keyword in front of the struct I'm passing to my recursive algorithm. If I understand correctly ( and I'm not sure I do ) this likely uses less memory since the variable is passed by reference and significantly reduced the load on the call stack. In my mind, this whole ordeal implies that the fact that I kept passing a struct variable by value to a recursive algorithm was piling on new references to the old struct values until the recursive function finally ended, before which point the random nature of the algorithm may or may not have already caused the stack to overflow.

What I think is going on is that since my struct contains lots of variables ( 15 integers, 10 floats and 9 booleans, this might be a dumb practice but it worked fine on 3.5 ) this ends up being too much for the call stack. The wikipedia article on "Call Stack" states that in some programming languages, when too many parameters are passed to a function to properly fit on the stack, memory should automatically be allocated to take care of those parameters. So basically the struct "hides" the amount of data needed to be passed. Maybe using a class, which is automatically passed by reference, would have been a smarter idea in this instance. But I had no way of knowing this would cause trouble while using .NET 3.5.

Something I have to point out is that this fix is definitely not a universal solution to the problem. The ref keyword works for me even though I make changes to the passed variable but still need the original version as is when the program returns to the part of the algorithm that called the recursive function. This is simply because I already stored the passed parameter into a local variable and made changes to that variable only, meaning the original struct remains unchanged.

What I still don't understand here is why this algorithm does not cause problems on 3.5 but does on 4.x. The .NET framework 4.0 migration issues page doesn't mention anything related to my issue and I'm quite frankly done googling for this issue only to stumble upon the same 3-4 threads... so if anyone with a deeper understanding of compilers and the .NET framework has any idea what causes this weird quirk of .NET 4.X, be sure to let me know.

Once again this isn't a support request, I'm just throwing my experience out there for anyone who might end up in a similar situation.

 

You're right, using 'ref' should reduce drastically the use of the call stack.

But I'm pretty sure that your problem is not related to mine.
My problem only occur when changing scene, if a lot of small chunks of memory have been allocated.
I guess that the GC is executed when changing scene, and it looks for the released memory chunks, and at that point I guess that they have a recursive algorithm in the GC which is exploding the stack when too many memory chunks have been allocated.

So, for my problem I think that the stack which is causing the problem is not broken by the user, but by the GC itself.


As for why does it work with .NET 3.5 and not .NET 4.x, I suppose that the stack is managed differently, and is probably smaller for .NET 4.x than for .NET 3.5, or there are more boilerplate code called in .NET 4.x than in .NET 3.5 which would inflate the stack.
That would explain why both our problems are solved by switching back to .NET 4.x.

 

@r3eckon If you send the project that crashes as a bug report, it'll probably be fixed. If you don't, there's a good chance it won't.

 

Here is a bug reports about crashes in .NET 4.x and not in .NET 3.5:
https://forum.unity.com/threads/uni...ne-only-with-net-4-x-100-reproducible.613483/

Unity has indicated this bug as a duplicate of this one:
https://issuetracker.unity3d.com/issues/unity-editor-crashes-after-using-circular-dependency

But I'm not sure it's a duplicate, because the first one has no recursive algorithm or circular dependency. Also, it occurs not in the C# code, but in Unity's code, there's a scene change and the next frame it crashes, without any C# code executing.


These bug reports are already a few months old, I hope Unity will fix them soon enough.

 

Same problem here. I am trying to change scenes in Editor from a blank/empty to my Main Game one and it crashes, no logs. I have completely removed all Unity from the system including cleaning out the registry and doing a fresh install.

 

This bug is the same as 1137077.
It has been fixed in Unity 2019.3.0a7, I've checked myself.