Visual Studio and Unity interlocked when breakpoint is hit in managed code

So this happened to me recently. I'm working on a native low-level rendering plugin which has a bunch of C# wrappers. When I set a breakpoint in C#, and attach the VS debugger to Unity, both VS and Unity lock up, with Unity unresponsible while VS is somewhat alive until you forcibly kill Unity.

It wasn't like that before. So I spent some time and did some tests. What I found is if I pass a handle of C# callback to my native plugin(which is a marshalled IntPtr), and the native plugin calls this callback on different threads other the main thread and the renderer thread, no matter the call is made thread-safe or not, it will cause this interlock behaviour. And I've created a simple test based on the official native plugin sample to prove my hypothesis: https://bitbucket.org/feng_ye2001/nativerenderplugincrashdemo

And I also found there should be some complications between the Mono runtime and the debugger: http://www.mono-project.com/docs/advanced/runtime/docs/soft-debugger/

FYI MonoDevelop on Mac works fine. Windows MonoDevelop and Visual Studio both have this issue. Tested from Unity 5.4 to Unity 2017.1.1.

Is there anyone who also experienced this?