Headless Linux Player Freeze on MonoManager ReloadAssembly

We're operating a game in open beta which makes use of dedicated headless servers on Ubuntu 16.04 hosts. We have a C++ application which manages headless server player instances on the particular host. Each headless player starts up, hosts a match, then shuts down, after which the manager launches a new instance. This all works as expected in our test environments, and normally works fine in production as well. However, sometimes we will hit a certain point where the new headless players freeze on startup, with the last line in the player log being "Begin MonoManager ReloadAssembly". Presumably the reload deadlocks for some unknown reason. This eventually leaves us with no player instances on that host. The manager will kill unresponsive server players but the problem persists in subsequent server player launches until the manager application itself is manually restarted, after which things begin working normally again. It does seem as if this is triggered when all server players are in use, but I'm not 100% sure this is related or not.

We have no idea what is triggering this behavior, or why it suddenly becomes persistent. It doesn't seem to be caching, as a simple restart fixes it without clearing the Unity cache files. Our players are built with Unity 2017.3.1p1 on a Windows build host. Any ideas on what might result in headless players deadlocking on this step repeatedly?

 

I suspect that something else is going on, and it just happens to be that ReloadAssembly is the last thing logged before that.
Is it possible that the manager application is holding file/process/socket handles after the related player instances have exited, eventually leaving the system with none available? Does changing ulimit values change the way the problem presents?

 

This is a long-shot, but did you ever figure out this issue? I'm facing the same problem (with a very similar Windows setup).