Question SAC training locks-up Unity PPO is fine

I'm on Win, Unity 2020.3.1f1 LTS, ML-Agents 2.0.0.1 exp, mlagents 0.26. PPO trains my project OK, but it seems slow and the results are modest. SAC looks more promising, but Unity freezes up after a few minutes when SAC-training. I changed about half of the settings in the yaml file, none of them fix the problem. Two settings make the lock-up happen faster: a) "time-horizon", if it's <2048 and b) "threaded", if it's set to "false". Setting the "time_horizon" to values >4096 makes no difference, Unity still locks up after a few minutes. It looks like in Unity some "critical" value is reached when SAC-training. If I knew which value it is, I'd print it out in the Debug. Anything else I can try, before I try other versions of Unity / mlagents? yaml file below.

Code (csharp):

1. behaviors:
2.   SBehavior:
3.     trainer_type: sac
4.     hyperparameters:
5.       batch_size: 2048
6.       buffer_size: 40960
7.       buffer_init_steps: 100
8.       init_entcoef: 0.7
9.       save_replay_buffer: false
10.       learning_rate: 0.0001
11.       learning_rate_schedule: constant
12.       tau: 0.005
13.       steps_per_update: 1
14.     network_settings:
15.       normalize: false
16.       hidden_units: 32
17.       num_layers: 2
18.       vis_encode_type: simple
19.       goal_conditioning_type: none
20.     reward_signals:
21.       extrinsic:
22.         gamma: 0.9
23.         strength: 1.0
24.         network_settings:
25.           normalize: false
26.           hidden_units: 16
27.           num_layers: 2
28.           vis_encode_type: simple
29.           goal_conditioning_type: none
30.       curiosity:
31.         gamma: 0.9
32.         strength: 0.01
33.         learning_rate: 0.0001
34.         network_settings:
35.           normalize: false
36.           hidden_units: 16
37.           num_layers: 2
38.           vis_encode_type: simple
39.           goal_conditioning_type: none
40.     threaded: true
41.     checkpoint_interval: 50000
42.     keep_checkpoints: 5
43.     max_steps: 1000000
44.     time_horizon: 4096
45.     summary_freq: 49152
46.  

 

I have experienced this issue with SAC before. Could you check your RAM when your Unity freezes?

 

Thanks, but it's not the RAM, ~7GB out 32 GB, no spikes. CPU usage is also modest ~20-30%

 

It remains stuck. As I wrote in my OP, setting "threaded" to false makes the lock-up happen a bit faster.

 

I wonder: could it be some "comm sync" problem? That is, Unity is producing more info that can be conveyed via the comm API? Does Unity and mlagents have to match some common time scale, frame rate or something like that? Is there a way to debug the communication? I tested the demo scenes (Basic, 3DBall) with SAC, they work fine. I tried training my agent with a copy of the Basic.yaml (only renaming the behavior), but Unity freezes with a few seconds.

 

I used Wireshark to monitor the Python-Unity communication and gain more insight. It's Unity that freezes, not Python. No abnormal or higher traffic just before the lock-up.

I started with a simple test: SAC-training the sample "Basic" scene, then deselecting "Play" in Unity:



I can see the mlagents buffers



Now I try SAC-training the project that freezes up. Initial handshakes look good.



I see ~3330 headers (higher than "Basic") on average. It stays that way until the lock-up.



Unity ran for 84 sec before the lock-up





Is there a way to analyze a Unity project with a VS debugger at runtime and try pinpoint the problem?

 

I managed to attach the MS VS debugger to Unity while it freezes. It looks like Unity is involved in raycasting when it gets stuck.

At the bottom of the function stack I see:
PhysicsScene2D_CUSTOM_Raycast_Internal_Injected

In another view I see:
b2DynamicTree::RayCast<b2WorldRayCastWrapper>
HandleOutOfBufferToReadFrom

Assuming it's coming from my code (not Unity's packages), is there a way to set breakpoints in my C# scripts in Unity, and have them analyzed in MS VS debugger to resolve this?