Agents act weird if set active by event

I have a ton of ml agents Walkers in a scene and want them to start at different times depending on where my player is. However they only function correctly if they are active when the scene starts. If I start them later they don't perform well. Any idea how to solve this issue?

I tried starting them all on scene start but it takes too big a hit on performance.

 

Another way of putting this is: How can I dynamically spawn new agents during inference?

 

I don't know how you are spawning agents now, but you can make prefabs with the GameObject that has the agent script attached and preload it with the relevant brain. Spawning from prefabs should not be an issue. It seems to work fine in my project.

Maybe there is dependency on some external script that causes these spawning errors?

I can't say much more without knowing more about the implementation unfortunately.