ML-Agents Server Setup?

Good day, as everyone is aware, once you start training your models your computer becomes sufficiently utilized, therefore, is it possible to set up a centralized server so workstations can take advantage of them?

If so, is there any documentation on this?

Thank you

 

Something like: https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-on-Amazon-Web-Service.md

 

Yes, pretty much! thank you kindly.

It seems that the "exe" and TensorFlow need to be on the same device.

If I wanted to run the EXE locally but have a service backend take the majority of the resource utilization, is that possible or even practical?

 

Sorry, I don't know the answer to that. Conceptually I think there is no issue here, as long as the latency between client and server is fast enough. I don't see any obvious way of doing this though within Unity. Sounds like a good feature request.

 

Hey thanks a lot for your input.. I couldn't find anything obvious myself. Maybe a code change somewhere im sure.

Cheers!

 

We don't currently have any solutions for training across multiple machines. There's a description of some upcoming work on our 1.0 blog post: https://blogs.unity3d.com/2020/05/12/announcing-ml-agents-unity-package-v1-0/ (see the "ML-Agents Cloud" section).

 

Thanks for the reply.

What about running the ML learn EXE continually even after the Unity engine stops...

 

I suppose you can always use something like 'netcat' to forward traffic from one machine to the other... I assume the connection is tcp based...

I suppose you can always use something like 'netcat' to forward traffic from one machine to the other... I assume the connection is tcp based...