Question Visual Observations in Python

Hi all!
beginner user of unity here. I was wondering if anyone can offer some examples/help as to how to get visual observations from a camera in the unity env to python. Currently I have vector observations implemented via CollectObservations, but I would also like to send visual observations so I can use those in the gym wrapper over my unity environment.

I have added the camerasensor component to my first person agent camera on the unity editor, but not sure what the next steps are. Seems like I need to implement the ISensor interface, but not sure where/how to do this so that the visual observations show up in python.


Thanks!

 

There are four example environments with visual observations, so probably a good place to start.

 

are they all using python to get the visual observations to run ML on? if so can you please link these examples? sorry I am having trouble finding them. Thanks!

 

So to clarify .... you want to train with your custom python training(or something like Stable Baslines, Rllib etc.), not using the ml-agents training (PPO/SAC)?
You were asking about implementation in Unity and I don't think this will change regardless of the way how you later train the environment created in Unity. In the latest releases you basically attach a camera, the visual observations component and that's pretty much it, unless you're trying to do some custom visual observations.
The examples are in the 'Project' folder in the github repo.

 

yeah, I am trying to write my own NN and then doing custom python training. Turns out my issue was that I was using the wrong version of mlagents(was using v0.16.1) instead of v0.17 which made it so that my visual observations didn't show up in python. Attaching a camera sensor to my agent works now, thanks for all your help!