Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Question Visual Observations in Python

Discussion in 'ML-Agents' started by mitochondriaCNN, Jun 19, 2020.

  1. mitochondriaCNN

    mitochondriaCNN

    Joined:
    Jun 19, 2020
    Posts:
    3
    Hi all!
    beginner user of unity here. I was wondering if anyone can offer some examples/help as to how to get visual observations from a camera in the unity env to python. Currently I have vector observations implemented via CollectObservations, but I would also like to send visual observations so I can use those in the gym wrapper over my unity environment.

    I have added the camerasensor component to my first person agent camera on the unity editor, but not sure what the next steps are. Seems like I need to implement the ISensor interface, but not sure where/how to do this so that the visual observations show up in python.


    Thanks!
     
  2. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    81
    There are four example environments with visual observations, so probably a good place to start.
     
  3. mitochondriaCNN

    mitochondriaCNN

    Joined:
    Jun 19, 2020
    Posts:
    3
    are they all using python to get the visual observations to run ML on? if so can you please link these examples? sorry I am having trouble finding them. Thanks!
     
  4. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    81
    So to clarify .... you want to train with your custom python training(or something like Stable Baslines, Rllib etc.), not using the ml-agents training (PPO/SAC)?
    You were asking about implementation in Unity and I don't think this will change regardless of the way how you later train the environment created in Unity. In the latest releases you basically attach a camera, the visual observations component and that's pretty much it, unless you're trying to do some custom visual observations.
    The examples are in the 'Project' folder in the github repo.
     
  5. mitochondriaCNN

    mitochondriaCNN

    Joined:
    Jun 19, 2020
    Posts:
    3
    yeah, I am trying to write my own NN and then doing custom python training. Turns out my issue was that I was using the wrong version of mlagents(was using v0.16.1) instead of v0.17 which made it so that my visual observations didn't show up in python. Attaching a camera sensor to my agent works now, thanks for all your help!