Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question An Agent which just sends visual observations (a CameraSensor component) to my custom trainer?

Discussion in 'ML-Agents' started by unity_95E2298061EFD837527E, Jul 2, 2023.

  1. unity_95E2298061EFD837527E

    unity_95E2298061EFD837527E

    Joined:
    Feb 10, 2023
    Posts:
    11
    I have a MainCamera object in my Unity simulation, which has a bird's eye view overlooking my Unity setup. I want to have a stationary ML-Agents agent object with the CameraSensor component whose only job is to send captured frames when my simulation runs to a custom trainer plugin that I have implemented in my ml-agents. How do I set up such an agent object in my Unity? Do I simply add a Camera Component to the Main Camera and attach the required script to it? Kindly help me out with this.
     
  2. yuvalstn-ception

    yuvalstn-ception

    Joined:
    Mar 2, 2023
    Posts:
    2
    I faced similar issues, if I'm not mistaken the cameraSensor component must be attached to the agent your using, and in the the cameraSensor component there is a field where you can insert camera to be used.
    I would love to hear about how you implemented your own custom trainer I'm having a hard time adding my own implementations of classic RL algorithms such as DDPG, and TD3.
     
  3. unity_95E2298061EFD837527E

    unity_95E2298061EFD837527E

    Joined:
    Feb 10, 2023
    Posts:
    11
    Hey there! Thanks a lot for the information! In my case, the custom trainer is not an RL algorithm but something completely unrelated (PredNet architecture, if you've heard of it). So, I'm not sure if my experience will be of any help.:( Additionally, still working on making it work without any errors. This is the tutorial I followed though: https://unity-technologies.github.io/ml-agents/Tutorial-Custom-Trainer-Plugin/. I hope this helps!