Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Implementing custom code for model training in ML-Agents package

Discussion in 'ML-Agents' started by unity_95E2298061EFD837527E, May 18, 2023.

  1. unity_95E2298061EFD837527E

    unity_95E2298061EFD837527E

    Joined:
    Feb 10, 2023
    Posts:
    11
    I have a model named PredNet, which is a deep convolutional recurrent neural network, and I wish to actually use the functionalities of the CameraSensor agent from ML-Agents to train this model of mine. I know that the ML-Agents package is solely for using RL algorithms, however, I was curious whether it provides any functionality to add your own custom training algorithm or model to train your Unity simulation on. If not, are there any other Unity packages that might help me with this task? Currently, I have the model (PredNet) fully coded in Python too, so additionally, is there any API/package by Unity to run external, Python-based models/scripts?
     
  2. Luke-Houlihan

    Luke-Houlihan

    Joined:
    Jun 26, 2007
    Posts:
    303
    The ml agents package is primarily targeting reinforcement learning tasks as you've identified and would probably not be suited for what you're trying to do. It is absolutely possible to use a custom algorithm with ml agents but I think you would be swimming against the current.

    Have you checked out unity's perception package? https://github.com/Unity-Technologies/com.unity.perception

    I feel like that would give you a better starting point to train from unity output without having to deal with RL by default. The marketing for unity perception makes it look like just a segmentation tool but it has alot more capability than just that. Here's an example of using it for robotic pose estimation - https://github.com/Unity-Technologies/Robotics-Object-Pose-Estimation
     
  3. unity_95E2298061EFD837527E

    unity_95E2298061EFD837527E

    Joined:
    Feb 10, 2023
    Posts:
    11
    Thanks a lot for the suggestion! While it does seem really promising, in my project I don't have a lot of disk space to work with, so I was hoping to use a more "online" training approach, in which I send the data from my running Unity simulation to my model, use it for training simultaneously, and then get the output and basically discard the data.