Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on March 30, 2023, between 5 am & 1 pm EST, in the Performance Profiling Dev Blitz Day 2023 - Q&A forum and Discord where you can connect with our teams behind the Memory and CPU Profilers.
    Dismiss Notice

Running a PyTorch (or even a TensorFlow) Model on Unity MLAgent

Discussion in 'ML-Agents' started by janimator, May 22, 2020.

  1. janimator


    Apr 5, 2013
    MLAgents Version: 0.15.1

    I've begun to dive into the unknown territory of trying to get a PyTorch model into an MLAgent within Unity. Looking at whatever forum threads I can find it seems that converting the model into a Barracuda model might do the trick. I was successful in converting PyTorch > Onnx > Barracuda and in the inspector, the model looks like this


    but I am not able to drag the model("brain") into the MLAgent via inspector. If I try to drag the model into the inspector I see the following error log.

    Code (CSharp):
    1. NullReferenceException: Object reference not set to an instance of an object
    2. MLAgents.Inference.BarracudaModelParamLoader.CheckModel (Barracuda.Model model, MLAgents.Policies.BrainParameters brainParameters, MLAgents.Sensors.SensorComponent[] sensorComponents, MLAgents.Policies.BehaviorType behaviorType) (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/
    3. MLAgents.Editor.BehaviorParametersEditor.DisplayFailedModelChecks () (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/
    4. MLAgents.Editor.BehaviorParametersEditor.OnInspectorGUI () (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/
    5. UnityEditor.UIElements.InspectorElement+<>c__DisplayClass55_0.<CreateIMGUIInspectorFromEditor>b__0 () (at <887a438259064e89922419bc7b8810ab>:0)
    6. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
    I've read on another forum something about naming the consts correctly might do the trick but I'm unsure what the steps are for that? <-- mentioned in this thread.

    My questions are is it at all possible to get my PyTorch model to work on an MLAgent? Is it even possible to get a TensorFlow model to work on a MLAgent? What can be done to achieve this? How can I ensure that the "consts" are named correctly?
    Last edited: May 22, 2020
  2. dani_kal


    Mar 25, 2020
    Hi!!!!Do you have find the solution ???
  3. dani_kal


    Mar 25, 2020
    sorryy for the you have found the solution??
  4. mcbauer


    Oct 10, 2015
    Interested in this thread as well.
  5. celion_unity


    Unity Technologies

    Jun 12, 2019
    We've clarified our policy around this a bit here. In short, you're welcome to try using models that you trained outside of ml-agents, but we're not able to provide much support for you if you do.

    In this case, that part of the code is looking for some constants stored in tensors with specific names. Since the original post, the error handling there has been improved a bit.

    The naming conventions for the expected tensors isn't documented, but you can at least find the names in TensorNames.cs
    mcbauer likes this.