Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Running a PyTorch (or even a TensorFlow) Model on Unity MLAgent

Discussion in 'ML-Agents' started by janimator, May 22, 2020.

  1. janimator

    janimator

    Joined:
    Apr 5, 2013
    Posts:
    52
    MLAgents Version: 0.15.1


    I've begun to dive into the unknown territory of trying to get a PyTorch model into an MLAgent within Unity. Looking at whatever forum threads I can find it seems that converting the model into a Barracuda model might do the trick. I was successful in converting PyTorch > Onnx > Barracuda and in the inspector, the model looks like this

    upload_2020-5-21_22-9-14.png

    but I am not able to drag the model("brain") into the MLAgent via inspector. If I try to drag the model into the inspector I see the following error log.

    Code (CSharp):
    1. NullReferenceException: Object reference not set to an instance of an object
    2. MLAgents.Inference.BarracudaModelParamLoader.CheckModel (Barracuda.Model model, MLAgents.Policies.BrainParameters brainParameters, MLAgents.Sensors.SensorComponent[] sensorComponents, MLAgents.Policies.BehaviorType behaviorType) (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/com.unity.ml-agents/Runtime/Inference/BarracudaModelParamLoader.cs:149)
    3. MLAgents.Editor.BehaviorParametersEditor.DisplayFailedModelChecks () (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/com.unity.ml-agents/Editor/BehaviorParametersEditor.cs:108)
    4. MLAgents.Editor.BehaviorParametersEditor.OnInspectorGUI () (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/com.unity.ml-agents/Editor/BehaviorParametersEditor.cs:68)
    5. UnityEditor.UIElements.InspectorElement+<>c__DisplayClass55_0.<CreateIMGUIInspectorFromEditor>b__0 () (at <887a438259064e89922419bc7b8810ab>:0)
    6. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
    I've read on another forum something about naming the consts correctly might do the trick but I'm unsure what the steps are for that? https://github.com/Unity-Technologies/ml-agents/issues/3398 <-- mentioned in this thread.

    My questions are is it at all possible to get my PyTorch model to work on an MLAgent? Is it even possible to get a TensorFlow model to work on a MLAgent? What can be done to achieve this? How can I ensure that the "consts" are named correctly?
     
    Last edited: May 22, 2020
  2. dani_kal

    dani_kal

    Joined:
    Mar 25, 2020
    Posts:
    52
    Hi!!!!Do you have find the solution ???
     
  3. dani_kal

    dani_kal

    Joined:
    Mar 25, 2020
    Posts:
    52
    sorryy for the mistake......do you have found the solution??
     
  4. mcbauer

    mcbauer

    Joined:
    Oct 10, 2015
    Posts:
    525
    Interested in this thread as well.
     
  5. celion_unity

    celion_unity

    Joined:
    Jun 12, 2019
    Posts:
    289
    We've clarified our policy around this a bit here. In short, you're welcome to try using models that you trained outside of ml-agents, but we're not able to provide much support for you if you do.

    In this case, that part of the code is looking for some constants stored in tensors with specific names. Since the original post, the error handling there has been improved a bit.

    The naming conventions for the expected tensors isn't documented, but you can at least find the names in TensorNames.cs
     
    mcbauer likes this.