Question Using an ONNX Model Trained on GameObjects with DOTS ECS?

Hello everyone,

I have been experimenting with Unity's ML-Agents and have trained a model using regular GameObjects that I've exported to an ONNX file. I'm now interested in transferring this ONNX file to a DOTS ECS version of Unity.

In theory, I understand that the ONNX model should be agnostic to the environment it was trained in, as it simply encapsulates the trained neural network architecture and weights. My model is trained to control joint angles and torques of the GameObjects.

However, I'm unsure of how I could apply the output of the model (specifically joint angles and torques) to Entities in an ECS setup. I understand that this might involve writing custom code to take the model output and apply it to the Entities.

Has anyone attempted this before or have any guidance on how this could be done? Are there any resources or updates that might be helpful for this task?

Any advice or insights would be greatly appreciated.

Thanks!

 

I havent tried it out but doesnt sounds like a big deal: just convert the OnactionReceived to entities and feed them to your system and the system output you feed to your gameobject. off course best thing is to have everything in DOTS.

IMO just rewrite your training env to ECS too and it should be just the same as with every other hybrid approach

 

I am a bit intimidated by having to fill out the observations scripts and building my own support for imitation learning in DOTS ML-Agents. I have managed to do well enough learning ML-Agents but the ECS version is incomplete. If you have any other suggestions or pointers, please let me know.

 

Thank you very much for replying by the way. It's greatly appreciated.