Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Resolved Image recognition

Discussion in 'ML-Agents' started by XSquare2, Mar 27, 2021.

  1. XSquare2

    XSquare2

    Joined:
    May 7, 2018
    Posts:
    8
    Hello, I want to use MLAgents to recognize patterns. The input data is 64 pixels. In my project there are several characters for casting spells, the player must draw this character and he will have this spell. I have a lot of questions. In general, is MLAgents suitable for this? How can I teach this? As I develop, I want to add a few more symbols for recognition, what should I do then? (retrain?) What to do in such cases when the pattern is recognized from the wrong pattern? Thanks and sorry for my English
     
  2. mbaske

    mbaske

    Joined:
    Dec 31, 2017
    Posts:
    473
    You can train your agent with visual observations, using the camera sensor.
    https://github.com/Unity-Technologi...ironment-Design-Agents.md#visual-observations

    I think what should work for recognizing symbols, is to use a disrete action space, where each action is associated with a specific symbol. So during training, you would present the agent with random symbols and reward it for choosing the matching action value.
    I haven't tried it, but my guess is you might be able to add more symbols by resuming a training run, without necessarily having to retrain from scratch. Provided that your agent's initial action space is large enough for additional choices.

    If you need more sophisticated image classification, possibly using a premade model, there's the option to use Barracuda inference outside of ML-Agents. Would be interesting to see a mash-up of the two: perhaps running YOLO in Barracuda and feed the outputs to ML-Agents?

    https://docs.unity3d.com/Packages/com.unity.barracuda@1.0/manual/SupportedArchitectures.html
    https://github.com/onnx/models#image_classification

     
    Last edited: Mar 27, 2021
  3. XSquare2

    XSquare2

    Joined:
    May 7, 2018
    Posts:
    8
    мbaske, thank you very much!