Search Unity

Why focus on reinforcement learning?

Discussion in 'AI & Navigation Previews' started by MookeeDev, Dec 14, 2018.

  1. MookeeDev

    MookeeDev

    Joined:
    Aug 2, 2018
    Posts:
    9
    Question about ML Agents:

    Why is Unity AI team so focused on reinforcement learning? Why not just give us a good way of running inference of a Tensorflow graph from C# (I know there is TensorflowSharp, but it's wacky to set up and looks like it's maintained by just a few people in their spare time, so no production support.)

    There are so many uses of neural networks outside of reinforcement learning. It seems like ML-agents project is just following the reinforcement learning hype.

    Imagine if it were easy to run custom neural networks in C# we could do so much more stuff that has never been done in indie games before.

    For example:

    These are just a few examples but there are many more potential use cases of NNs in games.


    Sorry if this question seems too combative, i still love Unity engine <3
     
    Tronyc likes this.
  2. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,223
    Not really answer to your actual question but wanted to point out that ML Agents IS using TensorflowSharp already.
     
  3. MookeeDev

    MookeeDev

    Joined:
    Aug 2, 2018
    Posts:
    9
    Yes I know. But the feature is experimental and it looks like not much progress has been made on it since the first release of ml-agents over a year ago. If you look at the TensorflowSharp github page, you can see it's mostly maintained by 1 person from Microsoft. It's good enough for prototypes, but we can't rely on it to be stable in production on all Unity supported devices.

    The whole ml-agents project is kind of made for reinforcement learning.(agents, environment, brain, etc..). And in my opinion, currently we could get more benefit just from normal neural networks.

    Also, I don't understand why devs spend effort in adding specific algorithms like immitation learning and curriculum learning. They sound cool, but they're really not that useful yet IMO. If we had proper support for running tensorflow graphs we could run any model we wanted and wouldn't need Unity specific implementations of RL algorithms.
     
    Last edited: Dec 15, 2018
  4. TashaSkyUp

    TashaSkyUp

    Joined:
    Dec 29, 2016
    Posts:
    6
    I think everyone probably comes at what is a "normal" neural network differently.. My self.. I had been watching the progress of deep learning and NN's intently for the last couple of years. I recently ended up with a large degree of available time so I though to myself that I should try to actually learn what NN's are about. I also love Unity3d, so I figured why not learn both of these things more deeply.

    I really like the ml-agents project. It does the things that i find useful.. I would dig more features of course, but who wouldn't? I would like it to be out of alpha/beta.. but who wouldn't? Maybe ill find something that I can not use the toolkit to do that I need and that a NN should be able to do.. Right now. that's not the case.
     
  5. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    3,315
    You might take a look at Microsoft Cognitive Toolkit. We use it in some of our procedural generation. Supported C# api and runs in Unity just fine. Unity missed the mark entirely on their ML tools, who knows what lead to their decision making on it all.
     
    MookeeDev likes this.
  6. AlanMattano

    AlanMattano

    Joined:
    Aug 22, 2013
    Posts:
    1,441
    Hi @mookeedev , welcome to this forum. I think Unity is optimising the engine first improving compilation time and working for more massive games using ECS, improving Terrain, the input system, etc... before jumping to Ai. I'm integrating Ai and is very interesting! Is possible to do all that you have mentioned with the Unity Ai ML-agent beta 0.6 at this point. Give Unity a year and probably the engine will be organized to handle Ai components from inside the editor in a more simple way. Look as for example the new UI or Input system how much time take Unity to make it. Looks like a slow process but is constant. Better constant and good than fast full of bugs. If you start now building what you have mentioned you will follow Unity progress.
     
  7. AlanMattano

    AlanMattano

    Joined:
    Aug 22, 2013
    Posts:
    1,441
    Because for complex scenarios like a game is much faster?
    (in order of magnitude)



    from Unity blog: https://blogs.unity3d.com/2017/12/0...urriculum-learning-new-environments-and-more/

    Probably Unity team want to have all the Ai script options running well before making the final Ai user interface.
     
    Last edited: Jan 23, 2019
  8. MookeeDev

    MookeeDev

    Joined:
    Aug 2, 2018
    Posts:
    9
    Thanks for linking the Microsoft Cognitive Toolkit. I've heard about it some time ago, but back then it din't have C# training support. (Also it used to be called CNTK). Now it looks like a good option.

    Looks like Microsoft has many different frameworks/tools for machine learning. There are also Accord.NET and ML.NET.
    I'll go look into it more, right now I have to figure out what exactly each one does and then try to integrate the ones I need in Unity. :)

    It seems like Unity ml-agents is primarily made to attract ML researchers to Unity and is not really aimed at existing game developers.
     
    Last edited: Jan 25, 2019
  9. MookeeDev

    MookeeDev

    Joined:
    Aug 2, 2018
    Posts:
    9
    Yes I know making a game engine is a lot of work, but there is probably a team inside Unity that makes just AI related stuff and my point was I don't agree with the direction they are going.

    Also you linked to a graph showing reward on some task when using curriculum learning. But that is a very narrow use case. It's like implementing Unity support for A* pathfinding before implementing support for general C# scripting. If you have support for C# you can implement A* yourself if you need it, just like you can implement curriculum learning if you have general support for specifying, training and running neural networks.

    It's much better to have proper general support, than to show off a few algorithms.
     
  10. MookeeDev

    MookeeDev

    Joined:
    Aug 2, 2018
    Posts:
    9
    Kellin_B and tbriley like this.
unityunity