Why focus on reinforcement learning?

Question about ML Agents:

Why is Unity AI team so focused on reinforcement learning? Why not just give us a good way of running inference of a Tensorflow graph from C# (I know there is TensorflowSharp, but it's wacky to set up and looks like it's maintained by just a few people in their spare time, so no production support.)

There are so many uses of neural networks outside of reinforcement learning. It seems like ML-agents project is just following the reinforcement learning hype.

Imagine if it were easy to run custom neural networks in C# we could do so much more stuff that has never been done in indie games before.

For example:

• Procedurally generate levels with LSTMs https://arxiv.org/pdf/1603.00930.pdf
• Make a bit smarter character in the game, that uses a neural network to generate chat output, instead of a linear pre-written dialog.
• Make a custom image classifier for an Augmented reality game.
• Make characters with acceptable-sounding voice that can say anything you want. AFAIK never been done in a game. https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html

These are just a few examples but there are many more potential use cases of NNs in games.


Sorry if this question seems too combative, i still love Unity engine <3

 

Not really answer to your actual question but wanted to point out that ML Agents IS using TensorflowSharp already.

 

Yes I know. But the feature is experimental and it looks like not much progress has been made on it since the first release of ml-agents over a year ago. If you look at the TensorflowSharp github page, you can see it's mostly maintained by 1 person from Microsoft. It's good enough for prototypes, but we can't rely on it to be stable in production on all Unity supported devices.

The whole ml-agents project is kind of made for reinforcement learning.(agents, environment, brain, etc..). And in my opinion, currently we could get more benefit just from normal neural networks.

Also, I don't understand why devs spend effort in adding specific algorithms like immitation learning and curriculum learning. They sound cool, but they're really not that useful yet IMO. If we had proper support for running tensorflow graphs we could run any model we wanted and wouldn't need Unity specific implementations of RL algorithms.

 

I think everyone probably comes at what is a "normal" neural network differently.. My self.. I had been watching the progress of deep learning and NN's intently for the last couple of years. I recently ended up with a large degree of available time so I though to myself that I should try to actually learn what NN's are about. I also love Unity3d, so I figured why not learn both of these things more deeply.

I really like the ml-agents project. It does the things that i find useful.. I would dig more features of course, but who wouldn't? I would like it to be out of alpha/beta.. but who wouldn't? Maybe ill find something that I can not use the toolkit to do that I need and that a NN should be able to do.. Right now. that's not the case.

 

You might take a look at Microsoft Cognitive Toolkit. We use it in some of our procedural generation. Supported C# api and runs in Unity just fine. Unity missed the mark entirely on their ML tools, who knows what lead to their decision making on it all.

 

Hi @mookeedev , welcome to this forum. I think Unity is optimising the engine first improving compilation time and working for more massive games using ECS, improving Terrain, the input system, etc... before jumping to Ai. I'm integrating Ai and is very interesting! Is possible to do all that you have mentioned with the Unity Ai ML-agent beta 0.6 at this point. Give Unity a year and probably the engine will be organized to handle Ai components from inside the editor in a more simple way. Look as for example the new UI or Input system how much time take Unity to make it. Looks like a slow process but is constant. Better constant and good than fast full of bugs. If you start now building what you have mentioned you will follow Unity progress.

 

Because for complex scenarios like a game is much faster?
(in order of magnitude)



from Unity blog: https://blogs.unity3d.com/2017/12/0...urriculum-learning-new-environments-and-more/

Probably Unity team want to have all the Ai script options running well before making the final Ai user interface.

 

Thanks for linking the Microsoft Cognitive Toolkit. I've heard about it some time ago, but back then it din't have C# training support. (Also it used to be called CNTK). Now it looks like a good option.

Looks like Microsoft has many different frameworks/tools for machine learning. There are also Accord.NET and ML.NET.
I'll go look into it more, right now I have to figure out what exactly each one does and then try to integrate the ones I need in Unity.

It seems like Unity ml-agents is primarily made to attract ML researchers to Unity and is not really aimed at existing game developers.

 

Yes I know making a game engine is a lot of work, but there is probably a team inside Unity that makes just AI related stuff and my point was I don't agree with the direction they are going.

Also you linked to a graph showing reward on some task when using curriculum learning. But that is a very narrow use case. It's like implementing Unity support for A* pathfinding before implementing support for general C# scripting. If you have support for C# you can implement A* yourself if you need it, just like you can implement curriculum learning if you have general support for specifying, training and running neural networks.

It's much better to have proper general support, than to show off a few algorithms.

 

Update! Unity actually did it. There is now a custom Inference engine in Unity that is cross-platform, can use the GPU and is included in the ML-Agents by default. Thanks Unity.

https://blogs.unity3d.com/2019/03/0...v0-7-a-leap-towards-cross-platform-inference/