Unity-compatible Neural Network libraries and/or frameworks?

Heya,

Anyone have any experience with getting some solid neural network library run nicely within Unity?

I've been going through a few options but none really seem to work out the box.

What I've tried so far:

Keras.NET, https://github.com/SciSharp/Keras.NET - this seems to be a relatively thin wrapper and depends on Python runtimes, etc, and I just couldn't get it set up properly. After setting up the correct Python version, installing NumPy, etc, it kept sometimes working and sometimes crashing. Seemed that every 2nd run would crash it when it entered the Python runtime.

KerasSharp, https://github.com/tcmxx/keras-sharp - kept getting some "This Tensor does not have a graph and is value only" error on trying to propagate the network after training, couldn't figure out how to fix that or how to work around it. Also this isn't up to date with Keras so that's always a minus.

ConvNetSharp, https://github.com/cbovar/ConvNetSharp - this worked pretty well out the box, but unfortunately it does not have transposed convolution nor unpooling so learnable upsampling is not possible, and that's something I'd really need. It could possibly be something I might be able to add myself, but I'm a bit hesitant with that as I'm not super experienced with convolutional neural networks.

So any suggestions and anecdotes are happily welcomed.

 

I've followed the Unity ML stuff, but haven't tried it as I've thought it's in very early stage of development. It's my understanding that Barracuda doesn't do learning yet and needs to be provided pre-trained TensorFlow models. I'd like to do the training in Unity though. Doing it outside Unity would principally be possible but represent a quite large extra hurdle.

I've quite a few projects I'd like to experiment with in regards of deep neural networks but the main one at the moment is teaching a strategy game AI to mimic human behavior and human strategies. This is mostly a learning project - e.g. I'm not aiming for any commercial uses - so I'm a bit more limited in how much time I've to set up additional development environments and so forth.

 

After some (failed) forays into TorchSharp, I tried Keras.NET again as it really seems like the most promising choice, plus Keras is pretty nice for us novice machine learners.

Sometimes the code I have runs once and produces legit output, but it crashes every time by the 2nd time it's ran (sometimes already on the first time).

All the code needed for the crash is:

Code (CSharp):

1. using Numpy;
2. using UnityEngine;
3.  
4. public class NeuralNetworkTest : MonoBehaviour
5. {
6.     void Start()
7.     {
8.         NDarray x = np.array(new float[,] { { 0, 0 }, { 0, 1 }, { 1, 0 }, { 1, 1 } });
9.     }
10. }

In case someone's interested, here's Editor.log: https://hastebin.com/raw/ilewiqofak and error.log: https://hastebin.com/raw/amuricayen

This appears to be some kind of a problem in how Unity reloads the libraries and how Numpy libraries are not set up to elegantly survive that experience. This seems related: https://github.com/pythonnet/pythonnet/issues/762

Unfortunately far as I can tell, no good workarounds have been described thus far. I suppose that my next attempt is to look into how I could save training snapshots from the game state inside Unity and use those in Keras that is ran separately and then import a TensorFlow model from Keras and run that in Unity. Not the ideal approach but seems like the only one at the moment.

 

@jaltsu,

Just wondering if you got any of these frameworks set up in a stable manner? been having the Python DLL issue, with keras.net . Tried with both the local Python installation and the Pythonincluded one.

And also how are you saving Keras models from In game states?

 

I've had fairly decent results with a combination of Tensorflow and Tensorflow.net (SciSharp) up to now. Basic networks are working, convolution is working. No success with deconvolution up to now. (Deconvolution is not implemented, the same seems to be the case with image resize, a manual nearest neighbour resize using concat runs into a NullPointer exception.) If it works out further I'll make a topic with some pointers and results.