I need help with replacing tenserflow lib with unity ml-agents for hand writing digit recognition

I have a quite big problem with my project. I make a learning app for kids. That app has a hand-write-digit recognition. To implement that I found a project on github with tenser-flow plugin. That project was working just fine out-the-box. I've made a complete application. Just before release I notice that I've forgot to switch to IL2CPP scripting backend. After switching tenserflow recognition stoped worked. I want to make recognition work on android and ios. I don't think it's possible to fix tenserflow but switching to unity ml agents may work.

So I have this method with texture2d as input and a number as output. Could you help me to write the same stuff with unity ml-agents?

Code (CSharp):

1. // Classify an MNIST image by running the model
2.     public int Evaluate (Texture2D input) {  
3.         // Get raw pixel values from texture, format for inputImg array
4.         for (int i = 0; i < img_width; i++) {
5.             for (int j = 0; j < img_height; j++) {
6.                 inputImg [0, img_width - i - 1, j, 0] = input.GetPixel(j, i).r;
7.             }
8.         }
9.  
10.         // Apply texture to displayMaterial
11.         displayMaterial.mainTexture = input;
12.  
13.         // Create the TensorFlow model
14.         graph.Import (graphModel.bytes);
15.         var session = new TFSession (graph);
16.         var runner = session.GetRunner ();
17.  
18.         // Set up the input tensor and input
19.         runner.AddInput (graph ["conv2d_1_input"] [0], inputImg);
20.         // Set up the output tensor
21.         runner.Fetch (graph ["dense_2/Softmax"] [0]);
22.  
23.         // Run the model
24.         float[,] recurrent_tensor = runner.Run () [0].GetValue () as float[,];
25.  
26.         // Find the answer the model is most confident in
27.         float highest_val = 0;
28.         int highest_ind = -1;
29.         float sum = 0;
30.         float currTime = Time.time;
31.  
32.         for (int j = 0; j < 10; j++) {
33.             float confidence = recurrent_tensor [0, j];
34.             if (highest_ind > -1) {
35.                 if (recurrent_tensor [0, j] > highest_val) {
36.                     highest_val = confidence;
37.                     highest_ind = j;
38.                 }
39.             } else {
40.                 highest_val = confidence;
41.                 highest_ind = j;
42.             }
43.  
44.             // sum should total 1 in the end
45.             sum += confidence;
46.         }
47.  
48.             //Display the answer to the screen
49.             label.text = "Answer: " + highest_ind + "\n Confidence: " + highest_val +
50.                 "\nLatency: " + (Time.time - currTime) * 1000000 + " us";
51.  
52.         return highest_ind;  
53.     }

 

hi @unity-huunity,
Have you tried using the Barracuda package directly? It allows you to import models from tensorflow and run inference on them in the unity runtime

 

thank you, I'll try it