Search Unity

  1. Looking for a job or to hire someone for a project? Check out the re-opened job forums.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Bug Conv2D errors when predicting

Discussion in 'Barracuda' started by hnphan, Jan 9, 2021.

  1. hnphan

    hnphan

    Joined:
    Nov 5, 2014
    Posts:
    7
    Hi!
    I've been able to run a model in Unity using Barracuda but as soon as I add a Conv2D type layer, inference will give me errors right away.
    • When Conv2D is added, I also have the input shape from (1x1x1x2) to (1x2x1x1)
    • I had to do a reshape for the output in Unity from (1x28x1x28) to (1x28x28x1) although the correct shape in python is 1x28x28x1 regardless of the usage of Conv2D or not.
    I validated the ONNX file by reloading it in Python and executed it and it works fine without issue (no need to reshape or anything) thats why I think there's a bug in Barracuda and not the ONNX file.

    I've included a toy decoder with MNIST (28x28x1) that I used
    • File that works: (NoConv2D_decoder.onnx)
    • File that has Conv2D and bugs out: (decoder.onnx)
    The decoder model that works in Python:
    Code (CSharp):
    1. latent_inputs = keras.Input(shape=(latent_dim,))
    2. x = latent_inputs
    3. actType = 'relu'#'tanh'
    4. for i in range(3):
    5.     x = layers.Dense(128)(x)
    6.     x = layers.Activation(actType)(x)
    7. x = layers.Dense(28 * 28 * chann)(x)
    8. x = layers.Reshape((28, 28, chann))(x)
    9. #x = layers.Conv2D(chann, 1, padding="same")(x)
    10. decoder_outputs = layers.Activation('sigmoid', name="outLayer")(x)
    The error message in Unity when Conv2D us used
    AssertionException: Assertion failure. Values are not equal.
    Expected: 1 == 28
    UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message, System.Collections.Generic.IEqualityComparer`1[T] comparer) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    UnityEngine.Assertions.Assert.AreEqual (System.Int32 expected, System.Int32 actual) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    Unity.Barracuda.PrecompiledComputeOps.Conv2D (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor K, Unity.Barracuda.Tensor B, System.Int32[] stride, System.Int32[] pad, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/BarracudaPrecompiledCompute.cs:648)
    Unity.Barracuda.StatsOps.Unity.Barracuda.IOps.Conv2D (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor K, Unity.Barracuda.Tensor B, System.Int32[] stride, System.Int32[] pad, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/StatsOps.cs:92)
    Unity.Barracuda.GenericWorker+<StartManualSchedule>d__30.MoveNext () (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/GenericWorker.cs:247)
    Unity.Barracuda.GenericWorker.Execute () (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/GenericWorker.cs:143)
    Unity.Barracuda.GenericWorker.Execute (Unity.Barracuda.Tensor input) (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/GenericWorker.cs:135)
    inferCNN.DisplayDebug () (at Assets/Scripts/inferCNN.cs:48)
    inferCNN.Start () (at Assets/Scripts/inferCNN.cs:33)


    The C# code in unity to load and predict:
    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using Unity.Barracuda;
    5.  
    6. public class inferCNN : MonoBehaviour
    7. {
    8.     private Model m_RuntimeModel;
    9.     private IWorker m_Worker;
    10.     private Tensor input;
    11.     private Tensor output;
    12.     private Tensor reshaped_output;
    13.     private TensorShape finalReshape;
    14.  
    15.     public NNModel modelAsset;
    16.     public RenderTexture outputRenderTexture;
    17.     public Material m;
    18.  
    19.     [SerializeField]
    20.     [Range(-1.0f, 1.0f)]
    21.     private float in0;
    22.     [SerializeField]
    23.     [Range(-1.0f, 1.0f)]
    24.     private float in1;
    25.  
    26.     void Start()
    27.     {
    28.         m_RuntimeModel = ModelLoader.Load(modelAsset, false);
    29.         m_Worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, m_RuntimeModel);
    30.  
    31.         finalReshape = new TensorShape(1,28,28,1);
    32.  
    33.         DisplayDebug();
    34.     }
    35.  
    36.     void DisplayDebug()
    37.     {
    38.         var temp = m_RuntimeModel.inputs[0];
    39.         string dbgString = "input shape: ";
    40.         for(int i = 0; i <  temp.shape.Length; i++)
    41.         {
    42.             dbgString = dbgString + temp.shape[i].ToString() + ", ";
    43.         }
    44.         Debug.Log(dbgString);
    45.  
    46.         input = new Tensor(1, 2, 1, 1); // when no Conv2D is used, the correct is (1,1,1,2)
    47.  
    48.         m_Worker.Execute(input); // infer
    49.         output = m_Worker.PeekOutput(); // 1,28,3,28
    50.         Debug.Log(output);
    51.        
    52.     }
    53. }
    54.  
    Barracuda 1.2.0
    Unity 2019.4.17f1
    Keras2Onnx 1.7
    Tensorflow 1.15
    Python 3.7
    Windows 10
     

    Attached Files:

  2. Mantas-Puida

    Mantas-Puida

    Unity Technologies

    Joined:
    Nov 13, 2008
    Posts:
    1,860
    Barracuda and Keras are assuming channels-last data layout, but ONNX uses channels-first layout for convolutions, so
    Reshape
    layer gets mistranslated on the Barracuda side. We are working on bigger update in this area and hope it will be out as 1.3.0-preview release few weeks from now.
    Meanwhile you could fix
    Reshape
    layer after loading the model in Barracuda:
    Code (CSharp):
    1.  
    2. Model FixReshape(Model model, string layerName, TensorShape newShape)
    3.     {
    4.         for (var idx = 0; idx < model.layers.Count; idx++)
    5.         {
    6.             if (model.layers[idx].name == layerName)
    7.             {
    8.                 model.layers[idx].pool = newShape.ToArray();
    9.             }
    10.         }
    11.      
    12.         return model;
    13.     }
    14.  
    15. //<...>
    16. m_RuntimeModel = FixReshape(ModelLoader.Load(modelAsset, false), "reshape/Reshape:0", new TensorShape(-1, 28, 28, 1));
    17.  
     
  3. hnphan

    hnphan

    Joined:
    Nov 5, 2014
    Posts:
    7
    Thank you so much, this fixed my issue!
     
  4. fguinier

    fguinier

    Unity Technologies

    Joined:
    Sep 14, 2015
    Posts:
    118
unityunity