Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug Conv2D errors when predicting

Discussion in 'Barracuda' started by hnphan, Jan 9, 2021.

  1. hnphan

    hnphan

    Joined:
    Nov 5, 2014
    Posts:
    7
    Hi!
    I've been able to run a model in Unity using Barracuda but as soon as I add a Conv2D type layer, inference will give me errors right away.
    • When Conv2D is added, I also have the input shape from (1x1x1x2) to (1x2x1x1)
    • I had to do a reshape for the output in Unity from (1x28x1x28) to (1x28x28x1) although the correct shape in python is 1x28x28x1 regardless of the usage of Conv2D or not.
    I validated the ONNX file by reloading it in Python and executed it and it works fine without issue (no need to reshape or anything) thats why I think there's a bug in Barracuda and not the ONNX file.

    I've included a toy decoder with MNIST (28x28x1) that I used
    • File that works: (NoConv2D_decoder.onnx)
    • File that has Conv2D and bugs out: (decoder.onnx)
    The decoder model that works in Python:
    Code (CSharp):
    1. latent_inputs = keras.Input(shape=(latent_dim,))
    2. x = latent_inputs
    3. actType = 'relu'#'tanh'
    4. for i in range(3):
    5.     x = layers.Dense(128)(x)
    6.     x = layers.Activation(actType)(x)
    7. x = layers.Dense(28 * 28 * chann)(x)
    8. x = layers.Reshape((28, 28, chann))(x)
    9. #x = layers.Conv2D(chann, 1, padding="same")(x)
    10. decoder_outputs = layers.Activation('sigmoid', name="outLayer")(x)
    The error message in Unity when Conv2D us used
    AssertionException: Assertion failure. Values are not equal.
    Expected: 1 == 28
    UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message, System.Collections.Generic.IEqualityComparer`1[T] comparer) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    UnityEngine.Assertions.Assert.AreEqual (System.Int32 expected, System.Int32 actual) (at <0847a0faf94444ccbaf1958021b27f54>:0)
    Unity.Barracuda.PrecompiledComputeOps.Conv2D (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor K, Unity.Barracuda.Tensor B, System.Int32[] stride, System.Int32[] pad, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/BarracudaPrecompiledCompute.cs:648)
    Unity.Barracuda.StatsOps.Unity.Barracuda.IOps.Conv2D (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor K, Unity.Barracuda.Tensor B, System.Int32[] stride, System.Int32[] pad, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/StatsOps.cs:92)
    Unity.Barracuda.GenericWorker+<StartManualSchedule>d__30.MoveNext () (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/GenericWorker.cs:247)
    Unity.Barracuda.GenericWorker.Execute () (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/GenericWorker.cs:143)
    Unity.Barracuda.GenericWorker.Execute (Unity.Barracuda.Tensor input) (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/Backends/GenericWorker.cs:135)
    inferCNN.DisplayDebug () (at Assets/Scripts/inferCNN.cs:48)
    inferCNN.Start () (at Assets/Scripts/inferCNN.cs:33)


    The C# code in unity to load and predict:
    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using Unity.Barracuda;
    5.  
    6. public class inferCNN : MonoBehaviour
    7. {
    8.     private Model m_RuntimeModel;
    9.     private IWorker m_Worker;
    10.     private Tensor input;
    11.     private Tensor output;
    12.     private Tensor reshaped_output;
    13.     private TensorShape finalReshape;
    14.  
    15.     public NNModel modelAsset;
    16.     public RenderTexture outputRenderTexture;
    17.     public Material m;
    18.  
    19.     [SerializeField]
    20.     [Range(-1.0f, 1.0f)]
    21.     private float in0;
    22.     [SerializeField]
    23.     [Range(-1.0f, 1.0f)]
    24.     private float in1;
    25.  
    26.     void Start()
    27.     {
    28.         m_RuntimeModel = ModelLoader.Load(modelAsset, false);
    29.         m_Worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, m_RuntimeModel);
    30.  
    31.         finalReshape = new TensorShape(1,28,28,1);
    32.  
    33.         DisplayDebug();
    34.     }
    35.  
    36.     void DisplayDebug()
    37.     {
    38.         var temp = m_RuntimeModel.inputs[0];
    39.         string dbgString = "input shape: ";
    40.         for(int i = 0; i <  temp.shape.Length; i++)
    41.         {
    42.             dbgString = dbgString + temp.shape[i].ToString() + ", ";
    43.         }
    44.         Debug.Log(dbgString);
    45.  
    46.         input = new Tensor(1, 2, 1, 1); // when no Conv2D is used, the correct is (1,1,1,2)
    47.  
    48.         m_Worker.Execute(input); // infer
    49.         output = m_Worker.PeekOutput(); // 1,28,3,28
    50.         Debug.Log(output);
    51.        
    52.     }
    53. }
    54.  
    Barracuda 1.2.0
    Unity 2019.4.17f1
    Keras2Onnx 1.7
    Tensorflow 1.15
    Python 3.7
    Windows 10
     

    Attached Files:

  2. Mantas-Puida

    Mantas-Puida

    Joined:
    Nov 13, 2008
    Posts:
    1,864
    Barracuda and Keras are assuming channels-last data layout, but ONNX uses channels-first layout for convolutions, so
    Reshape
    layer gets mistranslated on the Barracuda side. We are working on bigger update in this area and hope it will be out as 1.3.0-preview release few weeks from now.
    Meanwhile you could fix
    Reshape
    layer after loading the model in Barracuda:
    Code (CSharp):
    1.  
    2. Model FixReshape(Model model, string layerName, TensorShape newShape)
    3.     {
    4.         for (var idx = 0; idx < model.layers.Count; idx++)
    5.         {
    6.             if (model.layers[idx].name == layerName)
    7.             {
    8.                 model.layers[idx].pool = newShape.ToArray();
    9.             }
    10.         }
    11.      
    12.         return model;
    13.     }
    14.  
    15. //<...>
    16. m_RuntimeModel = FixReshape(ModelLoader.Load(modelAsset, false), "reshape/Reshape:0", new TensorShape(-1, 28, 28, 1));
    17.  
     
  3. hnphan

    hnphan

    Joined:
    Nov 5, 2014
    Posts:
    7
    Thank you so much, this fixed my issue!
     
  4. fguinier

    fguinier

    Unity Technologies

    Joined:
    Sep 14, 2015
    Posts:
    146
    Awesome