Search Unity

Question Classify 3 dimensional pointclouds

Discussion in 'Barracuda' started by Apfelbox, Apr 11, 2023.

  1. Apfelbox

    Apfelbox

    Joined:
    Apr 28, 2014
    Posts:
    39
    I have a NN Model in the ONNX Format that I want to use in Unity. An older version of the model worked fine but after we updated the model and changed the models input I'm no longer able to integrate it.

    The previous model would take 60 float values, in the inspector the input looked like this: dense_input shape(n:*, h:1, w:1, c:60)

    I got that model working with the following code
    Code (CSharp):
    1. public NNModel modelAsset;
    2. private IWorker worker;
    3. private Tensor input = new Tensor(1, 60);
    4. private void Start() {
    5.     var model = ModelLoader.Load(modelAsset);
    6.     worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, model);
    7. }
    8. public int RunModel(Vector3[] pointCloud) {
    9.     // pointCloud has 20 values -> 60 floats. Use each float as an input
    10.     for (int i = 0; i < pointCloud.Length;i++) {
    11.         int startIndex = i * 3;
    12.         input[startIndex] = pointCloud[i].x;
    13.         input[startIndex + 1] = pointCloud[i].y;
    14.         input[startIndex + 2] = pointCloud[i].z;
    15.     }
    16.     // Run model
    17.     worker.Execute(input);
    18.     // ...
    19. }
    The updated model instead takes 20 float arrays of size 3. In the inspector is looks like this: input_13 shape(n:*, h:1, w:3, c:20)

    my naive approach was to change the input Tensor to
    Code (CSharp):
    1.  
    2. private Tensor input = new Tensor(3, 20);
    3.  
    or
    Code (CSharp):
    1.  
    2. private Tensor input = new Tensor(1, 1, 3, 20);
    3.  
    According to the documentation, one can access the tensor input like this
    `tensor4D[n, h, w, c] = 1.0f;`
    Therefore I updated my code to
    Code (CSharp):
    1.  
    2. for (int i = 0; i < pointCloud.Length; i++) {
    3.         input[0, 0, 0, i] = pointCloud[i].x;
    4.         input[0, 0, 1, i] = pointCloud[i].y;
    5.         input[0, 0, 2, i] = pointCloud[i].z;
    6. }
    7.  
    If I run the code I receive an AssertionError on the line where I call worker.Execute(input);
    > AssertionException: Assertion failure. Values are not equal.
    Expected: 3 == 20
    I tried various things to "fill" the input Tensor but I just can't wrap my head around it. This is currently the most promising solution but I still fail to make it work. Most of my other attempts caused an OutOfBounds Exception during the loop.

    I'm grateful for every hint. Even if I have to change everything from the ground up...

    I'm using Barracuda 3.0 (Unity LTS 2021.3.19f)