Search Unity

  1. Get the latest news, tutorials and offers directly to your inbox with our newsletters. Sign up now.
    Dismiss Notice

[Barracuda] Imported pix2pix (generative image model), output looks incorrect

Discussion in 'Barracuda' started by hellovivian, May 22, 2020.

  1. hellovivian

    hellovivian

    Joined:
    Jan 7, 2018
    Posts:
    5
    Hi everyone,

    I successfully imported an ONNX file of pix2pix, but my output looks a little funky.

    I did export the pix2pix by running a docker container containing it and outputting a pretrained model (trained on the facades dataset).

    I imported the ONNX into Unity and tested it on a texture. Unfortunately, my output stays the same no matter what the input is. And it looks nothing like what the output is supposed to look like.

    Here is an image of what I got and what I expected. Screenshot from 2020-05-20 15-47-46.png Screenshot from 2020-05-21 18-09-35.png

    I'm looking into debugging this, but I'm struggling to understand how to debug in Barracuda. So far all I've been able to do is check that the floats in my tensor data are in the right intervals [-1,1] by accessing the Tensor's readonlyArray property.

    Does anyone have any idea what my problem looks like? Could it be a corrupted ONNX model or am I just not feeding in input tensors properly?

    Otherwise, if anybody has suggestions for debugging in Barracuda, I'd greatly appreciate that! This is day 2 of my adventure with it, and my first post on Unity Forums.


    Code (CSharp):
    1. public class testPix2Pix : MonoBehaviour
    2. {
    3.     public NNModel modelSource;
    4.  
    5.     public Texture2D testTexture;
    6.     public RenderTexture outputRenderTexture;
    7.  
    8.     Model model;
    9.     IWorker worker;
    10.  
    11.     // Start is called before the first frame update
    12.     void Start()
    13.     {
    14.         model = ModelLoader.Load(modelSource);
    15.         worker =  WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, model);
    16.         var inputTensor = new Tensor(testTexture, 3);
    17.  
    18.         worker.Execute(inputTensor);
    19.         var outputTensor = worker.PeekOutput();
    20.  
    21.         outputTensor.ToRenderTexture(outputRenderTexture);
    22.     }
    23.  
    24. }
     
  2. hellovivian

    hellovivian

    Joined:
    Jan 7, 2018
    Posts:
    5
    I figured it out!

    The problem was that when a Tensor is created from a Texture2D, it creates a tensor with values in between 0 and 1. However, for pix2pix, the values need to be between [-1,1]. I found this out with the help of these functions
    var inputs = inputTensor.data.Download(inputTensor.shape) and also by exploring the pix2pix model on the docker image I exported it from.

    That was why I was getting an image, though a subpar one.

    Now I get images such as the one I uploaded (which is as I expect) . I'm ecstatic! ^_^

    My suggestion to the dev team would be to either note this in the documentation (ie on this page) or include it as a parameter in the constructor.
     

    Attached Files:

  3. lialialiacole

    lialialiacole

    Joined:
    May 7, 2020
    Posts:
    1
    Hi @hellovivian! I'm doing something similar and trying to convert a Pix2Pix to .ONNX format. How did you do it? And would you be ok with sharing the .ONNX file?
     
  4. alexandreribard_unity

    alexandreribard_unity

    Unity Technologies

    Joined:
    Sep 18, 2019
    Posts:
    27
    @hellovivian thanks for the suggestion. It was something we are considering doing in the future.
     
  5. alexandreribard_unity

    alexandreribard_unity

    Unity Technologies

    Joined:
    Sep 18, 2019
    Posts:
    27
unityunity