[Barracuda] Imported pix2pix (generative image model), output looks incorrect

Hi everyone,

I successfully imported an ONNX file of pix2pix, but my output looks a little funky.

I did export the pix2pix by running a docker container containing it and outputting a pretrained model (trained on the facades dataset).

I imported the ONNX into Unity and tested it on a texture. Unfortunately, my output stays the same no matter what the input is. And it looks nothing like what the output is supposed to look like.

Here is an image of what I got and what I expected.

I'm looking into debugging this, but I'm struggling to understand how to debug in Barracuda. So far all I've been able to do is check that the floats in my tensor data are in the right intervals [-1,1] by accessing the Tensor's readonlyArray property.

Does anyone have any idea what my problem looks like? Could it be a corrupted ONNX model or am I just not feeding in input tensors properly?

Otherwise, if anybody has suggestions for debugging in Barracuda, I'd greatly appreciate that! This is day 2 of my adventure with it, and my first post on Unity Forums.

Code (CSharp):

1. public class testPix2Pix : MonoBehaviour
2. {
3.     public NNModel modelSource;
4.  
5.     public Texture2D testTexture;
6.     public RenderTexture outputRenderTexture;
7.  
8.     Model model;
9.     IWorker worker;
10.  
11.     // Start is called before the first frame update
12.     void Start()
13.     {
14.         model = ModelLoader.Load(modelSource);
15.         worker =  WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, model);
16.         var inputTensor = new Tensor(testTexture, 3);
17.  
18.         worker.Execute(inputTensor);
19.         var outputTensor = worker.PeekOutput();
20.  
21.         outputTensor.ToRenderTexture(outputRenderTexture);
22.     }
23.  
24. }

 

I figured it out!

The problem was that when a Tensor is created from a Texture2D, it creates a tensor with values in between 0 and 1. However, for pix2pix, the values need to be between [-1,1]. I found this out with the help of these functions
var inputs = inputTensor.data.Download(inputTensor.shape) and also by exploring the pix2pix model on the docker image I exported it from.

That was why I was getting an image, though a subpar one.

Now I get images such as the one I uploaded (which is as I expect) . I'm ecstatic! ^_^

My suggestion to the dev team would be to either note this in the documentation (ie on this page) or include it as a parameter in the constructor.

 

Hi @hellovivian! I'm doing something similar and trying to convert a Pix2Pix to .ONNX format. How did you do it? And would you be ok with sharing the .ONNX file?

 

@hellovivian, I am struggling with the same but for my custom model. Would you be so kind and share your project with facades? Cheers!

 

Could you please clarify which specific Pix2Pix file you are referring to? Additionally, I would like to know if you want to know if it was converted from Tensorflow or PyTorch.

 

Thanks for sharing this information. I really like your post very much. You have really shared a informative and interesting post with people..

TanyaSlot88