Search Unity

Question Barracuda + StyleGAN2 in Unity?

Discussion in 'Barracuda' started by TallyIsham, Dec 13, 2021.

  1. TallyIsham

    TallyIsham

    Joined:
    Dec 13, 2021
    Posts:
    2
    Hi! First-time Barracuda user, second-time gave dev here. I'm wondering if it's possible to use Barracuda/Unity to run StyleGAN2, in a PC game, on a player's CPU/GPU.

    Let's say we wanted to give players the ability to generate new characters by mixing and matching images in our game to create new results, or generate "children" out of a parent image, like so:





    Is this possible? I know that it's possible to get style transfer happening in real time using a CNN in Unity via Barracuda, but I'm not sure about this GAN. Is a real-time StyleGAN something that Barracuda can help accomplish in Unity?
     
    TallyIsham, Dec 13, 2021
    #1

  2. alexandreribard_unity

    alexandreribard_unity

    Unity Technologies

    Joined:
    Sep 18, 2019
    Posts:
    53
    Hi @TallyIsham you are correct we do support a number of style transfer and GAN models.
    I have not tried to run StyleGAN specifically, so I cannot grantee that the model would work.
    There is a few steps that you need to perform in order to make the model run in Barracuda.

    The first is to export the model to ONNX:
    * https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html
    * https://github.com/onnx/tensorflow-onnx
    You can also follow our guide here:
    * https://github.com/Unity-Technologi...lob/release/1.0.0/Documentation~/Exporting.md

    Be sure to export with fixed input size to make the process easier.

    Once you have managed to export the model to ONNX format, you can then try to import it in Unity.
    If it works, you are good to go, else feel free to poke us so we can help out.

    Let me know~
     
    alexandreribard_unity, Dec 14, 2021
    #2

  3. TallyIsham

    TallyIsham

    Joined:
    Dec 13, 2021
    Posts:
    2
    Thanks, Alexandre. Quick follow-up question. Let's say that the use case is something as simple as style transfer between two 500x500px images, to generate a third hybrid image, that happens on click. I'm trying to get a sense of the minimum hardware a user would need to have in order for this to run smoothly. The concern is that this wouldn't run on the average user's Macbook. For example, when I look at documentation for this pics2pics/Unity integration, "it's highly recommended to use a Windows system with GeForce GTX 1070 or greater", and the real-time style transfer Book of the Dead example on the Unity blog was shown on a machine with an RTX 2080 GPU. But because both of those seem like more intensive processes, I wasn't sure if they're perfect benchmarks for what we'd want to try. Do you think it's hypothetically possible that a simpler ML process could run on an average person's laptop? I know that, because to my knowledge no one's tried this with StyleGAN, to some extent the answer is "try it and see" but I'm curious if you have any data around this type of thing. Thanks again!
     
    TallyIsham, Dec 14, 2021
    #3

  4. alexandreribard_unity

    alexandreribard_unity

    Unity Technologies

    Joined:
    Sep 18, 2019
    Posts:
    53
    A few things on this.
    Indeed most style transfer models can be computationally intensive.
    For our Book Of The Dead demo the key issues is that we are running in realtime, on top of a fully rendered scene.
    The input is also quite larger than 512^2.

    For your application, I think you are not so strict on performance requirements and can let the user wait a few frames or so before getting the result.
    That allows you to split the inference over a few frames.
    Also your background rendering work might not be as demanding.

    To give you an idea, some very old numbers for Mobilenet on MacBook Pro GPU gives me around 8.1ms inference time for example.

    So overall, given the two caveats that I noted, I would say your application would be feasible performance wise on a MacBook.
     
    alexandreribard_unity, Dec 14, 2021
    #4

  5. CyberStreamer

    CyberStreamer

    Joined:
    Jun 28, 2017
    Posts:
    2
    I tried the Stylegan2, I export it into the onnx. But Barracuda cant read this model because Barracuda does not support dynamic weights of Conv layer. Below is the Error log.

    OnnxImportException: Unexpected error while parsing layer 331 of type Conv.
    Currently only constant tensors are supported for W input in node of type Conv. Instead 331.W is pointing to non constant node 322.

    Unity Version: 2022.3.0
    Barracuda Version: 3.0.0

    Not supporting dynamic weights means giving up many latest powerful models. Any solutions? Is there such a function in the future development direction? Thx in advance...
     
    CyberStreamer, Jun 7, 2023
    #5