Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Showcase Aify - Text to Image Utility [Editor Extension]

Discussion in 'Barracuda' started by AiKodex, Mar 6, 2023.

  1. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Your observations are correct. The match image feature works quite like a mask, distributing colors and initializing intensities based off the "Match Image Strength" parameter. The model throws the seed initializer into the mix after which the prompt carries forward the generation process.

    By game texture style transfer we mean that the model can try and make the input flat texture look more seamless, obfuscating any obvious patterns. It will do so by sifting through a dataset of AAA game textures. The model is yet to be trained for this.

    If by "Match Style" you mean AI style transfer or putting "filters" as we understand, we can definitely look into it.
     
    Dr_Evanzan likes this.
  2. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    [Announcement]
    A brief server maintenance is scheduled for UTC 17:30, with services expected to resume shortly at around UTC 18:30. We kindly ask for your patience and understanding during this time.
     
    Last edited: Apr 3, 2023
  3. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    [Announcement]
    Our servers are up and running again. We appreciate your patience during the brief interruption.
     
  4. Dr_Evanzan

    Dr_Evanzan

    Joined:
    Mar 4, 2015
    Posts:
    5
    I think so, but maybe some examples would help.
    Attached are 5 trees generated earlier by Ai.Fy from the same source image (another asset on the Asset Store):

    A: Tree A.png

    B: Tree B.png

    C: Tree C.png

    D: Tree D.png

    E: Tree E.png

    (Note: All had the same source image but the other parameters may be slightly different. These 5 were selected from 100+ trees generated and I was playing around with the settings and tags a little as I went.)

    I would say that Tree A and B had a similar style, what I would describe as a classic "Pixel Art" look.
    Tree D and E also have a similar style, a more defined hand-painted look.
    Tree C lies somewhere between these two extremes, a slightly pixelated look but more natural and less stylised than a classic "Pixel Art" style image.
    The source tree probably sits somewhere inbetween Tree C and D, as it has a somewhat hand-painted style but still with a slightly pixelated, computer game look.

    What I am looking for is the ability to pass in an image, such as one of these 5, as a style reference to constrain the style of variants generated rather than the general colours/shape (as with the current Match Image).
    At a basic level this would differentiate between Tree A and Tree E, which might also be possible using style tags (e.g. "Pixel Art" versus "Hand-Painted").
    However, ideally you would be able to use something like Tree C, which is harder to isolate using tags as it sort of sits inbetween several tags.

    Hopefully that is of some help?
    If so, do you think such a thing would be possible?
     
  5. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Thank you for sharing with us what you are looking for in a well put post. As we mentioned, we have been working on adding "styles". Pixel art style is one of them. With the new update, you will still be working with the "Match Image" feature, but you will be generating well defined styles. We will be listing the styles out when we have completed working on them.

    Meanwhile, here are the outputs of the new model that will be available in the update after the next matched at 0.5 closely with Image B. The prompt used was "pixel art, tree, bark, leaves, 8 bit, pixelized, stylized" and we used only about 5 generations before getting these results.

    pixel tree art 1.png

    pixel tree art 3.png

    pixel tree art 4.png

    pixel tree art 2.png

    Side note: You are of course free to use the above generations without restrictions.
     
    Lars-Steenhoff and Dr_Evanzan like this.
  6. andyz

    andyz

    Joined:
    Jan 5, 2010
    Posts:
    2,251
    Hi what is used behind the scenes for generating images? Just Barracuda & neural networks or other open source libraries and image content for learning from?

    i.e. "Text to Image Diffusion Model" is using which model - Stable Diffusion?
     
    Last edited: Apr 4, 2023
  7. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    We will utilize LoRA (Low-Rank Adaptation) models that run on Stable Diffusion. Earlier, we used textual inversion method tuned for game development / textures. We have already implemented LoRA models. We are working on setting up new servers that serve LoRA in the update after the next.
     
  8. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    [Updated]

    1.2
    - Now users can save concept art designs
    - Match Image now works with a range of dimensions
    - Regenerate images by entering image seed
    - Stack errors resolved
    - Concept art browser enhanced
    - Server Improvements
    - Editor look enhanced
    - Prompting Guidelines added in the documentation
    - Network Error Debugging
     
    Last edited: Apr 6, 2023
  9. ecarter

    ecarter

    Joined:
    Jan 3, 2013
    Posts:
    21
    have to ask, can these tool create the texture for a 3d model? for example, create teh texture for a 3d house that dont have textures, is that posible?
     
  10. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Yes! This is very much possible. The prerequisite for doing so is that you should have the house separated into different parts you want different textures to be on. Additionally, those parts should be UV unwrapped accordingly. Usually, most models are separated and have appropriate UV unwrapping if they are modular. Ai.Fy does not generate "baked" textures. However, Ai.Fy can modify them.
     
  11. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hello Enzo36t,

    Please upgrade to version 1.2 of the package. There is a document named ReadMe (Dependencies Installation) in which we have a step by step tutorial with pictures to install the dependency. You need to click on the "Plus Icon" on the top left of the package manager and then click on "Add package from git URL" and type "com.unity.barracuda". This should resolve the error.
     
  12. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Please click on the Plus sign and select "Add package from git URL" and type "com.unity.barracuda" and click on "Add" button. This gif should help you.
    ezgif-1-2ac7bc4d8f.gif

    Please tell us if this resolved your error.
     
  13. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Yes. Barracuda is needed to run the HWSR and the LWSR (upscaling models) locally. The rest of the models like the Text to Image and the Image to Depth can be accessed without Barracuda.
     
  14. lofwyre

    lofwyre

    Joined:
    Apr 19, 2007
    Posts:
    174
    Hi I'm getting blurry images using a phrase like "wide-angle, realistic, female warrior, full body, long hair, anime style". I saw in the docs NSFW option, can this be turned off somehow?
     
  15. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hello lofwyre,

    Not yet, the NSFW option cannot be turned off. However, with the prompt that you are using, you will soon be able to generate more uncensored images with the new LoRA (Low Rank Adaption) and a new model we are working on.
     
  16. simonkingandaoking

    simonkingandaoking

    Joined:
    Dec 27, 2021
    Posts:
    19
    build error :Assets\Aify\Demo\Resources\CanvasController.cs(9,19): error CS0234: The type or namespace name 'AssetImporters' does not exist in the namespace 'UnityEditor' (are you missing an assembly reference?)
     
  17. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hello simonkingandaoking,

    There are two methods to get rid of this error. If you are using the Ai.Fy generator in the build, then please comment out the line 8 through 13 and 81 in the CanvasController.cs script found in Ai.Fy > Demo > Resources folder.

    Else, you can simply move the Canvas Controller script to the Editor folder and you will be able to build your application. You can also delete the Ai.Fy asset folder if you are not using it in the final build.
     
  18. Tyrant117

    Tyrant117

    Joined:
    Aug 25, 2014
    Posts:
    12
    I've been trying to get a good set of game icons, can you give some sample prompts you used to generate yours. I lot of the time I will get very noisy abstract colors.
     
    zKici likes this.
  19. zKici

    zKici

    Joined:
    Feb 12, 2014
    Posts:
    438
    and how to upscale textures for characters already, would love to make them better please
     
  20. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hello zKici,

    To upscale textures for characters, please select the texture > go to the Ai.Fy window > you should see your texture in the selected textures category > Enable "Upscale Resolution" property in Postprocess settings > Choose your upscaler > Click on Generate Texture.

    Please note that if your texture is already 2k or more in size, and you want to upscale to 8k, the upscaler (LWSR, HWSR using the barracuda library) has a tendency to overload the graphics card and may cause a crash.
     
  21. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hello Tyrant117,

    Sure, here is an example:

    Prompt: stylized, icon, cap, cool hat, baseball cap, plain vignette background, minimalistic, vray, render, clean art

    Seed: 1

    Match Image:
    3572997178.png

    Match Image Strength: 0.2

    Result:

    1.png


    The trick to generating an icon using Ai.Fy is using a pre-existing icon image with a small value for Match Image strength.
     
    X2DGmDev and Dr_Evanzan like this.
  22. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    [Announcement]

    We have a very exciting announcement!
    We will be bringing out a new and improved model which has some incredible capabilities:

    - Next-level photorealism capabilities
    - Enhanced image composition and face generation
    - Rich visuals and aesthetics
    - Use of shorter prompts to create descriptive imagery
    - Greater capability to produce legible text

    You will still be able to use the previous models and will have a variety to choose from!

    Moreover, we will also be introducing the ability to generate icons with a central mask for better icon generation capabilities.

    We will also be introducing a higher quality server based upscaling method.

    Stay Tuned!
     
    shyamarama, thefreik, zKici and 2 others like this.
  23. dan_soqqle

    dan_soqqle

    Joined:
    Feb 2, 2022
    Posts:
    114
    Hi, just purchased this days ago. Its very interesting and cool.
    May i ask what diffusion model is being used? The docs did not specify.

    also wondering how you are able to manage the costs on the server since it appears inferencing servers may stil incurr some significant costs
     
  24. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Currently, the diffusion model we are using is SD 2.0 from the SD 1.5 (LoRA) we had previously. In the next update, which we hope to release in a few days, we plan to bring out SD 1.5, SD 2.0, SD 2.1 and SDXL.

    As for the servers, we have implemented spot instances for improving cold start performance. This helps to reduce server costs significantly.
     
  25. edzoro

    edzoro

    Joined:
    Jan 13, 2023
    Posts:
    1
    Hello,

    I bought this asset 2 days ago and I don't regret it. Although I can't find a way to make the match image work.

    I tested with Unity version 2020.3.19f1 with barracuda version 1.0.0 and also unity version 2021.3.15f1 with barracuda version 3.0.0 without success :'(

    I get this error :
    Code (CSharp):
    1. HTTP/1.1 500 Internal Server Error
    2. UnityEngine.Debug:Log (object)
    3. AiKodex.AifyEditor/<Post>d__78:MoveNext () (at Assets/Aify/Editor/Scripts/AifyEditor.cs:922)
    4. Unity.EditorCoroutines.Editor.EditorCoroutine/YieldProcessor:MoveNext (System.Collections.IEnumerator) (at Library/PackageCache/com.unity.editorcoroutines@1.0.0/Editor/EditorCoroutine.cs:80)
    5. Unity.EditorCoroutines.Editor.EditorCoroutine:ProcessIEnumeratorRecursive (System.Collections.IEnumerator) (at Library/PackageCache/com.unity.editorcoroutines@1.0.0/Editor/EditorCoroutine.cs:134)
    6. Unity.EditorCoroutines.Editor.EditorCoroutine:MoveNext () (at Library/PackageCache/com.unity.editorcoroutines@1.0.0/Editor/EditorCoroutine.cs:115)
    7. UnityEditor.EditorApplication:Internal_CallUpdateFunctions ()
    Though the others features works fine.

    As mentioned previously in this thread I tried to check the setting "allow https download" but was unable to find this settings in the player settings that Hydrotoker mentionned.

    This is very frustrating because I'm very thrilled by this feature.

    Hope that you can help me figure out what's the problem,

    Edzoro.
     
  26. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hello,

    Please flag the boolean EU decimal comma just below the Match Image section and try again. This solution worked out for most users who had trouble using the Match Image feature.
     
  27. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    [Updated]

    2.0

    - Users now can choose from different Engines and models to generate their image
    - With the new XL engine
    - Next-level photorealism capabilities
    - Enhanced image composition and face generation
    - Rich visuals and aesthetics
    - Use of shorter prompts to create descriptive imagery
    - Greater capability to produce legible text
    - Central mask for icons introduced
    - Server based upscaling now available
    - Example prompt shuffling
    - Error free builds
    - NSFW filter lowered (XL engine)
    - Editor look enhanced
    - Server Improvements
    - Documentation updated
     
    shyamarama, edzoro and Dr_Evanzan like this.
  28. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    [Announcement]

    Ai.Fy version 2.0 is live on the store!
    Here is a short tutorial we have made describing all the features inside the asset:



    We hope you like it the new capabilities.

    We are grateful to everyone who has supported us till now. If you are enjoying the service, updates and the tutorials, we'd much appreciate a rating on the store. It motivates us to keep building upon our work, open new pipelines for models like DALL-E 2, MJ and DeepFlyod, add more features and expand the capabilities of the asset.
     
    Last edited: Apr 22, 2023
    Dr_Evanzan, shyamarama and zKici like this.
  29. Artini

    Artini

    Joined:
    Jan 29, 2016
    Posts:
    181
    Thanks a lot for the update. Below is the abstract generated by Ai Fy V2_1
    Abstract01.jpg
     
    AiKodex likes this.
  30. Artini

    Artini

    Joined:
    Jan 29, 2016
    Posts:
    181
    One can get nice crystals...
    Crystal01.jpg
     
    AiKodex likes this.
  31. Artini

    Artini

    Joined:
    Jan 29, 2016
    Posts:
    181
    Cave generated by Aify
    Cave01.jpg
     
    AiKodex likes this.
  32. esmundoz

    esmundoz

    Joined:
    Sep 18, 2013
    Posts:
    12
    Get this error when installing the asset: "Assets/Aify/Editor/Scripts/AifyEditor.cs(19,13): error CS0234: The type or namespace name 'EditorCoroutines' does not exist in the namespace 'Unity' (are you missing an assembly reference?)"
     
  33. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Could you please specify the version number of Unity and Ai.Fy asset you have?
    Generally, to resolve the error you can open the package manager > Unity Registry (Top left corner dropdown) > Search for Editor coroutines > Enable/Install.
     
  34. DeamyV

    DeamyV

    Joined:
    May 15, 2022
    Posts:
    2
    I'm having some issues with getting the AI.FY window to pop-up in the Unity editor (2022 and 2023), I have used AI.FY before the 2.0 update and it worked fine but since the update I didn't get it to show up anymore. Any suggestions from your side @Xpert3D ?

    EDIT: I have also installed Barracuda V3 and Editor Co-routines V1

    EDIT 2: After replacing JsonConvert with the built-in JsonUtility.FromJson it seems to be building again and it showed up as an window now, FYI for other users using the package
     
    Last edited: Apr 24, 2023
    AiKodex likes this.
  35. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    We're glad you could figure it out.
    As of now, we have tested the asset on 2020.3 LTS (Official Release) and 2021.3 LTS (Official Release). For the 2020.3 the asset works perfectly. For 2021, there is an IWorker / Editor Coroutines error thrown which can be resolved by installing the Barracuda and Editor Coroutines Package.

    We still have to test and resubmit the packages for 2022 beta releases. Again, thank you for updated your comment which a solution. Hope this helps other users.
     
  36. Frpmta

    Frpmta

    Joined:
    Nov 30, 2013
    Posts:
    479
    Hello.

    Thanks for the asset.

    Some suggestions related to string processing of generated images name:

    I would like to request that any generated image with a repeated seed, instead of swapping any current saved image that has the same name, it should be renamed to for example 387231 (1) for example. That way you can be certain no old image of yours is being deleted even if the possibility of two repeated seeds is very low. Happened to me twice.

    Another suggestion: Every time you press the Generate Image button repeatedly while it is processing it will generate another image and replace the current one. Either make the button become grayed out while generating, or make it not replace the already generated images.
    I occasionally misclick. I also thought once I could press the generate button 5 times and then wait for 5 different results, but turns out they all get overwritten by the last generated image.
     
    AiKodex likes this.
  37. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Thank you for the suggestions. We'll bring out these changes as soon as possible.
     
    O_ and Dr_Evanzan like this.
  38. DeamyV

    DeamyV

    Joined:
    May 15, 2022
    Posts:
    2
    Are there any plans to be able to generate seamless textures through the asset, I've tried some prompts to get it to generate those but to no avail sadly. Do you think that would be a possible feature in the near future?

    Some info on the different sampler models and how they relate to the final image would be nice to have, maybe some hover-able tool tips with info in the AI.FY window would already be a great help for all of the features :)

    All in all. good asset but can use a bit more refinement on the above topics but I'm sure it will get there seeing your pro-activeness @Xpert3D
     
    Sabso and Dr_Evanzan like this.
  39. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hello DeamyV,

    Yes of course, since we are testing out new models everyday, there quite a few updates lined up for Ai.Fy.

    There is a bit of an issue with the XL engine when it comes to generation of seamless textures since the model is over-fitted slightly.

    The AIFY V1 engine handles seamless textures better. A prompt on the lines of "[object] texture, seamless, game art". Examples of [object] are wood, concrete, plaster, etc. These prompts should hopefully nudge the generation into the right direction.

    We have appended the list for the next update to include more information on each parameter so the developer can utilize the asset to the fullest. Thank you for the feedback.
     
    shyamarama and Dr_Evanzan like this.
  40. giollord

    giollord

    Joined:
    Mar 23, 2017
    Posts:
    17
    Hello!

    I bought AiFy a week or two ago and one of first things I tried was depth map generation, since it's quite useful feature for me. Unfortunately, it wasn't able to connect to the server, so I thought I can wait a bit - perhaps, some maintenance is happening. I tried again now, after about 2 weeks, and still it does not work for me. Prompt image generation works fine though.
    So, I select just generated image (or any other image) in "Project" tab, image automatically is shown in "Selected Textures", then I check "Generate Depth Map" and click "Generate Textures". Unity thinks for 1-2 seconds and prints "Cannot connect to destination host" in console. I also tried using VPN, but without success. Also tried to ping the server from local PC, even from remote VPS in case if my provider by some reason don't like the server address, but all packets were lost.

    Could you please check that your depth map server is reachable?

    P.S.: I think it would be really good feature to display and copy prompt for images, shown in Concept Art section
     
  41. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    We apologize for the inconvenience. The depth map server should be reachable now.

    Thank you for the suggestion. It is indeed a good idea to display prompts that were used to generate an image in the Concept Art Browser and have a button to copy them. We will add this in the list of updates we're bringing with the release of Ai.Fy 2.1.
     
  42. Brunomir-

    Brunomir-

    Joined:
    Feb 23, 2019
    Posts:
    9
    Assets\Aify\Editor\Scripts\AifyEditor.cs(19,13): error CS0234: The type or namespace name 'EditorCoroutines' does not exist in the namespace 'Unity' (are you missing an assembly reference?)
     
  43. Brunomir-

    Brunomir-

    Joined:
    Feb 23, 2019
    Posts:
    9
    Can someone help me with this error?

    Assets\Aify\Editor\Scripts\AifyEditor.cs(19,13): error CS0234: The type or namespace name 'EditorCoroutines' does not exist in the namespace 'Unity' (are you missing an assembly reference?)
     
  44. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    Hey Brunomir-,

    Please click on Window > Package Manager > Unity Registry. Search for Editor Coroutines and click on "unlock". Hopefully that should resolve the error.
     
    Brunomir- likes this.
  45. Brunomir-

    Brunomir-

    Joined:
    Feb 23, 2019
    Posts:
    9
    It did solve the problem, thank you
     
    AiKodex likes this.
  46. Brunomir-

    Brunomir-

    Joined:
    Feb 23, 2019
    Posts:
    9
    Where can I find a description of the differences between the engines AiFy V1, V2, V2_1 and XL?
    Also the description for the Sampler MOdels K_euler_a, K_euler, etc ?

    Thank you
     
  47. AiKodex

    AiKodex

    Joined:
    Jan 21, 2021
    Posts:
    316
    You can find the details regarding different engines and samplers in the documentation.
    We will be adding hover help options in the next update so one does not have to keep referring to the documentation to know the function of a setting.
     
  48. Brunomir-

    Brunomir-

    Joined:
    Feb 23, 2019
    Posts:
    9
    I couldn't find any specifications on that on the documentation, it just says:

    This extension offers 4 engines at the moment which are trained differently
    and are based on different diffusion models and LoRAs (low ranked adaption)
    AiFy V1 based on SD 1.5
    AiFy V2 based on SD 2.0
    AiFy V2_1 based on SD 2.1
    AiFy XL based on SDXL Beta

    and stull like that. I would like to read something more specific
     
  49. zkajo

    zkajo

    Joined:
    Jul 28, 2016
    Posts:
    9
    I have just purchased the package and when generating an image I'm getting an error: InvalidOperationException: Insecure connection not allowed

    Any idea what I can do to fix it?

    EDIT: Fixed, had to change allow requests in Player Settings.
     
    Last edited: Apr 29, 2023
    Rorep and gamayunovantonsakh like this.
  50. Artini

    Artini

    Joined:
    Jan 29, 2016
    Posts:
    181
    Anime style.
    Crow03.jpg
     
    AiKodex likes this.