Showcase Aify - Text to Image Utility [Editor Extension]

Unity's first Editors Extension asset containing neural network models and libraries that are capable of image generation through text (Image to Text), or images to enhanced images using AI (Image to Image) and deep learning.

You can create an unlimited number of images/textures in Edit Mode or Play Mode (runtime) inside the Unity Editor. This asset has a one-click, beginner friendly GUI and does not require any coding to use.

Please note: All the artwork you see on this forum post is AI generated.

The AI models that come with this asset are:

- SRGANs
1) Light Weight Super Resolution Generative Adversarial Network - Local Python Inference
2) Heavy Weight Super Resolution Generative Adversarial Network - Local Python Inference

- Depth Estimation Model for AI Depth and Displacement Maps - Server Inference
- Text to Image Diffusion Model - Server Inference

Dependencies:

This tool requires the Barracuda package from the package manager and an active internet connection for some features.

Additional Features and Applications:

️ Text to Image Converter: The desired image/texture should be described, and the results can be obtained within seconds through a single click, utilizing fast and efficient inference.

Examples for concept design:

● Octane rendered character portrait of mitsurugi, 3D, octane render, depth of field, concept art, vibrant colors, glow, trending on artstation, ultra high detail, ultra realistic, cinematic lighting, focused, 8k



● Interior design, open plan, kitchen and living room, modular furniture with cotton textiles, wooden floor, high ceiling, large steel windows viewing a city



● A highly detailed epic cinematic concept art an alien pyramid landscape , art station, landscape, concept art, illustration, highly detailed artwork cinematic, hyper realistic painting



Examples for Texture Design:



9 Seamless AI Generated Copy Right Free Textures with their corresponding generated normal maps


Rough Sketches into textures or icons: Convert a sketch into stunning and stylized game art using the Match Image Feedback Loop which is an image to enhanced image converter.



Creation of Materials: You can use the normal map and specular/smoothness map generators to add additional details to your material without an external 3D software.



Generate Game Icons: Generate unlimited 3D game icons for menu inventories, menu options, avatar icons, UI buttons and more!



Create Backgrounds and Characters for Visual Novel Games: This asset can create visual novel artwork without you hiring a 3D artist. Just put in a text prompt and let Aify take it over to generate art.



Upscale Game Textures: The asset offers the ability to upscale low-resolution textures to 4K textures, providing users with enhanced texture visuals. Using the barracuda inference python library, this tool can improve the overall quality of textures by bringing out details that may have been lost in low-resolution versions. With the assets efficient and user-friendly interface, users can quickly and easily upgrade their visuals with high-definition textures in a click.



Editor Script: The Editor Script displays all the options neatly in one panel. There is a batch mode as well as a list of all the settings you can choose from in the editor. The editor also has a list view of all the maps you generate with the post processing settings automatically in a single click.



Keeping it all in the editor: Keeping all assets in one workspace inside the Editor and having to switch to fewer services can have several benefits, such as:

- Improved Efficiency: When all assets are located in one workspace, it becomes easier to access and manage them. Users do not have to spend time switching between different services or applications, which can be time-consuming and lead to a loss of productivity.

- Streamlined Workflow: Having all assets in one workspace can help create a more streamlined workflow. This is because users can easily move between different assets, such as code files, images, and documents, without having to navigate between different services. This can help to speed up the development process and make it more efficient.

- Reduced Complexity: Using fewer services can help to reduce the complexity of the development process.

Please check out the documentation for an in-depth explanation and working of the asset.

 

Reading the asset store page: No sign up, No API Keys, no recurring payments, no subscription fees, no additional costs, no restrictions, just one-click easy to use unlimited inferences on our model.

It looks like a great set of workflow tools.
I'm curious, how are you able to provide this for free? (After initial purchase that is)
I have to imagine the upkeep, electricity, and wear & tear costs on the hardware wouldn't make this profitable. But you'd know better than me in this case - but it's definitely unconventional to offer a service like this for a one off fee.

 

Wondering the same question! Is this running locally on my device? Can it support mobile and mac?
I'll buy this plugin in no time if there are more details about it!

 

I bought the module. Currently, I am getting a lot of errors and I have tested it on 2 different computers and in 3 different projects. The image match simply does not work. I get an error every time. Same thing for generating a dept map. An error each time (my internet works perfectly).

I could not make a Rough Sketches into textures or icons, because it is not possible to make an image to image. You have to enter text. Unity crashed several times too because of memory problems (my computer is very powerful so it's weird).

I was not able to upscale like this image for example :

Maybe a more complete tutorial to show how to do what it's supposed to do :
Create inventories
Generate 2d Assets
Generate materials
Feedback loop generation
Create artwork
Upres textures

For the moment, I can't do what I want with the tool without errors. I will wait for updates and examples.

 

Hello,

Thank you for trying out this package.

Please could you elaborate on what type of errors you are getting when performing Image to Image conversion. Please also inform us about the version of Unity and Barracuda you are using. The errors could be due to a version mismatch. One of the developers encountered errors with generating an image with the text prompt. The issue was that in the setting "Allow downloads over HTTP" setting was disabled. We assume there could be an issue in uploading the image as well. Please try toggling this setting and try again. We are discovering errors like this and trying to rectify them.

You could also go through the troubleshooting section in the documentation. That may be helpful in identification of the problem (for example if the uploaded image was NSFW).

As of now, we know that Unity crashes if the images of dimension 1024x1024 (with round dimensions enabled) or more are tried to upscaled by 4x making them 4096x4096 at Light Weight Super Resolution (LWSR). Please tell us if this was the case for you.
We are also currently building a server based SRGAN upscale model with much better results rather than a locally run model.

Again, thank you for taking out the time to explain to us the issues so we can fix them for you in newer iterations of the product.

 

Hello Fyrburn,

Yes, indeed the asset does not have a sign up, or API keys on any other type of payments.
Our server is based on a Amazon EC2 P3 with Tesla V100 GPUs instances that provide high compute capabilities at a reasonable cost. We will be upgrading or making changes to the server settings regularly to ensure fast inferences on our model. Paired with this, we have implemented spot instances for improving cold start performance and reduce costs.

Our inspiration for this project was the thread where AI art for games is discussed in detail. We believe that this could be a very useful tool in generation of seamless textures, 2D game assets, UIs, concept art and generation of new ideas and a huge help for creators. We also offer upscaling of images inspired by the thread - Can indies use this AI texture upscaling to boost their game content? which would help developers use upscaled AI generated textures and boost the art designs. This has impressive implications which we would like developers to use with ease all inside Unity.

 

Thank you for your reply.

Unity : 2021.3.22f1
Barracuda : 3.0.0

No matter which image is used in "match image", I always get the error There was an error in generating the image. If I remove the image, it works (not always though). Also, I was wondering what do I have to write to this location to be able to do Feedback loop generation?


No matter what image I use too, I always get the error "There was an error in generating the dept map" when I try Generate Dept Map.



No matter what image I use as well, I always get the error "A Native Collection has not been disposed" when I try "Upscale Resolution Settings".



I can't do anything that is advertised except generate an image from a text. However, since my goal is to modify existing textures, I can't do anything at the moment unfortunately.

I don't find the option Allow downloads over HTTP in Unity 2021. I seems it was added in 2022.

 

I don't know if it's based on stable diffusion. but if yes, can you add a model for example from civitai.com
and we can choose a model from the editor, so that the resulting image can be better.

 

Yes, of course, that is on our agenda. We will be improving our model so that it can generate better results.

 

Thank you for taking the time to upload the screenshots and outline the issue.
We tried to replicate the errors, but were not able to unfortunately.

We used:
Unity: 2021.3.5f1
Barracuda: 3.0.0

The inferences for Match Image worked perfectly.
We also tried using transparent PNG images. The image generation filled out the transparent background, but the connection to the server was established and the response was received. Same was the case for JPG/JPEG images. We tried images of resolutions 256x256 as well and images with import settings set to Sprite (2D and UI)

We had no issues generating depth maps either. However, there is still an issue with sRGB import settings which makes the depth map perceive the image as slightly darker as pointed out by another one of the developers. This will be fixed in the upcoming updates. However, again, we did not experience connection issues.

For upscaling, we receive a few warnings for unreferenced and undisposed data and occasionally the error of undisposed native collection, but the upscaled image is generated and saved.


The gif above is the intended functionality on Unity 2021.3 with Barracuda 3.0.0.

However, we will keep working on looking for a possible solution to the problem.
Could you please try a non-transparent image of resolution 512x512 with default settings and tell us if that works with Match Image functionality?

 

Hello HelixNGC7293,
The upscaling models are developed with the help of pytorch which are converted to an Open Neural Network Exchange library (ONNX). Inferences on this model can be performed by the barracuda package manager dependency. These models are run locally. We provide two models namely the light weight super resolution (LWSR) and heavy weight super resolution (HWSR) models. The text to image models are run on a server due to their size and high computational requirements. Same is the case with depth maps.

As for mobile and mac support, we have not tested inferences on those platforms. We do not suspect there would be any issues with accessing the text to image model or the depth map generation model though as they utilize an external server.

 

Hi there, I bought this yesterday too - the idea of this is something I'd love to support! What would help a lot is some step by step examples of how to achieve each of the things in the docs. There are pictures which are about the same as those on the asset page, but the 'how' is missing for me. This is how I tried your 'sheep' example:

- I took a screenshot of your 'sheep' picture. The 'generate image' button remains ghosted. Is this intended?
- I type 'Sheep, game art, drawing, cartoon style' in the prompt box, now I can generate. It generates a slightly improved sheep under 'Selected Textures'.
- I click 'last' next to the match image, and it pulls the last generated sheep up for me to repeat the process. It wasn't really converging to an improved sheep though, but that may be my bad prompting.

Questions:
- Am I doing it right?
- Is it correct that I get only one image from the matching?
- I can get multiple images from 'browse concept art' - is it possible to then mutate those images? I can't seem to copy them, view them larger or move them into the 'match image' box

A good example of easy to follow 'recipes' is this one: https://topdown-engine-docs.moremountains.com/recipes.html. I think a recipe for each key function would be really helpful to get us underway.

Thanks!

 

Thank you very much for describing in detail the problem and the expected outcome. It is clear to us what you require help with and what you would like out of this asset. To quickly answer the questions you asked:

1) "I took a screenshot of your 'sheep' picture. The 'generate image' button remains ghosted. Is this intended?"

Yes, this is intended. Without a prompt, the model does not have a direction to take to enhance the image. For example, you want the sheep to have a particular art style, cartoonish, photorealistic, rendered, drawing style or for it to look fluffier, skinned, have tinted wool, and other photo details like have a background, shadow, and shaded.

2) "I type 'Sheep, game art, drawing, cartoon style' in the prompt box, now I can generate. It generates a slightly improved sheep under 'Selected Textures'."

A bit more detail could go into the prompt, for example, in the sheep example, we had something along the lines of "3D render of a sheep with big horns, red eyes, muscular goat, bison, octane render, plain white background, shadow, minimal, 8k, raytraced, vray". To get the image in this forum post, we needed to perform 6 iterations of the process. Some others can take up to 10 or more iterations with branching. If you want more variation in the image, you can decrease the Match Image Strength parameter and the CFG scale. If you want only a slight modification, you can increase the aforementioned parameters.

Another point to add is that there is an option for you to preview a larger image under the Generate Image button. That way, you can see the generated image right inside the Editor.

3) "Am I doing it right? - Is it correct that I get only one image from the matching?"

The prompting could be more specific and you could look into changing parameters to guide the image feedback, but yes, match image generates only one image at a time allowing you to "branch". Branching here refers to guiding of the feedback loop by re-iterating if there is a wrong artefact that has a tendency to propagate over generations.

4) "I can get multiple images from 'browse concept art' - is it possible to then mutate those images? I can't seem to copy them, view them larger or move them into the 'match image' box"

Yes, apologies, this functionality is still under development. We added the Concept Art feature just recently. We will be supplementing it with a save feature in the next update of the asset.

If you are on Unity 2020.3, you could uncheck the fit grid option above browse concept art and increase number on the slider to get a close to maximum resolution image (512x512) from the editor and take a screenshot. But yes, it would certainly be easier to have an option to save the image.

As per your suggestion, we will also be implementing a "recipe for each key function so that it is easier to utilize the asset to the fullest capacity!

 

Excellent Xpert3D, thanks for that thorough response!

- Requiring a prompt makes lots of sense, thanks for sharing the one you used. I got a much nicer result with that.
- It'd be great to be able to generate more than one image, say 3-5, then select each one to view it full size if needed. Then you'd have a button or context menu on each saying 'Set Match Image to this one' or whatever.

Viewing the image larger is very handy. That's one interesting head my sheep has...

 

Hi, that's a pretty cool asset, but I'm wondering, how will you guys be able to pay the server costs forever when the asset is single-time-pay model? What will you do if the sales go down or just stop for whatever reason? Can the people buying this asset be sure that they will be able to use this asset forever as is intended to be?

I still think that this is pretty nice, but I'd be a bit worried that is tied to your server so if you guys don't succeed you could be shutting it down and we'll be losing the money. Do you have a solution for that?

 

It is very strange, because I can generate an image with instructions, it works, but as soon as I try to use the same image for a "match image", it does not work. I tried a PNG and JPG as well.



I tried it at work and at home to make sure it wasn't a network restriction. It's really strange!

 

Hello VenkoGames,

We understand your concerns. We developed a one-time pay model at a competitive price after achieving a breakthrough in increasing our cloud computing efficiency tremendously. Our inference speeds are unparalleled across any product on the game engine market as of the time of this post. This may change with time, however, we will keep updating the asset over time to make it more competitive. The models are huge in size, but since they are stored on our servers, this Unity package occupies only 40MB of disk space. Added to this, one major advantage of using such servers is that it does not need high end GPUs to run the model.

We realized that the only other asset that offer text-to-image functionality requires a sign up, API key integration and subscription fees. This is not only cumbersome to set up but also limited in usage.

In the unlikely event that our servers fail to operate in the future, we are developing models that can run on your system locally. In fact, we already have two trained AI upscaler models called LWSR and HWSR that work locally, without the need for servers or an internet connection to upscale images.

Hope this answers your question.

 

Do not forget that the size of image in pixels for "match image" option need to be in the multiples of 64.
For example 512x512, 1024x1024.
This asset is really breathtaking - gives excellent results.

 

A quick example made with "match image" option and upscaled 4 times.

 

Thank you! We are glad that you liked the asset.
In the next update, that will hopefully be coming out next week, we will be releasing the "Match Image" feature which can take the height and width dimensions in multiples of 64 separately. For example, you will be able to generate images of dimensions 448x320 (landscape) , 512x256 (wide landscape), 320x384 (portrait) and so on. This can be further upscaled by a factor of 4 which will take them up to 1792x1280, 2048x1024, 1280x1536.

 

We were able to get rid of the Native Collection error at our end by implementing Editor Coroutines which will clear up the main thread for a smoother experience of the asset. This will be available by the next update (End of next week hopefully. Asset Store Reviews usually take 4 business days).

However, we still have not been able to figure out why the Match Image feature is not working for you. Could we please request you to contact us at our mail info@aikodex.com so we could fix this issue step by step for you?

 

These are great news - will wait for the update.
All the best for you.

 

im sorry but can i ask for a refund,it seems that i cant access the server in china even on a vpn

 

Hello simonkingandaoking,

Our servers do not have any geological restrictions and we have developers from China using the asset just fine. Please could you describe the issue you are facing in a bit more detail? Are you facing problems with generating an image, the Match Image feature, Browsing Concept Art Designs, Generating Depth Maps or Upscaling images?

Could we request you to please send us an email at info@aikodex.com so we could understand and solve your problem?

 

Hi, I Keep getting this error. Where can I allow http?

 

after some rebooting,i got it working! seems be some stange internet thing,wish future updates can include a local model maybe,i have a stable diffusion running locally tooill do more testing,most interested in the texture upscale

 

nope still couldnt connect stablly,maybe there is somewhere i can debug the connection,i only generated one image successfully,then i get
There was an error in generating the image. Please try again. If this problem persists, please check the documentation.
UnityEngine.Debug:Log (object)

however concept art is working fine

 

and there if often a memory leak:

 

Found it. In player settings

 

im in love with the upscaleworks like magic,is it local?because if i use heavy weight my unity crashes saying too much load on the gpu,however im pretty please with the light weight,ill do more tests

 

Hi, I installed Aify on a clean Unity 2021.3.5f1 project but I cannot modify existing textures with the "Match Image" feature. Got the same debug log message on different unity versions from AifyEditor.cs on line 560.

if (sDb64FromServer == null)
Debug.Log("There was an error in generating the image. Please try again. If this problem persists, please check the documentation.");

 

Hello,

Yes, perhaps you could try a VPN that is situated in Europe. We are doing some maintenance checks on servers hosted in the Eastern United States. There has been a huge influx of users that was not expected by us. Hopefully we'll resolve this issue tomorrow or day after at the most.

Thank you!

 

The memory leaks issue will be solved by the next update. This is on our agenda for the next update.

 

Thank you! Yes, the upscale feature works locally on your system using python inferences made possible by ONNX and Barracuda. The Barracuda library is still getting worked upon by Unity. Hopefully, the Unity crashes will reduce as the library gets upgraded. For now, we're working on an even better Super Resolution model which will be hosted on our servers.

 

Hello,

I have a few questions. Let's say I use this to develop some artwork for a game. And I want to be able to repeat my results. Is there a "Profile" or "Settings" that would enable me to repeat similar results ?
Example, An old lady wearing a black polka dot and bright yellow coat crossing the road. Now lets say I want the same old lady wearing the same coat, scaling a mountain cliff?

Also , Does your product support layering. That way I could develop "pieces" of artwork and Layer them.

 

Hello Da-Luk,

Please could you send us more details about the different Unity versions you tried. The project was built and submitted on Unity version 2020.3 with Barracuda 1.0.0. We have carried out the most testing on that version (due to it being the earliest LTS release that UAS supports and recommends).

Are you having any trouble accessing any other features? Please send us an email at info@aikodex.com so we can debug the issue step by step for you.

 

Hello Troynall,

Yes, there is a way to get the same image provided the prompt is the same. Every image that is saved is with the seed. If you enter the same prompt with this seed you will get the same result. You can imagine this like a key value pair or lock and key system. As of now, there isn't a way in the Editor UI to do this, but you can make an edit to the editor script to initialize the script with the same seed. You can also try switching off the Auto Randomize Seed settings if you often have to generate the same image. We are noting this down to be added in the next update.

Yes, we support layering, but with a depth map. Depth maps can not only provide mesh displacement but also layering.

Here is an example of what one can do with a depth map to segregate and separate the input image out into layers:

 

The more I play, the more I like it.
An example of stylized image.

 

Another example - the landscape...

 

maybe you can add a invioce check,only one device with the invioce can be used at the same time,incase this producte get leaked or somethingno checking or account looks dangerous to me

 

and how to upres textures like the blue robot you guys showed,i can only make them bigger but i dont see new details,its only a bit clearer,is there a way to add different styles to a skinned mesh texture like different colors or skin that kind of stuff,

 

Hello simonkingandaoking,

Thank you for your advice. We have taken measures to prevent asset leaks and would appreciate it if you could keep the asset up to date to ensure that it remains secure and protected against piracy.

 

You can use a few techniques to add variations to your skinned mesh textures. Firstly, it's important to note that the blue robot, on the left hand, has only an albedo texture and on the right half, it has a normal map, smoothness map, and high resolution albedo texture generated by Aify. You can run the normal map and the base texture through a pass of Match Image at 0.9 strength or more to add some detail without distorting the map significantly. This can help to create a clearer map and add minor details in the texture. Textures with big UV islands tend to respond better to Match Image than textures with small and compact UV islands. Settings may vary with different skinned mesh textures.

 

Hi, I'm considering buying it. It looks great. I just want to ask if it's possible for this asset to convert the texture I provide to a stylized version? I see the description mentions re-texturing and creation of materials and I wonder if it can convert an object's textures to stylized or not.

Btw the documentation link on the description page doesn't seem to work.

 

Hello ShizofMalkavian,

Thank you for your interest in Ai.Fy - Text to Image. As of now, AI style transfer is not offered in the asset. The closest to style transfer is the Match Image feature which also distorts the "shape" or the "look" or the "basic structure" of the image in the process. However, we plan to roll out art style generators / samplers like pixel art, photorealistic style, anime and game texture style by this month.

We tested out the link for the documentation and it seems to be live and running globally. Please inform us if this is not the case for you. We will send it to you as an attachment.

 

Could you add seamless textures mode, so that they tile perfectly? mid journey just added this too.

 

I see, so will those art style generators / samplers be added to this asset, or will those be in another asset?

 

Yes, hopefully we will be putting out an update for a game texture style transfer by this month. We will also study how midjourney accomplished seamless tiling and hope to integrate that as well.

 

The art style generators / samplers will be added in the same asset.

 

Hi,
I have been experimenting with Ai.Fy and it seem to offer lots of possibilities.

What do you mean by game texture style transfer?
Is this just the art style generators/samplers you mentioned earlier?

I was using Ai.Fy to generate variations on a 2D Tree earlier.
I noticed that that the source image for Match Image seems to function a bit like a mask, defining the shape/area where things are generated.
However, the generated images vary quite a lot in terms of the style they utilise and this is generally independent from the source image.

Being able to choose a particular style might be able to help with this to a degree.
However, I was wondering if it might be feasible to provide another image for a sort of "Match Style" type of operation?
Here the source image would determine the "textural" aspect of the image, if that makes sense, rather than acting as a mask for the image generation.
For example, you could create images with a Pixel Art style or a Cartoon style with a thick outline by passing in an appropriate source image.