Stable Diffusion Unity Integration

I have just released a new Unity package/repo for easy Text-to-Image generation with Stable Diffusion inside the Editor. This for now can generate textures for 3D models (any component with a MeshRenderer) and UI images (Image and RawImage).

Let me know what you think about it: https://github.com/dobrado76/Stable-Diffusion-Unity-Integration


Note: this is a Design only tool that works in the Editor. There is no dependency for Stable Diffusion at Runtime or in a Build.

Feel free to post any other interesting Unity tools related to Stable Diffusion.

 

That's great work. What obstacles did you have to overcome for the Editor version and is their a show stopper as to why this cannot be used at runtime?

 

I guess the main issue is that once the textures are generated, the Editor needs to refresh and import the assets in order to be able to use them properly. It is not the case for material textures, but it is for UI images (which needs to be imported and converted to Sprite (2D and UI) type. I have no idea how or even if that is possible at runtime.

Also, the Stable Diffusion image generation itself is relatively slow and that's something acceptable for artists when designing/creating a game or a level but that would be unsuitable for users in real-time/runtime.

The beauty of open-sourcing the framework is that if someone is willing to tackle these issues, they are free to do so but I am not considering spending time on that issue.

 

This looks like very interesting project indeed, and on the top of the tech time.

You have mentioned in repo, that images are saved for reference. Does that mean they save configuration of SD? Like for example steps, random seed etc? So for example you may want to regenerate same image for some purpose in the future?

Can you confirm, it generates also bump maps? You did mention them. In repo.

Also many thanks for sharing with a community. I bet you going get more attention and contributors.

 

Well, the short answer is that the image metadata will contain the generation parameters, like any other generated image in Stable Diffusion, and visible with any appropriate tool (or even with a simple text editor) in the ".png" file.

But the input parameters are set per GameObject so unless you generate several images for the same GameObject (which will be overwritten with the last generation for that object) you may or not have the saved values in the Unity Inspector.
Each game objects have their own GUID (unique ID) which I use for filename reference as "_GUID_.png". Note here the input seed (-1) and the generated seed fields are separate, so to keep a record of what seed was used for the generation.

The bump maps are calculated algorithmically from the generated image, by a custom function in Unity. They are not generated by Stable Diffusion. It's not perfect but it's a good starting point without effort.

 

Thanks that a really cool plugin you made, I been testing stable diffusion with dream textures in blender, and they are using an depth model of stable diffusion for texture projection, would that be something that can be done in Unity too?

https://github.com/carson-katri/dream-textures

 

I am just starting with this Unity project. So it is still very basic and limited but I will try to extend it a lot if I can.

As far as what is possible from the Unity Editor, the soft limit will probably be what I am allowed to do using the Stable Diffusion API interface. However, since the assumption is a working local Stable Diffusion server, it is safe to assume that there will always have the possibility of calling Python scripts directly, therefore there won't be any hard limit.

Thanks for the link, I didn't know about that project. Worth to have a look at it.

 

Yea that blender project also has the option to run locally after downloading the model

Dream Texture time-lapse in Blender - YouTube

 

I have tried this stable diffusion package but unity continuously gives me this error when I click on List Models button"
The remote server returned an error: (404) Not Found.
UnityEngine.Debug:LogError (object)
StableDiffusionConfiguration/<ListModelsAsync>d__5:MoveNext () (at Assets/Stable-Diffusion-Unity-Integration/Scripts/StableDiffusionConfiguration.cs:85)
"
I think the models Api url is not correct. I don't know how to get the correct url address to fetch the data from the models. Please help.

 

Have you read the part where you need to have installed the Stable Diffusion server on your PC? It does not connect to an online remote web server. And you have to adjust the address otherwise and I have not tested connecting to a remote server.

 

Yes. I have installed the stable diffusion on my system and it is running as expected. In fact, If i don't run the command, unity gives me some connection error. But when I run the command, it fetches the data from the local server stable diffusion site. What I think that the models api address is not valid and it gives the error which I have mentioned above.

 

You might not be running your server to accept API commands? You need to add "--api" in the COMMANDLINE_ARGS (your webui-user.bat startup file). And start the server with that batch file, not the webui.bat

 

Nice work! I'm trying to use your package at run time in my unity project to generate material textures. I do it simply by pressing a button to call the Generate() function in StableDiffusionText2Material script. And it always works perfectly at the first time when I enter play mode, but at the second time it always becomes extremely slow and finally gets error "the operation has timed out". What could it possibly because of? I'm new to unity and it's hard for me to largely modify your codebase.

 

Good question. The tool was initially implemented for Scene Design, not runtime. I have not tried it and I have no time for debugging potential issues of runtime generation. The main thing with runtime is that when you stop, Unity does not save the components fields change done during runtime. Any generation done at runtime will still show (materials are assets, not components) but the value of fields (prompt, cfg scale, seed, etc.) that were modified at runtime will be lost.
Since there is nothing I can do about that. I have made the decision to not spend time supporting generation at runtime.

 

Can these valuers be stored to a ScriptableObject and at runtime write a JSON object to parse into the SO in editor?