[Generative AI] Shap-E - Text To 3D

Shap-E, a conditional generative model for 3D assets, is developed by OpenAI. Shap-E - Text To 3D for Unity is exclusively offered by AiKodex in order to make this technology easily accessible to the Unity community.

Link to the Asset : Shap-E - Text-To-3D

Shap-E works in real-time, both in, Edit Mode or Play Mode inside of the Unity Editor. This asset has a one-click, beginner friendly GUI and does not require any coding to use.

You can read the updated documentation here.

Shap-E - Text To 3D features:

Text to Mesh AI Model: The main function of the asset is to generate 3D assets from texts. Simply enter the text to be converted into a 3D asset and click on generate.

⚡ Fast inferences: Models typically take 8-20 seconds to generate. This depends on the traffic. It may take lesser or more amount of time to generate the model depending on number of refinement steps, CFG and model format.

Formats: Generated models are offered in FBX, BLEND and GLB ensuring the developer has access to all avenues - desktop, web, mobile, VR and other platforms. Models exported to blend can be further changed into types like OBJ.

Mesh Decimation: Reduce the complexity of generated 3D models while maintaining their overall shape and structure. The asset can help remove unnecessary vertices, edges, and faces, optimizing the model's performance and resource efficiency. The mesh generated is clean quads so retopolgizing is possible in an external 3D application.

Mesh AutoLOD: The Automatic Level of Detail (AutoLOD) feature enhances efficiency by automatically generating multiple versions of the same model at varying levels of detail. This ensures that the generated 3D assets are optimized for different viewing distances and platform capabilities.

Mesh Painting: Add vibrant colors to your 3D models using the intuitive Model Painting feature. Easily apply colors to individual vertices, edges, or faces, giving your models a distinct and appealing visual aesthetic. Works by painting the mesh in the scene view.

Vertex Colors: In contrast to traditional UV mapping and texturing, the generated 3D models come with vertex colors. The color information is directly associated with each vertex, eliminating the need for complex texture mapping. The result is a more lightweight model that retains its aesthetic appeal without relying on external image textures.

Shap-E - Text To 3D Quota:

2,000 models per month can be generated with Shap-E. This limit is likely to increase in the months after launch.

We will be offering this service for one year (till 01-10-2024). OpenAI plans to develop this technology further. The date of service will be extended if there is improvement and widespread support and adaption of the technology which is highly likely.

Shap-E - Text To 3D Editor:

Keeping it all in the editor: Keeping all assets in one workspace inside the Editor and having to switch to fewer services can have several benefits, such as:

• -Improved Efficiency: When all assets are located in one workspace, it becomes easier to access and manage them. Users do not have to spend time switching between different services or applications, which can be time-consuming and lead to a loss of productivity.
• -Streamlined Workflow: Having all assets in one workspace can help create a more streamlined workflow. This is because users can easily move between different assets, such as code files, images, and documents, without having to navigate between different services. This can help to speed up the development process and make it more efficient.
• Reduced Complexity: Using fewer services can help to reduce the complexity of the development process.

Shap-E - Text To 3D dependencies

This tool requires the Editor Coroutines package from the package manager and an active internet connection.

Shap-E - Text To 3D limitations

As this tool is still under development by OpenAI, a few limitations exist:

• The primary limitation relates to the quality of models and how they appear. Generations sometimes deviate from the intended prompt. Given the technology's relative novelty, and with the backing of OpenAI, improvements in raw quality are expected. The generated mesh has quads and retopologizing can be done in an external 3D application.
• Instead of textures and UV mapping, the object uses vertex colors. In some cases, this can enhance performance, but you will not reduce polygons and have the same visual quality of textures held in place by UV maps. By default, vertex colors are not displayed on the model in any of the SRPs and Built-in Unity pipeline, but they can be easily displayed and used using the built-in particle shader, and the vertex Color node in HDRP.
• Substantial computational resources are required to operate and set up the service. Consequently, there may be periods of downtime. We will maintain backup servers to promptly address any issues that may arise and address this in here.

 

Hi, the docs has been deleted

 

Hello miracletechteam1,

We made a few changes just before release. We have updated the documentation. It should be available now.

 

where is the asset available? (didnt find any links in docs or post..)

 

Can't find this on the Asset store, please give us a link.

 

hello
this does not seem to be on the asset store
how can we purchase it?
i have need of furniture meshes

 

@mgear @unitygnoob008 @darnoldi @miracletechteam1
We're glad to see the excitement and sorry for the delay. We hope to launch this asset on the 7th of October. Stay tuned!

 

yes good news

 

It seems that there was yet another delay. The asset was duly launched on the 7th but had to be deprecated immediately due to a technical glitch. The asset will hopefully launch next week. We will inform everyone awaiting its launch on the forums.

 

Hello everyone,

We are pleased to announce that we have officially released Shap-E - Text-to-3D on the Unity Asset Store!
@mgear @unitygnoob008 @darnoldi @miracletechteam1

 

Cool!
Can we have the link? we can't find it on Unity Asset Store currently.

 

https://assetstore.unity.com/packages/tools/generative-ai/shap-e-text-to-3d-267180

 

We have updated the post as well as the website with the link now

 

hello, I just found this website:
https://app.masterpiecex.com

I was wondering, if your own model is trained on a royalty free mesh (CC0), assuming you have access to your own model's version of training data, it could be possible to increase the quality of output similar to this website, but you would know way more than me on AI model-training.

Right now your model lacks understanding of generalized topology, it seems to cube ("modern day 'voxel' ") flood-fill a space to get a general shape or outline, with many triangles, this is fine as it is nascent technology, however, I am seeing how other similar program operate, I think maybe it should have this free base meshes to flood fill in a controlled manner during its iteration phase, which it can then more accurately build a prompt based on understanding that certain base shapes always have reasonable topology, then additions, edits, deletions, modifications.

just some thoughts.

 

Will this do the rigging for humans and animals ? or is this just for static objects ?

 

Hello unitygnoob008,

Thank you for linking this interesting website. If the models on this website are AI generated, it looks very impressive. Perhaps we can contact them to know more / collaborate to improve the existing model.

Also we appreciate the time you've taken out share your thoughts on how we can improve the existing model. But please note that the model checkpoints have been ported from OpenAI's Shap-E (earlier version : Point-E) and they have used State-of-the-art methods to drive generation of the model on tokens. We are still looking to finetune models and just like LoRAs for Text-To-Image models, we will have Text-To-3D LoRAs as well.

 

Hello Mark_01,

As for now, this model does not auto-rig any of the models. It is strictly used for the auto-generation of static 3D models.

 

Thank you ;-) most world object in a game are static. thank you.