Unity AI

Blog Post: https://blog.unity.com/engine-platform/introducing-unity-muse-and-unity-sentis-ai

There were some teasers about this in the past but now there's some actual footage of the tools. Looks interesting but doesn't seem to do anything different than existing popular AI tools out there. Does anyone already use this and have more details?

 

I haven't seen inpainting for sprites with transparent backgrounds or transparent background sprite generation in general. But I also haven't really looked for any AI tools like that. Maybe it already exists. Seems like a cool way to prototype assuming it's not prohibitively priced. CEO strongly nodded that they'll be priced by usage according to the earnings call earlier this year.

 

I'm sure the devs worked very hard on these tools, and I don't want to S*** on them, but the results don't look great. Should these terrain textures that look like they were pulled from a 90s game be what they showcase?



The talking alien showcase felt kinda lifeless. If I were to make a realistic-looking game in Unity I wouldn't settle for the quality of content that was generated by Sentis.

 

Yeah exactly, that's why I wonder if anyone here has used it to give some info, but they are probably not allowed to talk about their experiences in the closed beta yet.

For example, the "chat" demo they have looks similar to ChatGPT (except trained on up to date unity docs and resources) but in terms of productivity I've found AI Code Tools such as Github Copilot (or Copilot-X) to be much more useful. Being able to ask specific questions about Unity and documentation certainly is useful though but that doesn't feel viable as a paid product (which I assume this will be)

In regards to art generation, Stable Diffusion is really powerful once you get into all the extensions and tools out there. What was shown in the demo seems very basic and unless they do a lot of prompt juicing behind the scenes to help users get what they want I think it will be just giving us those stale looking sprites and textures. MidJourney for example does so much prompt tuning behind the scenes that even really simple prompts from users can generate "good" looking images. But if you take a basic generator without any tuning then it gives images like in the unity demo (looks like something you can make in paint)

There's some game specific tooling out there such as https://layer.ai/
They are more tailored for game assets as it can do transparency, pixel perfect, etc. It supports generating assets from traces/sketches and also has in-painting directly in the tool.

their demo (couldn't figure out how to embed it):


https://res.cloudinary.com/layerai/video/upload/v1674225762/layer_hero_video_fzo5qn.mp4

Edit: just noticed unity lists Layer.ai as a solutions partner in their blog post

 

Isn't this just post IPO Unity jumping on the next biggest bandwagon?

Muse doesn't even seem to be anywhere near where it's at to ask ChatGPT for Unity assistance.

 

As others mentioned, image generation looks bad.

The poor quality of the textures may be due to the SD 1.5, which creates 512x512 images. Additionally, the generator is limited to creating only albedo textures.

Inpainting ignores the original style of the sprite, they haven't implemented Controlnet into their product.

We also don't know all the quirks this tool might require. With SD, you need to learn artists and how to compose them to get a good style.

I'm not impressed, feels like a rushed product.

 

Unity invested in AI developmet years before recent LLM hype, but I'm not sure if any of this is related to that.

 

And by "invested" we really mean "bought companies that looked shiny".

 

As far as I'm aware, Barracuda was developed in-house: https://github.com/Unity-Technologies/barracuda-release Earliest public release is from 4 years ago when no one gave a S*** about AI.

The question is if Barracuda is at all relevant at this point.

EDIT: ChatGPT4's opinion after searching the web via plugin:

"Unity Barracuda, as of 2023, seems to remain relevant in the context of large language models (LLMs) like ChatGPT, particularly for the gaming industry. Unity is focusing on deploying generative AI in games, which they believe will drive a significant transformation in the gaming experience1. With Project Barracuda, Unity aims to enable AI models to run locally on a device, which could help scale AI character implementation in games without incurring substantial costs1.

The use of generative AI, such as LLMs, in game development has the potential to enhance the richness and believability of non-player characters (NPCs) and the overall gaming world. This could result in unique game experiences with persistent and evolving contexts and characters for each player1. Unity's CEO, John Riccitiello, believes that game designers and developers could be up to 10 times more productive using AI, and while AI won't take anyone's job, those who utilize AI in their work may outperform those who don't1.

However, it's important to note that the capabilities of LLMs in game creation are not without their limitations. While they may eventually produce simple games, creating complex and compelling games is likely to remain a challenge. This is due to the dynamic nature of games, where interactions between characters can change the gameplay in real time, something that is fundamentally different from the types of content LLMs typically study and learn from1.

In parallel, advancements in integrating LLMs with development tools are evident. For instance, JetBrains has introduced AI features powered by LLMs in its Rider IDE for .NET development. These features allow developers to interact with the LLM to ask questions or iterate on a task, generate commit messages, and even generate files for a Unity solution written to developers' specifications2.

Given the rapid advancement of AI, it's challenging to predict its future capabilities. If AI continues to be integrated into game creation and operation engines, it might be capable of learning from the millions of games it helps create and operate, potentially leading to further advancements and improvements1. It's also worth considering the potential economic impact. If generative AI facilitates significant growth in the gaming industry, it could represent a substantial proportion of global value creation in the coming years1.

 

The more that's revealed about this AI stuff, the more it just looks like they plugged some ChatGPT outputs into a few basic inputs for some of the most mundane aspects of game development. We'll be able to set some color inputs, and get some bizzare AI generated character dialogue that's completely isolated from in game mechanics. A lot of fuss to integrate a bit of ChatGPT into Unity and marry the usual overhyped Houdini style modularity with some very narrow AI based points of input kneecapping both systems that already show little promise in creating great games.

Nothing here looks groundbreaking in the slightest.

I'm just hoping that in cleaning up the code to be utilized by AI, they inadvertently make it easier for users to work with the various systems.

 

Dare we dream...

I'm more worried they'll come to rely on the ChatGPT client they host in the Editor to save them from ever writing more documentation, or editing it to improve reference materials and better explain feature functionality and mechanisms. etc.

 

ChatGPT can't produce animation. Stable diffusion out of the box doesn't really produce materials.

 

Alot to unpack from the video!

Some interesting tools there; if the data sets trained on are legal and ethical and the outputs are decent-enough I can see myself using AI Animation generator alot for placeholders and blend fills and also the sprite generator/variant creator for also placeholders and background art. Would love to get on the beta to try them out.

What I find hilarious though is that the example for Unity Muse for 'How do I make a match-3 game' is exactly how you should NOT make a match-3 game; the combination of 2D Rigidbodies on cell tiles of the Tilemap system doesn't make any technical sense. This part was definitely not fact-checked by anyone who has used these Unity tools. I asked the same question to ChatGPT and it gave me a vague but fairly solid outline which is more aligned with reality and doesn't set you down a path of frustrations.

Also, the Unity 3.0 era terrain demo for 'Realistic Textures'.... Big yikes.

 

The marketing materials state they won't generate entire materials but textures to use in unity.


Second thing, base stable diffusion can generate textures. https://pixela.ai/

 

Shows it might be true, their claim that Muse is trained on the Unity documentation. The samples and examples throughout the Unity docs always use the least efficient way of doing anything. As did Brackey's vids.

 

I don't think its fair to mention Brackeys; he was an independent youtuber and his videos were bitesize focuses on beginner & intro topics, and weren't advertised as 'most efficient'. For what his channel aimed to do and who it was for; hit the mark perfectly.

Unity is a public corp with 5k employees and $billions so has a duty, scale and resources to provide the best standards and workflows for creating something with it.

 

Aren't they using Barium AI for pbr texture generation which they acquired earlier this year?

 

Barium AI is based on stable diffusion with some custom scripts and obviously a custom model.
https://forum.unity.com/threads/uni...-pbr-material-generator.1392916/#post-8905251

 

We're each responsible for the lies we tell and the truths we hide.

 

#deep

 

Yeah that's right, if it can make good animations for real that would be nice

 

It's not an easy problem to solve. We've been working on AI animation generation for the past ~3 years and are finally getting close to releasing a beta. We are able to generate animations that you can drop into your project without any cleanup. There has been a ton of research in order to get there. You can see one of our models generating a run animation here.

Regardless I'm interested in trying out Unity's animation generation solution. If anything it should be great for prototyping.

 

The stock already jumped 20% and I can't see any news other than this AI stuff. It serves as a good reminder that we are in a total hype. Mention AI and people throw their money at you. Take a closer look and those tools are quite underwhelming. Texture looks like crap while there are already free tools out there for a while that will make entire PBR materials. The animation looks an awful lot like a call to Mixamo.
I'd take Unreal's procedural modelling over any of this Unity AI stuff. I hope to be proven otherwise and theses tools are something we haven't seen on free websites a year ago.

 

Those generated textures are awful, especially when you can get pretty massive PBR packs for free already and that's what they were demonstrating in the moment.

 

My only concern/hesitation to being excited are the unknowns around pricing - I've been using a Personal license since 2019 but will be subscribing to Plus in the near future. It would be nice (if these new tools aren't free for everyone, which I currently doubt they will be) that Plus gets some benefits aside from the splash screen customization. However, if it's for Pro licensees only or requires costly subscriptions, then personally, it's out of reach / useless / I'm better off ignoring it. That said, I'm a hobbyist who has to make sacrifices just to get the occasional assets I really want (or that go on sale.. because I can be impulsive), but I understand that for pros or anyone else with plenty of money to burn on game dev it's a completely different perspective. For someone not blocked financially, just the fact these tools will exist at all would be exciting [for many].

For example, looking at the Layer.AI asset on the store, watching the videos ..as a coder, to me that looks incredibly useful to create mobile, 2D or simple games (especially if you don't know any artists with free time on their hands that you'd want to collaborate with), and it can be used for free, but you'd probably still decide to subscribe to a plan ($30/month being the cheapest on offer). I expect Unity will have to charge something as well, I mean maybe they can let 1,000,000 excited users access these new tools for free / just make them an integrated part of the Unity Editor experience for all.. but I don't know... supporting that kind of demand on hardware/servers would be costly.

 

Probably not real time. I think it is like stable diffusion for animation.

 

CEO strongly suggested the new AI services will be charged by usage. It remains to be seen if it'll materialize as multiple subscription tiers or some amount of $$$ per API call. They also definitely won't limit these to Pro. They want this to be accessible as much as possible.

It'd be interesting to see them add these services to Plus and above, but I highly doubt that'll happen. I fully expect these to be additional subscriptions.

 

I think for animation, tools like Cascadeur is the way to go. You still hand author them, but they have physics simulations and machine learning to help with secondary animations and getting to plausible poses quickly.

I would love it if this is the way the industry went and more tools like that would appear, but instead it seems we're going the "press a button to have a subpar animation you can't quite fine tune suddenly appear" route.

 

Exactly! CATS plus AI assistance is the kind of ideal.

Same with code: "Here's my attempt, how can this be done better, with both more design flexibility and performance?" prompting an array of suggestions and answers along with explanations of why it's better, and possibly even quick creations of test samples with timers to demonstrate this, and for the AI to learn from when it bullshits and hallucinates things into being that just aren't so.

 

There're rumors on twitter that Valve now ban games using AI tools (both art and text) if you can't prove you own the rights to the dataset.

Source 1
Source 2

 

Interesting, but does not show the games in question. Probably someone tried to submit a Mario clone or similar since it was apparently clear who the 3rd party owner is.
If you look at stuff for example by that "Layers" plugin (what a nonsensical, unremarkable name btw.), there's hardly a way Valve could tell that's AI generated and impossible to find the original data owner.
Similar with Unity's offer, I'd say.

 

I agree to an extent. What I'm concerned about is getting falsely flagged and then having to prove some kind of ownership of assets that might've come from contractors. There will inevitably be games that get banned unjustly due to some faulty AI detection system and/or due to human error. Steam's submission process' human reviewers have a long history of bullshit bans.

EDIT: Also, the story is about the use of AI tools in general, not straight up copyrighted material.

 

But those terrain textures are deeply embarrassing. There have been easy to use generative texture tools for years, not AI, but tools that have node based layouts for full PBR textures from expensive - substance - to much cheaper - Filter Forge etc. AI is only relevant if can do just as well AND require minimal work AND not rely on questionable data sets

 

According to the links, they don't ban games using AI but require a license for all art in their model.

Unity might be able to prepare such license. Probably only stable diffusion and mid-journey won't be allowed.

 

I think both of those sources may have been referencing the same game, and the reddit thread linked in one of them mentions that the game was uploaded with placeholder artwork that was very clearly AI generated. The developer mentioned they planned to replace the art before release but I suspect Valve believed otherwise.

https://www.reddit.com/r/aigamedev/comments/142j3yt/valve_is_not_willing_to_publish_games_with_ai/

One of the comments that was downvoted struck me as interesting:

 

Here's another thread from today: https://www.reddit.com/r/DefendingA...pressed_because_i_cant_use_ai_anymore_due_to/

And Jake Birket unofficially confirmed this is happening according to a source he trusts. Info about AI text also getting hit came from his source.

 

Technically, you don't own license to Firefly's training dataset either, just the output of it.

 

I'm pretty sure I don't own assets bought on the asset store as well. And I don't own games bought on Steam.

But it's not about ownership, but a copyright law.

 

In that case Midjourney also gives me copyright for commercial purposes. The issue seems to be centered on the training dataset ownership, not the copyright of the final output. I guess we need to wait for official announcement if this is true.

 

How sure are you that Unity's AI stuff isn't really a souped up stable diffusion model, when Unity, in their efforts towards being "transparent" and "ethical", decided they will not disclose anything?

 

Their "verified" service Layer.ai is 100% Stable Diffusion:



I'm fairly sure all sprite generation in the trailer is done with Layer.ai

 

Outside of the bounds of this conversation I have no idea who that is and neither does ChatGPT.

Spoiler

 

Therefore... HE DOESN'T EXIST! (cue x-files theme)

(but also his last name is Birkett with 2 "t"s)

 

Okay, now ChatGPT recognizes him but I still have no idea who he is.

 

He's the 11 years without a hit GDC guy:

And he's founded a couple of large gamedev meetup groups in Canada and UK. Decently well known in the indie space.

 

This nice fellow:
[Edit: Youtube embed redacted because it's the same as in this previous post]

(Apparently Google search is still safe from the ChatGPT garbage as well )

 

If the news about Steam requiring a license for images used for generating AI is true, then Unity must provide a correct license. Otherwise, no one can use their tools for PC games (and in the future, possibly in commercial games).

 

For Unity to be able to provide such a license they must obtain it first from their data sources, which already excludes (almost?) all existing models that are out there.

 

Wouldn't this also imply access to all images used in the training dataset? I don't see that happening for any LLM. Unity claim their training dataset is above board but I highly doubt it. (EDIT: Muse is accessing GPT4 API with some extra training data btw. And GPT4 dataset is also scraped from the web. End user does not have a license for GPTs training dataset.) Same with Firefly from which you couldn't realistically opt out. And most are based on Stable Diffusion, anyway. A tuned SD is still SD and Stable Diffussion is already being actively sued.

I think what Valve are saying is - don't spam Steam with AI games made with assets of dubious copyright. You can only use 100% custom trained LLMs for which you can prove the ownership of the training dataset in full. I'm not that well versed in training from scratch, but I believe this is only viable in AAA space from what I know. Correct me if I'm wrong.

EDIT #2 @JussiKemppainen snitched on their AI hobby game for science.

 

No, you don't need access to the entire dataset, similarly, you don't need the source code of Unity to release a game on Steam. Adobe claims Firefly is trained only on legally acquired images.

The issue with generative AI is that it takes someone's pictures (without asking), copies some pixels from it based on a mathematical equation, and then reuses that part of someone's picture to generate new images. You can't sell games with Hollywood actors in them, even If they were generated with AI. And you can't be sure that your custom face of a non-existent person isn't partially made out of a Hollywood actor. Unless you have a correct license that every picture is acquired legally, allows for generating new images, and that these images can be used commercially.

And yes, checking whether someone is using Stable Diffusion might be near impossible unless it's too obvious. Some algorithms can detect whether the image is generated with AI, though it is possible to avoid detection with rescaling and some post-processing effects.