Official Introducing Unity Muse and Unity Sentis, AI-powered creativity

There are lawsuit every day on AI. This is just sensational headline. Even if US rules against it, it will not apply on an international level. LLM are just too widespread at this point - there is so many viable open source LLM.

 

I doubt this. The US is super-strong on copyright laws even around the globe. Pesky trade deals.

 

The problem is not AI but the sources to train it. And nobody is going to forbid you from using open source software, but using it on commercial projects can put you in a lot of legal trouble that indie devs don't have the money to face.

 

And how are you going to prove that AI was used in the development process ? While I can see a point for image generation, AI code assist is no different than browsing internet for answer. For us End-Users, we have no risk. The one in legal trouble are those who train the AI. As indie dev, it is the least of your worries.

 

So hey this facial animation thing--when do we get access to this?

 

I'm somewhat disappointed this thread has gone silent from the Unity side after some early participation. There are plenty of unresolved questions here that are worth addressing.

 

hi there ,

been using muse for a couple of days now,

but i think its generating code with text format errors. i dont know if im allowed to share the code its generating or a screenahot so if someone could take a look at it it would be appreciated

by the way i didnt know where else to ask for help so i came here

 

That's just a class action lawsuit. In the US these things are a dime a dozen and are started by lawyers wanting to make a quick buck. We had one just last year for this game engine.

 

Yes, but also if a ruling comes out that you can't just scrape random stuff off the internet, or you need to disclose everything you used, etc etc, it means big trouble for all current models.

Not saying it's super likely to happen, but still.

 

And if they don't just settle out of court which is very common for class action lawsuits in the US as once again it's largely about the money.

"Integrated" sounds like lawyer speak. Has it been trained solely off of that data? Or does it include third party data?

 

These are precedent setting cases.

If you think judges don't understand that simple fact, you're making assumptions on the wrong side of history.

 

Hey I'm a bit confused. Not sure if the info's been mentioned somewhere and I just missed it. How can I get access to Muse (and maybe Sentis)? I signed up for the beta program hoping to be able to test them out but haven't seen or heard anything since then.

 

My wildest fear is Webgl , webgl is single threaded, do really sentis offer better performance than barrcuda 3.0? , at least in term of garbage collection? ... possible to support GPU inference on webgl?

I've applied for beta, please, someone let me put my hand on it, I will give you juicy feedbacks trust me

 

Yea, tbh it's rather dissapointing. It seems like Unity came up with many updates but fail to bring documentation or communications. It's like chaotic jungle out there.

I used muse using my work account, Muse failed to get things quickly. I need to fine tune my prompts every time. Plus, the longer the chat means the laggy it will be on my PC.

It's easier to use free version of ChatGPT at this point. Muse also failed to get the knowledge that exists in the documentation. Claiming that it's not even there. Even when I brought the link, it's just saying thanks but still giving the wrong example.

Unity only said it's close beta, but fail to give the feedback link at the interface and/or giving recurring updates on what to expect on this close beta.

 

that is exactly what i was going to say, the lack of a quick way to post feedback and how somewhat out of context some answers seem to be generated

 

Its not the dev problem, its the management and decisions problem... no one can deny that they have one of best software engineers in the world, better than google if you ask my opinion.... but again, its not the dev decisions to continue dev something or kill a package or any thing that their top management ask them to do

 

I'm curious why you think I would blame only the devs. Yes, the dev has problems... However, which company doesn't have dev problems?

That being said, it doesn't change the fact that the documentation and things is quite chaotic.

 

I'd be more inclined to believe any claims Unity might make about "AI" when they can get Search to provide some semblance of accurate responses in this forum.

 

There is a question. The video shows the creation of 3D animation.
- Is it possible to create animations for 2D characters?
- Will it be possible to link AI to work with Spine2D, for example?

 

Can you give us a release date for the closed beta for MUSE? I signed up the week it was announced, and cannot wait.

 

Is it event possible for a small no-name startup to get just Sentis acces, to migrate from Barracuda ?

 

I think it is closed to public right now. When it will open? And, is animation updates valid for 2D or it is just for 3D?

 

So the silence on this question from Unity would confirm it was trained on copyrighted data and that using it would mean we won't be able to release on Steam, right?

Nice.

 

I was wondering, in the promotional material for Muse, its advertised as being able to help create animations and materials to be used in engine. It seemed like from that, that you could add Muse directly to your game engine (almost like a traditional package).


I was wondering is this currently possible, and if not, is this a planned feature for the future?


Additionally, I wanted to get a more detailed idea of Sentis as well, as its a project that greatly interests me. Is the purpose of this AI Neural Network, primarily to give us the ability to add ChatGPT like NPC's into our projects and games? Also, other than doing something like that, what other potential use-cases could we have for this kind of technology

 

Currently, it is not possible to create animations and materials with Muse. It only answers questions with some google-able info and can generate some basic code/shader code (with varying results of accuracy).

 

Hello, I'm fascinated with the Muse tool as it brings documentation closer to users like me who aren't experts in this field. However, I have a series of comments that could enhance the user experience on the website.

1. On the web, the image layout has two black fades at the top and bottom of the text area. These elements look nice and are clearly aesthetic, but the issue is that they take up a significant amount of screen pixels, reducing the available space for the chat. This wouldn't be a problem if I had a high-resolution screen, but unfortunately, my screen is old and budget-friendly, so it has low resolution. These elements are more of an annoyance than anything else.
   
   
2. Other pages like Chat-GPT allow you to stop the scrolling of the screen while the text is being generated. This allows for comfortable reading of what has been written while the rest is being typed. This functionality is not available in Muse, so I have to wait for the text to finish typing to read it comfortably or get used to the text occasionally jumping. It's not a drastic issue, but having this solution would improve the overall experience.
   

I hope these comments are helpful. If this isn't the right place to share them, please let me know where I should post them. If any Unity Muse web developer happens to see this, I hope this feedback proves useful!

 

Well hope not that they also abandon this project in few years.

 

Any practical example of Sentis, showing for example creatures learning to steal boxes from one another?

 

To be honest, I am not too of hope, if seeing ML package not being touched for a while. While the announcement sounds exciting, I am yet to see some exciting ral use case examples myself.

 

If the US chokes it, then non western countries take the lead on THE money making AI tech of today.
Difficult to imagine the US letting that happen.

 

UK doesn't care, and puts effort, to stop blocking AI development. No matter what will happen, UK seems that will proceed with various AI tech, while rest of world will be fighting with arbitrary problems of AI existence.

 

I have been using Muse since its inception, but now, all of a sudden I get this message:
"Join the closed beta program to create with Muse, and accelerate development of your real-time 3D experiences in ways you never imagined! If you were recently accepted from the waitlist and received this message, please note that it may take up to 2 hours to process your account for access. When signing up for the Muse beta, please use the same email address associated with your Unity ID"

WHY?

 

Even not exciting would be nice, to get a sense of cost/benefits without spending time on it. Their GOAP had meaningful examples and could easily be changed to check for scalability.

I have a feeling that many teams are being jettisoned.

 

I changed my account email and now I lost the beta invitation because it's not longer the same email... Any solution to that problem?

 

Use your previous email.

 

A promo email today had some interesting stuff,



The videos are:
Muse Sprite:

Muse Texture:

 

Did you folks train your own model? What did you feed it?

 

An internet
I would be really surprised, if Unity would go through all Asset Store database effort and try to feed them in.
I can imagine handling these packages and whats not.
Where much easier (and controversial) way is to grab internet data.

 

I'm hoping it's ethically sourced. I don't currently have any use for this stuff (as my active/unannounced/commercial project is 100% AI-free, and I'll be busy with that until at least the end of 2024), but in the future I'd definitely like the option to leverage technology like this with a clear conscience.. and related tools being integrated directly within Unity Editor would be ideal. I'm certainly interested [continually] to know what I can and keep myself up to date on progress.

 

Un-huh. And how are you planning to monetise this feature?

 

Judging by Riccitiello's narrative in the last earnings call, they want to charge by usage. So probably similarly to OpenAI API access, but likely more expensive since this is just GPT 3.5 or 4.0 with some added Unity data on top.

The image gen will be probably similarly priced per X number of generations.

 

Makes sense, I wouldn't use it, my use of generative AI has dropped to zero.

 

It's $30+VAT tax:


Also, there seems to be a behavior tree functionality that's locked behind AI paywall?

 

I use ChatGPT every day to 'pair program' and bounce how to solve some challenges.

Gave Muse Preview a spin asking the same Qs ive given ChatGPT and its giving me worse answers. Eh, ill keep hounding it for the trial and see how it goes...

 

ChatGPT is also general purpose. I can shoot S*** about anything beyond just programming in Unity and I pay 10+ euro less than Unity is asking. Additionally ChatGPT can also search the web, and offer various additional functionality via plugins such as youtube video summarizer, etc. I can also hook into GPT API directly via IDE plugins and some months I'd be way below what Unity is asking.

I do pay another $10/month for Midjourney, but it's not really applicable to gamedev besides maybe some concept art. Steam still bans any use of AI unless you can prove dataset ownership, Unity tools fail this check by default since we have no clue from where the data is sourced. Will be interesting to see how Steam reacts to games with Muse generated sprite art still in the final build or at least part of early access builds.

Overall, I don't see the added value right now but I'll check out the trial unless it's another 12 month contract.

 

Looks like it is generally available now: https://unity.com/products/muse
They sent me a free month for participating in the beta program.

I don't think the current state of Muse is worth the $30/month. The main features right now (chat and sprite) are slower/worse than competitors. In many instances ChatGPT3.5 is better and always faster.. GPT4 is much better especially for coding. MidJourney also produced better images with much less effort. Muse chat performance feels like what you get from Llama2 7B model that was fine tuned on Unity documentation.

The Muse sprite feels like Stable Diffusion 1.5 without any fine tuning or additional settings.. it needs to be able to have different model selection. See how Leonardo.ai allows you to select different models for different styles. Anyways running your own Stable Diffusion plus plugging your own LoRA gives much better results, especially for a game since you have much more control over the style similarity between each generated image.

Lastly Muse chat needs better integration with editor and IDE. See Github copilot chat for an example of great integration. Copilot is integrated in the IDE as auto complete, in-line chat, and also a full chat. This also allows it to have much more context of your code base and doesn't require you to "copy and paste" code to ask questions.

It's a crowded space and $30 is too much for Muse right now. Copilot is $8.33/m, Midjourney is $10/m, GPTPlus is $20/m (which can generate images and textures too). You can also run Stable Diffusion or a Local LLM for FREE and in many cases get better results than with Muse.. especially when it comes to image generation.

I hope this surfaces to someone in the Muse team as this product has potential as it has the opportunity to be tightly integrated with Unity. I would post this in the Muse feedback forum but I can't seem to post there and can only read discussions.

 

tested 4 texture generations and 1 pbr preview,
that costs 203 points(!) for very low quality textures..

also would be quite important to clearly see how much each click costs..

 

Surely that AI has seen Mario images in some training data? (perhaps wikipedia?)

 

I think this is because they only trained the model on images licensed by Unity