Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Official Introducing Unity Muse and Unity Sentis, AI-powered creativity

Discussion in 'Announcements' started by LeonhardP, Jun 26, 2023.

  1. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,127
    AI-BetaBlogHeader-1230x410.png

    Today we’re announcing two new AI products: Unity Muse, an expansive platform for AI-driven assistance during creation, and Unity Sentis, which enables you to embed neural networks in your builds, enabling previously unimaginable real-time experiences.

    Whether you’re just getting started or already a Unity expert, Unity Muse and Unity Sentis are here to help you open up new avenues of creativity and innovation. We’re committed to working closely with you to explore, validate, and define our AI offerings, which is why both will be available via closed beta at this time.

    Read our blog post for more information
     
    M_MG_S, Enki_Y, DragonCoder and 10 others like this.
  2. JohanDking

    JohanDking

    Joined:
    Nov 7, 2021
    Posts:
    1
    This sounds really awesome

    I'm just wondering though since its going to be a closed beta, will there be issues if I use the tools for the game I'm currently developing when muse AI starts out.

    Other than that I can't wait for it
     
  3. Z_Kir

    Z_Kir

    Joined:
    Apr 9, 2017
    Posts:
    12
    Sounds good. Really gamechanger. But we can talk about it only after release.
     
  4. sbgmr

    sbgmr

    Joined:
    Apr 25, 2019
    Posts:
    1
    This should be very interesting. I look forward to seeing more. What makes this special isn't really the AI, but the Unity ecosystem working with it to include it in workflows. Hopefully this will improve productivity!
     
    andreiagmu likes this.
  5. Junior_Djjr

    Junior_Djjr

    Joined:
    Nov 3, 2017
    Posts:
    11
    Since the beginning of the year I was thinking of using something similar to GPT for my open world game, but the good solutions ran in the cloud and there were API costs, not good thing for immersive games. Unity Sentis is exactly what I was waiting for!

    I hope it has some good and easy training, something like Character.AI.
     
    brackkevin, andreiagmu and Sonicay like this.
  6. Pk125

    Pk125

    Joined:
    Jul 18, 2015
    Posts:
    2
    I'm all signed up! I'm in the beginning prototype phase for a casual family-friendly VR multiplayer game I'm making by myself and launching in 2 years with many more years of updates.
     
    andreiagmu and Sonicay like this.
  7. bart_artanim

    bart_artanim

    Joined:
    Jul 18, 2018
    Posts:
    19
    Can we find somewhere on what data Muse was trained? Or at least get the guarantee it's trained on legally unproblematic data? With how the AI landscape is rapidly evolving, and laws will likely follow with some delay, it would be good to know any generated data is always going to be in the clear if we end up using it in our applications.
     
  8. cloverme

    cloverme

    Joined:
    Apr 6, 2018
    Posts:
    189
    I'm assuming that the alien looking character from the Unity Sentis demo is an AI 2D animated model... similar to D-ID generated content, is that right or?
     
  9. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,810
    Great question! So, you cannot use it for a current commercial or external project. The beta is more meant to help test out and identify any issues.
     
  10. hibbygames

    hibbygames

    Joined:
    Mar 19, 2015
    Posts:
    48
    Very cool. I signed up a while ago, I want to play with anything AI related
     
    Sonicay likes this.
  11. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,810
    To create Unity Muse Chat, we licensed third party LLMs and integrated them with first-party Unity technical documentation (including manuals, release notes, tutorials, scripting guides, etc.) so creators can get the most relevant and up-to-date information.
     
    IMD and jfatroll like this.
  12. TH3_FHD

    TH3_FHD

    Joined:
    Sep 24, 2022
    Posts:
    1
    I came to hear about lots of concerns regarding the contents generated by AI being used in game development, do we need to provide addtional layer of licensing or anything of that sort if we are using sentis or muse for generating in game contents like materials or characters.
     
    Threeli, andreiagmu and jfatroll like this.
  13. LegacySystem

    LegacySystem

    Joined:
    Jan 28, 2021
    Posts:
    52
    In the video only 2D sprite, texture and animation generations are mentioned. I have couple questions.
    1. Will the animation generation only restricted for humanoid animations or will it include generic animation ?
    2. Do you also plan to work on model generation, particles, shaders etc. ?
    3. What is the underlying technology? GPT-4 ?
     
    lclemens and andreiagmu like this.
  14. bart_artanim

    bart_artanim

    Joined:
    Jul 18, 2018
    Posts:
    19
    I would assume that applies to the chat-based app/game development instructions. Does this extend to the other functionality you show such as character animation as well as the sprite and texture generation? Is that all based on first-party Unity content or for-purpose source assets from other partners?
     
    alsharefeeee and andreiagmu like this.
  15. jfatroll

    jfatroll

    Joined:
    Feb 22, 2018
    Posts:
    1
    Great job Unity! I'm so excited for Sentis to use with NPC conversations!
     
    andreiagmu and TreyK-47 like this.
  16. Paul-van-der-Laan

    Paul-van-der-Laan

    Joined:
    Oct 12, 2013
    Posts:
    57
    Which tool has been used for the facial animation of the alien head and does it automatically generate exact phoneme to viseme mapping?
     
    yoonitee likes this.
  17. Stealcase

    Stealcase

    Joined:
    Feb 26, 2018
    Posts:
    11
    Okay, but how does this answer square with your commitments to be transparent? This is from your AI guidelines:

    You are being Opaque while pretending to be Transparent.
    For Clarity:
    1. What Models are you using?
    2. What datasets were THEY trained on?
    If you only know the answer to Question 1, that's STILL useful.

    Until then, I'm just going to assume that
    • Muse is a ChatGPT wrapper
    • your facial animation system is just an Nvidia Omniverse Audio2Face wrapper
    • Your "Texture Generator" or "inpainting tool" is just a Stable Diffusion wrapper
    Which means they're all trained on dirty datasets containing millions of Copyright Infringing works.
     
  18. Seto

    Seto

    Joined:
    Oct 10, 2010
    Posts:
    243
    Hi. What's the difference between Barracuda and Sentis? Is Sentis just a rebrand of Barracuda? Barracuda is already an on-device AI model inference solution.
     
    Deleted User likes this.
  19. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,526
    My best guess is if using copyrighted works in a data set without explicit permission is deemed illegal, most AI tools will disappear (including Unity's), which is something to think about before basing your game and workflow around those tools.
     
    Threeli likes this.
  20. larskroll

    larskroll

    Joined:
    Dec 17, 2013
    Posts:
    51
    Exciting times. I've been using Unity since it was Mac-only, and this development is one of the most exciting ones I have seen in a long time. I'm particularly thrilled that there are capabilities for using AI in the executables, and that there seems to be a commitment for webGL support. Like others asking above, I am very interested in hearing about which training sets have been used to train the LLMs, since I also believe this is crucial for legal reasons.
    I have also applied for the beta, and hope to get in.
     
  21. Sterner

    Sterner

    Joined:
    Nov 11, 2012
    Posts:
    33
    Behavior AI is mention in AI section in asset store (https://assetstore.unity.com/ai-hub), but I couldn’t find any new AI-based assets related to it. Is it coming or just mentioned?
     
    Last edited: Jun 28, 2023
  22. takahashigy

    takahashigy

    Joined:
    Jun 28, 2023
    Posts:
    1
    I've already applied, how can I use or download the latest MUSE?
     
  23. vancura

    vancura

    Joined:
    Apr 8, 2010
    Posts:
    3
    From the original page:

    upload_2023-6-28_14-34-16.png

    I mean... Seriously, why? The person responsible for this image should revoke their design license :)
     
  24. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,526
    Wait, Atlas is on there? The “AI” that straight up searches for stuff on Sketchfab and then “generates” the models by… just downloading them from sketchfab?
     
    Tinetine9 likes this.
  25. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,810
    I can share some clarity on that. Unity Sentis is the new platform we announced today. Sentis is AI at Runtime, and version 4.0 of Project Barracuda (code name) is part of Sentis. Project Barracuda refers to the inference engine that is leveraged in Sentis. There has been a complete rewrite done to the inference engine to increase performance, enable extensibility to the latest models, and offer wider platform support.
     
    OmarVector, kankane and Wattosan like this.
  26. EricFFG

    EricFFG

    Joined:
    May 10, 2021
    Posts:
    182
    That stuff is a very bad look

    "Build confidently with Verified Solutions"
     
    Last edited: Jun 28, 2023
    TerraUnity and angrypenguin like this.
  27. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,810
    Didn't want to jump to reply to this before I had more info, but they have now been removed from the Asset Store for violating the Terms and Conditions.

    We are actively reviewing all verified solutions providers in the AI category as we remain committed to upholding the quality of our ecosystem.
     
  28. Threeli

    Threeli

    Joined:
    Jul 9, 2012
    Posts:
    15
    Basically my thoughts as well. Not a great choice overall.
     
    alsharefeeee likes this.
  29. So, which is it:

    - The whole "verified solutions" system is a scam, nobody verifies nothing and Unity actually just grifts for more subscription money regardless the garbage sold in this scheme
    - Unity actually thought this garbage is good enough and people just will let it slide?

    I know, we're eagerly waiting for legal to word the least damning bullshit about the situation...
     
    Deleted User, Tinetine9 and t-ley like this.
  30. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,526
    There is this: https://twitter.com/TheMirzaBeig/status/1673986872857567232

    I don't know the legitimacy, but it seems plausible.

    It seems they just do super basic checks in the style of "does this S*** function and does not explode the computer in the 5 minutes of testing we did".

    The weird part is that early in the document it sort of seems like they kinda understand what it does ("retrieving permissive license 3D models")
    upload_2023-6-29_4-14-49.png

    And yet they conclude that it's cutting edge, generative AI.
    upload_2023-6-29_4-18-8.png
     
  31. MirzaBeig

    MirzaBeig

    Joined:
    Dec 27, 2014
    Posts:
    602
    This is from the publisher forums.
    That QA report wasn't written by Unity, but it made it past their QA and Verified Solutions.

    I've posted an update.



    upload_2023-6-29_1-5-1.png
    ...
    upload_2023-6-29_1-2-45.png
    ...


    upload_2023-6-29_1-2-14.png
     
    Last edited: Jun 29, 2023
  32. ttermeer-reboundcg

    ttermeer-reboundcg

    Joined:
    Jul 12, 2017
    Posts:
    62
    I hate how all those verified solutions are marked as free in the asset store but all have hidden cost with some not even displaying their pricing on their website. They also do not specify what data is collected when you use their services.

    I am however very excited about Muse
     
    Last edited: Jun 29, 2023
    atomicjoe and Unifikation like this.
  33. bart_artanim

    bart_artanim

    Joined:
    Jul 18, 2018
    Posts:
    19
    While I welcome the responses we have received so far (thank you @TreyK-47) the responses that were given are all still very vague and non-committal. If any of the proposed tools are something we (me and my team) are going to integrate into our asset creation pipelines, we must have guarantees that everything is above board. This is not some sort of "gotcha", but a simple requirement for having any of the enthusiastic people on teams like mine to use any such functionality.

    I don't get the impression Unity realizes that being a responsible citizen in this domain is something where they could potentially shine, if it were up front and above board about the origins of its technology and in particular the data used in training.

    I know there have been various employees who have stated on a range of platforms that training included in-house data, or ethically source data (which I'm happy to hear), but this should not require an Easter egg hunt to figure out.

    I would welcome clear documentation from Unity (the entity, not random employees somewhere), available to all, which states such things as
    - Which third-party LLMs were at the origin of any LLM-based functionality?
    - For any module generating assets, be they animations, sprites, etc, what if any third-party models were used as a basis, and what steps were taken to ensure their original data was ethically sourced?
     
  34. toggy4353

    toggy4353

    Joined:
    Feb 10, 2023
    Posts:
    1
    so can we use the sentis and muse if we just go into a new unity project
     
  35. CyberStreamer

    CyberStreamer

    Joined:
    Jun 28, 2017
    Posts:
    2
    Hope Sentis supports dynamic parameters of conv which Barracuda 3.0 does not support.
     
  36. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
  37. yoonitee

    yoonitee

    Joined:
    Jun 27, 2013
    Posts:
    2,363
    Is Unity Muse a web-based thing? I haven't heard of any language models that would run well on an everyday laptop. i.e. would you need an internet connection to use AI Muse? Does it use Chat GPT as the backend?

    Sounds like a chat-gpt plugin which runs in Unity.

    Is Muse built yet or is it still in concept phase?

    They both sound very interesting even when you strip away the bombastic hype, lol.
     
    Last edited: Jul 1, 2023
    Unifikation likes this.
  38. innominate817

    innominate817

    Joined:
    Apr 30, 2020
    Posts:
    1
    Hi, I have multiple tutorials and packages that use the Barracuda inference library. I already filled out the form for the closed beta, but the welcome email did not indicate when to expect access to the closed beta.

    Is the Sentis package not ready for the closed beta, or is there something else I need to do (or can do) to get access sooner rather than later? I want to get familiar with the new package to update my Barracuda tutorials when Sentis comes out of closed beta.

    I am also curious to see what models Sentis supports out of the box that Barracuda 3.0 does not. I have other tutorials using different inference libraries to get around Barracuda's limited operator support.
     
  39. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,526
    Stable Diffusion runs just fine on a laptop, I don't see why an LLM wouldn't.
     
  40. yoonitee

    yoonitee

    Joined:
    Jun 27, 2013
    Posts:
    2,363
    Stable Diffusion is about 5GB. Chat GPT is about 175GB.

    You can run an LLM on your laptop (I have) but it will be dumber than a box of frogs.

    Actually more useful would be to download the entire Wikipedia which you could get into about 3GB. :)
     
    Last edited: Jul 3, 2023
  41. Andy-Touch

    Andy-Touch

    A Moon Shaped Bool Unity Legend

    Joined:
    May 5, 2014
    Posts:
    1,477
    What is the expected date for this to be out of Beta? This is basically a no-go for my company if we can not use it for commercial use for many years.
     
    Unifikation likes this.
  42. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,526
    So it’s exactly the same as chatGPT, because that is also the dumbest thing I have interacted with.

    Stuff like this exist: https://gpt4all.io/index.html
     
  43. Unifikation

    Unifikation

    Joined:
    Jan 4, 2023
    Posts:
    1,073
    The "who would dare use it" whilst it's providing an insecure feedback loop AND its results unusable question leaves us with a very low level of users and/or low level use cases for beta testing. So how's it going to get better enough to be good enough for actual usage.
     
  44. yoonitee

    yoonitee

    Joined:
    Jun 27, 2013
    Posts:
    2,363
    OK, try downloading that and tell me if it can write you some Unity code. If it can then they could sell it for a billion dollars.
     
  45. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,526
    I mean you can’t imagine a model of that size trained specifically for Unity coding to actually be as good / bad as chatgpt at that specific task?

    ChatGpt does a ton of things, all of them badly. I’m pretty sure you can run an LLM on a laptop that does a few things badly.
     
  46. yoonitee

    yoonitee

    Joined:
    Jun 27, 2013
    Posts:
    2,363
    Of course. But I'll bet you $5 that Muse will be a web-serviced thing. After all... they haven't said it's not.
     
  47. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,526
    Oh, I don't actually disagree with that, Unity does like its stuff to be web based even when they really don't have to be, it gives them more control and potentially higher sub prices (because you are paying to use the hardware too or something).
     
    yoonitee likes this.
  48. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
  49. yoonitee

    yoonitee

    Joined:
    Jun 27, 2013
    Posts:
    2,363
    :(Well that's going to set the singularity back a few years!:(

    On a different note, I have stuff on the internet. Does that mean Chat GPT owes me money? :cool:

    I typed my name into Chat GPT and it didn't know who I was. :(
     
  50. chilton

    chilton

    Joined:
    May 6, 2008
    Posts:
    561
    wondering the same.
     
    atomicjoe likes this.