Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Discussion Using KI for graphic generation

Discussion in 'General Discussion' started by The_wARmAcH1n3, Mar 17, 2024.

  1. The_wARmAcH1n3

    The_wARmAcH1n3

    Joined:
    Aug 5, 2023
    Posts:
    109
    I'm not very experienced with Text-to-Image KI. I tried it but was not happy.

    I'm asking if it is possible to upload an image to a KI and then change the image with the text to KI. Is it possible? Is there a image creater KI which can create the different animations frame of a 2D character too? What is the best KI for it?

    I'm asking me too if I can create intro/outro-scenes pictures with KI. How can I make it possible that the characters look at all the different pictures in the same look?
     
  2. Murgilod

    Murgilod

    Joined:
    Nov 12, 2013
    Posts:
    10,262
    ...do you mean AI? And no, you're not going to be generating character animation sets with AI, even for 2D characters.
     
  3. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,627
    I'll assume you're talking about AI and not chinese mystical principle.
    There are networks that can do that, but they're extremely limited. One approach is instruct pix2pix, other one is control net + Stable Diffusion.

    you'll have VERY hard time generating animation frames. The amount of work will balloon to the point where it is faster to just do it yourself.

    It is extremely difficult advanced use. Basically you'd need to design character you need, train a LoRA or Lycoris addon with it, then use it with the network. It is time consuming, and once again doing it yourself may be faster.

    The issue here is that image generators work best for initial draft, or quick test of an idea, not for a final product. If you're trying to use them to create final product image, I recommend to reconsider. Neural Network images have very recognizable visual style, especially when drawing faces, and using AI images in your game creates impression of it being "cheap".
     
  4. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    1,743
    From what I know, some last steps are missing unfortunately to allow using AI to full on make a consistent, playable character for a game. Midjourney recently presented a feature that will help in that direction: https://twitter.com/midjourney/status/1767316749547077824
    If it does not need to be animated, that may be enough and worth giving a try.

    Best to ask this in an AI focused forum! I feel like many forum-active devs here in particular are not all that open to the new tech.

    P.s. note that AI may help non-artistically inclined people a lot, but it's not without a learning curve. Hence why "prompt writer" is a new professional job description (albeit currently most of those positions are likely to be covered by traditional artists who were quick to adapt since the background does help).
    Not searching anymore with the German abbreviation for AI is a first step on that curve btw. :p
     
    Last edited: Mar 18, 2024
  5. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,627
    They aren't missing. They are simply not easy (lora + control net), not worth the effort and require advanced knowledge of neural networks. For example, to TRY making a consistent characters, you need several samples, and then your PC will have to work for a while in liftoff mode. If it fails, you start over. Which can eat hours. So It is not an easy thing to do, and it is faster to do it by hand. By using AI you do not save time, and it can reduce perceived value of the work. AI art creates impression of "cheap" once you learn how to spot it.

    In creative fields value comes from human effort. No human effort, no perceived value. AI art has recognizable style, and even if took insane amount of time to produce image, user will assume it was a prompt. No effort, hence no value. RPG maker effect (when RPG maker looks "cheap" by default) and negative reaction to asset flips follow the same logic.
     
    Last edited: Mar 18, 2024
  6. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,550
    Try out Unity‘s Muse texture and sprite solutions.
     
  7. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    4,000
    There is no information showing that animation frames can be made, so this is likely bad advice. There is a preview of an animation tool that sets up animations for humanoids, but again nothing about 2d sprite based frame by frame animation.
     
  8. The_wARmAcH1n3

    The_wARmAcH1n3

    Joined:
    Aug 5, 2023
    Posts:
    109
    Thanks for the answers.

    Of course I mean with the word "KI" (German word) the "AI".

    That are bad new for me. Maybe someone will create such AI, there would be a good market for this.
     
  9. Murgilod

    Murgilod

    Joined:
    Nov 12, 2013
    Posts:
    10,262
    It's not that simple, as explained. If you want art, hire an artist.
     
  10. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    1,743
    It will be possible in this decade, no doubt.
    Sora does it with photorealistic results already, just that it's not publicly available yet.
     
  11. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,627
    And then it'll become another RPG maker/DAZ immediately.
     
  12. Voronoi

    Voronoi

    Joined:
    Jul 2, 2012
    Posts:
    593
    You also need to think about the genre and audience for the work. In some genres, like Sci-fi and games, once the fans realize AI was used, it's a death blow. I would imagine because some fans are creative/artists and want to support productions that value artists. If it's an ad-driven, mobile game, I don't think those folks care as much how the art was made.
     
  13. UhOhItsMoving

    UhOhItsMoving

    Joined:
    May 25, 2022
    Posts:
    107
    With AI generated images, from what I've seen, people predominantly focus just on characters. However, that is only one thing, and AI models can generate far, far, far more things than just that. If we ignore AI generated characters, are people still likely to turn away a project because it has non-character AI generated assets? Or, a better question to ask is: if AI is used in a game, where does the line get drawn between acceptable use (i.e. "I will play this") and unacceptable use (i.e. "I won't play this")?

    Keep in mind that Newgrounds allows AI generated content, but (for our case) only in works where it's not the subject of the work. Also, keep in mind that "AI" is a buzzword today, and that players have long already played games with procedurally generated content, which is fundamentally the same thing as (or at least, the predecessor to) AI generated content.
     
  14. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,627
    Yup. Also, the issue that value comes from human effort is not immediately obvious, and is very counterintutive.

    Basically, someone is selling art and that's AI generated? Its value is zero. A book, but you had an AI co-writer? It goes into trash can. Book cover, however, is acceptable, but then image is a placeholder because someone couldn't afford an artist. When you're in a (digital) book store where every book cover is ai generated, then even MS paint cover immediately stands out from the rest and draws attention.

    This "value is human labor" principle does not apply to tools. If I want a car that moves me from point A to point B, I don't care if it was made by an army of robots, designed by an AI and there was zero human involvement, as I need a tool. If I want a car that is work of art though, then handmade quality becomes valuable.

    Another thing is, it is generally not a good idea to use an AI to clone someone's style, while that person is still alive. It is incredibly likely to upset people.

    I would say "this is likely". Characters are immediate giveaway, but if there are no characters, there's still recognizable styles, and then "handpainted > AI generated"

    Another issue with AI art is that it generalizes meaning for any object you want it to draw and picks the most common form. For example, you ask it to draw a car, it will draw a car, but that car likely will infringe someone's rights and is not safe to use. Because the network generalized and can draw about 5 cars total, slightly altering the base structure. Meaning every project using this network to make a car will have the same car. In time, this may become recognizable as well.

    The same thing appears in all artworks. Like you in case character you'll start to realize that you saw exact same armor layout before, this character is a spacemarine, this is master chef's armor, and this girl is wearing evangelion plugsuit with elements from Gantz.

    Safe topics seem to be landscapes, backgrounds, buildings, corridors and such, but then color and contrast may be a giveaway that something is not right and it was not made by a human.
     
    Last edited: Mar 19, 2024
  15. Murgilod

    Murgilod

    Joined:
    Nov 12, 2013
    Posts:
    10,262
    There's also the issue that the car on top has a different amount of lights per headlight.
     
  16. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,627
    Apologies for nuking images. I'll repost that picture here, so people know what you spoke of.
    upload_2024-3-19_5-27-44.png
     
    Last edited: Mar 19, 2024
    MadeFromPolygons and spiney199 like this.
  17. Voronoi

    Voronoi

    Joined:
    Jul 2, 2012
    Posts:
    593
    Actually, I posted because I just heard about a publishing house that was burned by an AI cover. This was a new sci-fi author's first book, which is a huge deal. If the first book does well, they will get another book, a better advance and be on their way.

    In this case, the illustrator hired to make the cover used AI without telling the publisher. The fans caught it after publication, immediately tanking the book. This poor author, who did nothing wrong and was not involved in designing the cover will be unlikely to get another book. The publisher also gets a black eye, when they had no idea the artist was using AI.

    The publisher now has a contract where the illustrators must state that AI was not used in the making of the cover art. I'm sure this issue is not the same for every genre, i.e. maybe romance or self-published books get a pass. But, this is not OK in the sci-fi world.
     
    marcoantap and Lurking-Ninja like this.
  18. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    1,743
    Those were obviously not fans.

    Cancel culture runs still strong, I suppose...
     
  19. Voronoi

    Voronoi

    Joined:
    Jul 2, 2012
    Posts:
    593
    Actually I meant "sci-fi fans", the author was new so did not have a fan base yet. But, I think it speaks to 'reading the room' or knowing your audience. Sci-fi fans value creativity and originality, so I would not be surprised if an AI generated cover caused a backlash. I think the same could be argued for indie games, and so I would caution against using AI for characters or any of the creative part of a game.
     
  20. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 5, 2024
    Posts:
    535
    Me and vast majority of my friends and family (after my private survey), the line for us is at zero. If we get to know (recognize or told, doesn't matter) that AI was used, then that's a no-play for us.
    If the creator uses AI and doesn't communicate it, or even lie about it, we simply put them on a lifetime ban (as soon as it becomes apparent, obviously).

    I have a handful of friends who actually embraces the generative AI-crap. Those are the ones who share Midjourney garbage on Facebook and stuff, the rest of us consider them the "flat-earthers of the group", don't know why.

    Anyway, we have enough games to not care about this too much, I wish any creator who uses AI to make code or art immediate bankruptcy, but no hard feelings.

    I just hope there are or will be more people like us out there and the entire industry won't degenerate into an AI-fest cesspool.
     
  21. Ryiah

    Ryiah

    Joined:
    Oct 11, 2012
    Posts:
    21,508
    Any AI? Because I've been thinking of downloading and playing around with the llama.cpp implementation for Unity that lets you run LLMs in builds to see if it's useful for anything (eg decision making logic in games). There are a few very small footprint models available (ie sub 1 billion parameter) that should be runnable on any system.

    https://github.com/undreamai/LLMUnity
     
    Last edited: Mar 19, 2024
    MadeFromPolygons likes this.
  22. icauroboros

    icauroboros

    Joined:
    Apr 30, 2021
    Posts:
    174
    If you can spare at least 4 gb of vram it could work, but don't expect instant replies.
     
  23. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 5, 2024
    Posts:
    535
    Any generative AI. But to be frank, I am personally not interested in locally-ran LLM or local NNs either. I just don't find them as interesting as proper game developer setup. We are very far away from any usable local decision making AIs which aren't immensely boring or limiting. Currently it's a waste of resources in an end-product IMHO. Prototyping or feeling around with the tech locally is different subject matter.
     
  24. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,627
    If you hire an artist to make a painting for a few thousand USD and the artist uses AI, you'll feel cheated. The situation is roughly the same. See earlier statement about "value comes from human labor".

    That is the problem, in my opinion. "Without telling" and "Big release".
    Basically if author self-publishes the story becomes different.

    Any AI that produces creative content. For example there's a group called ChalkEaters they made Doom x Animal Crossing song, and they made this about genshin impact:
    At 0:27 character clicks through dialogue, you can pause video and read it. This part is fun, because you're dealing with an easter egg, of sorts. At some point video creator, however casually mentioned that dialogue was made by ChatGPT, and that immediately killed the fun in it, replacing it with a disappointment.

    If you're trying to use LLM to run NPCs, by the way, that'll result in all of them acting roughly the same. ChatGPT retains its speech pattern even when you ask it to speak like lovecraftian character. For decision making you'll be probably much happier with decision trees. There are several AI-driven games on steam already, and they tend to go insane at random times.
     
    Ryiah likes this.
  25. Ryiah

    Ryiah

    Joined:
    Oct 11, 2012
    Posts:
    21,508
    I haven't planned anything particular yet. It's likely just going to be experiments to see what would work and what won't. I'm unlikely to end up with a system that the player is aware of let alone interacts with. I do want to try text adventure games where the AI is responsible for the parser and maybe world generation.

    Some of this might also be to have a use for the AI server that I'm building beyond just Silly Tavern. :p
     
    Last edited: Mar 19, 2024
  26. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,627
    There are several youtube videos where people try to use ChatGPT to try driving NPCs in skyrim. The issue is that even with OpenAI api access that results in long delays in response. Meaning the most advanced model running on the most advanced hardware is not good enough for expected level of interactivity.

    Regarding AI games.

    Steam has DREAMIO at the moment, the issue with that one is that it requires buying tokens to continue playing.

    There's also "AI Roguelite", which is the game that had problem with NPCs suddenly started orgies.

    My opinion is that those sorts of projects are currently trying to jump above what's possible. The systems we have right now resemble proper AI at a glance, but there's a difference, and the difference becomes huge when you try to use it as an AGI placeholder. I really like this quote:

    "What the large language models are good at is saying what an answer should sound like, which is different from what an answer should be."

    ---

    Also I think I previously explained in some posts how this sort of system can be wired to act as a sort of autonomous agent. Basically you let it control agent through text commands. In preamble you explain syntax to be used, add contents of memory, then run LLM with sensor data, also in text form. The LLM can elect to add more data into "memory" and issue more commands. Effectively the LLM will be playing zork, except it'll be knocked out cold after every command and forget everything it did not write in memroy.
     
    Voronoi, UhOhItsMoving and Ryiah like this.
  27. UhOhItsMoving

    UhOhItsMoving

    Joined:
    May 25, 2022
    Posts:
    107
    Yeah. But for me, as with anything, it ultimately depends on how it's used. I do have some of the same qualms others have with AI generated content (plus, AI results showing up in image search results, which I don't like at all), but I wouldn't turn away a project just for the use of AI generated content: for me, the major elements of the project still have to have a substantial amount of come predominantly from human effort. I don't want to play a game (even if it's free) where everything I'm experiencing was not the work of the "developer" releasing it.

    I also prefer that when people do use AI, that they disclose it; not to shame or expose them, but literally just to know, because it's naturally assumed that the work someone releases is... their work.
     
    Last edited: Mar 20, 2024
  28. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    1,743
    We are getting a bit offtopic, but here a cool AI-powered game:
    https://neal.fun/infinite-craft/
    For now that tech excels where mistakes are only adding to the experience and fun (which does definitely not exclude NPCs).
     
    UhOhItsMoving likes this.
  29. MarkHoffenberg

    MarkHoffenberg

    Joined:
    Mar 23, 2024
    Posts:
    2
    Like many mentionned already, AI isn't a miracle cure, it's a tool you have to learn to get good result, and yet still has a stigma no matter the amount of effort you put in it. Just like in my youth in france, people thought manga were made by computer, thus were soulless (yes even the ghibli one). Being wrong never stopped people casting judgement. Having a "souls" is a nebulous "no u" reversi uno card, you have to know the rules!

    The problem is as old as art, it's Evocation (the work stand by itself) vs Connection (the ability to imagine the artist intent). People who are in art for "connection" are frustrated to know there was no one behind a piece, even if they had praised it prior They will say it has no souls and you are dumb for linking something like that. Before ai is was "Math", or having a "formula", that made something "soulless", then it was the early machine, then the computer and now AI. Every generation will have its Bogeyman.

    Tron was denied Oscar in VFX because it was deemed cheating to use computer. Today Movie pretend it's all practical even though they have more VFX shoot than ever. Every time we invent a new way to get faster results (say blue screen, virtual sets, digital post production FIX IT IN POST :rolleyes:), the production time doesn't decrease accordingly, it stubbornly stay the same. In fact any significant works, no matter the media, tend to be created in the same amount of average time. Book writing don't require much equipment, George R. R. Martin still hasn't finish his Saga.

    The problem with ai art, even when it will be perfect you can jack your mind directly to the canvas, is that any work or art are refined by iterations. Human artist can produce the best art they can all they want, it will be covered in blue ink annotations as quickly as the art director say "not quite where I wanted it to be". Leonardo Da vinci was still working on the Mona Lisa before its death, it's technically not a finish piece, and he had been working on it for "decades". How many time George Lucas remastered the original trilogy of star wars?

    I expect any good art, even from ai, to be the same, you can't cheat the process. Ai art is shoddy and recognizable, only with people trying to cut the line, and get to the end, without following the process. Happened even without AI, shoddily written scripts, from wannabee authors, copying their favorite works, without thought put into it, still exist. Some people already master ai art, such that it doesn't look like the typical result, you just can't tell, like that guy who won a photography prize only to refuse it because he use AI. In the end, like collages and bought assets is used in artistic work (Wilhelm scream), you won't be able to tell a good work done by AI, because it's never about the AI. And what happen when AI is used as source assets, heavily modified by a human pass?

    Basically there is 3 levels of talents in art:
    - Skill, how good are you at executing things.
    It's just a moving target and was never truly important, just a mean to get to the next level.

    - Specificity, how good you are at coherence within constraints.
    Specificity is something that makes you an excellent technician, able to take any works

    - Identity, how good you are at being unique.
    Even with Ai you still have to figure out Identity, that's on you.