Discussion A.I. generated art for game development

This is something I have been discussing a lot lately, I really would like to know the community's position on using A.I. to generate game assets and images (as A.I. is available for both 2D and 3D), based on my discussions I can see that currently there is 2 main positions:

- We should use it because like any other technology it is a tool that allow us to get things done better, cheaper and faster.

- We should not use it because it essentially copies other artists arts, it take little effort and you should have no rights on whatever it creates.

As a software developer who has 10 years of experience writing code and 0 years on doing anything artistic A.I. art really got me excited, I could finally focus on my strength and generate any art I need, but seeing the controversy it feels like it could backfire, not only based on people's opinions but on any copyright law that different governments create around that.

 

I have impression that this position is held by people who do not really understand how image generators work and have no experience using them.

It is not this easy. As they say, there's no free lunch.

It takes time to learn how to use Stable Diffusion, for example, and how to coax neural network into producing something similar to what you would want. The tools are not simple. What's more, the tools are very rapidly evolving, for example, a short while ago someone introduced control net and now it is possible to pose characters. Well, sort of. The instrument can refuse to cooperate.

Basically, to get good results, you need to invest your time and learn. If you do not invest your time and learn and just slap a basic midjourney artwork, you'll produce "fodder art" which will look cheap to people familiar with Midjourney/Stable Diffusion, despite being detailed. You know how some people used Daz3D to make comics and visual novel characters? It is this sort of effect.

Also, be aware that learning how to draw is an option. Your software development skills should aid you while researching a new field. Any new field, art included.

 

The general consensus on gamedev forums is "use it and to hell with what the artists think; automation blah blah", so to get the alternate perspective you'll have to ask in multiple places

Personally, I am mostly negative to it, partly I do not have the hardware to get decent results from standalone version of Stable Diffusion, and refuse to support online services, like Midjourney, with money. It doesn't exactly "copy" artists' work, but it had artwork (scrapped from all over Internet) compressed into itself during creation. Since artists had no idea someone will ever use their work to train neural networks, no one could opt out of this practice until it was too late. So no surprise many of them beel betrayed. (AI advocates will go on about how "it Is tOTallyY LIke hUmANs learN to DraW", no I can't agree. Neural network doesnt form understanding of, say, anatomy and perspective from examples, and only draws what is statistically more possible. Thats why it is, for example, exellent with faces at front angle, but messes up eyes most of the time, because look direction on source pictures was all different)

About how it can help you, well, as neginfinity said, will depend on the context of what you need. It is trivial to produce something that looks like artwork, but for more specific tasks, you'll have to fight against the algorithm. Issues with eyes, hands, poses and even elementary logic (it can draw the belt overlapping an arm or more than 4 legs on a horse). At some points, even hacks like in-paint or img2img may be insufficient, so manual retouching will be needed. Hard to say where it is going, both legally and reception wise. In the end I feel like consumers will decide, not nesessary creatives themselves

 

I mean, there is a big discussion to be had if using copyrighted works in the training dataset is legal / ethical, especially in cases where you can make sure the works of some artist are over-represented in your output by adding "in the style of *x artist*".

But yeah, I think some kinds of artworks that are easily reproduced by AI-Gen will soon be considered *cheap*. They will become the same as "helicopter shots" for movies. It used to be the mark of a big production to have helicopter shots (like following a car from high above), but now with drones, every no-budget third rate movie has a shot like that, and people are starting to associate those shots with low budgets.

 

There's not much to discuss here, as the training process is similar to what human does while looking at a picture. If you ban robots from looking at someone's works and claim that this is copying, you should ban humans from looking at it as well. Because by this logic humans also "copy" the picture into their mind in the same way.

Styles are not copyrightable. And humans are perfectly capable of doing the same thing.

Another thing is, a single style is useless, people use blends of many styles that ends up resembling none of the original. So "style of X" does not even refer to the painter himself/herself. It is a bookmark with desirable behavior traits of neural network and it is meant to be combined with other bookmarks.

And that is the reason why I say "stealing artists works" is an opinion only held by people who haven't seriously used the tool. Those people think their style has value, and everybody is out to steal it. It is similar to novice programmers thinking that everybody is going to steal their "extremely valuable" code. In reality their style has no value alone and is used as a keyword to make the neural net draw faces better. And when it comes to proper painting, a real artist worth his salt will run circles around stable diffusion.

Also, in case of stable diffusion specifically, rather than using style merging voodoo it is easier to load a different checkpoint instead.

Spoiler

Effect of checkpoint switching. The prompt is minimal.

 

Neural networks compute difference between noise and a picture and store it, then remix those deltas. Nothing like breaking down the shape of object into simpler forms and forming understanding of general principles, like humans do (there is an element of it in drawing process, but it is a really small part). Humans do inevitably develop their own style, even they start from copying. AI have to utilize the style of existing artist for the best results. To say that the training process of neural network is similar is confirmation bias, and in your case, if you claim to "know how they work", intentional misleading.

Woah, slow down there with false moral equivalencies. Robots are tools and do not have any rights compared to humans (otherwise, there is no sense of using the machines at all, as it would be equivalent of slavery). We should, no absolutely NEED to restrict what machines can and cannot do (because, if you ask me, the technology in general becomes more and more a trainwreck with the passage of time).

There is a concept of unlawful use of someone's likeness
https://www.dmlp.org/legal-guide/using-name-or-likeness-another
Prompt engineers leech from the popularity and relevance of existing artists all the time. Check out the story of Greg Rutkowski. Maybe flooding the internet with cheap knockoffs of his work isnt exactly "illegal", but only because of it being gray area currently. (Also, pics "in the style of Rudkowsky" are good illustation of how "good" AI with anatomy and character interactions). Again, most humans who ever copied to study moved on with their art eventually

 

People been reproducing existing arts and their variations forever. In most cases there is no legal action against. Unless someone makes something like repaint of Mario animation nearly 1:1, with changed colors.

Art stations and other image stocks are flooded with iterations of other works. Appears there is no issue with it since years pasts. Before any AI even been to mentioned.

AI is not simple copy paste. It is unable to recreate exact image when asked directly. Unless specifically supported with such image of Monalisa for example, just to make her look cry.

Other than that, is just a tool and tech to adapt too. Learn it to use it. It isn't tool that will make what you want with two clicks. As mentioned already, it requires learning process, to create right prompt, to use right models and their variations. It is tool that still requires engineering level and artistic expertise.

Or be like Codak, whoo decided not to go with progress of digital cameras, when times arived and basically it is left over in some narrow niche of photo making market, almost unrecognisable anymore.

Law as usually will be behind the tech. By the time it's get to some consensus, tech will advance and law will be obscure. Most likely dead.

 

You could put a camera on a robot and train the model based on what it sees and you would have something very close to a human. The biggest difference is how humans, through their life experience (biological needs, cultural needs...), disseminate what is art and what is not. So supervised training AIs will have a human bias about "what is pretty to look at". But it has nothing to do with tracing other drawings, in fact contemplating, copying and deforming is the basis of human creativity.

Try putting a newborn human being in an empty white room, without even noise propagation, let it grow there for 18 years. Then give him a pencil and a sheet of paper. Let's then see how it "generates its own style". To think that humans would be able to create from nothing is absurd. One would have to assume some kind of mystical soul with no scientific basis.

 

You're right, if you're making such broad equivalences and presenting them as absolute truth, we don't have much to discuss.

 

Nope.

Humans do not do that sort of thing while looking around.

You see an image once, you have it in your brain. How well you can recall it depends on your visual memory, for example, in my case the image will be "blurry". But it is just there. Without any in depth analysis. You saw it, you copied it. It is encoded somewhere within your circuitry. If you're good enough, you can copy it back to paper. People with photographic memory exists, and there's opinion that images are actually stored in photorealistic quality, it is just you can't access them easily. Sure, there's a tiny number of people without ability to think in images (called "aphantasia"), but for majority of people you can evoke an image in your mind and play movies there.

Again, nope.

Because stable diffusion is tiny, it IS breaking down the shape of object into simpler forms and forming general principles. Just like you described.

Stable diffusion forms a library of patterns, and it is blending them based on input to produce final image. To notice that, you have to play the prompt while keeping the same seed and see several thousands of images. A casual user will never get there.

Basically, if you generate many pictures with the same seed, you'll notice that, for example, on the first image there's a crevice on clothes. In the second image the crevice forms into someone's neckline. Then it becomes a branch. Then a river. Its shape remains the same and position remains the same. Sometimes it is barely visible. Patterns like brick wall fade in or fade out of image based on prompt produced, but while fading in they do not move from where they are. There's a lot of those patterns, and the image is formed by overlaying them. You see this in greater detail when someone tries to use stable diffusion to animate a video, because patterns noisily move between frames and become more noticeabble.

And that is generalization in practice. Because it is impossible to store terabytes of data within 2 gigabyte file (minimum size of stabble diffusion brain), the net has to generalize. It wouldn't work otherwise.

Your visual cortex is a tool and has no rights compared to a full human. The process is identical therefore restrictions should be identical. Anything a human with a neural network can do, an artist is capable of doing and can do better. If robots are not allowed to generalize based on your works, then humans shouldn't be allowed to do it either, because the process is the same, and the end user of the data in both cases is a human.

Mankind is a trainwreck, not technology. On ChatGPT subreddit someone seriously tried to argue that GPT-3 has feeling, will remember everything told to it and therefore we should be polite to it so it won't kill us all in the future. That person was completely serious. That is a high degree of technological incompetence in practice.

People saying that "neural net is copying" are the same as that guy. In essence, those sort of groups will keep popping up and in practice they seek to stop technological advancement. If those guys gain power and succeed, the world will become a place I'd rather not live in.

Someone's likeness is their face and is not their drawing style. And you can't copyright your face anyway, because it was not created.

Before I tried stable diffusion I did not know that Greg Rutkowski existed. Like many other artists. Art, in the first place is a niche interest, and most people do not care about it.

What is happening is that those artists become known through prompt engineering. It is advertising.

Dude. People are not trying to make his knockoff, because in the first place it was never about rutkowski.

Making a "knockoff" will be worthless, and rutkowski can run circles around any neural network output, which is something you'd known if you checked his real artwork.

The idea is to have a convenient keywords that improves image quality. A convenient keyword in this case happens to be artists name. It doesn't matter what keywords mean, and whose name it is. What matters is its effect. It is a shortcut.

 

You are putting words in my mouth. Where did I said about "creating from nothing"? I meant that the human creative process is different from the neural network, and that the argument that they are the same as a defence of AI art is gaslighting and wishful thinking. Neural network model is unchanging after it is created, while humans learn during all their lives. It can never achieve understanding of factors such as shape language, physics and anatomy to be able to move away from simply reusing styles. Thats why I say it is more of equivalent of compression.

And about "believing in mystical soul", well, funny, because the AI community has many trait of a religious cult by itself. Good portion of the AI proponent's arguments is ascribing personhood and agency to a machine, and also believing humans/biological life be imperfect and in need or replacement. In other worlds, attempt to produce a god to rule us all, which is the root of the movement for general AI development. Not any real world concerns, just the idea that AI will know all, invent everything, solve every problem, etc, if you give enough money to tech companies. This is no more than faith. Assuming that there is a soul is as biased as assuming that human is equivalent of machine. Since in the end, our scientific understanding of reality is but a model, which sometimes works, but nothing close to completeness or absolute precision

 

I'm stating my opinion, which is a result of my life experiences and values.

If someone is unhappy about my opinion and beliefs, well, tough luck. You'll have to learn to live with me existing.

Also, writing this all down takes time. If, after me spending time to carefully break it all down, and explain piece by piece why I think this way, someone still isn't getting it, well, at this point this is no longer my problem.

Sufficient effort has been made, if the message isn't getting across, I can't help you. If after detailed explanation all I get is an attempt at sarcastic ad hominem in return, then a dialogue was never an option in the first place.

 

This is similar to a comparison that I have been doing, in the past decade tools to create things without any need of code exploded, Unity itself tries to make that happen, but no one really cared about developers (mostly because they were developing those tools to start with), I understand that it is hardly a fair comparison, but often you can write a whole piece of software with no code to be exactly what you want, but you can't create a whole comic book to be exactly what you want by using any kind of tool, even with A.I. more often than not you will need to use what you are given instead, so artists with that power will not go anywhere in the short term, I'm aware that many of them are using A.I. generated art to create a base to work with and adapt it to their needs, speeding up the process a lot.

 

You're thinking in the wrong direction.

There's no creative process in neural network, because it is incapable of doing anything by itself. The one performing the creative process is its operator. The network can produce a highly unexpected but interesting result, which can steer further process in a new direction, but that's the extent of it.

Neural network only knows how to draw. And the way it learns to draw is similar to the way humans learn to draw.

It is not a human. It is not a human artist. It is an external art knowledge module for your brain. If you're an artist, you've developed an organic equivalent in your brain. If you're not, then you can use the network.

It is not unchanging. Look up Textual inversion, Fine Tuning, and LoRA. You can alter it. You have to train it. Because it has no eyes and cannot do it by itself. It has no agency, remember? So if you want it to learn something new, you have to help it.

You're making a lot of accusations here, without supporting them with anything. Saying that the root of movement for making an AI is trying to produce a god is rich.

And in the end you yourself speak from position of faith.

 

I expect this thread to be shut down as not very Unity specific and usual arguments ensuing, but...

It remains to be seen how the legal situation pans out - perhaps there is just not enough similarity, in images generated, for any of the suits in progress to be successful at the end of the day (see https://stablediffusionlitigation.com/ Getty Images etc.)

If you use these tools you can expect to find them error prone because whatever is said they seem unable to learn basic anatomy! Artists will also be quite unhappy if you use them - it is understandably a threat to their work

 

ARTISTS do that while creating artwork. Important distinction. While you can say imagination is roughly similar to what neural network does, it is really a small part of drawing and painting process

If it was true, then Stable Diffusion would have no problem with drawing hands, eyes, four legged animals or character interactions whatsoever. If minute movement of the eye in the training data (as shown in your last picture) can confuse it, what general undestanding is there to speak of? Efficient compression algorithm is still compression

My visual cortex wont work without other systems of the brain, and brain is required to be full human. While it is perfectly possible to live without using artifical neural networks, even if it is harder. (Again, you are passing an assumtion that those processes are identical as absolute truth). Idea that machines should have same rights as person is idealistic to absurd degree. Then what, we should stop selling and owning them? Bye the capitalist economy, and the AI development along with it. Or maybe we should have car drivers compete with Olympic runners, because a human is in control in both cases.

It does extent to using their name.

Sorry, but you arent helping this case. Saying that we should treat humans and machines equally in the very same post, and also implying that the technology is perfect if it wasnt for those pesky humans. Machines arent perfect, they are merely more specialized. Such religious mentality is why AI can become a threat, in my opinion. Putting priority on the tool before its user will guarantee general AI, if it is possible, wont work in our interests

People wouldnt start generating such art if Rutkowski wasnt relevant in the first place

 

"We need an AI governer. Which will be immortal and capable of increase its intelectual capability infinitely"
Sorry for misquoting it, but the gist is there. And for some it is even not a joke

 

Regardless of the discourse here around legality, morality etc - you cannot stop progress.

Either get on the train, or get left behind. You can either be one of the artists who adapts and incorporates this into their workflow, or become one of the artists considered to be doing "retro" style workflows in the future.

If anybody here genuinely thinks that legal or moral objections will somehow stop this technology being used globally and in every industry, then you are not paying enough attention to what is happening in the software industry right now. Every agency and company I work with currently, has pivoted to incorporating AI to some degree.

It sucks, we have artists in my company and we dont like it, but its the reality.

So for those that hate it - Time to start learning how to use it instead of arguing against it.

 

You aren't seeing the forest for the trees here.

"would have no problem" is non-sequitur. Process is the same, memory capacity is inferior to a human artist, hence the artifacts. It still manages to generalize and be usable. It is usable because it generalizes. If it didn't generalize it wouldn't be able to draw at all. It has thinking capabilities roughly equivalent to a honey bee, after all. The model is TINY. The fact that it can process so many image is already miraculous, and there's no way to compress data into it. The training data is 100 terabytes, and 2.3 billion images.

If you want better quality, well, you need more memory, hardware power, and bigger training database. Midjourney offers that.

That's a strawman. Or you aren't listening. What rights? Stable Diffusion is not a person. And neither is your visual cortex.

However, if you're not allowed to use an image for training stable diffusion, you should not be allowed to use it with your visual cortex for training your mind. Meaning you should not be allowed to see any copyrighted artwork, because in your case fine-tuning of your circuitry is going on all the time non stop, meaning you're unlawfully copying every single image you see.

The artists are trying to implement absurd restriction. I'm demonstrating absurdity. It is worth noting that copyright laws have been steadily increasing duration of copyright protection in the past 2 hundred years, and this is the point where they're taking it too far. In the end the only thing it is gonna achieve is that another country will take the lead in neural network research.

You can't copyright names either. Look it up.

I have never said that, though. You're making a strawman. Either deliberately, or you need to work on reading comprehension.

 

The technological progress isnt an objective good, it is increasingly a liability (considering the waste of resources on unnesessary or even harmful projects). Sorry, but "you cannot stop progress" is a cargo cult mentality to be honest; we need to evaluate all the technologies on case by case basis. Many seemingly promising technologies fell into obscurity, so it can go either way. In the end, I believe the consumers will decide, but the pushback is justified

One of the reason that it image generation is hated by artists is that in many cases it is hard to incorporate it into existing content creation pipelines. Since it is quite rigid and limited. Either you settle with the first draft and close your eyes to errors, or lose the speed benefit while trying to fight against an algorithm.

 

So, the big difference now is that the "models" are modified in real time in our brains? What about babies who are not able to formulate words or understand the rules of physical world until they have experienced it for years? That doesn't seem like an argument to me at all.

Humans are not special in any way. To think otherwise has led us to the greatest outrages against our species.

I am not going to respond to the comparison between believing that humans are more than a bunch of interconnected neural networks, and that certain people behave in a certain way. Not related with anything i've said.

 

"You saw it, you copied it. It is encoded somewhere within your circuitry. If you're good enough, you can copy it back to paper." Actually, this argument is unbeatable. There is no point in continuing the discussion until no one can respond to this. I, for one, am not going to do so.

 

The hadrware power and dataset is clearly not the only issue here. Midjourney can output better resolution and operates with more parameters than Stable Diffusion, but its level of generalizing is still the same (as in same visual artifacts)

Spoken like generating art is some national security concern. We're speaking about private companies, not military. There were enough public domain images to train any neural network they like. Again, allusion that all innovation is nesessary because it is innovation. I rest my case here

 

I am honestly not sure what points you are trying to make?

Does it suck that artists are getting infringed? yep, agreed
Would it be nice if society evaluated each tech rationally on a case by case basis? almost definitely

Will consumers have any power over a tech with this far reaching use cases (remember what we are really talking about is applied ML)? Very minor (pricing, local/national availability and restrictions), but absolutely not enough to stop this being used altogether. Its already being used for far too much to be taken away, thats the reality.

So lets say for sake of conversation, they outlaw chat GPT or nerf it to the ground due to restrictions. They do the same to midjourney, stable diffusion etc etc. Every major consumer facing AI tool gets nerfed to the ground due to restrictions or outright outlawed. Then what? ML just disappears now that so many companies around the world have finally found ways to actually apply it to their needs? Absolutely never going to happen. And so now its "outlawed" - what you think it disappears? Do films games and music still get pirated?

So yeah in an ideal world I am sure whatever points you are making would be great, but it still comes down to this train is not going to stop, so I am not sure what good ranting about how it should slow down and be assessed is gonna help.

Almost all of this entire thread is pointless. (Which was my main reason for commenting, was not actually calling anyone out in particular)

Like dont get me wrong, if you or anyone else have actual real points about how this train can be stopped then I and I am sure everyone else, is all ears. But otherwise its just shouting at a train barreling towards (and eventually past/through) you.

 

AI progress is today, like industrial revolution, steam trains, electricity and computers, in their days.

None of these were stopped, once started, even had power to object.
For example, Great Britain whole is digged with water canals, for goods transportation. You can still travel them these days, but no more used as they originally were designed for.
Those who decided to object, and fight inevitable revolution of the steam train, went into abys. And mind, these wasn't some small powerless water canals companies. Other decided to adapt to the new transportation system.

Machine of AI revolutions is already spinning. It is most likely another binary jump in our progress.
Is not stoppable anymore, with (bonus) billions and large corpos behind, supporting the tech.
In worse case, there will be always gray market, which you can do nothing about it. Unless unplug an internet.

 

(This is not argue, is just a thought)
I always did wonder, what makes difference, if I get one good artist, or 10 artists, to make something in the style of A or B art, or even copy one to 1, vs using tools (AI related in this case). Besides time and cost, I don't think there is difference really.

Obviously, either artists or AI had to "study" they style, of other artists, before making related work. That it doesn't make their work illegal, just because they based on something that already exists.

 

Well, shilling for it and trying to peer pressure others into suppport helps even LESS

And a matter of fact, a popular topic on the Internet nowadays is exactly "ChatGPT got nerfed" As its analog based on Bing. (Nothing of value was lost if you ask me). Artstation also allowed opt out for ML usage. This is still something.

 

This idea does not work.

An invention will appear at a specific time, in multiple place of the world simultaneously when all prerequisites have been met. Even if you ban the invention in your country, it'll take root in another territory, and then you'll be left behind, while another takes the lead and reaps the benefits you missed.

Effectively the "artists" in this case seek to restrict AI development. That can easily have far reaching consequences if they succeed.

 

Bold assumptions there that a)all of the countries have access to same prerequisites and b) there will be "benefit" and not another pointless dependency. And even worse, shifting the blame to artists. If they'll suceed, more power to them.

 

All countries have access to the prerequisites.
There ARE great benefits, which are quite obvious.
There is no blame shifting.

Also, it reminds me of this comic.

Spoiler

https://www.smbc-comics.com/comic/2011-01-15

 

Getting back to the OP, I think using AI for something like a hero image or cover would be fine and easily doable. I think trying to get it to make all the assets you need for an actual, original game is a lot of work if not impossible.

Something to consider is if you really have no artistic skills, at some point you'll probably need to work with an artist or designer to finish up your game. Most artists are not going to be thrilled extending the vision that AI started for you, unless they are a big fan of AI assisted artwork. So you may limit your future pool of artists if you go too far in building an AI generated game.

I would liken this to my experience working as a freelance illustrator. Sometimes clients would come to me and ask me to 'work in another artists style' because they were too busy or expensive. That's a job I would never take, the client really wants the other artist and is basically going to complain the entire time that it doesn't look like so and so's work enough.

I can't imagine being approached to complete a work or fill in the details of what AI started. That would be just days of fixing hands and anatomy, etc. and I'd end up with nothing I would ever want to show anyone!

 

Closing, as this thread cannot stay on topic. There are many places to discuss the general topic of AI, please continue this discussion there.