AI art for games

Has anyone tried midJourney or Dall E2 for game art?

How did it work out for you?

I'm tempted to throw down the $50 for midJourney for concept art and textures.

 

Quick reference here's my "Doom Slayer in the style of Jack Kirby" result on the newbie channel:

 

Here's me trying to get concept art for a Half-Life snark rip-off

https://cdn.discordapp.com/ephemera...4f9c-a046-0b617baf5e80_progress_image_50.webp

 

I wonder what the legal implications of AI created art is. Has this been tested in court at all?

 

Well, the $50 a month version gives you full rights.

 

Yes but the ""AI"" (they're not AI, just pattern matching devices) generates content based on images it's trawled through on the web. Images that are copyright to other parties.

Basically it's very legally grey.

 

Not midJourney.

 

What data sets is it learning from if not from the web?

 

Interesting stuff. I also wonder the legal implication of this.

 

The AI doesnt copy parts of images verbatim, right?

In that case its the same as an artist re-drawing sn existing picture. There should be no copyright infringement but trademark infringement could happen ie if you end up with art containing a trademarked logo. Or font!

 

As to game art: you can make those images hanging on a wall in a frame or as character portrait. But they wouldn‘t serve well as environment or skinning textures.

 

Depends how similar it is. "Same but different" doesn't always hold up. I'm reminded of how the song Down Under by Men at Work was successfully sued by the writer of the song Kookaburra for having too similar a riff in their song.

 

For using as concept arts(to build an environment or character or maybe the game idea itself), I think there is no trouble since those images would not be used in the project. They can just be used to get an idea.

 

That's what I think too, it's a legal minefield and will take a long time till legislation acknowledges it as a problem.

Absolutely also midjourney. There 100% are images in the training set that are not in the public domain and near certainly weren't properly licensed, because it would have been impossible to organize and pay for, to license ALL images it scanned. The training set must have contained paintings by Frank Frazetta for example. Or from your own prompt, it must have scanned artworks from Doom to even know what the Doom Slayer is.

Without all the unlicensed images it scanned, midjourney would not exist. How is that NOT a derivative product made from copyrighted material? Just because they have committed millions or billions of cases of copyright violation doesn't make it right. It's not like a human looking at pictures, because it is not a human. It is software derived from copyrighted data.

 

Hmm, if the creation of this tool would be legaly problematic, how comes Google and Co. may crawl the whole internet and provide a search feature on that data (including images)?
That's also a service that would not exist without the data.

 

Its not problematic from a legal standpoint. Bare in mind everyone on these forums will be armchair lawyers on topics like this. Anyone saying "legally gray" or "illegal" or anything about legality of something, on these forums, should be taken with a heavy pinch of salt. (Me included )

Its highly unlikely that anyone here actually have the legal background, experience and qualifications to make any real assertions on the legality of anything, so yeah best not to take anything like that at face value

 

It is an interesting topic in every case.
Would love to see an atmospheric game with an abstract, painterly look (as that's what the AIs excell at) with graphics mostly generated by this.

I might try it out for background clutter in my project...

Since we are on the topic of AI, you guys heard of this voice generator/reproducer?
https://15.ai/
That is definitely a legal risk because it does not mash up thousands of indistinguishible sources but aims to reproduce one specific voice based mostly on input samples of that singular voice. (Though that requires a lawyer again: Is a persons voice protected somehow at all? )

However the concept is very interesting and would be cool as a software. Imagine needing your voice actor to only speak a couple minutes and then generate from that what you need at any point in the project (with the actors consent of course).
As silly ads would put it: "Voice actors hate this trick". Could well imagine contracts with voice actors in the future will include or deny the rights to reproduce their voice.

 

I tried midjourney trial, it is ridiculously hard to control and make it paint what you want. It might be worth it for background paintings, but not for primary artwork of the game.

Well, it doesn't store images within itself.

Also, copyright is handled in a specific way.
https://en.wikipedia.org/wiki/Rogers_v._Koons
AFAIK, if the work cannot be recognized as a derivative, it is not a derivative.

I.e. you're allowed to imitate but not allowed to copy. I.e. if you print mona liza, put it into a blender, and use resulting powder to make a new painting, that's not a derivative anymore.

 

But that's the thing, you can sure as hell identify the copyright materials that these ""AI"" are drawing upon.

If someone can look at an image made by these programs and go, "That looks like X", then one could extrapolate that to say, "this program used copyright material without permission".

Is it derivative? Or are these programs just mashing together material it found on the internet?

Do these programs "create"? Or are they merely copying?

And if said program isn't a 'creator', does that put the onus onto the designers of this program? Is this unadulterated trawling of information and media an infringement on their behalf?

This is starting to get more philosophical than legal, but it's pretty easy to see just how many legal questions can arise from this technology.

I guarantee that the first real court case that emerges from this technology will be one to pay attention to.

 

I think your conflating the meaning of "copying" and "deriviative" in regards to copyright.

This very much does not come under the umbrella of "copying" at all. BUT whether it is copyrightable to the user is another matter.

Also, there have already been legal cases surrounding use of technology like this - it hasnt made the slightest impact and most have not even been heard of because they simply did not make waves. For example the 2019 case with steven thaler (which deemed the AI generated image was not copyrightable to the user btw)

 

Alright, please find original image for the picture below. (warning: creepy)

Spoiler

The point is, you can't extrapolate. You should say "this is X" and not "looks like it". And if we use your logic, then if you draw anything, Disney can sue you right now for copyright infrigement in duck tales, because you saw it, therefore your picture of a cube is derived work.

Are you aware that artists use references?
One of the "How to draw manga" books demonstrated what is allowed with referneces.

Spoiler

#1 is the reference, #2 is plagiarism, #3 is allowed.

Yes.

In essence, in its extreme/ideal form, a neural network of this sort would be a machine for drawing all possible images. The thing is, to produce an image you want, you need an address or a clue to find it. The machine cannot store all those images, because their total number is infinity. Thus it creates them on request.

 

BTW, from their TOS, Midjourney technically owns all the art created using the service: https://gyazo.com/bc0cb1afb6ea6e340e6d0a7a7c9a6a3a
https://midjourney.gitbook.io/docs/terms-of-service

 

Interesting thought.
Reminds a bit of the idea that a truly random pixel generator of a simple 255x255 pixel image, would eventually produce a mona lisa grayscale at some point. Pretty much the art variant of the "infinite monkeys theorem".

 

i haven't used those specific apps but i've used some older ones to quickly generate faces and stuff like that.

As a solo dev I'd try to stretch the use further than just concept art though, if possible. Try to squeeze it for all it's worth. Perhaps you can get a lot of environment props, textures, backgrounds and more from it.

 

It is more complicated than that.

A simple Neural Network generator would represent an N dimensional space. Where each point is an image. N can be very large. For example, it can be 1000-dimensional space, with image at each point, where each dimension has some sort of meaning, but the meaning is unknown. A human would need to label some sort of data and attempt to derive a "vector" for desired characteristic. That's how "genderflip" and "make younger/older" filters work. They take an input, find an image similar to it in the N-dimensional space, then shift coordinates by the vector called "age", "gender" and so on.

With midjourney it seems to be more complicated. Because instead of one to one mapping it seems to be providing a list of possibilities, meaning same input might not provide the same output.

 

Can you use this for game dev? Absolutely, some people are even building entire automated pipeline right now, so that it generate during the night, and result is curated the day. You can use it for almost anything. It's pretty good to create variations. You can also use it to fill erased part with new prompt, or clean an image.

However there is some limit, it does not have consistency, it does not have permanence, ie you can tell him to reuse a character over a long series of request. It's very badbat giving you exactly what you want, it's best if you roll with suggestion and build on them.

What you can do, and that's the preferred workflow, is to kitbash result, and have a cleaning pass to correct eventual error.

Also consider you can use multiple tool and other AI to complement, like
- generating a face with midjourney and animating it with deep face transfer, while it's not perfect in the absolute (face rotation is a problem) you can create a worflow around like, capture only front facing face animation (like with performance mocap), use a markerless face detection soft for stuff like jaw movement, project the animation to a mesh, extract the animation to uv, clean the uv, wonderful you have a very life like stylized animation.
- have a passable quality mesh of a character face, done by hand or character generator, create many poses in different light conditions. Use a face generator to find the realistic version in the latent space, project all the poses. Use the result to train a deepfake ai, animate your cutscene, transfer the deepfake of the custom character into the cutscene, clean the small defect.

This won't make artist obsolete, at least not just yet you need the talent to correct and adjust the resulting stuff or create complex variations for permanent stuff. The ai can't do that because the architecture is still naive and wasn't designed for that.

If anything these ai are very powerful but very dumb interns, you still have to go after them to maintain the art direction and follow the vision, which they can't. It's similar of the job of animation director in anime, that will retouch sequences of various animator to maintain coherence in the production.

You all should also check xva trainer.

 

LOL, were you one there last night?

I tried for an hour trying to get it to make a painting in the style of Frezetta.

 

BTW, I could not get it to make tiled textures at all.

And I’m not so sure about the legal aspects.

No one was able to sue Winston Smith and all of his stuff contains lots of copyrighted material.

 

The legal aspect is mostly for the input rather than output, all art and creation are derivative by nature, created by a series of transformations of sources, ai automate this part. Lawyers on YouTube discussed that.

And the input is (partially) a question for new laws, not current law.

 

No. Did you spell him like that? If so, no wonder it didn't work.

A couple days ago on the server I scrolled along and saw 4 thumbnails of a work in progress prompt and thought "Hey, those colors, shapes and compositions look a lot like Frank Frazetta's paintings." and then I read the prompt and it had "in frank frazetta style" in there and I thought "Holy S***! This is actually pretty scary.". There is absolutely no way for the AI to get this close without having Frazetta paintings in the training data.

 

It can't do that by default. There might be some sort of keyword to enable that, but that's not certain. You'd need to alter it manually afterwards.

It definitely processed every available painting and probably online image. It is aware of many celebrities and have a rough idea of what Ciri or Gordon Freeman looks like. However the point of neural network is that it does not store paintings.

Same deal would be if you studied frank frazetta's paintings and figured out a way to replicate the style. That won't turn you into a walking copyright infringement, although the data will be imprinted into your brain.

 

I don't quite agree with that, but I don't want to argue semantics and I think you don't enjoy that kind of argument either, so we might just have to agree to disagree on this one. It's hard for humans to wrap their brain around how neural nets store their data because of the n-dimensionality of these structures, but I think the argument can be made that the individual fragments of image patterns it stores are still image-like enough to be legally treated as image parts in the traditional sense. It's just that they aren't taken verbatim from a single image, but averaged out between multiple images.

You remember the oldschool image morphing techniques from the 90's I assume, where people used to morph a face of a human into the face of an animal for example. Imagine taking these two photos from famous protected sources, and creating a morph animation with 100 frames where the 1st and last are verbatim the stolen pictures without any changes. How many of the 98 inbetween morph stages can you legally and ethically claim ownership over?
What if you stole 100 different photos that you have no right to use and layer them over each other with 1% opacity so that the end result is a messy blur? Can you claim ownership over that, even if it's entirely made from stuff you stole and weren't allowed to use in the first place? What if it's just 2 pictures with 50% opacity? How many do you have to layer for it to no longer be illegal?

 

Likely half of them.
https://en.wikipedia.org/wiki/Transformative_use
https://en.wikipedia.org/wiki/L.H.O.O.Q.

Most likely. See transformative use above.

Every single thing you see is imprinted into your brain and every single thing you create is based on things you've seen, many of which are copyrighted. And without seeing anything you can produce nothing. Same with the images.

That sort of thing requires proof. You need to rip the image net open and show the pictures within. Based on what midjourney does, however, you'd need to do an equivalent of copyrighting an idea of owl.

Spoiler

Pictured: Owl.

If something like that is possible, humanity is doomed.

 

Maybe another way to think abstract about this is: Imagine the data of an image being encoded with a plain ~20 characters long caesar chiffre. Each letter of the passcode affects a pixel.
Now without knowing the code (or adding your additional information of knowing the original image) you will just get a fraction of the pixels from the original image.
Does the result of this with user-selected passcode inputs, still represent a copyright infringement?

However am curious whether there will be some significant court case about this. Maybe if some large company decides to use this for some advertisement or widespread product.


EDIT: Just looked up, Dall-E 2 apparently consists of 3.5 Billion parameters that were learned from 400 million images.
That means 8.75 per image, so to claim that the original image is somehow in there is extremely far fetched.

 

"Viewing images is deemed to be equivalent to downloading images into human brain, court rules. On related note, Walt Disney sues mankind for copyright infringement and demands 1 billion in damages from everybody person involved".

I think it could happen.

 

@Martin_H that's like claiming legality over grammar, word and letter, because that's what ai does, it doesn't store the image, it store the common extracted grammar of all image. It's no longer true we can't wrap our mind over nth dimension, we understand how it works and have word for it like manifold and latent space, and can project it back to simpler space.

We know that the ai in nn work by composition of concept, with deeper level having higher concept, in fact those breakthrough happen because using architecture like encoder decoder, we force the ai to condense the data into high level description and that's what we used to make image, for example forcing hi level description of word to match hi level description of images, forcing the network to create its own joint token of meaning.

So basically, ai can not just do stuff only in it's training data, but you can present image in a style not on the training data, it will find a description in his own token language and can manipulate that back to reproduce it, like any artist can look at a Delacroix and break down the composition into triangle shape, texture, character, atmosphere, and with joint language training, we made sure these concept are associated to proper word.

That's how they are able to complete images in style they haven't seen before. In fact style transfer was the first big artistic application of nn. This property is call generalization, and that's the point of this new breed of ai. The nn don't store Franzetta paintings, it store the idea of what a franzetta painting is, ie it's artistic preference.

Earlier I told about naive application of generation in nn ai. Showing that it doesn't have consistency and sequence coherence. But hear the truth, ai can already do that, gpt3 is a transformer architecture, it produce coherent speech over a long sequence. I just explain these art generation model force them to come up with high level token of meaning, they are generally in sequence of block from top to bottom of an image.

We can extend that model by generating training that allow them to process multiple images as a coherent sequence. It's an ongoing research not to make it work, but to figure out how to manage the quantity of data, ie produce high quality sequence, because that's more data than text.

 

Cant wait to use this totally original character in my new game .

 

That's a strawman, though.

An artist would know how to paint Mario. That won't give them the right to use it. However, the knowledge of mario is still in their head.

 

But an artist knows not too just use mario, and I know mario is the most extreme example thats kind of the point, it has the ability to reproduce its input data to such a degree it can cause problems. An we all know mario so can see the issue.

Once you start using more generic terms like "space marine", does everyone using the AI recognize the Warhammer IP and knows what they can and can't use?

 

And that's why you need to have a few humans on your team. As a higher-level general purpose neurobiological platform, humans will able to identify potential issue your low tier high performance non-sentient inorganics might introduce into your project due to lack of comperhension of the world outside of the problem domain you've attuned your inorganics to.

 

what the computer is doing might as well be magic.

it's all going to boil down to convincing humans in a court whether or not something looks enough like something else. But that is the honest argument, which almost nobody actually cares about. The real argument will be the same as ever - who has the most firepower.

For tiny indie soloist making games that have slim chance to appear on any big radars, as long as your art is reasonably unique I think the risk is tiny and the reward is potentially huge to try and leverage any sort of productivity enhancing tools that you can.

 

An artist doesn't know nothing by default. We just had a brouhaha this week about an asset flip game using an obvious Kratos rip-off getting all the way onto the Microsoft Store.

The answer to your question is: it depends.

If I draw a picture of Mario, I own the copyright on that particular picture. I, however, do not own the copyright on the likeness of Mario.

The pictures midjourney produces are originals. The subjects they depict may or may not be. Therefore, you need to employ common sense and due diligence before using them, just as you would if you ordered a bunch of concept art from some random artist.

 

Also you can probably reproduce any character, even the one it hasn't learn by name, by localizing it in the latent space. It's a property use in face generation ai to produce many result, like kanga version of yourself, or color version of old photo portrait, or photo version of sculpture. Ie as long as you can describe it to the ai, and it has the vocabulary. Which lead to funny idea too, like finding the closest description of your face with an ai who has only learn car, and it produce a monstrosity made of car part.

 

Imagine using a character generator like meta human and accidentally reproducing Scarlett Johansson

Same problem really. In the space of possibilities there is point that are covered by copyright.

Also just space marine wouldn't give you Warhammer, with sufficient training it will generate something right on the middle of all possible space marine concepts most of the time, ie the archetype of space marine, you will have to be so specific in your prompt that it narrows the space to the copyright zone. Using mario as a prompt IS a specific narrowing.

If you ask any human to do generic platformer guy, vs telling him to draw mario explicitly, you will have the same result.

In fact you can probably use that as a property to have the ai explicitly stay away of copyright zone, because you can tell him to avoid the area at any distance of those latent space. And you can use a metric to measure similarity between different art, this has already been use to automatically classify art and detect mis classification in museums.

 

The difference is it is using the original media, albeit transformed. It is no better than taking a picture of Tom Cruise, then running image filters on it. The origin of an image is also protected, if it can be traced. It is always better to create new art, and then steer the likeness (a different legal issue) away.

It's controllable and better. Don't use AI imaging for it, it won't be legal to do so any more than running an image filter over things, especially if it can be recognised or even traced.

 

Yeah no, it's way different than an image filter, I can trace any actors likeness to a metahuman set of parameters too. It's more like an artist figuring out what parameters give you likeness of actor, the ai is figure what weight need to be activated to give you an image, it's the same, metahuman's parameters are actually a latent space in the strict sense.

The issue is still using copyright materials as an input like I said already about legality, still a gray area as of now, it's a matter of legislation, because that's a new case, it would be an agreed upon convention, not a philosophical absolute. And that would be pointless, because the result would be the same, the ai, even with no copyright input, can reproduce copyright input with sufficient description. And style can still be copied anyway.

The machine isn't copying, in the initial training it cannot reproduce the image, it also has to see all other images too. Ie its ability to reproduce an image depends on the complete training, not just showing an image once. Hence the reverse of a filter.

For example let say you have an ai able to recognize copyright likeness, it must store a description of that likeness in his latent space, is it copyright infringement? Is my memory of celebrities copyright infringement? That memory is what allows identification and reproduction. Is someone retelling a movie to his friends copyright infringement?

 

I used to write contracts from scratch for billions of dollars of commodities. UCC, Black's Law dictionary and how to review case law at findlaw or similar databases are not that different from using github and understanding the framework you are considering adding to your project.

In this case the final image is total reasonable to generate, share and sell. Tools for the creation of art can certainly be the convolution of arrays of 0's and 1's. That big bogeyman standing it's shadow over your creativity? Spit in it's face, tell it to go back and hide in it's grifting unproductive hole and get back to your prime purpose and directive for choosing to be here at this time. Remember though..if it hasn't got your hallmark of individual creativity on it yer probably a poseur.

 

I have organized conference proceedings around copyright for illustrators and have some general knowledge from that, but I'm certainly not a copyright lawyer. From what I understand, you can't 'copyright a style' and you can't copyright a concept. So an illustrator imitating another illustrators style is unfortunate but does not automatically run afoul of copyright.

What's protected is the final image that an artist makes. So while I could paint in 'the style' of Frazetta, I can't steal a particular instantiation of a figure, face, horse, etc. that is substantially similar to Franks. AI can probably get around that as the images are typically not direct copies of the original.

I think where AI will run afoul of copyright is that in order to imitate a style, it needs to consume the images the artist created to build a database. A human can be 'inspired' by looking at Frazetta, but does not require access to the actual images, it could be just from memory. I don't see how anything an AI produces can escape the fact that it used/consumed the actual, copyrighted images to do so.

 

That is, as far as I know, correct.

Once a human has looked at an image, said human has consumed it and has a copy in their mind. The copy is easily accesspible only to those with eidetic memory, though, but it is there for everybody else.

There is a legend that young mozart has copied a protected piece of music ("Miserere") after hearing it once. The legend is not true, but this sort of thing can be done.

And speaking of lookalikes, if you find Scarlett Johanson's complete lookalike and use their photos (with permission), can you be sued for using a likeness? This is kinda related.

 

Good luck at tracing out the one out of 400 million images in Dall-E!