Good uses for machine learning?

Nowadays there are some pretty well-developed algorithms that allow code to efficiently predict outcomes, classify things, and so on. These are rarely used in games, except for games that are entirely built around that (such as Black & White or Creatures), where (in my opinion) they don't work very well anyway.

So I'm pondering where machine learning could be effectively applied to enhance the player experience in regular game designs. Here are some thoughts to get the ball rolling.

1. Helping the Player

a. Choice Helper — In any sort of what-to-build-next context, an AI could learn to predict what building/unit/tech/upgrade he would choose next, and pre-select that in the selection dialog. This way, the player only has to smack OK most of the time, or could even turn on some sort of auto mode that bypasses the dialog entirely.

b. Companion AI — This one's harder, but if you have a companion NPC, it could be useful for it to predict what the player is going to do at various points, so the NPC can do something helpful (or at least get out of the way). If you always like to sling fireballs when facing frost minions, your companion should get behind you. But if you usually stand back and buff the companion, then it should charge right in.

c. Trainable Pet — this is stepping a little bit into Creatures/Black&White territory, but pets in games goes all the way back to Rogue/Nethack days, and usually they're pretty stupid. If I have a pet, I want to be able to train it to do stuff (whether it's useful or just fun).

2. Challenging the Player

a. Predicting Player's Moves — this one is like the Companion AI, but used for evil; the AI could learn your favorite moves in order to counter them. This might be especially effective in something like a fighting game, and should prevent the player from finding one combination that always works, since if you keep doing it, the AI (just like a human opponent) will come to expect it.

b. Predicting Outcomes — not quite the same thing, but a similar effect: enemies could learn the likely outcome of their own decisions, so over time they make choices that work better against you and your style of play. This helps avoid the common AI failure where an enemy makes the same dumb mistake over and over again.

Does this inspire any thoughts in anybody? If your NPCs or enemies were actually controlled by paid actors in Elbonia (who don't speak the player's language but are otherwise ordinarily smart), what would you do with them, and how would this affect your game design?

 

The idea of having AI learn how best to help/assist the player is intriguing. And novel! If one were clever, I expect you could design an entire game around that one idea. Or perhaps, the game mechanics might have to be extremely deep such that the AI doesn't end up 'playing' the game for the human.

Gigi

 

AI opponents are old, but learning opponents are rare. I actually wrote a custom AI for Newbie Chess, but like all chess engines, there isn't a whit of learning in it (though there is a bit of randomness, both to add variety and to give new players a fighting chance). I think that's fairly typical.

And yeah, we don't want to take over and leave the player with nothing interesting to do. I guess this relates to the micromanagement topic over in the sim game thread. But I know I've certainly gotten to places where it just feels like busywork... Thinking things like "Let's see, we've got a granary here, and a library, so now it's time for the barracks" over and over. I'd be fine with the computer at least pre-selecting barracks for me. Presumably I've got other things to do (exploring, attacking enemy cities, etc.), plus in some cases I'll want to override it due to special circumstances.

 

This thread's been live a whole 45 minutes and no one's mentioned Left 4 Dead's AI Director.

Its actions are heavily hard-coded (though parameterized), but that may be ultimately inevitable because any AI has to work within the context of a specific game and its particular elements.

There's been some academic research into this kind of dynamic management for story-based games (e.g., RPGs and adventure games). I'm skeptical about its applicability in games that rely on story. Would it make sense to try to build a robot to crank out Monets and Rembrandts? I understand the appeal -- there aren't enough humans to create all the content that players want. Machine-generated content could fill the gap while aligning the content to the player's play style. Is there a place for machine learning here?

 

Hey Tony, thanks for bringing that up! The AI Director is quite nice, and I think will become more and more common in high-end games (especially action games). Level designers try to manage the player's emotional state anyway, to keep them engaged and having fun, but when this has to be done entirely ahead of time, with no feedback on what (or how) the player is doing, this is a very chancy business. Something like the AI Director can do the same thing much more effectively, since it can take current state into account and adjust on the fly.

I'm not sure I see a big role for machine learning (ML) in it, though. But maybe such a director could be even more effective by predicting how the player will react to things it might do (certain numbers and types of spawn for example). E.g., this player does fine on Common mobs, but seems to totally fail at Witch mobs, and they're currently weak and probably going to die if we throw another Witch at them now, so let's give them something else instead.

Of course some players like a challenge like that. What you really want to know is whether the player is having fun, or getting frustrated. There's been a little bit of work on frustration detection, based on controller (esp. accelerometer) inputs and/or reading facial expressions, but this hasn't made it into a commercial game as far as I'm aware.

As for story generation, that's another topic entirely, I think. And I agree, it's a very tough nut to crack.

 

Your second category has existed in games for a significant amount of time.
Many games, though specifically RTS's, use Neural Nets at design time to determine the best solutions to problems, creating decision trees that drive the AI decisions within the game.

An example, you have an RTS with 3 units: a tank, a soldier and a helicopter.
You tell the Neural Network about these units, then it performs tests of different combinations of units against each other.
Perhaps 1 tank vs 1 tank, then 1 tank vs 2 tanks, then 1 tank vs 1 soldier, 20 tanks vs 5 soldiers and 2 helicopters etc.
(sometimes this is easier accomplished by putting the Neural net inside the engine, allowing it to use the game to perform its tests, especially in modern games)

Once it has ran through the tests (sometimes thousands of times for combinations in the millions), it produces a set of outcomes that it structures into either decision trees or graded tables (or however you need the data really).

Then you program your AI to make use of the pre-learnt data, giving it a huge depth of knowledge when it comes to the ins and outs of your specific game. The downside to this precomputed data is that you have to update the Neural Net every time you alter parameters/update the game. It results in lightening quick decisions with a huge amount of complexity though. Not ideal for every game, but it's my favourite kind of AI system

 

Fair point — I should have distinguished online learning (i.e., while the end-user is playing) vs. offline learning (what happens during development). I'm mainly interested here in online learning. Neural networks are generally too slow to be of much use for that, though there are some new training algorithms (developed in part right here at my local university!) which make it dramatically faster for some cases.

 

Perfectly understandable, there's nothing stopping you from using both in conjunction though
In fact it's pretty difficult using a Neural Net trained system without having it react dynamically to situations at runtime.

 

I played a game where cavemen can be set to follow your character, and by repetitively performing different actions they would learn to do that task and become a semi-autonomous entity.

 

That's fair enough in some cases, but not in others. Decision trees and Bayesian classifiers don't really fit the curve-fitting model, and they're both quite effective for some kinds of problem.

Yes, you're absolutely right about the inputs and outputs; those are crucial to getting any algorithm to do something useful.

I generally agree about the tuning too, but I wouldn't let that stop me. At least, not if there are good reasons to work ML into the design — which is the point of this thread.

 

Sounds interesting. I don't suppose you can dredge up the name of this game? I'd like to check it out.

 

The issue, I think, with machine learning, is that machines don't learn like humans. It takes a long time, and a lot of exposure to samples, before a machine noticeably picks up on anything. And most human players aren't going to want to sit around and participate in the AI's scientific experiments while it figures out whether running headfirst into a bottomless pit is a good strategy.

 

Worked quite a lot with interesting algorithms in academia, not only machine learning. But the problem with algorithms such as Artificial Neural Networks(ANN) and Genetic Algorithms are that they often need quite some time to 'learn'. There are however ANN implementations like NEAT that are quite interesting for AI in real-time.

A lot of the algorithms are really interesting for procedural generated content though.

An algorithm that i find very interesting at the moment though are Monte Carlo Tree Search, as it "learns" in real-time by doing play-out's at random and finding the move with best rewards. The problem here is finding the most efficient way to do a simulation of the game and finding the correct rewards.

 

It's awesome that you mention Monte Carlo Tree Search. I'm interested in it, too. It's so new that there's a lot of untapped potential for game AI. I've been following the papers and mcts.ai, but I haven't yet done anything with it myself. I understand Total War: Rome II uses it, as described in this AIGameDev article. It seems great for RTSes: random sampling with statistical history to take advantage of best-so-far solutions, perfect for an environment with so many possible moves. I wonder though how quickly it can adjust to a player making a complete 180 in playing style. The number of moves involved in changing playing style in an RTS might be enough to allow it to adjust, but it might get thrown by something like an RPG (e.g., AI enemies in Baldur's Gate or The Witcher). If you've played with the algorithm, do you have any insights about this? (I have in mind an MCTS implementation that's pre-trained at design time but learns at runtime.)

 

Well that is one of its strengths. It only looks a the current state, so it does not care about that. Of cause you can create implementations which saves information from previous states, but as such that is not the 'norm'.

Created a simple MCTS for pacman. Worked OK, but needs to improve my reward calculation.

 

Point taken. I had in mind an implementation where subsequent traversals are guided by statistics accumulated in previous trials.

Unrelated to that, thanks again for bringing up MCTS. I'm putting the finishing touches on a procedural quest generator, and I think I'm going to revisit the algorithm, especially given papers like this one on procedurally generating stories with MCTS.

 

Caveman Craig.

I was obsessed with it for a while, I kept creating mock game designs based on the concept... hell, I'm still kind of obsessed with the concept of characters mimicking your actions and becoming autonomous. I made many suggestions to the developer, but he was one of those hard headed (albeit talented) youngsters who didn't listen much, he thought it was perfect how it was.

Then I became fascinated with the idea of these autonomous entities learning by watching other autonomous entities that were more highly proficient... Something mind blowing about there coming a point when you are no longer able to control all of the actions of your own minions.

 

Well heck, that does sound neat. If you ever decide to pursue it, count me in (for whatever level of participation you like).

 

When I stop sucking at scripting in Unity, get through the stuff I'm working on I might. It could be a while but I'm going to do it eventually.

 

I think it was the first Need for Speed game that was learning from the player... so long since I played those games and mostly because a friend loved them.

The funny thing was, I learned how to cheat the physics engine and get the car to drive on walls and stuff like that. It turned into its own game for us but then, one day... we were the criminals and got chased. They police had a roadblock, so we drove on the wall to avoid it but then came police cars from the opposite direction... on the wall This escalated with AI cars that intentionally started to drive into rocks to flip through the air to avoid us when we were the police and it all tuend into a stunt, cheating mess

My lesson from that game was that you have to be careful when you give the game learning capabilities. Unexpected things can start to happen and it can get even worse when the game allows the AI to follow online and potentially teach other clients its new skills.

I'm currently considering some sort of learning system for the game I am working on... It would be nice to get enemies to behave mote naturally, adapt to the players favorite tactics and force him (me) to change his way to play.