The coming age of conversational NPCs

Executive summary: game NPCs will soon be able to carry on real, spoken conversations with the player. I'm interested in how this will impact game design.

So, let me set the stage by pointing out a couple of recent developments:

1. Apps like Siri, Amazon Echo, and Google Talk have already got us talking to inanimate objects. We have an Echo in our kitchen, and the whole family uses it on a regular basis. These aren't really conversations; it's a voice command interface. But... all of us have occasionally caught ourselves saying "please" or "thank you" to it.
2. A fully conversational Barbie is coming soon. It's not perfect, but kids don't (generally) care.

I believe these are the tip of the iceberg, and it's going to be in toys first where this really takes off. But video games are just expensive toys for grown-ups. It won't be long before the same technologies start to appear there. And it will probably be small indie studios, rather than the big AAA studios, who innovate here first.

So! Imagine you could just speak (you know, with your voice) to NPCs in a video game. And they would understand you and respond by voice. They're not geniuses (though they may be savants), but they can understand any common questions or commands and respond appropriately.

How would this impact the design of your game? People already get attached to NPCs sometimes, despite their clearly 2-dimensional character. How would this change when characters know you, remember things you've seen and done together, and reminisce with you later? What about subordinates that can follow verbal orders? And what about enemies that can engage in a bit of repartee before (or during) the big duel?

Any thoughts?

 

I don't like to talk out loud while I play games. It disrupts my entire family and keeps them awake if I am gaming late at night. Currently, I have to listen to my 15 year old play Minecraft with her friends, talking, making weird noises, for hours. I am very close to tossing her mic in the garbage can. Having her play in her bedroom on her laptop only makes her talk louder. lol If I could type to the NPC, I would be much happier.

So..very cool concept and I am sure we will see this sort of thing in game someday, but I am not looking forward to it. We need more human interaction, not more electronic interaction.

 

I'm with @Teila.. I guess I'm too old school. I don't even like dealing with things like teamspeak when I'm gaming. For me, it takes away from the immersion. I can see where spoken dialog with NPCs could increase immersion, but could also lower family/roommate faction. I prefer old school where I can pick a choice during interactions or way old school where I can type responses - especially if NPCs are smarter and can formulate understanding and not look for a 100% specific response.

 

It's going to be a long time before people are interested in actually speaking to characters in games versus playing then add they are now.

Voice driven games are usually flops. It's a cool idea but not sustainable right now.

 

Meh, I say you're all a bunch of old-fashioned fuddy-duddies.

No, seriously though, I do appreciate the feedback. I just think that, looking back on it 10 years from now, it's going to be in the same bin as "Why do we need to have moving pictures in our games? I much prefer text-based games, because the pictures I make in my head are so much richer than anything you can display on a screen." There's some truth to it, but it doesn't stop progress from marching on.

And, kids who grow up confiding in Barbie are going to simply expect to be able to speak to videogame characters at least as well. Show your granddaughter (in the year 2025) an old game where the characters can't talk, and she's going to be like "Hello... HELLO?!? Why won't he talk to me?!?"

​

And @LaneFox, voice driven games are usually flops now because they suck. Decent voice interfaces are very new and getting perceptibly better every year; it's only in the last few years that they've gotten barely competent, and the real bend in the progress curve hasn't even hit yet. That will happen when millions of talking Barbies, and other conversational toys, start hitting the market. This will provide the combination of market forces, forgiving audience, and massive amounts of real-world data needed to make this really take off.

So. For the sake of argument, can we just suppose for now that I'm right and this technology is coming? We can either stay firmly in the familiar and let others innovate with it, following grudgingly along after the fact; or we can think about how we would innovate with it, and make others follow. I'd like to get the latter discussion going. If you don't believe it's going to happen, then consider it just a bit of science-fiction fun.

Thanks,
- Joe

 

I don't think either of us said it was going to happen and in fact, I did say it would.

Just not something I plan to put into our games. Our goal is to encourage cooperation and interaction between players so it doesn't fit into our plan. We will leave that to you "younger folks" with your new fangled ideas.

Oddly enough, my teens prefer the older games to the new ones. They spend MUCH more money on those. So not sure about "kids these days" since I think they are more discriminating than we are in some ways. They have the ability to compare the older games to the newer ones and find the newer ones lacking.

I would not assume that just because some new cool fad comes out that you will get the next generation of gamers.

I remember Furbies....my kids threw them away after a few months and now remember them with distaste. I can name dozens of other such "next amazing electronic thing" that never caught on. I lived in the days of "intelligent houses", that now people hate because of all the maintenance. lol

And oddly enough, books stay popular, maybe on Kindle now, but people buy them like crazy. My daughter has a stack she wants for Christmas. I have faith in today's kids. I spent two hours in the car with 5 teens this past weekend and their conversations were enlightening. You know what? They were talking about some hand drawn artistic game on Kickstarter that they love and is so much better than the new games...no action, more story..but so cool. Three of these teens were boys. Then they talked about genetic mutation among insects in Japan. And then about how violent and bad so many movies have gotten...unnecessary violence that didn't add to the story or to the development of the character.

Wow..at there age, I wasn't talking about that stuff. lol

 

lol! That is almost the exact statement I kept making while sticking with Simultronics' DragonRealm before moving into graphics based MMOs. I avoided graphics based MMOs for a couple of years stuck in the mindset that I didn't want some artists painting pictures and showing my imagination where I was.

 

Conversations with NPCs aren't naturally enhanced. In fact they can be downright rubbish. A player who's a bit boring (lets face it, a lot of people are) wouldn't have a particularly brilliant conversation and might end up being bored by the responses as well. It's a novelty.

When you speak to a shopkeeper in real life, how much do you really want to drag that out? Also, it lacks creative writing. When the hero says something cool in a scripted event, people can laugh or appreciate it.

By using AI and real conversations you would end up laughing at your own jokes essentially.

So while there probably is a place for this, it isn't naturally a better thing.

 

@JoeStrout - How far are you willing to extend the speculation? There's a world of difference between a Teddy Ruxpin that remembers your favorite color and Daniel Day-Lewis in-character on improv night.

I do think knowledge awareness (or at least much better appearance of such) is near. But generating compelling natural speech will take longer. Once that's in place, though: VR dating sims and the end of the human race.

How times have changed! Seems like not long ago that video games were "only for kids."

 

It's not hard to imagine scenarios where having to speak to NPC's would actually lower immersion. If the protagonist is Lara Croft or a gruff space marine, and I had to speak to characters in my own voice, that's kinda funky. Even if the protagonist was very similar to myself, I'd be a terrible "voice actor."

 

My six-year old niece and three-year old nephew are precisely the reason I wear noise isolating headphones practically all day long. I even have a fan running in the background for some extra white noise. I still hear them on occasion but I'd imagine they're louder than your daughter.

Why not buy her an affordable headset (or just a microphone) that is capable of picking her up so she isn't trying to talk over the fans in her laptop? I know that can't be very fun for those listening to her either.

 

@JoeStrout I didn't say it wouldn't happen, I said its going to be a long time - like probably 10 years.

@hippocoder makes a good point too, it really dials back the quality you would be getting from scripted narratives.

 

We're decades before it's even remotely feasible in games. The major catch to siri and google is that these are grafted onto big data. Their ability to parse language is second to the massive databases fueling them. Even ignoring the computational expense, the operating a database is nowhere near possible, never mind the labor involved in filling that database.

The more fundamental issue is: what value does it bring? Unless you're working on a wiafu simulator and want desperate japanese guys to marry your game characters, will it actually make the game better? Maybe you can say it will give games a greater sense of world building (ignoring the massive amount of labor involved with feeding in information), but what happens when the player asks an NPC to do something? Will it even be enabled to do anything other than talk? Will it have a personality and tell the player to F*** off? Will there be multiple personalities?

There is no way an indie could make this. Even if it was a third party library/service (nearly unlikely at this point), just implementing it in any reasonable manner (that actually had a reason to be in a game and wasn't horribly out of place) would probably take a team the size of most AAA studios.

 

Do you know the TV show "black mirror"? I get a feeling you might like it.

I think there could be something interesting made with this technology that works well with a 3D VR experience. I could see a bigger indie studio make an interesting game where you play a police negotiator that needs to talk to crazy people that have hostages or want to jump from a bridge. I'd imagine such a setting would excuse a few not so natural npc responses that I have no doubt will occur.

I bought a game on steam where you play a mage and have to say magic words into your mic to make spells happen. It is an interesting gimmick but ultimately I barely played that game. In general I prefer not to talk if I don't have to. Even when I play Insurgency online, have my headset on and I'm home alone I usually use text chat instead of voice chat. One of the reasons for that is that I'm not a native English speaker and I don't speak it nearly as well as I can write it. I could see how that would be a problem with speech driven games in general if the game ends up misunderstanding people. I often have big trouble understanding other people that use voice chat myself, either because of their S***ty recording setup or their heavy accent. If it was working correctly I could see great value in using such technology to transfer voice to text clientside, transmit that data instead of the voicechat stream and then turn it into speech again with speech synthesis on the client side. If working perfectly and properly tuned that could boost immersion a lot I think. Also it could apply a bit of content filtering to auto-block voice spamming or insults about your mom.

Considering how this could be implemented in games like GTA, Skyrim or Fallout I would prefer to not have it in the game at all and focus on properly voice acted content with great writing instead.

 

It is not her fan. lol She is in her room alone and now she can be noisy. At the desktop, which is more powerful so her preferred computer, she is in the family room with the rest of us.

 

Indeed, but there's no reason games won't hook into exactly the same big data. When you use the Nuance SDK, the actual audio recording flies through the intertubes to their servers, where the actual speech recognition is done, and the parsed speech (including several alternatives) is sent back. When you speak to Siri or OK Google, the natural language processing is done in the cloud too.

As an engineer it strikes me as a horribly inefficient way to do it, but it works — most of the time. Reliability will go up, and eventually, I expect there will be offline options too, but in the meantime, I fully expect games to work this way.

Yes, thank you, that is the discussion I'm trying to start!

I should certainly hope so. I was trying not to interfere with the creative process by throwing out my own ideas too soon, but what the hey:

• To my wingman or cohort in a fighter jet/spaceship sim:
  • "Let him go, stay on the leader!"
  • "Red Three, draw their fire; Red Four, you're with me!"
  • "Get out of here, Biggs, you can't do any good like that!"
• To my companion in an RPG:
  • "Hang back and keep me healed, I'm wading in!"
  • "Wait here, but if I'm not back in three minutes, come find me."
  • "Take out the spellcaster!"
  • "You help Furg, I'll hold them off!"
  • "Have you seen the Count anywhere recently?"
• To a companion in an action game:
  • "Hide on that ledge and get your Plasma Lance ready; I'll lead 'em in!"
  • "I'm going in, cover me!"
  • "Quit using that pea shooter. What you need is one of these."

Those are mostly commands, and yes, in most cases I would expect the NPC to do what it's told (given the usual status of the player as the hero/leader of the story). This sort of thing can actually change the gameplay, or at least make it a lot easier to achieve what would otherwise require a tedius UI.

Of course there should be non-command dialogs too, particularly for players that enjoy role playing:

• "Why do you hate orcs so much?"
• "Whew, we almost bought the farm there, didn't we?"
• "Nice shooting, Ray!"

and so forth, all with appropriate responses from the NPCs. This sort of interaction doesn't really change the gameplay, but then again, neither does a high poly count or a rich backstory.

Sometimes, if that's in character.

You mean in one character? Or just different personalities for different characters? If the latter, then yes, of course.

This year, yeah. 10 years from now? I don't think so. There will be third party libraries/services, and they will be no harder to use than 3D modeling and rendering tools.

 

Honestly, if it's that level of engagement, I really want to play with my fellow human beings.

 

For a positive, though, I would expect a VR game to be much more immersive with this technology. It would be a case of "is it right for the game" rather than it being right for all kinds of games (it really wouldn't IMHO).

 

It could also help with games like Elite Dangerous that require many functions to be managed by the player. You can map everything to a controller but then you have to memorize dozens of button combinations to do stuff that you don't use often but need anyway. Things like "computer - activate landing gear" would make sense in the context of the game, make the input scheme more accessible and might even boost immersion a tiny bit. The "computer" keyword could be used to filter the sentence out from voice chat so that other players don't have to hear those things.
Thinking about it, that game possibly would be a good early candidate for an actual implementation since it is online-only anyway. So not really a problem of having the voice processing in the cloud (even though I'd never want to put something like that in a game that doesn't run clientside).

 

At least then you can hurt someone's feelings if they play like S***.

Let's just assume player stupidity for a minute, and say they leave Rick and Morty on in the background. How should an agent interpret getting schwifty?

Half of the time you should probably get a quizzical look, while the other half should be a slap in the face.

Being able to use natural language for commands is a great idea in theory, but it comes down to the whether the commands are complex enough to warrant doing without a UI. For simple commands, normal interfaces are faster. This is also the area where most of the current solutions fail miserably, and they have been for decades.

Somehow, I can't imagine creating personalities will even be remotely as easy as modeling a complicated character. God forbid it ends up being impossible to map input variables to the type of personality you get out, so it requires constant testing to figure out if it's the type you want.

Now if you actually want these to be characters in a world, you would then need them to build relationships. Then there is the world's knowledge base which has to be created by somebody. Unless every game that uses a conversation system is also running as a front-end for Dwarf Fortress, I can't see any indie project being capable of making something out of it.

 

I believe that's already a thing.

But issuing voice commands is a much different topic than having conversations with NPC's I'd say.

 

Yes. It's a step in that direction, technologically, but doesn't really get at the design possibilities that true conversations will open up.

Perhaps it will help fire imaginations to think of the Holodeck in ST:TNG.



Here the characters are playing a video game. An amazingly advanced VR game, sure, but still, it's just a game. They know the characters are all being played by the computer and don't (with rare plot exceptions) have any real consciousness behind them. In one episode, Picard, playing the Dixon Hill character, has this exchange with an NPC:

JESSICA: I'm not sure who wants me dead. My husband, my stepdaughter.
PICARD: Or a lover, perhaps?
JESSICA: Perhaps. Or perhaps it's Cyrus Redblock. I need you to find out. Name your fee.
PICARD: Twenty dollars a day, plus expenses.
JESSICA: Agreed.
PICARD: I haven't said yes yet.
JESSICA: Oh, you'll say yes, Mister Hill. (she kisses him) If it is Redblock, he must think I've got what he's looking for. But believe me, I don't.
PICARD: I'll take your word for it. ​

He does this by just saying whatever leaps to mind. Now suppose there were no conversational interface. Jessica (the NPC) walks in and says "I'm not sure who wants me dead. My husband, my stepdaughter." And then she freezes, and a menu appears in the air.

• Why should I help you?
• Or a lover, perhaps?
• Tell me about your husband first.
• Get out of my office.

And Picard uses gestures or some sort of pointer or gamepad to select the response that's most similar to what he wants to say. (I've been generous here by having what he would actually say in the menu — more likely, that option would not be among the available choices, so he would have to settle for something else.) And then Jessica gives her canned response.

Now, how can you argue that this would be a better game-playing experience than the conversational one?

My thesis is that in 10 years, the conversational interface will be standard, and nobody is going to want to pick responses from a menu anymore. It will just seem too limited and lame to bear. So the question is, what can we do with this new capability? How will it affect game design?

(And yeah, I realize this is like asking people in 1980 what they would do with realistic 3D graphics in game design... it could be we can't imagine it before it happens, but let's try anyway!)

 

Perhaps having a limited number of choices is actually beneficial to keeping an experience focused? Otherwise, it's up to the player to be creative and "role play", which is not for everyone.

Plus it would seem that the thread of the story could go off into uninteresting tangents. If it wasn't possible for the thread to go off in the wrong direction, then the illusion of a free form experience is broken anyway.

 

I'd like to start by bringing up something from SF Debris' review of "The Game" from Star Trek: The Next Generation...

Conversational AI is certainly a step towards holodecks. Instead of writing NPCs with static dialogue, we instead define a character; this is a person who has a history, personality traits including interests, quirks, vices, and virtues, and has a current context that relates to their personality, including goals that they want to meet in their 'life.'

The problem is the agency of this intelligent agent. I know very few people who just sit about talking about themselves, the narcissistic real-life equivalent of repeatedly shouting "WELCOME TO CORNERIA!" People usually are also doing something. People each have their own story that is unfolding.

The point of making conversational AI is to make a more life-like character. However as long as NPCs are secondary characters who have no impact on the player, as long as the player remains king, you will never have a lifelike NPC. This runs counter to everything video games have taught us in the last thirty years - the player is king - they're the most able, the most desirable person in their world, and their decisions affect everything over the course of their journey. ...Even if that includes how specifically you want to inbreed Chocobos.

The point is, you're cresting the tip of an iceberg. Conversational NPCs are well and good, and I look forward to games that use them, particularly Eastern-style RPGs. The problem is, unless you figure out a creative, resource-efficient way to design a world in which the player is one part of many, but without the player becoming irrelevant to the work, they won't live up to their full potential, and will either make themselves look flat by being literally all talk, or make the world look flat by fiat of being more advanced than the rest of the work they exist in.

 

As much as it seems like it's about creating better NPCs, this is actually about creating better worlds. If there was any use case for this, it's in MMOs and in completely generated worlds. Neither of which really care much about artistic vision.

 

I completely disagree; a game like Final Fantasy VII with this sort of system in place would be, to say the least, 'interesting.' "Shinra pulled the Sector 7 plate and kidnapped Aeris! What are we going to do?" Well, you could go the canon path of scaling the collapsed plate segment, by buying batteries...or you could perform some good ol' social engineering to get uniforms, legitimate ID cards, and literally walk in the front door, not like Barret "Leeroy Jenkins"...but calmly, casually, and even spring Aeris from Hojo's little experiment...before the Turks have you face the elevator music.

Such a system does not remove the linearity of game storytelling; what it would instead do is broaden the parts that are in the player's control. You're not shoehorned into, "I'm in this part of the game, so now I have to do that." It's more, "I'm in this part of the game, I have these options...but do I have more? Let me chat some dudes up and see..." You might actually have more options.

Also MMOs in my opinion are a dead genre, in the way that Latin is a dead language - based on what I see of the state of the industry and that genre at this point in time, I don't think that they can express any new ideas. All a conversational NPC will do is obfuscate that, not mitigate it.

TL;DR - Can you imagine FFVII as a holodeck simulation? Holy hell.

 

To try to spur this into a conversation about artistic vision, these sort of options don't necessarily make the story/the world stronger. Part of playing a character is coming to terms with how much of it is just a player avatar and how much is a discrete character. In this case if Cloud, Barret, and Tifa walk into a bar, you should expect some insurance claims, regardless of what the player does. Never mind that whole "evil force actively trying to control the protagonist" thing that sort of becomes a major plot point (which actually has more control over the plot than the player).

MMOs in the sense of the inefficient server architecture that dumps hundreds to thousands, and then some, in the same worldspace, then sure. There's still some room for games that have massive playerbases that interact concurrently and with one another, just without all the randos on the screen at an given moment.

 

The problems with conversational NPC's are not that different IMO then IVR systems. We have had IVR systems that make you speak your choices for over a decade yet most people hate them. Sure some of it is the tech, which btw I don't think is really going to be that much better a decade from now. But some of it is that it's not actually an improvement. It's not getting you to your objective any faster or easier and certainly not less error prone.

Take a competitive game. I'm trying to get to some goal, and your conversational NPC is just another obstable. Ya it might be an interesting conversation the first time, but generally I just want to get past it. And it's far less work and faster to click or escape through it.

So conversations need to actually add value to the player. Even if the tech was perfect I don't think a lot of people really want to talk to an NPC. We all know it's a computer. We listen to them, they give us information, but at the end of the day I have no desire to have a pretend conversation. It's actually more like a puzzle if anything.

And another thing is talking is often just not convenient. I get home sit down to play a game. I don't want to talk,I've been talking all day. Or maybe like me you stay up later then your wife/gf and don't want to wake them up. Or you are playing a game somewhere in public. Go to asian countries and most people play games in groups.

Plus you have social issues. Real life conversations are full of things that would probably offend some group somewhere. But that's what makes real interactions interesting. If you limit conversations to stuff everyone can agree one, it's just not interesting. People hang around people who think like they do. I think you have to find some way to mirror that within a game to make conversations interesting. Right now companies design their dialogs around not offending all the right groups. They need to look at that differently and find another solution, or interesting in game conversations just won't happen.

 

I think we can all agree that we want the holodeck and it would be amazing! The problem that TNG conveniently skips though is that the computer basically reads the mind of the person instructing it instead of following the instructions to the letter and filling in the blanks with statistically valid or random parameters. Even if the tech was there and working as good as it conceivably could I'd imagine the most frequent sentence said to the machine would be an angry "No! Not like that!".

I also think that the only real way to make real conversations happen is simulating a real person with their own experiences, neurochemical parameters and and personal goals. That would mean such a system would approach a level of complexity that it is no longer possible to debug it or predict what the final experience will be. I feel like that is 100% the opposite direction of where AAA is heading. You'd end up with games (or maybe just "experiences") that all had to be rated "M for mature" because real people can be scary things and if you simulate people there is no telling how dark the content that a player experiences could turn out. I'd say this all would be best suited for "movie like but interactive and open ended" experiences that don't follow the paradigms of games that are currently popular. Imagine Saw 1 as a game, but you are one of the characters chained to a pipe in that first room and the other one is the AI. The possibility space for physical interaction is finite in this environment and can be solved with traditional content creation. Now if the AI of the other person has endless possibilites to talk about stuff and knows his own backstory and how things in the world work etc. then I could see this being interesting. And only having one AI character would at least give a chance for playtesting to test in depth enough to ensure that at least most of the players will have an interesting non-buggy experience with it. I don't see how it could be done with 100+ real AI characters that all have a different pool of information to draw from. And I fear that the transitional period between a sophisticated real conversational AI solution and what we have now will be jarring uncanny valley stuff that could turn quite a few people off before the tech gets good enough.

Possibly the way to go is start with things that in the context of the game make sense to be "broken". For example in Fallout 4 you have a dog, some might enjoy giving commands to the dog via voice interface. The dog doesn't need to say smart things, he just barks at you from time to time and no one is going to be mad if a dog doesn't perfectly execute their complex commands.

 

Yes, you're right, and in fact the episode I quoted actually goes sideways shortly after, when Picard decides it's not the sort of game he wanted to play at all.

As I recall, there ensued a long conversation with the computer about exactly what sort of program he wanted, and adjustments were made. In real life, at least in the 21st century, it's more likely that this would be solved by reading reviews and more carefully selecting the game before you start playing. Some games will be long on dialog and intrigue; others will devolve quickly into shooting or whatever. But I think even in the latter case, players will come to expect to be able to converse with the NPCs at appropriate times.

You raise a good point about the uncanny valley. This is why I think the rapid progress in the next decade will begin with children's toys. Children are much more willing to suspend disbelief and tolerate mistakes for the sake of their pretend play. (They're also, in my experience, much more aggressive about playfully exploring the limits of the system by purposefully giving challenging inputs.)

In fact, many of the objections I've seen in this thread may have, at their root, an unwillingness of adults to engage in pretend play. After all, when role-playing, that's exactly what you're doing. Kids are great at this but most of us lose it as we get older. We feel silly. Maybe this is why pencil-and-paper RPGs aren't more popular, and why even in RPG MMOs, most players don't actually role-play.

But I suspect that conversational NPCs may help. If there are no real people around to see you pretending to be a 1950s detective or elf maiden or whatever, and all the other characters in the game relentlessly stay in character, then I think more people may find they can do it (and that it's a lot of fun).

 

@JoeStrout - This is something you're going to have to prototype, as difficult as that is to do. The best way to see the strengths and pitfalls of this idea, isn't so much to describe it with words, but to try to make it reality. Writing a conversational NPC in a time-efficient manner is something that I foresee being a particularly interesting challenge, and some of the work from @TonyLi with defining an NPC based on personality traits may be useful for this.

 

If you want to use something similar now, you can go to luis.ai. It is the system used by Microsoft's Cortana and is intended for commands. It wouldn't work very good for conversation, but for ordering troops into battle, it could work great.