Discussion Help Wanted: Setting up ML Agent for Board Game

Hello,

I'd love to use Unity's ML Agents to train for a turn-based board game. I have 13 years of Unity development experience, just no ML experience. I'd love to hear some ideas on how to approach this. In a nutshell:

Players choose a Hero. Each Hero has a unique board with abilities on it as well as different cards that are played by paying a Card Points cost. This is a 1v1 type of game.

There's also multiple phases to 1 turn that the player takes.

1. Draw card and earn a Card Point
2. Upgrade your Hero's abilities (if you have the applicable cards)
3. Choose attack against the opponent
4. Opponent gets chance to defend
5. Hero has option to discard any cards they don't want
Opponent's turn to do 1-5. Rinse and repeat until somebody wins.

This is a very broad overview, but any general ideas on how to approach the AI would be great! Some questions I've had about it:

1. How can we relate that upgrading an ability had some kind of reward for it?
1a. If that ability is chosen and does extra damage, is there some way to help the Agent learn the cause and effect of that choice?
2. What would a Rewards structure look like for this? Would we give rewards for each decision? Or only when the turn is finished?
3. Would doing separate Agents for each phase the player is in? But then how to appropriately give agent Rewards based on the outcomes?

Thanks in advance!

 

I am making a turn based board game as well! One of the advantages we have with this is that we can do A LOT more training samples. Just make sure everything moving has delta time and you should be able to get it to train many games a second.

Do not think of rewards being coupled with decisions, treat them separately. Give a reward either when the agent "wins" the game, or by some other metric that you deem "successful." You could do this at the end of a turn (after evaluating what the agent did to improve its situation), at the end of the game, or at any point really. You may have to adjust where and how much rewards (proportionally) are given as you train your agent. The agent will figure out how decisions lead to these rewards on its own.

(Aside: My view is that if you can get it to work, only giving the reward when the agent wins is ideal, as this removes any bias or misconception that you may have about the best way to play. With many training steps, this may work. However, you may need to add in other rewards to push the agent in the right direction.)

You probably only need one agent. Masking can be used if there are different decisions at each step. You can use teams if it is multiplayer.