Building an AI system for enemys and bosses

Hello everyone. I'm working on a topdown 2D action roguelike at the moment. It's my first big project.
Here's a link if you want a better understanding what I mean and where I'm at - https://twitter.com/Kaktuspirat

I want to start creating many different enemies soon. Right now there are only two enemies with very simple AI.
The enemies will have states like idle, pursue, attack, wandering, flee, stun, defeated and patrol.
Some enemies, especially bosses will have more than one attack.
Because there's so much overlap when it comes to these states, I want as much as possible to be reusable.

There are so many options to create a system like this I'm not really sure how to start.

• I could do everything myself by creating a finite state machine or by using mecanims behaviours.
• I have Playmaker but haven't used it so far. It may be helpful for something like this?
• Then there are plugins like Behaviour Designer or other behaviour tree tools.

I'm about to do dive into these different options but I'm sure many of you already have experience with stuff like that. What offers a good workflow for creating enemys and bosses? What is the easiest to get into and maintain? Is something like Behaviour Designer overkill for my requirements? Does it make sense to use Playmaker for AI in a project that doesn't use Playmaker at the moment? I'm also not afraid of getting my hands dirty and building something myself. At the same time I also don't shy away from paying some money if it saves time and trouble.

I'd love to hear your angle on this.

 

There are multiple possible approaches. If/else tree, AI behaviors as components, coroutines, state machine, Behavior Tree.

See this post for an example:
https://forum.unity.com/threads/enemy-ai-help.880048/#post-5787295

The easiest to maintain is code. (opinion)
The easiest get into is a visual tool.

The reason for that is that you cannot diff or bisect visual tool AI.

Best idea would be to evaluate and see how it works.

No. Visual Scripting is a crutch that is mostly useful for people that can't program. If you can program, you don't need it.
However, you should consider looking for an asset that implements Behavior Trees. As those offer visual aid regarding what your AI is doing right now. Those are not exactly visual scripting.

You can implement fairly complex behavior using coroutines, in which case they'll act like code-based behavior trees.
Behavior trees give visual feedback regarding what your AI is currently doing, and are worth investigating.
If/else tree is still a viable tool if the AI is not very difficult.
It is also possible to split AI into MonoBehavior based components deriving from some interface, and have component based configurable behavior.
I would not recommend visual scripting like playmaker, though.

 

So Playmaker is out then. I thought so. Never felt like learning it just to be able to do something I already can do with coding. When I bought it a few months back I thought it would offer me what I will actually need a behavior tree for.
I will look more into behavior trees for now as it seems to be a tried and proven practice. It feels a lot cleaner than coming up with something myself, that will likely get a little messy and hard to maintain at some point.

 

The reason why I suggested behavior tree asset is because while you can design full full equivalent of behavior tree with coroutines, it is very hard to debug them due to excessive recursion (the problem is not recursion, but recursion in a coroutine). And coding state machine manually results in less-readable code. Behavior Trees, however, provide at-a-glance depiction of what your AI is doing.

^^^ Example from Unreal.

The reason why I suggest to avoid playmaker is because intent of such toolkits is usually to avoid coding, and and while trying to pursue this goal, they make development process slower and less efficient. Blueprints and visual tools have less readability than code, and you seem to be able to program. So you probably won't need them.

 

I would avoid nested if/else blocks, it could end up like this mess

We are working on and off on top tier AI and we are doing a hybrid between utility AI and behavior tree AI. We use node canvas which we are pretty happy with, it does its thing. Also we use A* pathfinding project for navigation, and .UMotion Pro for altering animations (If you want to make AAA like AI you will need to customize the animations)

 

Have you seen the Pluggable AI with Scriptable Objects tutorial videos?
I have 10-15 enemy types in my upcoming game (Smash Dungeon) that use a couple of Monobehaviours (different movement types, melee/projectile based attack etc) for AI but for bosses I used the Scriptable Object method in this video. I found using this I could break the attack styles in to phases as the health changed - so phase 1 might be stalk & melee, phase 2 stalk, melee, charge, phase 3 throw something etc.

It worked well for bosses but was a bit overkill for basic enemies hence my mono approach for those.

 

I'm kinda sold on behavior trees now. It sounds like learning to use those is really beneficial in general. The Pluggable AI with Scriptable Objects sounds interesting too though. I'll have a look at that too when I finished working.

 

I managed to make some decent patrol, wander, melee attack patterns using the pluggable AI but I struggled extending this to multiple attack patterns. Any insight/help would be appreciated for how you accomplished this. Additionally how did you manage attack cooldowns with all the logic being in scriptableobjects?

 

I ended up with a mixture of Monobehaviours & Scriptable objects to help deal with cooldown timers etc.
I had a StateController mono which kept track of which plugable AI. It also had a timer on which the plugable AI could read and write to which gave cooldown timers etc.
I found as the AI got more complex I was better off adding additional monobehaviours to cater for the per instance variables and using the scriptable objects more as a decision maker on what to run next.