Question Low FPS with a large number of CharacterController.

I plan to have about 1000 NPCs on the stage, everyone does something, for example, walks from point A to point B, sometimes jumps if necessary.
I attached a CharacterController to them and for the test I just spawned 1000 units.
script minimum

Code (CSharp):

1. controller.Move(MoveDirection * Time.deltaTime * speed);

fps immediately decreased to 5-8...
That is, it's just an empty object with a CharacterController, more and more there is nothing on it, while such a low FPS, and I didn't even attach a script to them that will direct them...

What can be done in this situation?
How can you move 1000 units without dropping fps?
(they should definitely have a controller since the terrain is generated, slopes, cliffs...)
(3D Project)

 

The answer is simple: don't use GameObjects. They're not supposed to scale to thousands of characters. Use entities instead (DOTS).

As a general rule of thumb, things in games aren't done the same way if you have 10s of them, 1000s of them, or 100k of them. For instance, a crowd in a stadium wouldn't be done using GameObjects and AnimationControllers, but wouldn't be done using entities either: you'd use instancing and VATs (vertex animation textures).

 

This is exactly what I'm doing right now...

 

You mentioned you're using CharacterController? that's a component for GameObjects, so you're not using ECS/DOTS right now.

If you mean you're doing a stadium crowd, use VATs:
https://stoyan3d.wordpress.com/2021/07/23/vertex-animation-texture-vat/
https://medium.com/chenjd-xyz/how-t...ters-with-20-draw-calls-in-unity-e30a3036349a

Note this is intermediate/advanced stuff, unless you know some shader basics and quite a bit about how GPUs work you'll find it rather confusing.

 

Now I have completely changed the approach to the implementation of the plan ... and yes, I use vertex animation, but this is not enough, so I turn to entities ...

 

I'm a bit confused here.... vertex animation textures are by far the fastest approach. ECS won't help much here. I guess you're not using VATs but regular skinned mesh renderers?

VATs are "vertex animation" like skinned mesh renderers or blendshapes, but they don't work the same way. VATs encode animation clips in texture data (RGB channels store XYZ vertex positions) and use a special vertex shader to deform mesh vertices in the GPU.

 

no no, you misunderstood, vertex animation works fine, but due to the sheer number of game objects and their children, there is a performance drawdown.

my objects looked like this

ObjectPrefab
--ChildLOD0
---- Animator, CharacterController, Mesh...
--ChildLOD1
---- Animator, CharacterController, Mesh (Low Detail 60%)...
--ChildLOD2
---- Object Vertex Anim
....

child lod objects (not to be confused with the lod system, just called them as well) turn on as the camera approaches, vertex animation is not applied in lod0 and lod1 objects, because there are too many vertices and shaders will not pull them, and IK is also needed, and if already further than lod2 objects are included in vertex animation. thus, the prefab contains both a detailed animated object and a vertex object for distant viewing and mass use. but... a large number of such objects certainly affects performance...

 

You’re confusing “vertex animation textures”(VATs) with skeletal animation via skinned meshes (what the Animator and SkinnedMeshRenderer components do). They’re two completely different things.

The point of VATs is that they perform animation in the GPU in a very cheap way that makes it compatible with instancing, which is what in turn allows you to render thousands of animated meshes in a performant way without the need of having a GameObject for each individual character. I’d recommend taking a look at the links I provided in my previous post to understand this.

To be clear: what you want is not doable using GameObjects with CharacterController /Animator components, since they don’t scale past a couple dozen characters. Also, the built in LOD system won’t be of much use since it happens in the CPU.

 

you misunderstood me again - since inverse kinematics cannot be used in vertex animation, and there are too many vertices for the shader in a high-poly model, this technology is not used, this happens with an object when it is close to the camera, respectively, there are many such objects in the camera they won’t hit, and when the camera moves away and the object becomes smaller, then more objects get into the camera, and since they are smaller, vertex animation is used ...

the fact is that my object can be close to the camera, but also distant that several thousand such objects are placed on the screen.

thus, sometimes (when zoomed in) I need both animation with inverse kinematics and a high-poly model, and when it moves away, low-poly and vertex animation is already applied there.

the problem is that all these model options (all lods) are simultaneously in one object and the necessary ones are turned on / off.

eventually there are a lot of unused (turned off) objects on the stage, but even though they are turned off they consume performance.

 

The problem here is not animation: it is the fact that you have one GameObject per character. There’s absolutely no point in using VATs if you animate each character mesh individually since you get the worst of both worlds: bad performance (since they’re individual meshes, and probably individual drawcalls unless dynamic batching kicks in) and limited animation capabilities (no state trees, no IK, no events, etc).

Again: the point of using vertex animation is that it allows you to use instancing and get rid of the GameObjects. Keeping the GameObjects around and switching from skeletal to vertex animation based on LOD won’t help much with performance, you’re saving on CPU skinning but that’s it.

If you need to swich back and forth between VATs and GameObjects, consider batching your vertex-animated meshes together into a few instanced drawcalls, then splitting them into GameObjects when they get close (use a GameObject pool for this).

 

vertex animation is the same Gameobject with a mesh renderer, only the shader changes the position of the vertices according to the positions of these vertices recorded frame by frame, but this object also needs to be moved, rotated, texture switched in the shader for another animation, and you also need to take into account the gravity system, etc. .
it's not just a stadium with a crowd of people...

 

Store your character data in arrays: Translation, rotation, uv offset (for texture atlasing), animation offset, velocity, etc. One array entry per character.

You can efficiently process all your characters in parallel using the jobs system, using multiple threads and vectorized code (if using Burst compiler). That still allows you to move each individual character around and store state for each of them. This is *a lot* faster than having each character’s data scattered around in memory and processed one by one in the main thread, which is what you get with GameObjects.

Such an approach is generally known as a data-oriented or data-centric approach, very common in particle systems: store data in a compact, efficiently accessible and cache-friendly way (aka arrays) and then have code that transforms that data taking advantage of its layout and the fact that most operations only affect one entry in each array, so it’s often easy to parallelize.

Coincidentally, that’s exactly what the DOTS (data-oriented technology stack) is for. And why DOTS also has two physics engines available (Unity Physics and Havok), so that each “entity” (which is just a glorified index in a bunch of arrays, no GameObjects involved) can have its own gravity, collisions and so on.

Now, to use this approach you can’t have SkinnedMeshRenderers and Animators components around, since that’s a completely different (much slower) paradigm. Once you have all your per-character data updated in an efficient way in the CPU, you need to also render them efficiently: having each character be a separate mesh doesn’t cut it in terms of performance, hence why you should use instancing.

Problem is: you can’t use instancing with SkinnedMeshRenderers, and you can’t use a data-oriented approach with GameObjects either. You only can do both with VATs. Afaik, DOTS doesn’t have a built-in equivalent of Animator, so it’s either VATs or roll your own animation system using DOTS.

Your current approach of having individual GOs for each character and swap between skeletal animation and VATs based on distance LOD won’t help, because neither updating or rendering that many GameObjects individually is viable.

Hope I have painted an accurate picture of it all, and that it helps!

 

I'm already completely confused, that is, I can't use animation and inverse kinematics in conjunction with entities?

the main reason for switching to entities was that I have many objects that are inactive, but at the same time they are always present in the prefab, the main object that moves, let me describe in more detail the scheme by which I now have prefabs arranged and why it heavily loads the system...

here is the device of the prefab itself (I replaced the word LOD with a variant, not to get confused)
also I replaced CharacterController with my control script.

ObjectPrefab
+ Script Movement (Move, Gravity, collisions)
+ Script DefinitionVariation (determines which option to enable depending on the camera range)
(DefinitionVariation turn on/off the desired variations)
--ObjectVariant0
---- Animator, IK Foot and Hand, Model 100% detailing
--ObjectVariant1
---- Animator, IK Hand, Model 60% detailing
--ObjectVariant2
---- Mesh Renderer + Shader Vertex Animation, Model 10% detailing
(if DefinitionVariation determines that the object is not visible to the camera, then turns off all options)

as a result, even if the objects are outside the camera and all options are turned off, the prefab moves with a bunch of turned off objects, but even though they are turned off, they load the system.

as I thought, the entities will change this problem, but I already doubt it, I'm confused. How do you think - what is the best direction for me to choose to solve this problem?

(so that you understand what I have planned, I will describe a little - in short, imagine a simcity game, a city, a couple of thousand people move around, but you can choose any character and control it from a third person, while everyone else on the map continues to move in their positions) I know it’s hard to do what was planned, but I still hope that I will succeed, the main thing is to understand with what help this is best done ...

 

Nope. Animator (which is in charge of skeletal animation and IK) is to be used with GameObjects. There's no GameObjects in an entity-based system -that's the point- so no Animator.

I guess Unity will add some sort of animation system compatible with DOTS at some point, but we are not there yet, at least not that I know.

Ok, now I get a clearer idea. Most games would just divide the city into sectors/areas of some sort, and dynamically despawn/respawn characters as they enter/exit the sectors around the player. There's absolutely no need to update the entire population of the city every frame, even if you want them to live their own lives (go to X place, do X activity, etc). No one is going to see them to that anyway, so why bother? you're only going to see what the people currently around your player's camera is doing. If you decide to follow a specific guy around, he'll be updated when moving between areas of your city, so he'll never be despawned and you'll get to see him do his stuff. But that guy you saw crossing the street 2 minutes ago? he's gone.

Entities won't help with that, neither will do VATs or instancing. You just need to rethink your game for a moment and find ways to make it look like everyone in the city is being constantly updated, other that just looping trough everyone every frame and updating them.

 

but this is the problem, I need that even though they were not visible, their positions were calculated, I will not go into details for what it is needed, but according to the idea it is necessary.
(this is an online project and other players can be in the same city and interact with other NPCs)

I don’t think that calculating only positions will take a lot of performance, but somewhere you need to put the disabled objects when they are not visible or when they are not used, if you deal with this issue, then there will be no problems ...

any ideas how it can be implemented?

 

There’s no “correct” way to do this, it all depends on the specifics of your game. You could for instance keep some minimal data about all your characters in a single object (only positions, for instance) and update these in a job in the server. Then clients would only spawn GameObjects for those citizens that are actually in the same region as the player.

There’s lots of techniques you can combine to do this: a regular grid for determining “regions” in the city, a pool to spawn/despawn GOs, etc

 

but the camera is constantly moving and scaling, as in any RTS, so those objects can be visible now, but not in a second, and in a couple of seconds they are visible again, creating and deleting objects with such dynamics will not work ...

so I’m thinking in what way to do it, that I even began to study entities in the hope that this would solve my problem, and now I don’t know what else to think of ...

 

That's why the "object pool" pattern exists: you don't destroy/create objects, you take them out of the pool when required and return them to the pool when no longer needed. This way you reuse the same objects over and over, instead of constantly allocating/freeing memory.

I should also point out the obvious: you should not be checking every single character every frame to see if they're visible: you place them into some spatial partitioning structure (a regular grid is the simplest, though you could also use hierarchical ones like BVHs) and you test visibility for entire regions instead of individual objects.

 

at the expense of the visibility zone, I know everything is calculated normally there.

but I didn’t know about the pool, thanks for reading ...

 

I answered without reading the article about pools, put it off the next day, it turns out that in fact my system works like this for me, only in the article these objects are in the sheet, and I have them right in the prefab hierarchy...

That's what I thought - is it possible to achieve this result with the help of compute shaders?

I just haven't studied them yet. just wondering if it's possible to do something similar with them?

 

If you're using pooling that certainly means you're not updating all 1000 GameObjects all the time, which is your original problem?

Compute shaders won't help here anymore than DOTS would. They're designed to process large amounts of data in a highly parallel way, but that data must be laid out in flat arrays (RawBuffers or StructuredBuffers). It's very similar to DOTS in spirit, the only difference being that processing happens in the GPU.

Processing data in the GPU is extremely fast, but bringing the results back to the CPU is extremely slow. So compute shaders are most useful when you want to process data and then use it for rendering immediately without the data ever going back to the CPU. In your case this would likely force you to implement most of your game logic in the GPU.

At the end of the day, you need to get rid of as many GameObjects as possible, and move character updating to a faster system. Your best bet imho is to have all your character data laid out in a data-oriented way (basically arrays) and process it all in parallel using jobs. Then, identify the ones that are visible by the player and spawn GameObjects for them (using a pool).

 

I read how pools are made and this is the same as what I did, there is one object (prefab) in a few more, but they are disabled and turned on as needed, but - an object without anything and an object with other objects turned off are different things, turned off objects those attached to the prefab also consume resources, this is very noticeable when there are a lot of prefabs on the stage and they have a lot of disabled objects such as animator for example.

I conducted tests and compared, for example, a thousand objects

1. Prefab (just transform)
2. Prefab
-- Object With Animator (but object off)
-- Object With MeshRenderer (but object off)

the difference is that with objects turned off, FPS sags by 30% than there are none at all ...

 

here is a visual example of what I need, but it's a pity there is no code for this example...
https://www.reddit.com/r/Unity3D/comments/sdvked/100_000_agents_with_compute_shader/

 

This isn't pooling at all.

Even if the Animator and the MeshRenderer objects are off, the root Prefab (which I guess it's where the CharacterController is, and the object actually updating your character's behavior) is still active. You're just enabling/disabling child objects, but at the end of the day you still have one GameObject for every single character in your city, and they're all updated one by one in the main thread.

The point of a pool is to reuse GameObjects. If you have 1000 characters in your city and only a maximum of 50 are visible at any given point in time, you create a pool of 50 prefab GameObjects. You update your 1000 characters using a system other than GameObjects (I suggested using a data-oriented approach and jobs for multithreading), determine the ones that are visible, and then take GameObjects from the pool to represent them. If a character becomes invisible, you return its GO to the pool to reuse it when some other character becomes visible.

This allows you to update 1000 objects efficiently regardless of them being visible or not, and still have Animators for those that are visible.

 

You don't need compute shaders for this at all, if you only plan on updating 1000 or even 10000 characters. The downside of using compute shaders -as explained before- is that you can't take the data back to the CPU in an efficient way, so you wouldn't be able to use animator or IK with this.

You can however use jobs/burst just fine, and still benefit from Animator/IK using a pool.

Here's a simple boids system implemented using jobs:
https://github.com/komietty/unity-jobsystem-boids

 

but after all, all 1000 objects can be visible at the same time, so it turns out that all 1000 objects should also be in the pool ...

and the example above states that it was done with compute shaders, the result is pretty impressive considering they can be manipulated

Now I'll look at your example and try to implement it with my idea ...

 

You're completely missing the point. I think we already established the problem is not rendering 1000 objects (because you don't have to use Animator, IK etc if they're points in the distance: you could just use instanced billboards), it's updating 1000 objects. In order to do this in an efficient way, you can't have 1000 GameObjects with a character controller script in them. It only makes sense to use GameObjects for the ones that are visible (and by visible I mean those than actually require using Animator and IK because the player is looking at them from up close, so those very far away from the camera wouldn't use GameObjects either).

You stated before you need to be updating all 1000 objects all the time regardless of them being visible or not. This is what jobs and pools allow you to do: decouple how you update and how you render them. You'd be updating them in an efficient, parallel way (using jobs is best, since compute shaders won't allow you to access the updated positions in the CPU) and then use GameObjects to render those that require complex animation/IK/whatever.

Now if you need to use Animator and IK for all 1000 characters, then sadly it's impossible to do.

 

CPU->GPU: OK.
GPU->CPU: No-no.

The problem with compute shaders is that you wouldn't be able to use Animator or IK at all once you've updated your characters, because these work in the CPU and your character positions are being updated in the GPU. So when using compute shaders you're back to GPU-only animation techniques like VATs.

Edit: the guy that did the video states he used VATs for animation in the video comments:

If you want a GPU-based particle/boids system (which is what the video you posted essentially shows), Unity has a built-in one: VFX graph.

 

example boid (5000) works for me with 10-20fps, it's like not the result that I need ...

Add: a simple change in position and then gives out 30fps with so many ..

 

Should work much, much faster than that. Make sure you've installed the Burst compiler and that your jobs debugger is disabled.

 

I'm figuring it out now, at the moment I'm confused by the number of batches, although the SRP batcher is on...

add: burst is also installed

 

This problem has nothing to do with batching.

If all 5000 spawn in the same position (like in your screenshots), it will be extremely slow because this is a boids example, which means each object's trajectory depends on all other objects around it. Having them all in the same place means they all have to check everyone else.

Set your area size to 200,200,200, your distance threshold to 20, and slowly add boids by pressing "A". I can get to 5000 boids at 60 fps, on a laptop computer.

Also, get used to check your profiler to determine what the bottleneck is.

 

no, the same thing, maybe my PC is "very outdated"?

OS: Windows 10 (x64)
Proc: AMD FX(tm)-8320 Eight-Core (8 Core)
RAM: 16Gb
GPU: NVIDIA GeForce GTX 1050 Ti (4 Gb GDDR5)

Editor unity: 2022.2.10 (URP)

 

but I took a clean project and installed the boid example there and these are the results...

I figure out what's what...

 

This is an example of a boid that works perfectly, I get 60fps stably, this is an excellent result...

it remains to understand one thing - can I apply the technology from this example for my needs? (I mean to achieve the same result as in the example of 100k agents)

it's just that I've been sorting through technologies for days and trying to figure out what I need to use specifically, what scheme to build in order to get the effect of a controlled crowd ...

on the one hand, the VAT technology is good, but for more low-detailed objects, on the other hand, with a large approximation, they should have animators with inverse kinematics ... based on what was discussed in this topic, I have the following scheme

1. Object coordinates must be processed in a separate script, only path calculations
2. this data is sent to objects for rendering on the screen (if it is within the screen)
2a. if they are small (camera is far away) vertex animation is used.
2b. if they are large (the camera is close and many of them won’t fit), ordinary game objects from the pool with animators are used (I think there will be no more than 50 of them)

the concept of what needs to be done is built a little. thanks for the tips,
I'll run some tests to get an approximate result ..

 

I LOVE YOUUUUUUUUUUUU SO MUCH THANKS MAN