Large scale battles ( at least 100,000 extremely high quality 3D animated characters with AI)

Hello friends
I am looking for a way to reach a level of performance similar to Ultimate Epic Battle Simulator 2 game for my medieval 3D high quality Real-Time Strategy ghame.
my game can handle 8000 extremely high quality 3D animated characters with AI ( Navmesh agents with realtime navmesh updates for updating castle walls for navigation ) and I use MeshAnimation package for reaching 8000 extremely high quality 3D animated characters with AI at 35 frames per second. without MeshAnimation package I can get around 3300 extremely high quality 3D animated characters with AI at 20 frames per second. MeshAnimation package bakes animations into meshes for gaining performance.
however I need at the very least 12.5X more performance to reach 100,000 extremely high quality 3D animated characters with AI at 35 frames per second ( 60 frames per second on better machines)
when I have around 10,000 of the same NPCs my frame rate drops to less than 10 frames per second. ( mainly due to my RAM capacity which is 16GB)
can anyone help me reduce RAM usage and get my performance to 100,000 extremely high quality 3D animated characters with AI at 35 frames per second?
does unity have such settings which I can reduce RAM usage?
I looked at unity profiler and found nothing which causes frame drop.
I dont want to use Entity Component System at least until the official version 1.0.0 release. *(please dont ask me why)
*Edit : I have no Update methods in my codes , I update the game using Coroutines every few seconds.)
my computer specs :
CPU : AMD Ryzen 7 5800H
GPU : Nvidia Geforce RTX 3060 Laptop 6GB
RAM : 16 GB DDR4 3200 MHz with Timings : (22-22-22-23-1T)
SSD : 2TB Nvme PCIE Gen 4.0 6900 MB/s
SSD2 : 1TB Nvme PCIE Gen 4.0 3000 MB/s
Monitor : 1080p @ 144 Hz
OS : Windows 11
Unity version : 2022.1.14f1

 

To reach such mass of AI its quite hard in terms of "how unity handle certain things", i mean unity helps to make things much more easier for certain cost.

ECS would be my gues, since you dont want to use it, my question would be, where is your bottleneck?
GPU / RAM / CPU?

at 100.000 entities probably all of them.

My Ideas are as following:

- Addressables
- In case you dont see all AI in one screen, Multiple Scene with dynmaic Load/Deload (works wonders)
- Shadow Settings (Work with it, especially in a crowd simulation)
- Avoid Garbage (GC)
- Use the right Data Structure (this would be ECS) but since you dont want to use it now, i would suggest, create your own kind of ECS Lifecycle, one Manager which have an "Array" of all entities (hopefully you dont have the need to add/delete dynamically AI from your Array, if so, you will have to use a list probably).
You can diversify this idea even further by having two Managers, one for close Range where AI Tick once every second (for example) and the AI further away ticks every 15 seconds (for example).
- Since you use AI with health and some other values (i would gues), use SO which saves Memory instead of assigning it for ea. and all AI in the inspector.
- Hash values for example in terms of AI - Animations, use Animator.StringToHash instead of animator.SetTrigger("STRING), this is one of many examples you can use across many use cases which improves performance.
- Redcure Hierarchy complexity
- Use Billboards if possible
- Avoid having "ANY" MonoBehaviours on your AI if possible, handle them through your Manager, in that case you dont have any Unity Dependencies which the Engine check, as for example for the Script and any Mono Events as Update/FixedUpdate/LateUpdate. Use Plain C# Files.
- In case of "LOADS" AI, there is also an Idea i have read:

 

Thank you very much for these information.
I will do all the above then test my performance again.

 

I tested my game while disabling my "Unit" script which was the script attached to all AI and player NPCs so if I noticed any improvement in performance I rewrite my own ECS way but there was no performance difference at all and my game crashes when it uses 64 GB RAM and wants to use more than 64 GB RAM ( 16 GB RAM + 48 GB pagefile on NVME SSD). my CPU usage is around 55 percent while having 7172 characters in my scene and my GPU is running at 100 percent clock speed and 100 percent usage while the camera is on the army and at 12 percent usage while the camera is rendering a place of the map which does not have any characters but in both cases my game slows down eventually until it crashes at 64 GB memory usage.
I think my game is extremely RAM intensive. how to fix it? is there any way to force unity use more than 64 GB memory in my game? my pagefile is 384 GB large and my physical DDR4 RAM is 16 GB. and is there any secret caching of data or something similar happening in the background which uses RAM? ( if it is possible in unity) and how to disable caching if it is happening?

 

By using so much RAM you have something wrong in your Data Structure or the Assets you are using are overusing RAM.

We are in development of an RPG since quite a while and we have a huge Medieval Town with High Quality 3D Models.
Without using any adv. optimisation, with a Behaviour Tree State Machine, i spawn 1000 AI having used 1GB of RAM (or less).

Or ofc. you dont use Adressables (BUT) even then its way too much RAM you use, in that case i would love to see what Profiling tells you, where is the RAM going?

 

I think 100.000 entities is just way too much, no matter what you do, you are going to have performance issues, i promise you.

 

Not true.
Ultimate Epic Battle Simulator 2 can handle millions of characters and it is using Unity without Entity Component System.
An example of 6 million characters in Ultimate Epic Battle Simulator 2 :

 

You misunderstand the point. Ultimate Epic Battle Simulator 2, probably can do this, but you, a lone dev cant compare or have the knowledge that they do. Regardless of engine (and ive used Unity/UE and in-house engines), you cant come here and ask for a easy answer, for your lack of knowledge, or lack off staff.

You are not a studio, or team of many. What you ask takes many people, systems, and custom content and tech, you do not have. Good luck implimenting what you ask, i await your results...

 

it does not need staff or a ton of knowledge. it is about a few multithreading tricks on animation and pathfinding and a few tweaks in unity memory allocation setting.

 

Use a pathfinding solution that uses dijkstra. That solves it, since most agents go to the same place. So you need to "solve the navmesh", not each agent. It's also not hard to program.

 

If every agent go to the same place, than you can use Flow Field Path-finding.

 

Would something like the DOTS boids example work for this style of game?

 

Im still awaiting the results...

 

Are you LODing and 2D imposter'ing your characters and animations?

As that's what most large-scale battle games do distant soldiers are reduced to 2D sprites.

Have you considered creating blocks or groups of soldiers as AI agents as opposed to individual soldiers.

 

Update : my project file size is 637 GB now and 400 GB of it is only baked animation clips into meshes as .asset files using MeshAnimator v1 package in the asset store. maybe upgrading to v2 would help reduce the size of my total assets. and my bottleneck is RAM. when opnening my project. it uses 314 GB RAM (64 GB RAM + 250 GB pagefile) and I have set my page file to 512 GB on 2 NVME SSDs. my project used to use less than 64 GB RAM a month ago because my character types were much fewer (60 GB or less for animatedmeshes .asset files). LODs make it worse since it should bake multiple LODs and a ton of more meshes (will result in even higher .asset file sizes) and may make my project take 1 terabyte of space. CPU is used at 4 to 21 percent most of the times (Ryzen 9 7950X) and GPU is at 10 to 35 percent load most of the times (RX 7900XTX Sapphire Nitro+ ) and VRAM usage is18 GB out of 24 GB but RAM usage is maxed out and making my build throwing memory allocation errors. (my build size is 52.8 GB, it was 7.13 GB a month ago and worked perfect). 1 month ago I had around 10 character types. now I have over 60. and I need over 400 character types (military units, lords, ladies, ministers, workers, miners, farmers, bandits, peasants, merchants, animals etc) and I need a better solution other than MeshAnimator asset in assetstore since MeshAnimator mentioned it is best for low number of character types but high number of the same characters in the scene. any suggestions on which animation baking asset to use?

 

That's a very extreme project you have there. I have no experience with animation baking assets, so I cannot recommend one. But I understand how those work, they trade cpu time that would be needed to play regular animations for memory. Like almost any algorithm, one can usually modify things to either have more calculations or more memory usage. In this case the balance seems off, because you are using too much memory. Maybe you can reduce the details of your baked animations, improve compression or stream them (via something like addressables). If the data size is so large, you are basically defeating the purpose of the animation baking. Maybe you can analyze which animations are used most often and only bake those, keep the rig and mix them with regular animations, basically trade back some memory for more cpu usage. Overall I think a game of such a scale is only possible following a data oriented design approach, which does not necessarily mean using ECS. (I would use that though)

 

Also you mentioned that you haven't created any LODs for your units yet. If you want to make this scenario run, I would start there. The higher LOD levels are the ones, that need the animation baking that you were doing, not the LOD0. After all if a player can see tens of thousands of units on the screen, they would necessarily mostly be at higher LODs. Then your baked animation data would also be at a much more reasonable size, because it is based on decimated meshes, you probably dont need 30 fps animations, but for example 15 fps instead. And close up units at LOD0 can be skinned meshes using regular animations with all the detail you want, because of how a (perspective-)camera works, you would be guaranteed not to have too many of them. Of course it still is a balancing act and you should use the unity profiler to see what's actually happening with memory and CPU, instead of the operation system CPU/GPU load statistics..

 

Your idea worked. I deleted my baked assets then reduced the bake quality settings by a lot then rebaked them. then when the camera is near 1 or more particular characters (for example the distance between the camera.main and those characters are less than 80 meters ), those caracters render with skinnedmeshrenderers and have realtime GPU Compute Skinning 4bone based animations, high quality particle systems for blood, smoke, dust of feet etc and extremelty realistic and accurate physics and can render hundreds of characters in realtime, else if the camera is too far away and is rendering the characters (for example tens of thousands of characters), then thoese characters disable their realtime child and enable their baked child gameobject with baked animations and regular mesh renderers and simple physics and very low quality particle system for blood (if hit by the enemy arrows or other weapons), and when the camera is very very high and rendering those characters, LODs come to play and baked animations run at low fps, physics are extremely basic and even particle systems are disabled and when the camera is not rendering those charactyers ( for example the camera is rendering another part of the map which does not have characters ), the animations, particles and other non_necessary scripts are disabled and thoese characters have only pathfinding, AI, group AI and very very basic physics. This made my baked asset sizes from 314 GB down to 34 GB. and can run very very smooth and fast. I think about ECS and DOTS a lot but I am waiting for them to be released and their animation packages be released too. ECS is now in pre_release stage and does not support 3D animations. Thank you very much for your idea.

 

Your idea worked. I deleted my baked assets then reduced the bake quality settings by a lot then rebaked them. then when the camera is near 1 or more particular characters (for example the distance between the camera.main and those characters are less than 80 meters ), those caracters render with skinnedmeshrenderers and have realtime GPU Compute Skinning 4bone based animations, high quality particle systems for blood, smoke, dust of feet etc and extremelty realistic and accurate physics and can render hundreds of characters in realtime, else if the camera is too far away and is rendering the characters (for example tens of thousands of characters), then thoese characters disable their realtime child and enable their baked child gameobject with baked animations and regular mesh renderers and simple physics and very low quality particle system for blood (if hit by the enemy arrows or other weapons), and when the camera is very very high and rendering those characters, LODs come to play and baked animations run at low fps, physics are extremely basic and even particle systems are disabled and when the camera is not rendering those charactyers ( for example the camera is rendering another part of the map which does not have characters ), the animations, particles and other non_necessary scripts are disabled and thoese characters have only pathfinding, AI, group AI and very very basic physics. This made my baked asset sizes from 314 GB down to 34 GB. and can run very very smooth and fast. I think about ECS and DOTS a lot but I am waiting for them to be released and their animation packages be released too. ECS is now in pre_release stage and does not support 3D animations. Thank you very much for your idea.

 

Nice to hear, good job on implementing those changes so fast.

Yep, even though I develop with the tech stream 2022.2 version where Entities 1.0 is released, I'm also waiting for the animations. My game project is kind of the opposite of yours anyway, all focused on first person closeup with story and only a few characters, so ECS seems less necessary. Do you have some sort of homepage for your game?

 

I have a Youtube channel which I have a lot of videos from my game and a lot of other videos from stronghold crusader extreme and 2 videos each from one of my other 2 game unity projects (one is realistic supercar street racing and the other is horror/fps/survival/mafia based mixed ).
here is my Youtube channel link :
https://www.youtube.com/channel/UCwv4J_p69A0jaDB9zotZdwA
I have not uploaded any videos from my latest change which I discussed in this thread but still the performance (frame rate and number of troops) is good in older version of my project.
Thank you very much.