When Animated GPU Skinned Instancing Goes Wrong

Hi there, I'm trying to implement a simple bare bones version of this paper to make a horde system.

I believe I've got the main features down and should have something workable but I'm not getting results anything like what I expect. The mesh comes out a mess.



You can download the assets I'm using here if you want to reproduce the issue.

The steps to this implementation are as follows:
Bake all bone transform matrices to a texture - AnimationBaker.js.
Pass per-vertex and per-instance data to shader - skinnedInstancing.js
Read animation texture to rebuild bone transform matrix - SkinnedInstancing.shader loadBoneMatrix()
Use conditional branching to skin the vertices by deforming vetex position by the bone transform matrix multiplied by the bone weights. - SkinnedInstancing.shader Vertex function

I pass bone weights and bone indexes to the shader by using the spare uv channel and vertex colours.
Also just giving each instance a different colour by instanced_properties.

Grateful for any advise or help on this, I've tried everything I can think of to solve this on my own and have the feeling I'm missing something simple. I've checked the forums and the internet for anything to shed any light on it but I think I might have read almost everything on the topic by now.

Here's the code:

AnimationBaker.js

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using System.IO;
5.  
6. public class AnimationBaker : MonoBehaviour {
7.     Texture2D animations;
8.     Animator animator;
9.     bool generated=false;
10.  
11.     void Start()
12.     {
13.         animations = new Texture2D (256,256,TextureFormat.RGBA32,false);
14.         animator = gameObject.GetComponent<Animator> ();  
15.         generated=false;
16.     }
17.     void Update()
18.     {
19.         if(!generated)
20.         {
21.             Transform[] subBones = transform.GetChild (0).gameObject.GetComponentsInChildren<Transform>();
22.             Matrix4x4[] binds= new Matrix4x4[subBones.Length];
23.             for(int i=0; i<subBones.Length; i++)
24.             {
25.                 //Debug.Log (subBones[i].right);
26.                 binds[i] = subBones[i].worldToLocalMatrix.inverse*transform.localToWorldMatrix;
27.  
28.             }
29.             animator.Play ("Movement");
30.             animator.SetFloat ("moveSpeed", 1);
31.             animator.speed=0;
32.             //animator.
33.             //loop through each animation and each frame and store bone transforms
34.             for (int y = 0; y < 40; y++)
35.             {
36.  
37.                 animator.Play("Movement",0,(float)y/40);
38.                 animator.Update ((float)y/40);
39.                 animator.speed=0;
40.  
41.                 for (int x = 0; x < subBones.Length*4; x++)
42.                 {
43.                     Color color = Color.black;
44.                     Matrix4x4 matrix = subBones[(int)Mathf.Round(x/4)].worldToLocalMatrix;//Matrix4x4.TRS(transform.TransformPoint(subBones[(int)Mathf.Round(x/4)].localPosition), rotation, subBones[(int)Mathf.Round(x/4)].localScale);//subBones[(int)Mathf.Round(x/4)].worldToLocalMatrix;
45.                     Vector3 p =  subBones[(int)Mathf.Round(x/4)].position;
46.  
47.  
48.                     //Quaternion rotation = Quaternion.Euler(subBones[(int)Mathf.Round(x/4)].eulerAngles.x, subBones[(int)Mathf.Round(x/4)].eulerAngles.y, subBones[(int)Mathf.Round(x/4)].eulerAngles.z);
49.  
50.                     //matrix.SetColumn(0, new Vector4(subBones[(int)Mathf.Round(x/4)].right));
51.                     //matrix.SetColumn(1, new Vector4(subBones[(int)Mathf.Round(x/4)].up));
52.                     //matrix.SetColumn(2, new Vector4(subBones[(int)Mathf.Round(x/4)].forward));
53.  
54.                     //matrix.SetColumn(0, new Vector4(subBones[(int)Mathf.Round(x/4)].position.x,subBones[(int)Mathf.Round(x/4)].position.y,subBones[(int)Mathf.Round(x/4)].position.z,0));
55.                     //matrix.SetColumn(1, new Vector4(subBones[(int)Mathf.Round(x/4)].rotation.x,subBones[(int)Mathf.Round(x/4)].rotation.y,subBones[(int)Mathf.Round(x/4)].rotation.z,0));
56.                     //matrix.SetColumn(2,  new Vector4(subBones[(int)Mathf.Round(x/4)].localScale.x,subBones[(int)Mathf.Round(x/4)].localScale.y,subBones[(int)Mathf.Round(x/4)].localScale.z,0));
57.  
58.                     //matrix.SetColumn(0, subBones[(int)Mathf.Round(x/4)].right);
59.                     //matrix.SetColumn(1, subBones[(int)Mathf.Round(x/4)].up);
60.                     //matrix.SetColumn(2, subBones[(int)Mathf.Round(x/4)].forward);
61.  
62.  
63.                     matrix *= binds[x/4];
64.                     matrix.SetRow(3, new Vector4(p.x, p.y, p.z, 1));
65.  
66.                     //for each pixel assign transform values clamped to 0-1
67.                     if(x % 4 == 0)
68.                     {
69.                         //set color clamped to 0-1
70.                         color = new Color( (matrix.GetColumn(0).x+2.5f)*0.2f ,(matrix.GetColumn(0).y+2.5f)*0.2f , (matrix.GetColumn(0).z+2.5f)*0.2f , (matrix.GetColumn(0).w+2.5f)*0.2f );
71.                         Debug.Log (matrix);
72.                     }
73.                     if(x % 4 == 1)
74.                     {
75.                         //set color clamped to 0-1
76.                         color = new Color((matrix.GetColumn(1).x+2.5f)*0.2f ,(matrix.GetColumn(1).y+2.5f)*0.2f , (matrix.GetColumn(1).z+2.5f)*0.2f , (matrix.GetColumn(1).w+2.5f)*0.2f );
77.                     }
78.                     //for each pixel assign transform values
79.                     if(x % 4 == 2)
80.                     {
81.                         //set color clamped to 0-1
82.                         color = new Color((matrix.GetColumn(2).x+2.5f)*0.2f ,(matrix.GetColumn(2).y+2.5f)*0.2f , (matrix.GetColumn(2).z+2.5f)*0.2f , (matrix.GetColumn(2).w+2.5f)*0.2f );
83.  
84.                     }
85.                     if(x % 4 == 3)
86.                     {
87.                         //set color clamped to 0-1
88.                         color = new Color((matrix.GetColumn(3).x+2.5f)*0.2f ,(matrix.GetColumn(3).y+2.5f)*0.2f , (matrix.GetColumn(3).z+2.5f)*0.2f , (matrix.GetColumn(3).w+2.5f)*0.2f );
89.                     }
90.                     animations.SetPixel(x, y, color);
91.                 }
92.             }
93.             byte[] bytes = animations.EncodeToPNG ();
94.  
95.             animations.Apply (false,false);
96.             File.WriteAllBytes(Application.dataPath + "./MyAnimations.png", bytes);
97.             //AssetDatabase.CreateAsset(animations, "./Assets/Animations.png");
98.  
99.         }
100.         generated=true;
101.     }
102. }
103.  

SkinnedInstancing.js

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class SkinnedInstance : MonoBehaviour {
6.  
7.     public Transform prefab;
8.     public int instances = 1000;
9.     public float radius = 100f;
10.     MaterialPropertyBlock props;
11.     MeshRenderer meshRenderer;
12.  
13.     void Start()
14.     {
15.         MaterialPropertyBlock props = new MaterialPropertyBlock();
16.         // for each vertex for each bone weight store 0-1 weight value and bone index
17.         Mesh mesh = prefab.gameObject.GetComponent<MeshFilter>().sharedMesh;
18.         Color[] colors = new Color[mesh.vertexCount];
19.         List<Vector4> boneIndex = new List<Vector4>(mesh.vertexCount);
20.         //Loop through verts and assign bone weights from the skinned version of the mesh
21.         for(int i = 0; i < mesh.vertexCount; i++)
22.         {
23.             float o,p,l,k;
24.  
25.             o = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].boneIndex0/51;
26.             p = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].boneIndex1/51;
27.             l = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].boneIndex2/51;
28.             k = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].boneIndex3/51;
29.             Vector4 index = new Vector4 (o, p, l, k);
30.             boneIndex.Add (index);
31.  
32.             float r,g,b,a;
33.             r = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].weight0;
34.             g = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].weight0;
35.             b = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].weight0;
36.             a = (float)GameObject.Find("PlayerMesh").GetComponent<SkinnedMeshRenderer> ().sharedMesh.boneWeights[i].weight0;
37.             colors[i] = new Color (r,g,b,a);
38.         }
39.         mesh.SetUVs(1,boneIndex);
40.         mesh.colors = colors;
41.  
42.         for (int i = 0; i < instances; i++)
43.         {
44.             Transform t = Instantiate(prefab);
45.  
46.             t.localPosition = new Vector3(Random.insideUnitSphere.x * radius,-0.01f,Random.insideUnitSphere.z * radius);
47.             t.localRotation =  Quaternion.LookRotation(Vector3.forward)*Quaternion.AngleAxis(Random.Range(-180,180),t.up);
48.             t.SetParent(transform);
49.  
50.             float r = Random.Range(0.25f, 1.0f);
51.             float g = Random.Range(0.25f, 1.0f);
52.             float b = Random.Range(0.25f, 1.0f);
53.  
54.             meshRenderer = t.gameObject.GetComponent<MeshRenderer>();
55.  
56.             // Get the current value of the material properties in the renderer.
57.             meshRenderer.GetPropertyBlock(props);
58.             props.SetColor("_Color", new Color(r, g, b));
59.             meshRenderer.SetPropertyBlock(props);
60.             meshRenderer.gameObject.GetComponent<MeshFilter>().sharedMesh = mesh;
61.         }
62.     }
63.  
64.     void Update()
65.     {
66.      
67.     }
68. }

SkinnedInstancing.shader

Code (CSharp):

1. Shader "Custom/SkinnedInstancing" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _AnimTex ("Animation (RGB)", 2D) = "white" {}
6.         _BumpScale("Scale", Float) = 1.0
7.         _BumpMap("Normal Map", 2D) = "bump" {}
8.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
9.         _Metallic ("Metallic", Range(0,1)) = 0.0
10.     }
11.     SubShader {
12.         Pass{
13.             Tags { "RenderType"="Opaque"  "LightMode"="ForwardBase"}
14.             LOD 200
15.          
16.             CGPROGRAM
17.             // Upgrade NOTE: excluded shader from OpenGL ES 2.0 because it uses non-square matrices
18.             #pragma exclude_renderers gles
19.  
20.             // Physically based Standard lighting model, and enable shadows on all light types
21.             #pragma vertex vert
22.             #pragma fragment frag
23.             #pragma multi_compile_fwdBase
24.             //#pragma multi_compile_shadowcaster
25.             #pragma multi_compile_instancing
26.  
27.             // Use shader model 3.0 target, to get nicer looking lighting
28.             #pragma target 3.0
29.             #pragma glsl
30.             #include "UnityCG.cginc"
31.             #include "Lighting.cginc"
32.             #include "AutoLight.cginc"
33.             #include "UnityInstancing.cginc"
34.  
35.             sampler2D _MainTex;
36.             sampler2D _BumpMap;
37.             sampler2D _AnimTex;
38.             half _Glossiness;
39.             half _Metallic;
40.  
41.             // Add instancing support for this shader. You need to check 'Enable Instancing' on materials that use the shader.
42.             // See https://docs.unity3d.com/Manual/GPUInstancing.html for more information about instancing.
43.             UNITY_INSTANCING_CBUFFER_START(Props)
44.                 UNITY_DEFINE_INSTANCED_PROP(float4, _Color)
45.             UNITY_INSTANCING_CBUFFER_END
46.  
47.             struct appdata_t
48.             {
49.                 float4 vertex : POSITION;
50.                 float4 color : COLOR;
51.                 float3 normal : NORMAL;
52.                 float2 uv : TEXCOORD0;
53.                 float4 bIndex : TEXCOORD1;
54.                 //float4 bWeight : TEXCOORD2;
55.                 UNITY_VERTEX_INPUT_INSTANCE_ID
56.             };
57.  
58.             struct v2f {
59.              
60.                 float2 uv : TEXCOORD0;
61.                 SHADOW_COORDS(1) // put shadows data into TEXCOORD1
62.                 fixed3 diff : COLOR0;
63.                 fixed3 ambient : COLOR1;
64.                 float4 pos : SV_POSITION;
65.                 half3 normal : NORMAL;
66.                 UNITY_VERTEX_INPUT_INSTANCE_ID
67.             };
68.  
69.             float4 _MainTex_ST;
70.  
71.             float4x4 decodeMatrix(float3x4 encodedMatrix)
72.             {
73.                 return float4x4(    float4(encodedMatrix[0].xyz,0),
74.                                     float4(encodedMatrix[1].xyz,0),
75.                                     float4(encodedMatrix[2].xyz,0),
76.                                     float4(encodedMatrix[0].w,encodedMatrix[1].w,encodedMatrix[2].w,1)
77.                                     );
78.             }
79.             // Read a matrix (3 texture reads) from a texture containing
80.             // animation data.
81.             float4x4 loadBoneMatrix(float3 animationData, float bone)
82.             {
83.               float4x4 rval={ 1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,0,1 };
84.  
85.               int baseIndex = animationData.x;
86.               baseIndex += 4 * (bone/256);
87.  
88.               float baseU = baseIndex;
89.               float baseV = (animationData.y/256);
90.  
91.               float4 mat1 = tex2Dlod(_AnimTex, float4(baseU,baseV,0,0));
92.               float4 mat2 = tex2Dlod(_AnimTex, float4(baseU+(float)(1/256),baseV,0,0));
93.               float4 mat3 = tex2Dlod(_AnimTex, float4(baseU+(float)(2/256),baseV,0,0));
94.               float4 mat4 = tex2Dlod(_AnimTex, float4(baseU+(float)(3/256),baseV,0,0));
95.  
96.               mat1 = (mat1-0.5)*5;
97.               mat2 = (mat2-0.5)*5;
98.               mat3 = (mat3-0.5)*5;
99.               mat4 = (mat4-0.5)*5;
100.               // Only load 3 of the 4 values, and decode the matrix from them.
101.               rval *= float4x4( mat1, mat2, mat3, mat4 );//decodeMatrix(float3x4(mat1,mat2,mat3));
102.               return rval;
103.             }
104.  
105.             v2f vert (appdata_t v)
106.             {
107.                 v2f OUT;
108.                 UNITY_SETUP_INSTANCE_ID(v);
109.                 UNITY_TRANSFER_INSTANCE_ID(v, OUT); // necessary only if you want to access instanced properties in the fragment Shader.
110.  
111.                 float4x4 finalMatrix = {    1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,0,1};
112.                 // Load the first and most influential bone weight.
113.                 finalMatrix = v.color.x * loadBoneMatrix(float3(0,1,0),v.bIndex.x*51);
114.                 // Conditionally apply subsequent bone matrices if the weight is > 0.
115.                 if(v.color.y > 0)
116.                 {
117.                   finalMatrix += v.color.y * loadBoneMatrix(float3(0,1,0),v.bIndex.y*51);
118.                   if(v.color.z > 0)
119.                     {
120.                       finalMatrix += v.color.z * loadBoneMatrix(float3(0,1,0),v.bIndex.z*51);
121.                       if(v.color.w > 0)
122.                         {
123.                           finalMatrix += v.color.w * loadBoneMatrix(float3(0,1,0),v.bIndex.w*51);
124.                         }
125.                     }
126.                 }
127.  
128.                 float4 vAnimatedPos     = mul(float4(v.vertex.xyz,1),finalMatrix);
129.                 float4 vAnimatedNormal  = mul(float4(v.normal.xyz,0),finalMatrix);
130.  
131.                 OUT.pos = UnityObjectToClipPos (vAnimatedPos);
132.                 OUT.normal = UnityObjectToWorldNormal(vAnimatedNormal);
133.  
134.                 OUT.uv = v.uv;
135.                 half nl = max(0, dot(OUT.normal, _WorldSpaceLightPos0.xyz));
136.                 OUT.diff = nl * _LightColor0.rgb;
137.                 OUT.ambient = ShadeSH9(half4(OUT.normal,1));
138.  
139.                 return OUT;
140.             }
141.  
142.                half4 frag (v2f IN) : SV_Target
143.             {
144.  
145.                 UNITY_SETUP_INSTANCE_ID(IN); // necessary only if any instanced properties are going to be accessed in the fragment Shader.
146.  
147.                 fixed4 col =  UNITY_ACCESS_INSTANCED_PROP(_Color);
148.  
149.                 // compute shadow attenuation (1.0 = fully lit, 0.0 = fully shadowed)
150.                 fixed shadow = SHADOW_ATTENUATION(IN);
151.                 // darken light's illumination with shadow, keep ambient intact
152.                 fixed3 lighting = IN.diff * shadow + IN.ambient;
153.                 col.rgb *= lighting;
154.                 return col;
155.             }
156.             ENDCG
157.         }
158.             // shadow casting support
159.             //UsePass "Standard/DEFFERED"
160.             //UsePass "Standard/DEFFERED_DELTA"
161.             //UsePass "VertexLit/SHADOWCASTER"
162.     }
163.  
164.     //FallBack "Standard"
165. }

 

Any Progress on your findings? I am also looking into offsetting my crowds to the GPU by using Graphics.DrawMesh() instancing functions. Any light would be helpful. I'll pm you with my progress from replicating. But the first thing i noticed was that using your shader on the man character vs a stock shader changes the size of the Character. Your shader scales the mesh to a very large size. Unfortunately My shader coding skills are novice at best.

 

Hi Reaper3223 thanks for the interest and having a go with the shader.

Currently I don't use Graphics.DrawMesh() instancing functions. I let the normal rendering pipeline do its thing in terms of instancing.

The shader should be used on a standard mesh which can be instanced, the animations are looked up from a pre-baked texture that stores the bone transforms. The vertex shader uses that information to move the verts to the desired position of a particular frame of animation.

The scaling comes from the animation texture where the bone transforms need to be scaled down to fit a 0-1 range and need to be scaled up in the shader. So if there is no animation texture it will return 1 for all it's texture lookups and the added scaling on that will make a mesh much larger.

The animationBaker script goes on a skinned mesh renderer and will capture the animations into a texture for you to use in the custom shader.

The problem is the transformations come out the other end as nonsense. So either I'm storing the bone transforms in the texture wrong or reading them incorrectly int the shader or some fundamental concept i'm missing.

Some other thought's I've had on this are instead of trying to do Matrix Pallet Skinning which maintains a skeleton hierarchy allowing for adding customization attachments and other benefits. I would take a simpler approach and store the relative position of each vertex in the texture instead as in this video

.

 

Thats because youre using skinned mesh renderer. You should be using regular mesh renderer if you do your own skinning.

 

What are the potential pitfalls going this route I wonder.. Do you think a 10,000 person goal could be feasible? Lighting /Shadows shouldn't be a problem if your not using DrawMesh() right? This is very interesting. I would love to replicate a system similar to the Assissin's Creed Unity Crowd System. Have you seen this?

https://www.gdcvault.com/play/1022411/Massive-Crowd-on-Assassin-s

 

Yes standard shadows and lighting will work and 10K characters should be feasible too. The trick is its just instancing a standard mesh but you animate it in the vertex shader.

Yes I've played Assassins Creed Unity and this should form the basis of a system similar to that.

 

Would you be interested in collaborating on this?

 

Sure, although I've done as much as I can and am truly stuck at this point. I don't understand why the bone matrix transforms don't get applied properly. That's why I posted on the forums to see if anyone else could shed some light on it.

If you get that working we could work together on fleshing the rest out. I'd happily make some characters etc. I started this because of the game I made for Ludum Dare you can see here https://ldjam.com/events/ludum-dare/40/Demon-Hunt

 

@jbooth @bgolus @Keijiro-UTJ are the people most likely to know where to start with this! ask them politely and maybe they will be able to help

 

Excellent. I tried to pm you but your profile doesnt appear to allow starting a direct conversation with you. In either event. I'll let you know when the Repo is up and ready for Unity Collab and get you connected.

 

If you really want this to get finished you should put it on git, far more likely to get decent developers pitching in that way and get it finished very quickly

 

@AVitt , Tried to start a conversation with you to send the Link to the GitLab Repo Project I setup. Send me a pm if you can so we can setup the GIT.

 

@Daemonhahn That's a good idea, thanks for the advice.

@Reaper3223 sent you a pm or started a conversation with you

 

Successful Testing of Writing Animation to Texture and Playing animation from Texture on Mesh Renderers. You can see some De-Scaling Distortion in the Mesh Renderers dealing with the Floating points its hard to get the Data to Scale to 0-1 then back without some noise in the values.

 

Nice work! Is that using the Adam scripts and shaders or mine?

 

The texture to animation was based on yours then modified for the new approach. everything else is its own blend of following the adams presentation and me trying different normalizing/denormalizing. I didnt get it right yet. The Method used in the presentation used direction vector and I didnt see any distortion in his playback using his method. But my current method is just straight value conversion so due to the floating point issues and it being hard set vertices rather than direction vector moved it has blobby look up close. But before I got back and start getting the Write/Read from texture perfect. I'm gonna switch gears to getting it onto the GPU shader side of things as this current state will work for prototyping purposes.

 

Really really interesting thread.
I'm looking for something similar but I'm not able to give you an help (I can't writing script for something complex like this).
I hope you'll make this happen.

 

great start though!

 

Lighting is a problem actually because you have to calculate the normals in the shader. For instance if you look at the animation instancing repo from Unity which uses the same approach (https://github.com/Unity-Technologies/Animation-Instancing), they punted on this problem completely.

Blending is kind of a big feature to give up also. Given that for a lot of projects where you just need a few hundred skinned meshes at most, IMO the correct solution is for Unity start by correctly jobifying their animation stuff, because that's where most of the resource usage comes.

 

Blending shouldn't be an issue in a normal use scenario where the Close Animated Characters are still handled by the Animator and only the distanced Characters are offloaded to the GPU.

Have you seen the Other forum I started for this "4,000 Adams at 90 FPS"?
It has the link to the Public Repo Source Project Files and a windows standalone downloadable for testing. Any furthering of this work would be greatly appreciated by me and I'm sure the Unity Community.

Shadows for the Real-Time Lights are not an issue. But yes you are correct the Lighting logic for the Shader will have to be added. I'm just very new at HLSL and pre occupied with Synstasis at the moment to devote more time to this side project currently.

Unity handles the Runtime shadows but the Current shader doesnt have Lighting logic yet, but can be added from someone more shader experienced.



//Forum Link for Open Source Repo (GPU Animated Mesh Renderers)
https://forum.unity.com/threads/4-000-adams-90-fps.520685/

 

Ok so blending is not an issue because you basically punt on it. That wouldn't be acceptable in every context, and IMO not really a great solution. It means you need to have two rendering systems at play. I'd rather bite the bullet and solve the underlying problem, or use another approach.

You say lights are not an issue, but until it's no longer an unknown as to how exactly you would handle it, it's definitely an issue. If it was just a matter of implementing a well known approach, that would make it a non issue in my book.