Search Unity

get skinned vertices in real time

Discussion in 'Editor & General Support' started by benblo, Dec 10, 2008.

  1. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    I need to get the vertices of a skinned mesh as they are deformed... problem is, the SkinnedMeshRenderer component only exposes the sharedMesh, which isn't updated (it's the asset itself, not the mesh instance).
    Am I missing something?

    The MeshFilter component exposes both the mesh and sharedMesh, but from what I see it's not in synch with the SkinnedMeshRenderer.
    I found trace of a SkinnedMeshFilter, which sounds like it would be just what I need, but I fear it's an obsolete piece of code (doesn't compile, SkinnedMeshFilter is unknown... I never heard of it before either).

    I'd be okay with reinventing the wheel, ie manually calculating the vertices positions using the bones, if anyone knows the algorithm... some sort of barycenter of the bones' positions balanced by their weights?
     
    ModLunar likes this.
  2. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Okay... so apparently it is a simple barycenter or whatever it's called, script below will draw gizmos on each vertex in real time.

    If someone knows a less crazy solution, I'm still interested though... I'll log a "gimme the mesh instance dammit" feature request when it's not 2am.

    Code (csharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4.  
    5. public class SkinnedVertices : MonoBehaviour
    6. {
    7.     Mesh mesh;
    8.  
    9.     class Bone
    10.     {
    11.         internal Transform bone;
    12.         internal float weight;
    13.         internal Vector3 delta;
    14.     }
    15.     List<List<Bone>> allBones = new List<List<Bone>>();
    16.  
    17.     void Start()
    18.     {
    19.         SkinnedMeshRenderer skin = GetComponent(typeof(SkinnedMeshRenderer)) as SkinnedMeshRenderer;
    20.         mesh = skin.sharedMesh;
    21.  
    22.         Debug.Log("{0} vertices, {1} weights, {2} bones",
    23.             mesh.vertexCount, mesh.boneWeights.Length, skin.bones.Length);
    24.  
    25.         for (int i = 0; i < mesh.vertexCount; i++)
    26.         {
    27.             Vector3 position = mesh.vertices[i];
    28.             position = transform.TransformPoint(position);
    29.  
    30.             BoneWeight weights = mesh.boneWeights[i];
    31.             int[] boneIndices = new int[] { weights.boneIndex0, weights.boneIndex1, weights.boneIndex2, weights.boneIndex3 };
    32.             float[] boneWeights = new float[] { weights.weight0, weights.weight1, weights.weight2, weights.weight3 };
    33.  
    34.             List<Bone> bones = new List<Bone>();
    35.             allBones.Add(bones);
    36.  
    37.             for (int j = 0; j < 4; j++)
    38.             {
    39.                 if (boneWeights[j] > 0)
    40.                 {
    41.                     Bone bone = new Bone();
    42.                     bones.Add(bone);
    43.  
    44.                     bone.bone = skin.bones[boneIndices[j]];
    45.                     bone.weight = boneWeights[j];
    46.                     bone.delta = bone.bone.InverseTransformPoint(position);
    47.                 }
    48.             }
    49.  
    50.             //if (bones.Count > 1)
    51.             //{
    52.             //    string msg = string.Format("vertex {0}, {1} bones", i, bones.Count);
    53.  
    54.             //    foreach (Bone bone in bones)
    55.             //        msg += string.Format("\n\t{0} => {1} => {2}", bone.bone.name, bone.weight, bone.delta);
    56.  
    57.             //    Debug.Log(msg);
    58.             //}
    59.         }
    60.     }
    61.  
    62.     void OnDrawGizmos()
    63.     {
    64.         if (Application.isPlaying  enabled)
    65.         {
    66.             for (int i = 0; i < mesh.vertexCount; i++)
    67.             {
    68.                 List<Bone> bones = allBones[i];
    69.  
    70.                 Vector3 position = Vector3.zero;
    71.                 foreach (Bone bone in bones)
    72.                     position += bone.bone.TransformPoint(bone.delta) * bone.weight;
    73.  
    74.                 int boneCount = bones.Count;
    75.                 Gizmos.color = (boneCount == 4) ? Color.red :
    76.                     (boneCount == 3) ? Color.blue :
    77.                     (boneCount == 2) ? Color.green : Color.black;
    78.  
    79.                 Gizmos.DrawWireCube(position, boneCount * 0.05f * Vector3.one);
    80.  
    81.                 Vector3 normal = Vector3.zero;
    82.                 foreach (Bone bone in bones)
    83.                     normal += bone.bone.TransformDirection(mesh.normals[i]) * bone.weight;
    84.  
    85.                 Gizmos.DrawRay(position, normal);
    86.             }
    87.         }
    88.     }
    89. }
     
    ModLunar likes this.
  3. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Damn... now how do I get the skinned normals?? sounds trickier...
     
    ModLunar likes this.
  4. Jonathan Czeck

    Jonathan Czeck

    Joined:
    Mar 17, 2005
    Posts:
    1,713
    I think the skinned mesh data is written to a special write-only portion of memory for performance, or something. I brought this up a while ago but can't find the thread. At any rate, it's like how it is for a reason.

    Cheers,
    -Jon
     
    ModLunar likes this.
  5. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,603
    especially with the higher end cards and the pure shader engine, its likely not written anywhere outside the GPU RAM, so out of your reach.

    it only works if it falls back to CPU animation mode where the data are required to be on RAM (basically only Intel CPU to DVI adapters, that do not have any acceptable hardware vertex transformation performance if any support at all *X3100+, the 900 / 950 don't have it*)
     
  6. StarManta

    StarManta

    Joined:
    Oct 23, 2006
    Posts:
    6,525
    It shouldn't be. Normals should be exactly the same as vertices as far as the math is concerned.
     
  7. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Thanks Jon, it makes more sense... still sucks though.


    To transform the vertex I do this
    Code (csharp):
    1. Vector3 position = Vector3.zero;
    2. foreach (Bone bone in bones)
    3.     position += bone.bone.TransformPoint(bone.delta) * bone.weight;
    I just don't see what math to do to transform the normal... I'm probably just too tired though, it's 4am here ;)

    Edit: just played Night of the Cephalopods, pretty cool :)
     
  8. StarManta

    StarManta

    Joined:
    Oct 23, 2006
    Posts:
    6,525
    It should be the same, just using TransformDirection instead of TransformPoint.
     
  9. llavigne

    llavigne

    Joined:
    Dec 27, 2007
    Posts:
    977
    Unity compute skin on the cpu only.
     
  10. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Well... that defies my sense of space (which is pretty crappy to start with), but you're right it works, thanks a bunch!
    FWIW, I edited the script above to gizmo normals.


    So... which is it? UT, any chance to have a definitive answer on this? (David I know you're out there, stop hiding ;))
     
  11. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,536
    Skinning is currently done on the CPU.CPU.
     
  12. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Thanks Aras... so why can't I get the deformed mesh? (dammit please)
    Could you explain what Jon hinted at? and more importantly, will that ever change? (after Windows dammit)

    Edit: because right now it kinda works but I seem to have a small offset between the rendered mesh and some vertices I compute, and some normals are way out... plus this reinventing of the wheel is really inelegant, and I like my code pretty.
     
  13. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,536
    Because the vertices indeed are directly written into a "dynamic vertex buffer". That memory usually sits on AGP or PCIe bus, and writing to it is okay, but reading is slow. Possible, but slow.

    Will it ever be implemented - probably. What we could do, I guess:

    * Whenever skinned result is requested, compute skinning at that time. So each reading of the skin would compute the whole result.

    * Or, the ability to say to the skinned mesh renderer "I'll need skinned results, keep them around please". Then it would not skin directly into AGP/PCIe memory, but instead into a temporary buffer. Which then would be copied to AGP/PCIe. And temporary buffer could be returned to your code.

    Now, when that will be implemented - good question. I don't know. If enough people shout "I need it dammit", then it will be sooner. The problem is that people shout "I need this dammit" for thousand different things, and we have to sort them somehow :)
     
    ModLunar likes this.
  14. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Thanks! So I guess both parties were right, it's kind of in between, good to know it's doable in theory.

    BTW, any idea why I have an offset (I'm working on LateUpdate)? does my algorithm seem correct?

    Gee, this is surprising! :D
    After all you only had 15000 bugs reported in the last month... slackers.
     
    ModLunar likes this.
  15. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,536
    I'm not sure if it's correct, specifically the "deltas" part. Maybe it is, I did not write down the math and see if it matches.

    Basically, the skinning is this:

    Code (csharp):
    1. result =
    2.   matrices[bone0]*value*weight0 +
    3.   // ...
    4.   matrices[boneN]*value*weightN;
    Where 'matrices' would be a matrix array. Each matrix in the array is:
    Code (csharp):
    1. currentBoneMatrix * bindPose
    The 'bind pose' matrices is nothing more than just an inverse matrix of the bone at mesh export time - i.e. the matrix that transforms from mesh-as-it-is to local space of the bone. Then current bone matrix takes from local space and transforms into world space.

    Now, the above skinning code can of course be rewritten as:
    Code (csharp):
    1.  
    2. finalMatrix =
    3.     matrices[bone0] * weight0 +
    4.     // ...
    5.     matrices[boneN] * weightN;
    6. result = finalMatrix * value;
    7.  
    here, matrices[bone]*weight just multiplies all components of the matrix by a number. Basically, it's fine to "blend" the matrices first. Whichever is faster, depends on the situation.
     
  16. andeeeee

    andeeeee

    Joined:
    Jul 19, 2005
    Posts:
    8,768
    This would actually be useful even if it is slow. For example, you might want to emit particles from the surface of a skinned mesh object to simulate an explosion, disintegration or whatever. I would have found this useful on a couple of occasions.
     
  17. metervara

    metervara

    Joined:
    Jun 15, 2006
    Posts:
    203
    This won't work for me, seems it's not possible to multiply a Matrix4x4 with a float?

    /P
     
  18. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,536
    That was pseudocode :) Multiplying a matrix by a float here is multiplying all matrix elements by that float.
     
  19. metervara

    metervara

    Joined:
    Jun 15, 2006
    Posts:
    203
    thanks :)

    Adding the matrices together is also done on the element level then?

    /Patrik
     
  20. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,536
    Yes.
     
  21. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    So in your pseudocode, result and finalMatrix are Matrix4x4, right? As for value, I don't get what it is :? !

    And once I have that result, how do I get my vertex normal?
    Code (csharp):
    1. Vector3 position = result.MultiplyPoint3x4(mesh.vertices[i]);
    2. Vector3 normal = result.MultiplyVector(mesh.normals[i]);
    3.  
    Sounds right? should I use MultiplyPoint3x4 or MultiplyPoint?

    Also, pardon my ignorance but does currentBoneMatrix (which I assume is bone.localToWorldMatrix, bone being a Transform) change every frame? or can I do currentBoneMatrix * bindPose on Start?


    I left this issue standing because it was good enough (just had that offset), but now I want to tidy things up... it was only a month ago, yet getting my head into this again is hell! Never do tomorrow what you could do today, right?
     
  22. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Mmmmokay, so assuming value is the unskinned vertex (mesh.vertices), then finalMatrix * value is a Vector4, which (apparently) can be cast to a Vector3, so I guess result would be the skinned vertex... right?

    Unfortunately, none of my trials lead to anything. On the bright side, I've seen my model distorted in all sorts of funny ways today. I'm actually stunned that I such a good result at the beginning with that clumsy delta thing.
     
  23. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Hello, anyone?... pretty please?
    metervara, did you get anywhere with this matrix stuff?
     
  24. metervara

    metervara

    Joined:
    Jun 15, 2006
    Posts:
    203
    I got it working, Aras's pseudocode works like a charm :)

    I omitted the variable value as I couldn't see why that was needed, and it works fine without it. I'm using MultiplyPoint3x4 MultiplyVector to get the position+normal (just like you), but since I'm not using value it's with finalMatrix instead of result.

    Code looks like this:

    Code (csharp):
    1.  
    2. weights=mesh.boneWeights;
    3. for(int i=0;i<mesh.vertexCount;i++){
    4.     BoneWeight weight = weights[i];
    5.     Matrix4x4 m0 = matrices[weight.boneIndex0];
    6.     Matrix4x4 m1 = matrices[weight.boneIndex1];
    7.     Matrix4x4 m2 = matrices[weight.boneIndex2];
    8.     Matrix4x4 m3 = matrices[weight.boneIndex3];
    9.     finalMatrix[i] = Matrix4x4.identity;
    10.     for(int n=0;n<16;n++){
    11.         m0[n] *= weight.weight0;
    12.         m1[n] *= weight.weight1;
    13.         m2[n] *= weight.weight2;
    14.         m3[n] *= weight.weight3;
    15.         finalMatrix[i][n] = m0[n]+m1[n]+m2[n]+m3[n];
    16.     }
    17.     vertexData[i].pos = finalMatrix[i].MultiplyPoint3x4(verts[i]);
    18.     vertexData[i].normal = finalMatrix[i].MultiplyVector(normals[i]);
    19. }
    And to generate the matrices[] Array:
    currentBoneMatrix I'm creating using Matrix4x4.TRS() and bindPose comes from mesh.bindposes

    /Patrik
     
  25. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Thanks man, works perfectly now, my stupid offset is finally gone!

    Here's my final script... I think if one only needs the vertices and normals, it can transparently replace a static mesh.
    Code (csharp):
    1. using UnityEngine;
    2.  
    3. /// <summary>
    4. /// Compute a skinned mesh's deformation.
    5. ///
    6. /// The script must be attached aside a SkinnedMeshRenderer,
    7. /// which is only used to get the bone list and the mesh
    8. /// (it doesn't even need to be enabled).
    9. ///
    10. /// Make sure the scripts accessing the results run after this one
    11. /// (otherwise you'll have a 1-frame delay),
    12. /// or use the OnResultsReady delegate.
    13. /// </summary>
    14. public class SkinnedMesh : CsBehaviour
    15. {
    16.     internal Mesh mesh;
    17.     internal SkinnedMeshRenderer skin;
    18.  
    19.     internal int vertexCount;
    20.     internal Vector3[] vertices;
    21.     internal Vector3[] normals;
    22.     public System.Action<SkinnedMesh> OnResultsReady;
    23.  
    24.     void Start()
    25.     {
    26.         skin = GetComponent<SkinnedMeshRenderer>();
    27.         mesh = skin.sharedMesh;
    28.  
    29.         vertexCount = mesh.vertexCount;
    30.         vertices = new Vector3[vertexCount];
    31.         normals = new Vector3[vertexCount];
    32.     }
    33.    
    34.     void LateUpdate()
    35.     {
    36.         Matrix4x4[] boneMatrices = new Matrix4x4[skin.bones.Length];
    37.         for (int i = 0; i < boneMatrices.Length; i++)
    38.             boneMatrices[i] = skin.bones[i].localToWorldMatrix * mesh.bindposes[i];
    39.  
    40.         for (int i = 0; i < mesh.vertexCount; i++)
    41.         {
    42.             BoneWeight weight = mesh.boneWeights[i];
    43.  
    44.             Matrix4x4 bm0 = boneMatrices[weight.boneIndex0];
    45.             Matrix4x4 bm1 = boneMatrices[weight.boneIndex1];
    46.             Matrix4x4 bm2 = boneMatrices[weight.boneIndex2];
    47.             Matrix4x4 bm3 = boneMatrices[weight.boneIndex3];
    48.  
    49.             Matrix4x4 vertexMatrix = new Matrix4x4();
    50.  
    51.             for (int n = 0; n < 16; n++)
    52.             {
    53.                 vertexMatrix[n] =
    54.                     bm0[n] * weight.weight0 +
    55.                     bm1[n] * weight.weight1 +
    56.                     bm2[n] * weight.weight2 +
    57.                     bm3[n] * weight.weight3;
    58.             }
    59.  
    60.             vertices[i] = vertexMatrix.MultiplyPoint3x4(mesh.vertices[i]);
    61.             normals[i] = vertexMatrix.MultiplyVector(mesh.normals[i]);
    62.         }
    63.  
    64.         if (OnResultsReady != null)
    65.             OnResultsReady(this);
    66.     }
    67. }
    And here's a little test script that draw the vertices and normals, turning your model into a porcupine:
    Code (csharp):
    1. using UnityEngine;
    2.  
    3. public class SkinnedMeshDebug : CsBehaviour
    4. {
    5.     void Start()
    6.     {
    7.         SkinnedMesh mesh = GetComponent<SkinnedMesh>();
    8.         mesh.OnResultsReady += DrawVertices;
    9.     }
    10.     void DrawVertices(SkinnedMesh mesh)
    11.     {
    12.         for (int i = 0; i < mesh.vertexCount; i++)
    13.         {
    14.             Vector3 position = mesh.vertices[i];
    15.             Vector3 normal = mesh.normals[i];
    16.  
    17.             Color color = Color.green;
    18.             Debug.DrawWireCube(position, 0.1f * Vector3.one, color);
    19.             Debug.DrawRay(position, normal, color);
    20.         }
    21.     }
    22. }
    Thanks a lot Aras metervara for introducing me to the wonderful craziness of matrices :D !
     
  26. marblez

    marblez

    Joined:
    Oct 6, 2008
    Posts:
    14
    Thanks for all the code so far! It has been really helpful. I've (modified and) used these scripts to combine skinned meshes for characters but my animations on my characters stop after I combine.

    Have you run into this problem? If so, how did you solve it?

    I thought of saving the animation so I can recreate it on the parent node (the node with all of the combined meshes and materials) but there doesn't seem to be a way to access the keyframes from the Animation Curve.

    How can I combine meshes/materials and not lose any animations on them?
     
  27. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Yes, CombineChildren is for static meshes only. I'm surprised it works at all because it needs a MeshFilter, and skinned meshes don't have that component by default, did you add it? Or are you not using CombineChildren?

    My first maybe-not-so-useful advice would be to combine modeler-side, maybe use a 3DS/Maya script or whatever. I don't know that it would be a good idea to solve it Unity-side.

    But it should probably be possible though, and definitely an interesting challenge! First you'd have to recreate the bone influences, I don't think CombineChildren does that. Second step would be the animations... are you sure they're lost? My guess is the transforms are still animated but they don't deform the mesh because the SkinnedMeshRenderer isn't linked to the combined mesh.
    But if they are lost, you'd have to combine them too (good luck with that ;)). AnimationClip can SetCurve() but not get it (shame, that), but you can access the complete curve (keyframes and all), but only in the editor (shame, again), so maybe you could combine everything at edit-time? I've never tried to save a script-created mesh, don't know if the data gets serialized... if it doesn't, a) hint hint UT, and b) you'd have to store all the vertices, influences and curves on your own, then recreate everything on Start. This would seem so much trouble that again, I'd try to do it modeler-side.

    Lastly, please note that I'm a big pompous ass because I honestly don't know crap about animation, I can barely open Cheetah (discovered 2 days ago that it's able to open .3ds files natively), so all the above comes from a scholar knowledge of the API, not practical experience.
    That said, enjoy your pain and let me know how that went :D !
     
  28. marblez

    marblez

    Joined:
    Oct 6, 2008
    Posts:
    14
    Yeah I took that script as well as the great code you've submitted and used it to merge all the meshes into 1 MeshFilter... but now I realized yesterday (and your post confirms) that I need to merge the meshes into a SkinnedMeshRenderer with all the bone influences... that would probably explain (as you mentioned) why it's not animating... nothing is really lost, it just wasn't done correctly.

    Unfortunately, our design limits us so that we can't do it modeler-side (I don't think). We want to do it programmatically so we can create avatars for our game that have customizable clothing. Meaning, swap pieces in and out and then when the player is done, we'll merge it all down, for performance reasons. And since each clothing piece is animated separately we need those animations to work after we merge...

    Double shame... Anyone from UT can explain why we don't have access at the Keyframes or curves?

    No worries, mate. I appreciate the feedback and you've been incredibly helpful in the way of scripts so far!
     
  29. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Oh man that's brilliant, didn't think you were aiming at something that dynamic! In that case, you only need to extract the curves at import-time and serialize them, then you'll have all the ingredients to do your combine magic at runtime... which you still have to figure out. Oh, the fun :D.

    I'd love to get my hands on that once you're done! One of our client has also asked us about an avatar S*** (ok, not surprising, who isn't doing/planning an MMO these days :?), but we don't really know what's even possible yet... I smell there's gonna be some dynamic skinning somewhere.
    Like doing a sort of inflatable body à-la Mii/Xbox 360 seems doable, but putting say a coat on top of the guy seems more difficult already... and I hadn't even thought of combining the guy and its clothes!
     
    ModLunar likes this.
  30. marblez

    marblez

    Joined:
    Oct 6, 2008
    Posts:
    14
    Obviously, I wouldn't be able to share any finished work with everyone publicly but I'm not so selfish that I wouldn't want to share back with the community which has so graciously shared with me. :) I have seen from multiple different posts throughout the forums (as I was looking for answers to how to solve this) that there are many others trying to do this exact same thing. I think I've already given away the steps we are taking to do this and it is mostly based on code that is freely available in the forums, including a great deal of code you've submitted. I will be happy to follow up on this thread whether we are successful or not which should encourage others that the answers are out there (or in here, depending on how you look at it).
     
  31. benblo

    benblo

    Joined:
    Aug 14, 2007
    Posts:
    469
    Sure I didn't mean the full-blown avatar system, just the bone-combine and animation-combine parts.
    Although if you happen to have any good links to share regarding avatar systems, I'm all ears (white paper, postmortem, whatever).
     
  32. dawvee

    dawvee

    Joined:
    Nov 12, 2008
    Posts:
    276
    This is funny, I only just worked out this same process a few months ago in a face matching setup.

    I designed a script to deform a face mesh according to a set of control points (rigged as bones) and then dump the new vertex data into another mesh rigged and animated for facial expressions. My code is really similar though:

    Code (csharp):
    1. Vector3[] verts = new Vector3[deformMesh.vertices.Length];
    2. for (int i=0;i<verts.Length;i++)
    3. {
    4.     //This applies the skeletal subsurface deformation algorithm V(f) = Sum (M*M^-1*V*w) where we've pre-calculated M*M^-1 for each bone.
    5.     verts[i] = Vector3.zero;
    6.     Vector3[] boneDeform = new Vector3[4];
    7.            
    8.     //Calculate the partial deform for each bone influencing the vertex
    9.     boneDeform[0] = (boneTransform[deformMesh[0].boneWeights[i].boneIndex0].MultiplyPoint3x4(deformMesh.vertices[i]) * deformMesh.boneWeights[i].weight0);
    10.     boneDeform[1] = (boneTransform[deformMesh.boneWeights[i].boneIndex1].MultiplyPoint3x4(deformMesh.vertices[i]) * deformMesh.boneWeights[i].weight1);
    11.     boneDeform[2] = (boneTransform[deformMesh.boneWeights[i].boneIndex2].MultiplyPoint3x4(deformMesh.vertices[i]) * deformMesh.boneWeights[i].weight2);
    12.     boneDeform[3] = (boneTransform[deformMesh.boneWeights[i].boneIndex3].MultiplyPoint3x4(deformMesh.vertices[i]) * deformMesh.boneWeights[i].weight3);
    13.  
    14.     //Now sum the partial deforms to find the new vertex position.
    15.     for (int q=0;q<boneDeform.Length;q++)
    16.     {
    17.         verts[i] += boneDeform[q];
    18.     }
    19. }
    20. faceMesh.vertices = verts;
    21. faceMesh.RecalculateBounds();
    It would be fantastic if we could read the skinned vertex positions directly from Unity, though. The above code is incredibly slow on a detailed mesh - in my case I had to break up the loop and multithread it to get acceptable performance on around 10k vertices.

    Edit: I should have looked closer at the previous code examples. Is it generally faster to lump the matrices into one matrix before multiplying the vertex position? I'll have to try that out.
     
  33. n0mad

    n0mad

    Joined:
    Jan 27, 2009
    Posts:
    3,732
    Hello there,


    just wanted to add a potential solution to your animation saving problem, marblez.

    Here is the workflow :

    1) Just do the modeling+animation work on 3rd party tool side, as usual.
    2) Export a animated version of bones only, without skins.
    3) Export separate non-animated versions of all of your different skin parts, only keeping the effective bones in each file, and of course, the mesh.
    4) under Unity, just combine your fresh imported unanimated skins to a one-piece Mesh type, as you like, with some code. There are plenty of them, and Benblo's one seems to be good at it.
    5) Instantiate your animated bone under your target gameobject.
    6) set your bone SkinnedMeshRenderer.sharedMesh to your recently code-generated mesh.

    7) profit :)



    That works for me.



    Cheers, and thank you again, all of you, for your great sense of community sharing.
     
  34. Evan-Greenwood

    Evan-Greenwood

    Joined:
    Aug 6, 2008
    Posts:
    96
    Just wanted to say thank you Aras and Benblo (and others).

    This is indeed a feature I would like to have... (I'm trying to slice up a skinned mesh)
     
  35. LKIM

    LKIM

    Joined:
    Feb 17, 2011
    Posts:
    40
    I'm having speed issues as well here. I assume that I'm going to have to break this up and multi-thread it to get appropriate performance, but I just wanted to check if anyone has had any success doing this quickly. Basically what I need to do is get the positions of all the vertices (over 10K of them), then move the bones, and get the positions again.
    This is taking about 7-8 seconds each time I need to get the verts (for a total of about 15 seconds) using the code that benlo posted. With the current implementation it also totally locks up the application during those 15 seconds, but I'm sure I can fix that with some threading trickery.

    Any suggestions?

    Thanks,
    Liron
     
  36. Rafes

    Rafes

    Joined:
    Jun 2, 2011
    Posts:
    764
    Soooo... 2 1/2 years later, any updates? I've got this spikey thing animated by a bunch of bones and I need it to roll around as it deforms, so I'd love a mesh collider. Any thoughts? Did anyone manage to create a good bit of code as discussed above?

    Cheers,
     
  37. MarkD

    MarkD

    Joined:
    Sep 18, 2013
    Posts:
    159
    This was exactly what I needed!

    If you don't mind I converted it to JavaScript.
    I'l post it here for the people in need for the JavaScript version.

    Code (csharp):
    1.  
    2. #pragma strict
    3. /// <summary>
    4.  
    5. /// Compute a skinned mesh's deformation.
    6.  
    7. ///
    8.  
    9. /// The script must be attached aside a SkinnedMeshRenderer,
    10.  
    11. /// which is only used to get the bone list and the mesh
    12.  
    13. /// (it doesn't even need to be enabled).
    14.  
    15. ///
    16.  
    17. /// Make sure the scripts accessing the results run after this one
    18.  
    19. /// (otherwise you'll have a 1-frame delay),
    20.  
    21. /// </summary>
    22.  
    23.     @HideInInspector
    24.     var mesh: Mesh;
    25.     @HideInInspector
    26.     var skin: SkinnedMeshRenderer;
    27.  
    28.  
    29. @HideInInspector
    30.     var vertexCount:int=0;
    31. @HideInInspector
    32.     var vertices:Vector3[];
    33. //@HideInInspector
    34.   //  var normals:Vector3[];
    35.  
    36.  
    37.  
    38.  
    39.  
    40.  
    41.     function Start() {
    42.  
    43.         skin = GetComponent(SkinnedMeshRenderer);
    44.  
    45.         mesh = skin.sharedMesh;
    46.  
    47.  
    48.  
    49.         vertexCount = mesh.vertexCount;
    50.  
    51.         vertices = new Vector3[vertexCount];
    52.  
    53.       //  normals = new Vector3[vertexCount];
    54.      
    55.        //animation example
    56.        for (var b:int= 0; b < mesh.vertexCount; b++){
    57.             var cube : GameObject= new GameObject.CreatePrimitive(PrimitiveType.Cube);
    58.             cube.name=b.ToString();
    59.             cube.transform.localScale.x=0.1;
    60.             cube.transform.localScale.y=0.1;
    61.             cube.transform.localScale.z=0.1;
    62. }
    63.     }
    64.  
    65.    
    66.  
    67.     function LateUpdate(){
    68.  
    69.         var boneMatrices: Matrix4x4[]  = new Matrix4x4[skin.bones.Length];
    70.  
    71.         for (var i:int= 0; i < boneMatrices.Length; i++)
    72.  
    73.             boneMatrices[i] = skin.bones[i].localToWorldMatrix * mesh.bindposes[i];
    74.          
    75.  
    76.  
    77.         for (var b:int= 0; b < mesh.vertexCount; b++){
    78.  
    79.              var weight:BoneWeight = mesh.boneWeights[b];
    80.  
    81.  
    82.  
    83.               var bm0:Matrix4x4 = boneMatrices[weight.boneIndex0];
    84.  
    85.               var bm1:Matrix4x4 = boneMatrices[weight.boneIndex1];
    86.  
    87.               var bm2:Matrix4x4 = boneMatrices[weight.boneIndex2];
    88.  
    89.               var bm3:Matrix4x4 = boneMatrices[weight.boneIndex3];
    90.  
    91.  
    92.  
    93.              var vertexMatrix:Matrix4x4 = new Matrix4x4();
    94.  
    95.  
    96.  
    97.             for (var n:int= 0; n < 16; n++){
    98.  
    99.                 vertexMatrix[n] =
    100.  
    101.                     bm0[n] * weight.weight0 +
    102.  
    103.                     bm1[n] * weight.weight1 +
    104.  
    105.                     bm2[n] * weight.weight2 +
    106.  
    107.                     bm3[n] * weight.weight3;
    108.  
    109.             }
    110.  
    111.  
    112.  
    113.             vertices[b] = vertexMatrix.MultiplyPoint3x4(mesh.vertices[b]);
    114.             //   normals[i] = vertexMatrix.MultiplyVector(mesh.normals[i]);
    115.            
    116.             //animation example
    117.             var fetch= GameObject.Find( b.ToString());
    118.             fetch.transform.position = vertices[b];
    119.             }
    120.  
    121.  
    122.            
    123.        
    124.  
    125.     }
    126.  
    127.  
    128.  
     
  38. Eric5h5

    Eric5h5

    Volunteer Moderator Moderator

    Joined:
    Jul 19, 2006
    Posts:
    32,139
    Unity has SkinnedMeshRenderer.BakeMesh now.

    --Eric
     
  39. MarkD

    MarkD

    Joined:
    Sep 18, 2013
    Posts:
    159
    Yes, but I used that before this script, the problem is that the vertex offset is always wrong, when my cubes follow the vertices in this script it is spot on, when I use bakemesh they are oriented over the wrong axis, creating the cubes to move in the wrong direction.
     
  40. kilik128

    kilik128

    Joined:
    Jul 15, 2013
    Posts:
    843
    How My God you try kill me ! ? how i can understand that's !
     
  41. Alima-Studios

    Alima-Studios

    Joined:
    Nov 12, 2014
    Posts:
    47
    using bakemesh... too slow for realtime
     
    ModLunar likes this.
  42. CrazyRocksStudios

    CrazyRocksStudios

    Joined:
    Sep 29, 2015
    Posts:
    15
    Hi guys !

    I have a riddle for You ;)

    How to partially transform a point by matrix like: [ V = m.MultiplyPoint3x4(V) * weight ] and then back to it's original position ?? If there is no "weight" or weight = 1 - It's just m.inverse used to do this.

    I need it to transform several meshes to their skeleton space, make some adjustments to normals and positions and then transform their back for normal animation proccesing.
    Just can't figure it out - I feel, the solution is simple but maybe I'm just tired ..

    Please help, Thank You :)

    Stan
     
  43. YoungXi

    YoungXi

    Joined:
    Jun 5, 2013
    Posts:
    47
    1. for (int n = 0; n < 16; n++)
    2. {
    3. vertexMatrix[n] =
    4. bm0[n] * weight.weight0 +
    5. bm1[n] * weight.weight1 +
    6. bm2[n] * weight.weight2 +
    7. bm3[n] * weight.weight3;
    8. }
    I got Matrix4x4 .get_Item() called over 100 thousands for less than 10k vertices. Is there anyway to avoid get_Item() call?
    I've compared List.get_Item() with array[], the second one is much faster. (I really thought List was nothing but a wrap of array[] like vector in cpp, never thought calling List[] would cause me problem in a large loop).
     
  44. YoungXi

    YoungXi

    Joined:
    Jun 5, 2013
    Posts:
    47
    Code (CSharp):
    1. Matrix4x4 vm = new Matrix4x4();
    2.                 vm.m00 = bm0.m00 * weight.weight0 + bm1.m00 * weight.weight1 + bm2.m00 * weight.weight2 + bm3.m00 * weight.weight3;
    3.                 vm.m01 = bm0.m01 * weight.weight0 + bm1.m01 * weight.weight1 + bm2.m01 * weight.weight2 + bm3.m01 * weight.weight3;
    4.                 vm.m02 = bm0.m02 * weight.weight0 + bm1.m02 * weight.weight1 + bm2.m02 * weight.weight2 + bm3.m02 * weight.weight3;
    5.                 vm.m03 = bm0.m03 * weight.weight0 + bm1.m03 * weight.weight1 + bm2.m03 * weight.weight2 + bm3.m03 * weight.weight3;
    6.  
    7.                 vm.m10 = bm0.m10 * weight.weight0 + bm1.m10 * weight.weight1 + bm2.m10 * weight.weight2 + bm3.m10 * weight.weight3;
    8.                 vm.m11 = bm0.m11 * weight.weight0 + bm1.m11 * weight.weight1 + bm2.m11 * weight.weight2 + bm3.m11 * weight.weight3;
    9.                 vm.m12 = bm0.m12 * weight.weight0 + bm1.m12 * weight.weight1 + bm2.m12 * weight.weight2 + bm3.m12 * weight.weight3;
    10.                 vm.m13 = bm0.m13 * weight.weight0 + bm1.m13 * weight.weight1 + bm2.m13 * weight.weight2 + bm3.m13 * weight.weight3;
    11.                
    12.                 vm.m20 = bm0.m20 * weight.weight0 + bm1.m20 * weight.weight1 + bm2.m20 * weight.weight2 + bm3.m20 * weight.weight3;
    13.                 vm.m21 = bm0.m21 * weight.weight0 + bm1.m21 * weight.weight1 + bm2.m21 * weight.weight2 + bm3.m21 * weight.weight3;
    14.                 vm.m22 = bm0.m22 * weight.weight0 + bm1.m22 * weight.weight1 + bm2.m22 * weight.weight2 + bm3.m22 * weight.weight3;
    15.                 vm.m23 = bm0.m23 * weight.weight0 + bm1.m23 * weight.weight1 + bm2.m23 * weight.weight2 + bm3.m23 * weight.weight3;
    16.                
    Instead of using a for loop and [], directly access the m00 - m33 will avoid the indexer's method call. This will improve the performance quite a lot! And the m30 - m33 could be ignored, if you look into Matrix4x4::MultiplyPoint3x4 method, it never uses the m30 - m33.
     
  45. CodeKiwi

    CodeKiwi

    Joined:
    Oct 27, 2016
    Posts:
    56
    I was trying to get the skinned vertices and found this post. I tried the scripts by @benblo using a mesh with 8k verts. The main problem is that getting the vertices, normals or bones weights creates a copy. This creates about 3GB of data per call and runs at about 0.2 fps on my machine.

    Performance tests:
    Standard skinned mesh: 1,400 fps
    Orig code: 0.2fps
    Store single copy of mesh data: 56 fps
    Store data and matrix suggestion by @YoungXi: 120 fps
    With bakemesh: 1,100 fps
     
  46. gbernal

    gbernal

    Joined:
    Oct 25, 2016
    Posts:
    2
    Sorry to bring this thread back. I'm not sure If I follow the results from your test. What approach gave you the best results?

    Thank you
     
  47. CodeKiwi

    CodeKiwi

    Joined:
    Oct 27, 2016
    Posts:
    56
    I uploaded the scripts for reference. I forgot to get the data from the baked mesh and to transform the points to world space. My new stats are:

    No animation: 2,000 fps
    Standard animation 1,500 fps
    Only calling bake mesh 1,300 fps
    Baked mesh without transformed points but calling GetVertices() and GetNormals(): 540 fps
    Baked mesh with transforms (could be improved): 230 fps
    Optimised code: 100 fps
    Orig code: 0.2 fps

    So the optimised code is about 500 times faster than the original but bake mesh is about 2-5 times faster than that. If I needed the data in realtime I’d probably use a compute shader to transfrom the points after calling bake mesh.
     

    Attached Files:

    Ony likes this.
  48. gbernal

    gbernal

    Joined:
    Oct 25, 2016
    Posts:
    2
    Awesome! thanks, @High_Noon, I will check it out now.
     
  49. arturmandas

    arturmandas

    Joined:
    Sep 29, 2012
    Posts:
    196
    @CodeKiwi this is a great asset, thanks. Works really fine for blendshapes, allowing me to spawn/transform objects at vertices of deformed SMR in realtime. Good job!