Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Dual quaternion skinning for Unity

Discussion in 'General Graphics' started by mr_madcake, Oct 22, 2017.

  1. Holoboost

    Holoboost

    Joined:
    Jul 6, 2020
    Posts:
    4
    Hi there, this looks really awesome. Does this technique work on mobile (oculus quest), using opengles? Is there any way to run this on unity 2018 LTS?
     
  2. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94
    Hi

    I haven't tried but theoretically it should work. Have you encountered some issue with it?
     
  3. Holoboost

    Holoboost

    Joined:
    Jul 6, 2020
    Posts:
    4
    No, but I haven't because I didn't know if it would work on 2018 LTS due to the multi_pragma_compile issue. But now that you mentioned it should, I can give it a shot. Just wondering how much a performance hit it is compared to normal skinning, since we are heavily GPU bound right now,
     
  4. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94
    It's hard to give a clear number as it scales differently with different things. From my benchmarks I could not get worse performance drop than 30% compared to built-it skinning in purposefully built worst-case scenario, but it could vary wildly depending on the project.

    Also CPU hit from extracting bone transform matrices affected performance the most, there should be no big issue with GPU.

    If you want good performance I strongly recommend using IL2CPP compiler as is properly optimizes bone transform matrix extraction (I made the benchmarks with IL2CPP). Mono uses some really weird shenanigans that slow down the script A LOT.

    This line in particular is A LOT slower in Mono and in profiler it reportedly calls to String.memcpy() for no apparent reason.

    If you don't need the bulging compensation you can remove this part of the shader for a little extra performance. This will leave you with traditional DQ implementation.
     
    Last edited: Jul 14, 2020
    hopeful likes this.
  5. vladibalan

    vladibalan

    Joined:
    Sep 14, 2014
    Posts:
    6
    I have a mesh with 4332 vertices (so vertNumber/textureWidht = 4). If I use GetVertexTextureHeight in Update it is rendered garbled at runtime (I did not investigate yet why a textureHeight of 5 instead of 4 makes it do that).
    upload_2020-7-18_17-17-2.png

    Anyway, i'm posting this in case anybody else has this problem. I've reverted the line in Update to the previous form to solve the issue: this.materialPropertyBlock.SetInt("skinned_tex_height", mf.sharedMesh.vertexCount / textureWidth);
    Observation: there are multiple meshes using DualQuaternionSkinner (different vertex count, all of them worked but this one). Again, I don't know yet why, the only difference is that all the other meshes have a texture height ratio lesser than 4 (0,1,2,3).
    PS: I am using Unity 2019.4.4f1 Personal.
     
  6. sihan66

    sihan66

    Joined:
    Aug 26, 2020
    Posts:
    2
    Get the mesh of DQ blending skinning:
    I have implemented the dqls in my own project. And I want to get the mesh of human model of dqls in obj format. However, I found i get the obj model of lbs. How can i get the mesh of dqls? Thank you.

    ![dqls in unity](https://user-images.githubusercontent.com/58064039/91283618-68203d80-e7bd-11ea-8e36-63409b4b9d02.png)
    ![export lbs in obj](https://user-images.githubusercontent.com/58064039/91283629-6bb3c480-e7bd-11ea-8394-135c43a4dad0.png)
     
  7. sihan66

    sihan66

    Joined:
    Aug 26, 2020
    Posts:
    2
     
  8. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94
    Hi.

    I am a bit confused about what are you trying to achieve.

    OBJ format does not support animation, it is a static model. For animated models you would usually use FBX.
    Are you trying to make a static model from a single frame of animation in Unity?

    My script only works in Play mode, so if you are trying to export the model in Edit mode, you would get the model with built-in linear skinning.
     
  9. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94
    Could you try this version of the script with the same model?
    I would like to make sure that these issues are not connected.

     
    vladibalan likes this.
  10. Ldnicon

    Ldnicon

    Joined:
    May 22, 2016
    Posts:
    4
    Any updates on the distortion of meshes?

    I've tried DualQuaternionSkinner.cs on different models, but they all look extremely distorted.
     

    Attached Files:

  11. Cold_Frank

    Cold_Frank

    Joined:
    Dec 15, 2020
    Posts:
    6
    Hey, I am testing different solutions for better skinning and I found something like this: OptimisedCentresOfRotationSkinning

    Could this be useful in improving your solution?

    Cheers!
     
  12. Wothanar

    Wothanar

    Joined:
    Sep 2, 2016
    Posts:
    122
    this dosnt worked to me i used version 2020.1.7 of unity, then this error happens



    line 604 of DualQuaternionSkinner.cs this.mf.sharedMesh.MarkDynamic(); // once or every frame? idk.

    I follow the guide you have on github , but it didn't worked. i set as read enable true the skinned mesh renderer character have Body And Head separated , cant work this way? I'm just don't understand what its causing it. why didn't work ?. can you be more detailed in the instructions?
    and here in my picture

    https://gyazo.com/a09ce30fb5fc1af9b2dcfe1b73448b0b


    it looks like empty in compute shaders and I cant set anything manually, so whats wrong with this? i was so excited about this but now I think I should continue with mesh animator, actually did this gpu instantiate the skinned mesh renderers? as mesh animator does? coz mesh animator its what I'm using now and works fine for GPU instancing Skinned mesh renderers
     
  13. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94
    If you still have this issue, you can send me the minimal example project and I will take a look.
     
  14. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94
    Optimized CoR indeed provide improved skinning, but AFAIK there's some issues with licensing and I'm not sure I can legally implement and share it other than for education purposes.

    The technique I used is an alternative to optimized CoR that I developed as my master's thesis. Unlike CoR it does not require any precomputations, but the quality is worse (still better than baseline DQ, but not as good as optimized CoR). Initially I implemented DQ in unity just to have it here, but when writing the thesis I decided it would be convenient to use my already working script to demo the new technique.
     
  15. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94
    Anyone having their models look completely butchered and distorted by the script can try this version (quoted message). I might have hard-coded some values in the script that some GPUs are not able to work with. If the issue is what I think it is, the quoted version should be compatible with GPUs failing to run the main version.
     
    vladibalan likes this.
  16. alloystorm

    alloystorm

    Joined:
    Jul 25, 2019
    Posts:
    75
    I've been scratching my head for quite some time trying to figure out why my rigged models exported from Daz3D don't look the same in Unity comparing to Daz. Now I realized this is what I always needed...
    However I imagine it would be difficult to adopt your approach since I use a lot of custom shaders done in ShaderGraph, I guess this is going to be tricky trying to integrating your stuff in my shaders...
     
  17. alloystorm

    alloystorm

    Joined:
    Jul 25, 2019
    Posts:
    75
  18. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Thank you for sharing that code. I've been hitting a brick wall trying to make DAZ models look good in unity. I finally found out about Dual Quaternion Skinning, and this was the key.

    I made a working HDRP/URP/LWP implementation using the new 2021.2 Mesh API, that allows direct buffer access and modification. It was a lot more straight forward than I originally though (thanks to the helpful tips from Qleenie as well)

    To get this to work, it's only a few lines added to the DQBlend.compute shader.

    RWByteAddressBuffer Vertices; //a direct byte buffer which can be used to directly modify the vertices of a mesh.


    All we need to do, is assign the skinned vertices we already have:
    For that we first need to find the exact addresses of the vertex buffer.
    To do this, you can use mesh.GetVertexAttributes(); and print out the array.
    This will give you offsets, strides, etc to get the exact positions of the data you need.
    In my case, this was

        uint vidx = id.x * 40; //0 + 12(vert) = 12
    uint nidx = vidx + 3 * 4; //12 + 12(norm) = 24
    uint tidx = nidx + 3 * 4; //24 + 16(tan) = 40 (tangents are itself 4*4)


    Keep in mind that this might depend on the imported mesh, so the best way to would be transfer the data from CPU so that no values are hard coded like this.

    After that, all we need to do is put the skinned vertices into the array.

        Vertices.Store3(vidx, asuint(vertinfo.position.xyz));
    Vertices.Store3(nidx, asuint(vertinfo.normal.xyz));
    Vertices.Store4(tidx, asuint(vertinfo.tangent));


    The rest ist just assigning the Vertices Buffer in the DualQuaternionSkinner

        GraphicsBuffer vertexBuffer;

    void GrabMeshFromSkinnedMeshRenderer()
    {
    ...
    //all of this only needs to be done once:
    mesh.vertexBufferTarget |= GraphicsBuffer.Target.Raw; //needed to get the raw byte buffer
    vertexBuffer = mesh.GetVertexBuffer(0); //retrieve the vertex buffer of the mesh
    this.shaderDQBlend.SetBuffer(this.kernelHandleDQBlend, "Vertices", vertexBuffer); //assign the vertex buffer
    ...
    }


    That's pretty much it. Except one thing:
    this.mf.sharedMesh.MarkDynamic();

    has to be moved from update to init.

    Calling it every frame invalidate the direct buffer (as will any other modification of the buffer from the CPU side), and cause the buffer to be recreated. If that happens, you will not see any modification to the mesh. I don't think this method is necessary for the original version, as the original never modifies the vertexBuffer itself, and instead deforms via shader. From my OpenGL days, I guess this like setting the hint for a vertex buffer object.

    Additionally, I made a version of the ApplyMorphs function that will change one morph by removing the previous weight, and then adding the new one, all done on the already uploaded array. This saves a lot of performance by not rebuilding the complete set of morphed arrays when only one morph changed.

    I hope this helps people running into similar issues as me.
     
    vladibalan, ekakiya and Qleenie like this.
  19. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Since my figure is using over 200 fast changing morphs (JCM to compensate for bulges), it was necessary to optimize the morphing system for memory and speed.

    As already mentioned above, my first optimization was to only update morphs that changed. Because calling ApplyMorphs() with a full update caused massive slowdown, only updating changed morphs already helped a lot.

    However, there were more problems arising from the morph system. The biggest problem being that every morph has a complete delta image for every single vertex. With ~80k vertices and 200+ morphs, all in all, this would be more than 1 gb of vram just for the blendshapes, which wasn't viable.

    Rewriting the morph structure to include a vertex index solved that problem. The compute buffers for morphs can be scaled down to only contain vertex deltas that are greater than distance zero. In the compute shader, the vertex array can then addressed via the morph's vertex index.

    Not only did that save a ton of space (I'd say about 95% smaller in vram, about 1gb less), but also because using the indices to update on the compute shader, it also greatly reduced processing, because there are a lot less vertices that need to be processed.
     
    vladibalan likes this.
  20. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Bounds: Because all the vertices are stored in VRAM and not on the GPU anymore, there is no easy way to calculate the bounds based on the vertices.
    Using the compute shader to do it wouldn't work, because you would run into racing conditions when all threads try calculate and write the min/max at the same time. There is a way to synch up threads via memory barrier, but that would cost quite a bit of overhead. Alternatively running a compute shader on one thread to just iterate over every vertex to create the bounds will cost even more performance.

    What I went with now is just completely ditch any vertex based bound calculations. Since it's going tobe axis aligned bounding boxes at the end, precision is not very important.

    Instead I'm collecting an array of transforms at key bones (head, hands, chest, shins feet), and create an AABB from each with an arbitrary margin, and then add in turn to a complete AABB, and submit that to the bounds of the mesh.

    This seems to work perfectly with almost no cost, and frustum culling can be done again.
     
    vladibalan and Qleenie like this.
  21. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    525
    Calculating the mesh boundaries sounds like a good approximation! I am doing it in compute shader, as I also do some subdivision for mesh collision stuff, but that's a bit expensive, as you need the mentioned memory barriers / InterlockedMin and InterlockedMax. About the morphs: I looked into my code, I am doing it exactly same (this is also somehow similar as SkinnedMeshRenderer seems to do it).
     
  22. Cold_Frank

    Cold_Frank

    Joined:
    Dec 15, 2020
    Posts:
    6
    Hello, @schema_unity would you be able to share your package with the code and changes you describe above? Pretty please :)
     
  23. iconnary

    iconnary

    Joined:
    Feb 9, 2020
    Posts:
    6
    I'd also love to check out what you've put together here @schema_unity -- I've been looking for a solution to some of these problems for several years now.

    Thanks
     
  24. mr_madcake

    mr_madcake

    Joined:
    Jul 17, 2017
    Posts:
    94

    IIRC (might be wrong) when importing an animated mesh, Unity goes through all the animations and computes AABB once so that all attached animations stay within the volume of AABB. Re-calculating would be needed only if you have unpredictable procedural animation, or you need a very precise AABB.

    IIRC my script does not do this, but it shouldn't be too difficult to implement.

    If you need fast (min/max/avg) in a shader, this pdf provides a lot of info on how it can be optimized, though it's not an easy task to get right. GPU performance can be very counter-intuitive.
     
    vladibalan likes this.
  25. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Yeah, the main purpose of bounds is to frustum call the model. Using a per vertex and per frame update probably will get the most accurate result, but the overhead for the additional accuracy will probably be never worth it, because the whole reason to frustum cull is to avoid going through the data on a vertex level (be it for drawing or not).

    The only advantage of a more accurate bounding box would be to not draw objects that are slightly off camera. Unless a scene has a ton of skinned objects, it will likely not make a difference in performance for those edge cases. And if there indeed are a ton of skinned objects, the overhead from going through each vertex for a bounding box might be heavier than what you would gain culling it. So especially on a lot of models you would want a really fast bounding box calculation.


    Currently my code is a bit intertwined with some other systems, like Odin Inspector, and some of my other systems that extract vertex data from skinning for their own calculations.

    However, I'm planning together with @jcluts to release a cleaned up version of madcake's DualQuaternion implementation for Unity.2021's mesh API, featuring blend shapes, up to 6 bones, and blendshape vertex/normal/tangent override.
     
    vladibalan and hopeful like this.
  26. SpringHeel

    SpringHeel

    Joined:
    Jan 17, 2021
    Posts:
    1
    Hey @schema_unity, is this something you're still working on? Are the code snippets you posted the only code needed to make your solution work in a HDRP project? I pasted them in, but no luck unfortunately. --I'm definitely no shader programmer

    Thanks for your efforts and explanations so far.
     
  27. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Yeah, the code snippets should technically be enough to get it working. The problem might be in the mesh you're displaying/modifying. I recommend rewriting the DQ code to get rid of all the now unneeded texture code.

    Your skinned object should have both a skinned mesh renderer as well as a normal mesh renderer. The skinned mesh renderer is a reference for the DQ code. It is turned off on start and the mesh renderer is what is actually displayed. Furthermore, I recommend not making changes to a shared mesh buffer (sharedMesh), as changes will persist between edit mode and play mode until a complete mesh reload (usually restarting unity). It's best to crease an instance of the mesh at the start and work with that. This instance would be the only thing targeted in the DQ code also. You will only need the skinnedMeshRenderer to read the bindposes and bones into the shader at startup.

    Is there any error you're getting?
     
  28. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    525
    To add on what @schema_unity wrote you can also write to the SkinnedMeshRenderer, without the need of an extra MeshRenderer. It's a bit more complicated, but with this you are able to use skinned motion vectors (if this is relevant for your use case).
     
  29. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Is it possible to prevent the default skinning method of unity from running when using skinned meshes? And if you would do that, wouldn't you have to write motion vectors manually? Just wondering if the cost of having to run skinning twice on the same object can be avoided or if there is approach that simply replaces linear skinning.
     
    vladibalan likes this.
  30. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    525
    I tune down all parameters of SMR, e.g. use one bone only, and overwrite the mesh each frame. Motion Vectors are generated by SMR automatically, if option is enabled. Would of course be better if we'd be able to provide own Motion Vectors in MeshRenderer, but I am not sure if this is possible.
     
    vladibalan and schema_unity like this.
  31. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Taking a stab at it I implemented my own double buffering to keep track of velocity, then had a custom function in the shader graph grab the buffer.
    I first thought it was just a matter of setting the custom velocity of a stack lit shader.
    Turns out, that no matter what you put in there, the motion vector calculation doesn't actually happen. Trying it out on a SkinnedMesh not only applied the default motion vectors but also my custom vectors on top of it, so I'm guessing the purpose of the custom velocity is to apply any additional deformation motion on a mesh that is already using a skinned mesh renderer. I'm not sure if the normal mesh renderer lacks per vertex motion vectors all-together or if I'm missing something.
    I've not seen any post or documentation that goes into it very much, unfortunately.

    Additionally, everything about the skinned mesh pipeline gets more confusing the deeper you go into it, to say it nicely. The way it is implemented allows for almost no customizability, unless injecting the vertex manipulation just before the frame gets drawn, effectively overwriting the default skinning. It's possible to remove skinning calculation by removing the vertex bone weights and just setting the object itself as the root bone and its only bone.
    However, this does get rid of the motion vector calculation as well, as that is likely done in the skinning pass.

    While I got this to work now, and motion vectors with blur do look nice, it really bothers me that an extra skinning pass has to be done, even if it's just on one bone per vertex, but I don't have a better solution right now.

    The other thing that bothers me is how the skinned mesh renderer switches buffers, which makes it very hard to additional passes like softbody processing without running into possible errors. And I don't know if it is relevant, but having the skinning done so late in the frame might be subject to pipeline issues.

    Using an extra buffer would make it possible to do all vertex manipulation beforehand, and then just copy it over before the frame draws, however, it introduces another full verte buffer on top of the skinned motion vector buffering. But at least that buffer would be consistent. I might add this as an option to my code just to see how much it impacts things (probably doesn't even matter).

    What we need is a new version of the skinned mesh renderer. A version that is modular, where you can just switch out all or part of the geometry processing. This would instantly solve all problems. Not only that it would solve problems for so many other Assets on the store that suffer from how closed such a central component is. Cloth simulations, soft body, collision detection all would profit greatly from a new open version of SMR.
     
    hopeful and Qleenie like this.
  32. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    525
    You can still do softbody and/or cloth stuff with this setup, just do it after DQS pass, and before writing to the VertexBuffer of mesh. I am doing it like this without issues in production. Writing to the mesh works best/ only in "OnBeginFrameRendering" on SRP, so I do it either directly affter calculating DQS, or after calculating softbody on top of DQS positions.

    I'd be interested how you remove the bone weights. Just set smr.sharedMesh.boneWeights.boneIndex0 - smr.sharedMesh.boneWeights.boneIndex3 to "-1" ?
     
  33. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33
    Oh yeah, I'm doing softbody on top. However, I'm not using any vertex buffer in-between, so I'd be writing directly to the skinned mesh graphics buffer, which always gets overwritten in unity's skinning pass after post update. So I do most of the stuff in "OnBeginFrameRendering", which might not be the best idea pipeline wise, so I might invest in an additional buffer, just so I can run things earlier in the frame.

    I'm doing it already for a cloth simulation asset that uses jobs instead of the Mesh API (Magica Cloth). It's working fine with an extra buffer to write to. They are doing jobs instead of compute shaders (they only use them to write to the mesh), so it's quite interesting.

    I think deactivating bone weights might get rid of motion vectors, too. I'm not sure when exactly the writing of the velocity takes place. If it is done in the skinning pass, or if it's done in actual rendering after.

    Depending on, I guess they could be manually written. I haven't tried to take away the bone weights yet. Setting the root bone to the object itself might maybe do the trick, but maybe you have to actually remove the bone weights by setting them to -1.
    I'm not sure how unity's skinning is implemented exactly. It seems it went through a lot of stages over time.
     
  34. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    525
    I am already setting the root bone to the GO where the renderer is on. This does not seem to prevent the skinning. I am seeing 0.08 ms in the Profiler (on 3080), which is not much, but every tiny bit counts;
    I actually was able to reduce to half by setting:
    Code (CSharp):
    1. mesh.boneWeights = new BoneWeight[mesh.vertexCount];
    This should at least overwrite the first 4 bone weights. As I am using >4 bone weights, I guess there is still room for improvement, but I was not able to write to the NativeArray bone weight structure without errors; it's either complaining that weights are zero on vertex (if I set all weights to zero), or that there are zero weights (if I set the count to zero).
    So it seems impossible to get rid of skinning completely.
     
  35. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    525
    A little bit better:
    Code (CSharp):
    1. byte[] bytes = new byte[mesh.vertexCount];
    2. for (int i = 0; i < bytes.Length;  i++)
    3. {
    4.    bytes[i] = 1;
    5. }
    6. NativeArray<BoneWeight1> newBoneWeights = new NativeArray<BoneWeight1>(new BoneWeight1[mesh.vertexCount], Allocator.Temp);
    7. NativeArray<byte> newBoneCounts = new NativeArray<byte>(bytes, Allocator.Temp);
    8. mesh.SetBoneWeights(newBoneCounts, newBoneWeights);
    This should set the skinning to one bone per vertex, with no weights. I guess the compute shader for GPU skinning is still running due to the bone count of one per vertex, this seems not to be avoidable.
     
  36. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    33

    I see. I'll probably do that as well, since there doesn't seem to be any downside. I wonder if it would be possible to gain access to the skinning compute shader or at least its data, and then setting its vertex count to 0. Might be extremely hacky, though.