Dual quaternion skinning for Unity

Hi there, this looks really awesome. Does this technique work on mobile (oculus quest), using opengles? Is there any way to run this on unity 2018 LTS?

 

Hi

I haven't tried but theoretically it should work. Have you encountered some issue with it?

 

No, but I haven't because I didn't know if it would work on 2018 LTS due to the multi_pragma_compile issue. But now that you mentioned it should, I can give it a shot. Just wondering how much a performance hit it is compared to normal skinning, since we are heavily GPU bound right now,

 

It's hard to give a clear number as it scales differently with different things. From my benchmarks I could not get worse performance drop than 30% compared to built-it skinning in purposefully built worst-case scenario, but it could vary wildly depending on the project.

Also CPU hit from extracting bone transform matrices affected performance the most, there should be no big issue with GPU.

If you want good performance I strongly recommend using IL2CPP compiler as is properly optimizes bone transform matrix extraction (I made the benchmarks with IL2CPP). Mono uses some really weird shenanigans that slow down the script A LOT.

This line in particular is A LOT slower in Mono and in profiler it reportedly calls to String.memcpy() for no apparent reason.

If you don't need the bulging compensation you can remove this part of the shader for a little extra performance. This will leave you with traditional DQ implementation.

 

I have a mesh with 4332 vertices (so vertNumber/textureWidht = 4). If I use GetVertexTextureHeight in Update it is rendered garbled at runtime (I did not investigate yet why a textureHeight of 5 instead of 4 makes it do that).


Anyway, i'm posting this in case anybody else has this problem. I've reverted the line in Update to the previous form to solve the issue: this.materialPropertyBlock.SetInt("skinned_tex_height", mf.sharedMesh.vertexCount / textureWidth);
Observation: there are multiple meshes using DualQuaternionSkinner (different vertex count, all of them worked but this one). Again, I don't know yet why, the only difference is that all the other meshes have a texture height ratio lesser than 4 (0,1,2,3).
PS: I am using Unity 2019.4.4f1 Personal.

 

Get the mesh of DQ blending skinning:
I have implemented the dqls in my own project. And I want to get the mesh of human model of dqls in obj format. However, I found i get the obj model of lbs. How can i get the mesh of dqls? Thank you.


 

Hi.

I am a bit confused about what are you trying to achieve.

OBJ format does not support animation, it is a static model. For animated models you would usually use FBX.
Are you trying to make a static model from a single frame of animation in Unity?

My script only works in Play mode, so if you are trying to export the model in Edit mode, you would get the model with built-in linear skinning.

 

Could you try this version of the script with the same model?
I would like to make sure that these issues are not connected.

 

Any updates on the distortion of meshes?

I've tried DualQuaternionSkinner.cs on different models, but they all look extremely distorted.

 

Hey, I am testing different solutions for better skinning and I found something like this: OptimisedCentresOfRotationSkinning

Could this be useful in improving your solution?

Cheers!

 

this dosnt worked to me i used version 2020.1.7 of unity, then this error happens



line 604 of DualQuaternionSkinner.cs this.mf.sharedMesh.MarkDynamic(); // once or every frame? idk.

I follow the guide you have on github , but it didn't worked. i set as read enable true the skinned mesh renderer character have Body And Head separated , cant work this way? I'm just don't understand what its causing it. why didn't work ?. can you be more detailed in the instructions?
and here in my picture

https://gyazo.com/a09ce30fb5fc1af9b2dcfe1b73448b0b


it looks like empty in compute shaders and I cant set anything manually, so whats wrong with this? i was so excited about this but now I think I should continue with mesh animator, actually did this gpu instantiate the skinned mesh renderers? as mesh animator does? coz mesh animator its what I'm using now and works fine for GPU instancing Skinned mesh renderers

 

If you still have this issue, you can send me the minimal example project and I will take a look.

 

Optimized CoR indeed provide improved skinning, but AFAIK there's some issues with licensing and I'm not sure I can legally implement and share it other than for education purposes.

The technique I used is an alternative to optimized CoR that I developed as my master's thesis. Unlike CoR it does not require any precomputations, but the quality is worse (still better than baseline DQ, but not as good as optimized CoR). Initially I implemented DQ in unity just to have it here, but when writing the thesis I decided it would be convenient to use my already working script to demo the new technique.

 

Anyone having their models look completely butchered and distorted by the script can try this version (quoted message). I might have hard-coded some values in the script that some GPUs are not able to work with. If the issue is what I think it is, the quoted version should be compatible with GPUs failing to run the main version.

 

I've been scratching my head for quite some time trying to figure out why my rigged models exported from Daz3D don't look the same in Unity comparing to Daz. Now I realized this is what I always needed...
However I imagine it would be difficult to adopt your approach since I use a lot of custom shaders done in ShaderGraph, I guess this is going to be tricky trying to integrating your stuff in my shaders...

 

I came across this new Hybrid Renderer, and it looks like there is a chance to create your own deformation node to perform the skinning.

https://docs.unity3d.com/Packages/com.unity.rendering.hybrid@0.11/manual/mesh_deformations.html

 

Thank you for sharing that code. I've been hitting a brick wall trying to make DAZ models look good in unity. I finally found out about Dual Quaternion Skinning, and this was the key.

I made a working HDRP/URP/LWP implementation using the new 2021.2 Mesh API, that allows direct buffer access and modification. It was a lot more straight forward than I originally though (thanks to the helpful tips from Qleenie as well)

To get this to work, it's only a few lines added to the DQBlend.compute shader.

RWByteAddressBuffer Vertices; //a direct byte buffer which can be used to directly modify the vertices of a mesh.

All we need to do, is assign the skinned vertices we already have:
For that we first need to find the exact addresses of the vertex buffer.
To do this, you can use mesh.GetVertexAttributes(); and print out the array.
This will give you offsets, strides, etc to get the exact positions of the data you need.
In my case, this was

    uint vidx = id.x * 40; //0 + 12(vert) = 12

    uint nidx = vidx + 3 * 4; //12 + 12(norm) = 24

    uint tidx = nidx + 3 * 4; //24 + 16(tan) = 40 (tangents are itself 4*4)

Keep in mind that this might depend on the imported mesh, so the best way to would be transfer the data from CPU so that no values are hard coded like this.

After that, all we need to do is put the skinned vertices into the array.

    Vertices.Store3(vidx, asuint(vertinfo.position.xyz));

    Vertices.Store3(nidx, asuint(vertinfo.normal.xyz));

    Vertices.Store4(tidx, asuint(vertinfo.tangent));

The rest ist just assigning the Vertices Buffer in the DualQuaternionSkinner

    GraphicsBuffer vertexBuffer;



void GrabMeshFromSkinnedMeshRenderer()

{

...

//all of this only needs to be done once:

mesh.vertexBufferTarget |= GraphicsBuffer.Target.Raw; //needed to get the raw byte buffer

vertexBuffer = mesh.GetVertexBuffer(0); //retrieve the vertex buffer of the mesh

this.shaderDQBlend.SetBuffer(this.kernelHandleDQBlend, "Vertices", vertexBuffer); //assign the vertex buffer

...

}

That's pretty much it. Except one thing:

this.mf.sharedMesh.MarkDynamic();

has to be moved from update to init.

Calling it every frame invalidate the direct buffer (as will any other modification of the buffer from the CPU side), and cause the buffer to be recreated. If that happens, you will not see any modification to the mesh. I don't think this method is necessary for the original version, as the original never modifies the vertexBuffer itself, and instead deforms via shader. From my OpenGL days, I guess this like setting the hint for a vertex buffer object.

Additionally, I made a version of the ApplyMorphs function that will change one morph by removing the previous weight, and then adding the new one, all done on the already uploaded array. This saves a lot of performance by not rebuilding the complete set of morphed arrays when only one morph changed.

I hope this helps people running into similar issues as me.

 

Since my figure is using over 200 fast changing morphs (JCM to compensate for bulges), it was necessary to optimize the morphing system for memory and speed.

As already mentioned above, my first optimization was to only update morphs that changed. Because calling ApplyMorphs() with a full update caused massive slowdown, only updating changed morphs already helped a lot.

However, there were more problems arising from the morph system. The biggest problem being that every morph has a complete delta image for every single vertex. With ~80k vertices and 200+ morphs, all in all, this would be more than 1 gb of vram just for the blendshapes, which wasn't viable.

Rewriting the morph structure to include a vertex index solved that problem. The compute buffers for morphs can be scaled down to only contain vertex deltas that are greater than distance zero. In the compute shader, the vertex array can then addressed via the morph's vertex index.

Not only did that save a ton of space (I'd say about 95% smaller in vram, about 1gb less), but also because using the indices to update on the compute shader, it also greatly reduced processing, because there are a lot less vertices that need to be processed.

 

Bounds: Because all the vertices are stored in VRAM and not on the GPU anymore, there is no easy way to calculate the bounds based on the vertices.
Using the compute shader to do it wouldn't work, because you would run into racing conditions when all threads try calculate and write the min/max at the same time. There is a way to synch up threads via memory barrier, but that would cost quite a bit of overhead. Alternatively running a compute shader on one thread to just iterate over every vertex to create the bounds will cost even more performance.

What I went with now is just completely ditch any vertex based bound calculations. Since it's going tobe axis aligned bounding boxes at the end, precision is not very important.

Instead I'm collecting an array of transforms at key bones (head, hands, chest, shins feet), and create an AABB from each with an arbitrary margin, and then add in turn to a complete AABB, and submit that to the bounds of the mesh.

This seems to work perfectly with almost no cost, and frustum culling can be done again.

 

Calculating the mesh boundaries sounds like a good approximation! I am doing it in compute shader, as I also do some subdivision for mesh collision stuff, but that's a bit expensive, as you need the mentioned memory barriers / InterlockedMin and InterlockedMax. About the morphs: I looked into my code, I am doing it exactly same (this is also somehow similar as SkinnedMeshRenderer seems to do it).

 

Hello, @schema_unity would you be able to share your package with the code and changes you describe above? Pretty please

 

I'd also love to check out what you've put together here @schema_unity -- I've been looking for a solution to some of these problems for several years now.

Thanks

 

IIRC (might be wrong) when importing an animated mesh, Unity goes through all the animations and computes AABB once so that all attached animations stay within the volume of AABB. Re-calculating would be needed only if you have unpredictable procedural animation, or you need a very precise AABB.

IIRC my script does not do this, but it shouldn't be too difficult to implement.

If you need fast (min/max/avg) in a shader, this pdf provides a lot of info on how it can be optimized, though it's not an easy task to get right. GPU performance can be very counter-intuitive.

 

Yeah, the main purpose of bounds is to frustum call the model. Using a per vertex and per frame update probably will get the most accurate result, but the overhead for the additional accuracy will probably be never worth it, because the whole reason to frustum cull is to avoid going through the data on a vertex level (be it for drawing or not).

The only advantage of a more accurate bounding box would be to not draw objects that are slightly off camera. Unless a scene has a ton of skinned objects, it will likely not make a difference in performance for those edge cases. And if there indeed are a ton of skinned objects, the overhead from going through each vertex for a bounding box might be heavier than what you would gain culling it. So especially on a lot of models you would want a really fast bounding box calculation.


Currently my code is a bit intertwined with some other systems, like Odin Inspector, and some of my other systems that extract vertex data from skinning for their own calculations.

However, I'm planning together with @jcluts to release a cleaned up version of madcake's DualQuaternion implementation for Unity.2021's mesh API, featuring blend shapes, up to 6 bones, and blendshape vertex/normal/tangent override.

 

Hey @schema_unity, is this something you're still working on? Are the code snippets you posted the only code needed to make your solution work in a HDRP project? I pasted them in, but no luck unfortunately. --I'm definitely no shader programmer

Thanks for your efforts and explanations so far.

 

Yeah, the code snippets should technically be enough to get it working. The problem might be in the mesh you're displaying/modifying. I recommend rewriting the DQ code to get rid of all the now unneeded texture code.

Your skinned object should have both a skinned mesh renderer as well as a normal mesh renderer. The skinned mesh renderer is a reference for the DQ code. It is turned off on start and the mesh renderer is what is actually displayed. Furthermore, I recommend not making changes to a shared mesh buffer (sharedMesh), as changes will persist between edit mode and play mode until a complete mesh reload (usually restarting unity). It's best to crease an instance of the mesh at the start and work with that. This instance would be the only thing targeted in the DQ code also. You will only need the skinnedMeshRenderer to read the bindposes and bones into the shader at startup.

Is there any error you're getting?

 

To add on what @schema_unity wrote you can also write to the SkinnedMeshRenderer, without the need of an extra MeshRenderer. It's a bit more complicated, but with this you are able to use skinned motion vectors (if this is relevant for your use case).

 

Is it possible to prevent the default skinning method of unity from running when using skinned meshes? And if you would do that, wouldn't you have to write motion vectors manually? Just wondering if the cost of having to run skinning twice on the same object can be avoided or if there is approach that simply replaces linear skinning.

 

I tune down all parameters of SMR, e.g. use one bone only, and overwrite the mesh each frame. Motion Vectors are generated by SMR automatically, if option is enabled. Would of course be better if we'd be able to provide own Motion Vectors in MeshRenderer, but I am not sure if this is possible.

 

Taking a stab at it I implemented my own double buffering to keep track of velocity, then had a custom function in the shader graph grab the buffer.
I first thought it was just a matter of setting the custom velocity of a stack lit shader.
Turns out, that no matter what you put in there, the motion vector calculation doesn't actually happen. Trying it out on a SkinnedMesh not only applied the default motion vectors but also my custom vectors on top of it, so I'm guessing the purpose of the custom velocity is to apply any additional deformation motion on a mesh that is already using a skinned mesh renderer. I'm not sure if the normal mesh renderer lacks per vertex motion vectors all-together or if I'm missing something.
I've not seen any post or documentation that goes into it very much, unfortunately.

Additionally, everything about the skinned mesh pipeline gets more confusing the deeper you go into it, to say it nicely. The way it is implemented allows for almost no customizability, unless injecting the vertex manipulation just before the frame gets drawn, effectively overwriting the default skinning. It's possible to remove skinning calculation by removing the vertex bone weights and just setting the object itself as the root bone and its only bone.
However, this does get rid of the motion vector calculation as well, as that is likely done in the skinning pass.

While I got this to work now, and motion vectors with blur do look nice, it really bothers me that an extra skinning pass has to be done, even if it's just on one bone per vertex, but I don't have a better solution right now.

The other thing that bothers me is how the skinned mesh renderer switches buffers, which makes it very hard to additional passes like softbody processing without running into possible errors. And I don't know if it is relevant, but having the skinning done so late in the frame might be subject to pipeline issues.

Using an extra buffer would make it possible to do all vertex manipulation beforehand, and then just copy it over before the frame draws, however, it introduces another full verte buffer on top of the skinned motion vector buffering. But at least that buffer would be consistent. I might add this as an option to my code just to see how much it impacts things (probably doesn't even matter).

What we need is a new version of the skinned mesh renderer. A version that is modular, where you can just switch out all or part of the geometry processing. This would instantly solve all problems. Not only that it would solve problems for so many other Assets on the store that suffer from how closed such a central component is. Cloth simulations, soft body, collision detection all would profit greatly from a new open version of SMR.

 

You can still do softbody and/or cloth stuff with this setup, just do it after DQS pass, and before writing to the VertexBuffer of mesh. I am doing it like this without issues in production. Writing to the mesh works best/ only in "OnBeginFrameRendering" on SRP, so I do it either directly affter calculating DQS, or after calculating softbody on top of DQS positions.

I'd be interested how you remove the bone weights. Just set smr.sharedMesh.boneWeights.boneIndex0 - smr.sharedMesh.boneWeights.boneIndex3 to "-1" ?

 

Oh yeah, I'm doing softbody on top. However, I'm not using any vertex buffer in-between, so I'd be writing directly to the skinned mesh graphics buffer, which always gets overwritten in unity's skinning pass after post update. So I do most of the stuff in "OnBeginFrameRendering", which might not be the best idea pipeline wise, so I might invest in an additional buffer, just so I can run things earlier in the frame.

I'm doing it already for a cloth simulation asset that uses jobs instead of the Mesh API (Magica Cloth). It's working fine with an extra buffer to write to. They are doing jobs instead of compute shaders (they only use them to write to the mesh), so it's quite interesting.

I think deactivating bone weights might get rid of motion vectors, too. I'm not sure when exactly the writing of the velocity takes place. If it is done in the skinning pass, or if it's done in actual rendering after.

Depending on, I guess they could be manually written. I haven't tried to take away the bone weights yet. Setting the root bone to the object itself might maybe do the trick, but maybe you have to actually remove the bone weights by setting them to -1.
I'm not sure how unity's skinning is implemented exactly. It seems it went through a lot of stages over time.

 

I am already setting the root bone to the GO where the renderer is on. This does not seem to prevent the skinning. I am seeing 0.08 ms in the Profiler (on 3080), which is not much, but every tiny bit counts;
I actually was able to reduce to half by setting:

Code (CSharp):

1. mesh.boneWeights = new BoneWeight[mesh.vertexCount];

This should at least overwrite the first 4 bone weights. As I am using >4 bone weights, I guess there is still room for improvement, but I was not able to write to the NativeArray bone weight structure without errors; it's either complaining that weights are zero on vertex (if I set all weights to zero), or that there are zero weights (if I set the count to zero).
So it seems impossible to get rid of skinning completely.

 

A little bit better:

Code (CSharp):

1. byte[] bytes = new byte[mesh.vertexCount];
2. for (int i = 0; i < bytes.Length;  i++)
3. {
4.    bytes[i] = 1;
5. }
6. NativeArray<BoneWeight1> newBoneWeights = new NativeArray<BoneWeight1>(new BoneWeight1[mesh.vertexCount], Allocator.Temp);
7. NativeArray<byte> newBoneCounts = new NativeArray<byte>(bytes, Allocator.Temp);
8. mesh.SetBoneWeights(newBoneCounts, newBoneWeights);

This should set the skinning to one bone per vertex, with no weights. I guess the compute shader for GPU skinning is still running due to the bone count of one per vertex, this seems not to be avoidable.

 

I see. I'll probably do that as well, since there doesn't seem to be any downside. I wonder if it would be possible to gain access to the skinning compute shader or at least its data, and then setting its vertex count to 0. Might be extremely hacky, though.

 

is the topic still relevant? Are there similar ways?

 

My way is bit more obeying the laws of the game engine and tackling those with the blend shapes.

Let me give you an example; I know every move any of my characters can do. Any loss of volume (which you can pinpoint) is compensated with the blend shapes. Is it fast to render, yes. Fast to create, depends. What I do in Maya is that with any maximum bend I take a quaternion skinning copy of the mesh. Then I switch to linear skinning with my model. Now I see the difference.

Next, I modify the blend shape for that specific body part to restore the loss of the volume.

What I have as a blend shapes;

L&R thigh bends, both knees and both elbows. I have no need to rotate any other joints beyond the limits. If my cloth needs to follow, I use a Maya wrap deformer to create the same blend shapes for them.

I know this is not what the topic is about, but having tested DQ skinning, it's just slow.

 

I'm having the same issue for "Unsupported conversion of vertex data" when setting multiple weight1s, which I think is caused by unclean information within the mesh, recreating a mesh and typing it can solve such problems.
Here is my solution:

Code (CSharp):

1.         Mesh newMesh = new Mesh();
2.         newMesh.MarkDynamic();
3.         newMesh.vertices = oldMesh.vertices;
4.         newMesh.uv = oldMesh.uv;  
5.         newMesh.triangles = oldMesh.triangles;
6.         newMesh.normals= oldMesh.normals;
7.         newMesh.colors = oldMesh.colors;
8.  
9.         newMesh.RecalculateBounds();
10.         newMesh.RecalculateNormals();
11.         newMesh.RecalculateTangents();
12.  
13.         //OYM:Now you can input more than 4 weight1 a vertex
14.         newMesh.SetBoneWeights(newBoneWeightCount, newBoneWeight);