Dual quaternion skinning for Unity

Hey, I'm glad you like it. And thanks for posting the demo.

Calculating both linear and DQ will indeed get more expensive. Not 2x, as some things can be repurposed, but still noticeable.

Vertex alpha sounds like it might conflict with something else. I'd imagine many shaders already use it for different purposes. If I get to implementing it (though no promises), might get the user to choose where the weights are stored from multiple variants.

Meanwhile, have you tried the bulging compensation option? Knees and elbows are it's "favorite" areas (meaning it works best on them).

Demo:

 

Thanks for the reply! (Please note I'm on Unity 2019, so I'm using the latest version that works with that.)

I've tested the Bulge Compensation. It's better for the knees and elbows, but there are two issues.

1. The knee still has a protrusion. In the gif below the white is linear skinning in front of the DQ model. Notice the Bulge Compensation doesn't deflate the knee entirely.
2. Bulge Compensation is deflating the glute area, and I want that to keep that volume. (You can't see that affect in the gif below.) So I'm back to the request of masking / weighting the Bulge Compensation or Linear skinning using vertex color or a bitmap.

Again, thank you for creating this project. Please take my feedback as suggestions and not as demands. Even as is, I think our project will move forward with this plugin. I'll let you know if it works on mobile platforms.

 

Please, clarify, which part of the knee do you consider improperly deflated. This link provides demonstration of the skinning artifacts I considered.

As you can see, linear skinning causes the middle of the knee to loose a lot of volume, while DQ animated it properly. That is why you see the difference. This effect might be possible to replicate, but I don't see why you would want it.

On the other hand, if what you mean is that setting the compensation intensity to 1 is still not enough, I can increase the limit, or you can even do it yourself. The limit is artificially set in UI and the method can properly handle values above 1 without any modifications required.

The masking of Bulge Compensation is on my To-Do list already. It would be easy to implement, once I get some spare time.

For the blending between Linear and DQ - I'll consider it, but no promises there.

Edit:

I tried to launch a project with the latest version of the script in Unity 2019.3.13f1 to see how hard it would be to backport, but it surprisingly worked without an issue. If it doesn't work for you, please upload a minimal project somewhere so I can take a look at it.

Depending on how old the version you are using is, it might have serious issues.

 

@mr_madcake do you know offhand what happens to meshes that have animated blendshapes which get converted to the DQ skinning setup? I know blendshapes are supported with this asset, but do they animated how the original model would've having keyed blendshapes?

Until COVID-19 sort of took over, I was experimenting with iClone and Faceware motion capture. Curious whether DQ skinning could be used for these models to improve their animations while keeping the facial animations via blendshape.

 

Not sure I understand the question, but blendshapes are applied before the skinning takes place. The skinning method is unaware of their existence. First you apply the blendshapes, then the resulting model is passed to the skinning code.

Regarding the application of blendshapes, I'm 99% sure the default skinning system does it in exact same manner.

 

Perhaps I can provide you a model to better illustrate what I am trying to accomplish. What would be the easiest way to provide you with something?

 

Just upload to google drive, mega or similar. Though it would be better to have a minimal project, not just a model.

 

@mr_madcake sorry for the delay, but here's a link to a sample project all setup: link

In the SampleMocap scene, there are three characters:

• The (reference) character was exported from Reallusion's 3DXchange. It does not have any DQ skinning and is used to demonstrate the facial mocap via blendshape keying. The DQ Skinner script is found on the CC_Base_Body game object.
• The (DQ) character is a duplicate of the character mentioned above but is configured using DQ skinning. The DQ skinner script does inherit the blendshapes for the skinned mesh, but none of them animate. Also, the character turns into some sort of mess as well which I'm not too sure why.
• The (DQ working) character was exported from Reallusion's iClone which is one step prior to 3DXchange export. This is a reference of what the same character should be looking like with DQ skinning. Unfortunately, it isn't useable in my workflow as it buggers up the blendshape naming. My workflow requires the 3DXchange character to be working as I am using custom imported characters made outside of Reallusion's character suite (so I am hoping to resolve the second character mentioned in this list).

Pressing play in the scene will have all three characters idle and do a facial mocap loop.

 

After looking into the implementation, would a conversion to HDRP "only" mean to adjust the shader (Hacked Standard)? I guess the rest (including compute shader) could stay same. I have no clue about shader programming, but as I somehow need a dual quaternion shader, I might give it a try, if it's "only" about changing the rendering shader (I saw other projects successfully "hacking" the Lit and LitTesselation shader of HDRP).

 

seems that in general blendshapes only work "offline" with this solution; as soon as I touch blendshapes while running in play mode, mesh gets destroyed, and I need to reimport my asset.

 

both issues confirmed, will investigate

 

sounds good!
If you have some time, could you revisit my question above about the shaders ? I'd really like to try out to implement the HDRP shader in the same way, if you think it's feasible. I saw in another project, that only relatively small changes in some include were done to make a custom shader work (the include handles the vertex placements). I am struggling a bit to understand how the current "standard" hacked shaders work; I looked into the implementation, the only change I see is that you pass in "uint id : SV_VertexID;" in all stages; but there must be more ?

 

Hi, I failed to setup DQ skinning. I followed 'How to setup' in github. Unity version is 2019.3.12f1 and this character(from Daz3d) has lots of blendshapes. btw, an error like the picture below appears and DQ is not applied. I think I made a mistake, so please teach me how to set it up in more detail.

 

The script work only in play mode. Your configuration seems to be good.

 

The errors should not be there, though. I'm afraid without the project files I won't be able to help, as I haven't seen such errors happening before.

Did you do anything to the skinned mesh through your own scripts? What did you do, when the "instantiating mesh..." error appeared?

If you send me a minimal example of a project, where this error happens, I will look into it.

 

All modifications are marked like this:

Code (CSharp):

1.         // ----- DQ modification start -----
2.  
3.         ---inserted code---
4.  
5.         // ----- DQ modification end -----

There are 7 such blocks. Before each block begins, the original #pragma vertex is commented and replaced with a new #pragma vertex within the inserted block.

Example (replaced #pragma vertex vertBase):

Code (CSharp):

1.             #pragma multi_compile_fwdbase
2.             #pragma multi_compile_fog
3.  
4.             //#pragma vertex vertBase
5.             #pragma fragment fragBase
6.             #include "UnityStandardCoreForward.cginc"
7.  
8.             // ----- DQ modification start -----
9.  
10.             #pragma vertex vertSkinnedForward // original #pragma vertex (above) was commented
11.  
12.             struct VertexInputSkinningForward
13.             {
14.                 float4 vertex   : POSITION;
15.                 half3 normal    : NORMAL;
16.            ...

All it does is replaces the vertex function with a custom one, that extracts skinned vertex positions from a texture. There are no other changes.

 

Thank you for replying. here is example project link : https://drive.google.com/file/d/1eJL1Je-OxxANLy0qLpfy_Tg3wAI5qx9v/view?usp=sharing

An error occurs when setting bone orientation.

 

Then can I check if DQ is applied only when I play the animation?

 

Yes. To see the difference, you need to duplicate your mesh and materials. One with default materials and the other one with the DQ materials and configuration (DQ script)
Then start play mode. Normally you will see the difference depending on your animation.

 

The difference is most obvious with twisting a limb.

 

The materials (G8F_Sclera, G8F_Torso, etc.) all use Standard shader. For the script to work, you must use
"MadCake/Material/Standard hacked for DQ skinning".

The errors are caused by a bug in the script, but it should not prevent the script from working. The problem resolves itself, once you enter Play mode. I will fix it in later version.

 

Thanks so much for the explanation!
I nearly got a first basic version running in HDRP pipeline.
But somehow, no matter what I do, a very small number of vertices don't get (dis)placed correctly.
I am using plain shader graph with no custom code, and managed to put all the infos in the uv of the mesh (which might also speed up a bit, as we don't need to calculate during update anymore). However, here it seems the issue is; not sure if I got it right. This is my code to pass in info via UV to shader graph:

Code (CSharp):

1.         Vector2[] vertIds = new Vector2[this.mf.mesh.vertexCount];
2.  
3.         for (int i = 0; i < this.mf.mesh.vertexCount; i++)
4.         {
5.             float x = ((float)(i % width)) / width;
6.             float y = ((float)(i / width)) / textureHeight;
7.  
8.             vertIds[i] = new Vector2(x, y);
9.          
10.         }
11.         this.mf.mesh.uv3 = vertIds;

This should be similar of the lookup in hacked standard shader, but I do it only once inside of "GrabMeshFromSkinnedMeshRenderer". One thing which I noticed is that the textureHeight here is different from the texture height in "Update()"; in update, you are using "this.mf.mesh.vertexCount / textureWidth", while in the method "GrabMeshFromSkinnedMeshRenderer" the height can be different due to

Code (CSharp):

1. if (this.mf.mesh.vertexCount % textureWidth != 0)
2.         {
3.             textureHeight++;
4.         }

But I also tried to align both, issue gets not completely resolved. Do you have any hint / clue what the reason could be?

 

Got it at working in HDRP, albeit the issue remains, but it gets smaller or vanishes if I increase the texture size for the shader; still not sure what the real issue is, also asked in shader graph forum.
Joints look very good, everything works smoothly (I am now on version 1.8, as I need lots of blendshapes).
@mr_madcake I would also be interested in testing the bulge compensation, any chance you'll be able to repair the blendshapes?

 

I am planning to work on it whenever I get the time. Alas, currently I have no way of knowing when will it be.

 

I briefly looked into the code and it turned out the bug was quite simple. Both blend shapes and exceptions popping up when changing bone orientation in edit mode have been fixed in v2.14

 

The different textureHeight formula in Update() is indeed a bug. I am a bit confused how it remained hidden for so long, as texture height missmatch should have produced obvious problems.

The texture height in Update() matches the texture height in GrabMeshFromSkinnedMeshRenderer() since v2.15

 

Sounds great. Very eager to test the bulging compensation, hopefully the remaining small issue with HDRP is also gone with this. Unfortunately can only test next week.

 

Which version of 3DXchange are you using?

This might be the cause:
https://www.reallusion.com/FeedBack...des-the-Root-Bone-in-Exported-CC-Characters_1

In your example the root bone is detected incorrectly for the second model (CC_Base_Waist)

 

@mr_madcake currently using 3DXchange 7.61. Seems that issue was from 2016 and (fingers crossed) resolved.

 

You make it work in HDRP? I'm trying to figure out how... I tried to add the modificated lines from the hacked standard to a new 'Lit Hacked' shader but it's not working...

 

Somehow yes; but only version 1.8 of DQ Skinner; there seems to be a change somewhere (how materials are handled), which break my current approach, not sure why.
To make it work, I built an own shader with shader graph (using vertex displacement). I am pretty sure it would also be possible to hack the lit shader of HDRP, but I did not want to go this route because of future updates.
Also, my approach works not perfect in all cases; on some models, some vertices are displaced wrongly. I have not figured out, why this is the case, but also did not bother too much, as I can work around this with adjusting the texture size in the script (4096 seems to work without issues).

 

Please help, I think this problem is from Shader.


'MadCake/Material/Standard backed for DQ skinning' was used as shader. The character splits like a picture. I wonder if I made a mistake. version is 2019.3.12f1. here is my test DQ link : https://drive.google.com/file/d/1lJmJfkwYML5SS1VS1dMCcLbq43jVwDJk/view?usp=sharing

 

This is interesting. It’s the exact same issue I got in my HDRP port.

 

PS: This somehow seems to related to the vertex count and the used internal texture size. There is a mod operation to look up the vertex displacements. This gets 0 on every 1024th vertex. I guess this is the reason, but not 100% sure.

 

For using HDRP, I wrote the code above but failed. if you don't mind, Please share the code, I really want to use a custom shader like this(https://assetstore.unity.com/packag...dvanced-skin-shader-2-standard-edition-148546)

 

this might get more difficult (integrating into another shader.) What I did to make it work:
- used an older version of DQ skinning (1.8, in the 2.x versions there is a change which seems to break my approach)
- added the code I posted above in the script for DQ skinning to write the vertex positions into UV map
- wrote an own shader (with shader graph) to use the vertex positions for displacement on the textures which are passed over from DQ script
- changed texture size to 4096 width, because of the bug discusses above; this also does not work for all meshes, but in my case it gave good results

If I find time, I'd try to make the most recent version work with HDRP, but this will only work with a custom shader using vertex displacement. The part for the displacement can in theory be added to an arbitrary shader graph, as long as the master node supports displacement. it's just using https://docs.unity3d.com/Packages/com.unity.shadergraph@8.1/manual/Sample-Texture-2D-LOD-Node.html to read the pixel displacement from the textures passed to shader graph.

 

Does this technique work with IK as well?

 

IK is just another way to change the pose of the character. Using IK should work just the same as any other animation.

 

The model in your project looks quite high poly and I've seen this happen to models that have crazy vertex counts. Not sure what exactly it is, but it does not happen if you either lower the model vertex count or get better hardware.

On my setup the animations in your project work properly (I manually tweaked the pose for this screenshot):


It would be helpful, if you ran something like GPU-Z to figure out what exactly is the bottleneck.

On the slight chance it has to do with Unity version - I ran it on 2019.3.15f1

 

I tried DQ on 2019.3.15f1 as you set it up. But it breaks as shown in the picture. I can't find any trouble in GPU-Z, my hardware is Ryzen 3700X, 2070s. would you tell me DQ options you used?

 

I ran your project as-is, without changing anything. I also noticed, that there is an error during post-processing your model on import. I reduced

AssetPostProcessorReorderBones

to this minimal example to show where the error happens, but I have no idea what it means and why it happens:

Code (CSharp):

1. using UnityEngine;
2. using UnityEditor;
3.  
4. public class AssetPostProcessorReorderBones : AssetPostprocessor
5. {
6.     void OnPostprocessModel(GameObject g)
7.     {
8.         SkinnedMeshRenderer smr = g.GetComponentInChildren<SkinnedMeshRenderer>();
9.         smr.sharedMesh.boneWeights = smr.sharedMesh.boneWeights;
10.     }
11. }

The line (9)

smr.sharedMesh.boneWeights = smr.sharedMesh.boneWeights

produces an error:

Code (CSharp):

1. Unsupported conversion of vertex data (format 4 to 0, dimensions 4 to 4)
2. UnityEngine.Mesh:set_boneWeights(BoneWeight[])
3. AssetPostProcessorReorderBones:OnPostprocessModel(GameObject) (at Assets/DQ/
4.         AssetPostProcessorReorderBones.cs:9)
5. UnityEditor.AssetPostprocessingInternal:postprocessMesh(GameObject)
6.  

The basic DQ skinning does not depend on post-processing, but the bulging compensation does. Though the results displayed on your screenshots should not be possible regardless of the settings and post-processing of the model (or lack of such).

 

Sorry for noob question, I've opened AssetPostProcessorReorderBones, but I don't know how to fix it. I'd really like to use high vertex counts. Is there any chance that this problem will be resolved in the near future?

 

you may try to increase texture size in "textureWidth" (line 60 in main script). For my case, a value of 4096 helped to mitigate all issues, but if I change vertices, issues are also present with this size, so you might want to try other sizes (power of 2). I think there is a bug in the implementation somewhere related to either the pixel lookup, or creation of the textures, but this is complicated stuff I don't fully understand; just guessing it is an issue with the mod operation being used.

 

Thank you for replying, sadly, to change "textureWidth" to 4096 was not helpful btw, even if I mitigate this problem, If I build it and run it on another computer, could it still have the same problem again?

 

It looks like the GPU is not able to perform one of the operations the code is asking it to do.

 

Thanks for reporting this. I will try experimenting with textureWidth.

Please, explain in more detail what do you mean by "if I change vertices"

 

I don't understand what causes this problem so I can't really say.

Regarding AssetPostProcessorReorderBones, it seems to be either a bug in Unity or an issue with your model.

This error makes no sense at all. I read the value from a variable and immediately assigned it the same exact variable. If the original value was valid, idk how could it possibly become invalid after assignment.

 

Try this version

Verify that it says

const int numthreads = 8

on line 59 of

DualQuaternionSkinner.cs

Note, that it is not the only change. It is just to ensure that you have the right version.

 

for example by changing the smoothing angle value in the import settings (model tab) of an imported fbx; I guess this changes the number and order of vertices. If I do more smoothing, the artifacts gets worse.
Edit: Please be reminded that I am working on an own branch with HDRP adjustments; just posted this as it produces exactly the same artifacts as others are reporting here with the unmodified version.

 

I just double-checked to be sure and "smoothing angle" import setting only affects normals. It does not change the number of vertices and their order is sorted in the

AssetPostProcessorReorderBones.cs

script after the import.

This is very weird and might be a manifestation of a much deeper and obscure bug. Please, try the same model and import settings on my original version from github. It might have the same flaw.