CoR: Real-time Skeletal Skinning with Optimized Centers of Rotation

I recently wanted something similar to Blenders maintain volume skinning. I liked the method described in Real-time Skeletal Skinning with Optimized Centers of Rotation by Disney and implemented it in Unity. I’ll abbreviate it as CoR (Centers of Rotation). It’s similar to dual quaternion skinning but without bulging artifacts.

IMPORTANT: Disney research owns the patent for this technique so only use it for research or technical work on a non-commercial basis. I'm not related to Disney in any way. The code has a MIT license.

Normally I’d just add more volume bones on the elbows, knees and twist bones. Then control them with scripts or the new constraint components. Controlling it with scripts reduces the animation size, it can work with humanoid characters and prevents ragdoll or IK issues. The main thing I like about maintain volume skinning is that I can get better results with less work.

This was also my first time trying compute shaders. They’re a lot easier than I was expecting and reduced the preprocessing step from 10 minutes down to about 0.2 seconds. I’m also using a compute shader for skinning based on Unity Internal-Skinning.compute.

The algorithms performance seems to be similar to standard linear skinning. Unfortunately I can’t get standard linear skinning close to the native implementation even using the Internal-Skinning.compute shader. The bottleneck seems to be getting data from the compute shader. It would probably be faster to just skin it in a material shader (similar to some instanced skinned meshes examples). But then I’d have to modify each material shader.

The sample character is from a modified version of sintel-lite . I only roughly skinned it so it's not as good as the original. The code from skellington also helped me validate the output.

If it could be used commercial I’d upload it to the asset store as a free asset. I might try contacting Disney’s research lawyers again in a few weeks. Someone from Unity would probably have better luck at contacting them. It would be great if this type of skinning was available as a native skinning method (maybe as an optional package).

To use:
Add SkinnedCor component to a skinned mesh.
Click "Create CoR Asset" button
Extra: set a hd mesh (might help with low res mesh, I don't use it)
Extra: set weight map image black = linear skinning, white = cor skinning e.g linear skinning on face
Click "Compute Shader Pre Process" (need to click again if the mesh changes)
Run the game, move the "CoR Weight" slider to see the differences between linear (0) and cor skinning (1).

Download: CoR.unitypackage

 

This is quite cool -- Maybe post this in the Animation forum instead?

I'm not sure if Unity has looked into this yet, but it might be quite handy if they did. -- I hope you get some kind of response from them.
I, for one, totally appreciate you for all your efforts in shaders/animation. Just wanted you to know.

 

I tested your package and it works great. How do you think it would be possible to keep blendshapes with CoR? Are you considering adding this? As a side note, I saw that mr_madcake managed to add bledshapes in his Dual quaternion skinning for Unity. Maybe you could do it together? I think that the opportunity to test such a good solution could be useful for many people who do small personal or educational projects.

Cheers!

 

Thanks for the feedback. I haven’t tried creating blend shapes in a shader before. The CoR bones don’t work well in detailed features like the face so it normally uses standard linear skinning for those areas. Personally, if I needed face morphs I’d just have a different skinned mesh using the standard unity blend shapes. It feels like blend shapes in CoR might just work the same as linear skinning. I’m guessing it could be useful for corrective shapes but I’ve never really needed to use them before. Unfortunately, I don’t have much free time so I probably won’t be able to add it.

 

Got it, thank you for your reply

 

Hi!
Tested this package today, works like a charm! Huge thanks to you!
A couple of questions though:
1) What is "sigma" setting?
2) Does it expect a weight map that is painted in UV1 channel?
3) Is it okay if a character consists of a number of skinned meshes? Like, legs are a separate mesh, torso is a separate mesh and so on. Do I add this component to every skinned part that I want to have this effect?
Thanks!

 

Hi, the pdf explains the algorithm in more detail. Disney seems to keep changing the location. It's now at https://la.disneyresearch.com/publication/skinning-with-optimized-cors/

1) What is "sigma" setting?

Sigma (σ) from the pdf.
Similarity Function: σ is the parameter that controls the width of the exponential kernel.
4 Results and Comparisons: We found that our method is not sensitive to parameter selection. In our experiments, it produced visually indistinguishable results with any e ≤ σ ≤ 0.1. ... For this reason, we use (σ,...) = (0.1, ...)

To be honest, I'm not really sure what sigma does but it seems to work fine at 0.1 (probably ok to hide the sigma setting on the SkinnedCoR component).

2) Does it expect a weight map that is painted in UV1 channel?

The weight map (linear skinning vs CoR skinning weight) is optional. The attached character doesn't have any animated face bones so all vertices can use CoR skinning.
The UV channel is the same as the standard diffuse texture UV e.g. TEXCOORD0 in the shader.
I used an image because it's easier to setup and debug, but using a vertex color would probably be more efficient.

3) Is it okay if a character consists of a number of skinned meshes? Like, legs are a separate mesh, torso is a separate mesh and so on. Do I add this component to every skinned part that I want to have this effect?

I think I originally had the character as parts but combined it to made the demo easier. Multiple parts should work as you suggested, just add a component per skinned mesh.
One issue I've had with multiple parts is a noticeable seam e.g. head using linear skinning, body using CoR skinning. I was thinking of trying an extra vertex weld pass between the neck and head but never tried it e.g. average vertex positions and recalculate the normals.

 

Yeah, couldn't notice any difference as well. Probably it's okay to hardcode the "0.1" value and hide this option.

Haven't thought of that. Thanks for the idea!

Tried it on a mixamo rig. Applied to torso only, leaving legs and everything else untouched. Looks okay, no seams so far.

 

@camerondus has a wip repo base on this project (currently removes the need to get the vertex data on the CPU and is ~76% the speed of the built in unity GPU skinning). Thought I'd include it incase anyone wants to get new updates or maybe for creating pull requests.

https://github.com/Milk-Drinker01/CoR-Skinning

 

hey guys!
I just updated the repo again. im now getting about 83% the speed of the built in gpu skinning. Though, it uses a couple hundred extra mb of ram with 225 skinned characters on screen. next updates will focus on functionality rather than speed. be sure to check it out!

 

I've had several approaches to preserving blendshapes and CoR at the same time. I tried to split the mesh as @CodeKiwi suggested, but then there are terrible gaps between the parts of the mesh without CoR and the one with CoR. Maybe you @camerondus have any idea how to keep skinned mesh renderer with blendshaps or how to dynamically load them without lag?

I tried to do this by pulling out the current mesh snapshot with all the blends at the moment from the original Skinned Mesh and project it as a base for CoR reloading and it works but doesn't allow you to make changes to the blends in real time - this is better suited to updating from time to time, but I would need to do it in real time.

 

I ended up testing blendshapes a while ago but didn’t get any good results. The main problem is that CoR requires some expensive precomputation to work e.g. took days to process before moving it to the compute shader.

Options:

• Ideally there's a way to apply the blend shapes using the preprocessed CoR data but I’m not sure how to calculate it. It also might not have a solution.
• If you just want morphs on a fixed area (e.g. characters face) then there is a CoR / Linear weight painting option. So you could blend the face to linear skinning and apply the morph to that. If it was a separate mesh then the vertices should align but the normals might be a bit off.
• Probably the more difficult option would be a corrective shape e.g. arm muscle bulge. I’m not sure how to calculate this. If you only have a few morphs then it might be easier to have an instance per morph and then merge them. This would use a lot of memory and be fairly slow.

 

And what do you think about using the new 2021.2 Mesh API, that allows direct buffer access and modification? @schema_unity writes about it in this thread:
https://forum.unity.com/threads/dual-quaternion-skinning-for-unity.501245/page-4#post-7985886

 

The new Mesh API looks really interesting. One of the main bottlenecks I've had in the past was transferring data between the mesh and compute shader. Sounds like it could provide some decent performance improvements.

 

1)is this project still on? does it work with the newest unity LTS version?
2) does it work with daz3d characters that have multiple clothes(Shirt,pants,helmet) as skin meshes

 

I haven’t updated it in a long time and I’m not actively working on it but @camerondus did add some improvements recently so it should work with the current LTS version. It should also work with multiple skinned meshes, I think the character and clothing in the demo / video are also separate skinned meshes.

It should work with any skinned characters including daz3d. A few things to watch out for with daz3d characters.
1: They appear to be very high poly, this could be a performance issue and will probably take a long time to pre-process.
2: I think the daz3d characters use a lot of shape morphs to fix geometry. As @Cold_Frank mentioned, this method doesn’t really work with morphs.

 

yup, as @CodeKiwi mentioned it does work with the latest LTS versions. my additions also make it possible to be used with every rendering pipeline, since i integrated it with shadergraph. the repo has built in shaders using shader graph - https://github.com/Milk-Drinker01/CoR-Skinning

It also does work with characters that have multiple skinned meshes. Some of my optimizations group the data thats sent to the gpu - it will do all of shirtA, then all of shirtB, skinA, skinB, etc, to send less data to the gpu. to take advantage of that, minimize the amount of variation you have in the meshes