Is there some method to add blendshape in editor?

Hi, developers,

Do you know if there's some method to add blendshape by script?

For now, I find that the only routine to add blendshape is to use external softwares and let the unity model importer to make blendshape when imported.

I really hope there's some API (or hidden API) to let us, for example, add a deformed Mesh as new blendshape.

 

Hi TMPxyz,

We are working on the API at the moment, it's not yet finish so it won't be part of 5.0 for sure.

 

Thanks for the reply,

I'll see if I can find some workaround then.

 

@TMPxyz what's the complication you're experiencing with the established process used for blend shapes, beyond having to rely on a 3D modeling package to perform the morphs/blend shapes?

 

Hi, theANMATOR2b,

Nope, I don't mean that I have trouble with current blendshape workflow,

I'm just trying to make a plugin to add blendshape in unity editor, and I took a short investigation and found no API to add blendshape in editor script. So I'm considering to take a workaround for now.

 

You can't really add a blend-shape in a Unity script. But you can alter existing blend-shapes through a Unity script. At the moment, it's just a given that you create blend-shapes in whatever 3D program you are using, and then use them inside of Unity.

 

Hi, RichardKain,

Do you mean that we can use scripts to alter the blend-shape set in modeling software? I didn't find related APIs but SetBlenderShapeWeight().
Could you shed some more light on that? Thanks

 

No, I merely meant that the value of an individual blend-shape can be accessed and altered in Unity through the Skinned Mesh Renderer component. For creating blend shapes themselves, you would have to create and alter the blend shape in a program such as Blender, 3D Studio Max, Maya, etc...

While I can understand the desire to alter a model through script, most blend-shape applications are faster and easier to edit in an external 3D program. There are often thousands of vertices that would need to be shifted around for even a basic blend shape. (depending on the complexity of the model) There are also no generalized approaches to applying such transformations through a script. Any such script would have to be customized for every model, as the polygon density and distribution of any model will vary wildly from any other model.

Once you have your blend-shape created in an external 3D program, the values for those shapes are readily available through scripting in Unity. Just don't expect to be using scripts to alter the model itself.

 

Thanks for the detailed explanation.

 

If you create a blend-shape in an external 3D program, it will show up naturally when the model gets imported into Unity. I did this with my own models and Blender. When Unity imported the .blend file, it would automatically detect any blend-shapes that I had added to the model.

Those blend shapes are visible as a list in the Skinned Mesh Renderer component. (the default rendering component for any imported model that has an armature or blend shapes) Each of the blend-shapes in that list has a value from 0.0 to 100.0. Adjusting that slider will "animate" the blend-shape from it's least effect to it's greatest effect. For this reason, it is a good idea to exaggerate your blend-shapes a little bit when you are creating them in your 3D program.

These 0 - 100 values can be easily accessed and manipulated through scripting by accessing the Skinned Mesh Renderer component. Using this method, it is possible to adjust these values in a more dynamic fashion, and allow the animation of your blend-shapes to react to different events happening in the game. (instead of just having static, pre-defined animations) One common and extremely useful blend-shape fueled effect is to adjust your character's "breathing." By creating a blend shape of your character's chest and stomach expanding outward, you can make your character appear to be breathing by adjusting the slider for that blend-shape. If your character is running around a lot, you can increase the level to which that slider is extended, as well as the speed at which it changes, to make it seem like your character is breathing heavier or faster.

 

Can we have a preview version of the API, similar to WebGL, with warning that the API is subjected to change, just to get it out there so the developers on the asset store can have fun with it?

Right now I have to write my own vertex morphing scripts in LateUpdate that took the vertices array out of the mesh, accumulate the morph in a cache-line unsavvy way, then put them back into the mesh, that's a huge chuck of memory between mono and the native code.

I was thinking of writing a fbx exporter to export the runtime generated mesh and blendshape to a temp fbx then ask unity to import it back in, but that only work in editor, not the runtime, so I am stucked.

 

Hi sunnyDavid,

It not yet ready to be shipped, but I can give you a preview of the scripting API.
We did extend the mesh class to add blend shape support like this. And of course this all available at Runtime too.

Code (CSharp):

1.  
2. public struct BlendShapeFrame
3. {
4.     public Vector3[] vertices;
5.     public Vector3[] normals;
6.     public Vector3[] tangents;
7.  
8.     private float m_Weight;
9.  
10.     public float weight { get { return m_Weight; } set { m_Weight = value; } }
11. }
12.  
13. public struct BlendShape
14. {
15.     protected string m_Name;
16.  
17.     public BlendShapeFrame[] frames;
18.  
19.     public string name { get { return m_Name; } set { m_Name = value; } }
20. }
21.  
22. public class Mesh
23. {
24.     public BlendShape[] blendShapes;
25. }
26.  

 

That would be great! Oh, does each BlendShapeFrame contains the full list of vertices? you know sometimes only a small portion of the mesh need to be morphed.

And are the normals and tangents optional? Some stylized shaders doesn't need all of them.

 

Yes you need to define the full list of vertices

yes they are optionnal

 

Ok great! Can't wait for it to be released!!!

My only suggestion is that maybe the API allow us to get and set each BlendShape one by one, instead of a huge array of BlendShapes. In that case we can reuse the same vertices array. If I want to set 50 BlendShapes, instead of allocating 50 X 10,000 verts X Vector3 of memory, I can just reuse the same vertices array 50 times.

 

From the Unity 5.0 feature page if you missed it here.

 

Correct me if I misunderstand the data structures here.

It looks that ONE blendshape is consists of multiple BlendshapeFrame, is it like the multiple Keyframes in an AnimationCurve?

But I cannot see there's time / inTangent / outTangent attribute in the BlendshapeFrame structure, so how do you define the time and curve for them?
Or maybe you had just omitted these attributes to make it look simpler?

 

You can specify the blend weight for each BlendShapeFrame to get more control over how the blending occur.
So for each blend shape you can define intermediate BlendShapeFrame, it like adding extra keyframe to shape your animation curve.

 

Oh, I think I might get what you mean.

So we still use the AnimationCurve to tweak a float value ranging between [0, Count_of_BlendShapeFrame], which controls the interpolation between two BlendshapeFrames.

And the time and tangents are defined on the KeyFrame of AnimationCurve.

That's a great idea. I'll looking forward to the new APIs.

 

There is no animation curve involved in the process.

You define a set a blend shape[smile, sad, angry] and for each shape you can define as many BlendshapeFrame as you want. For most case you will have only one BlendShapeFrame

But let say that for shape smile you don't like the way Unity interpolate your shape at 50% because some vertex become too visible so you can add an intermediate BlendShapeFrame at 50%. Between 0-50% Unity will blend the default shape with BlendShapeFrame.weight = 50, and between 50%-100% Unity will blend blendshapeframe 50% and 100% together.

If there is only one blendshapeframe the weight is assume to be 100%

I hope it's clearer

 

The blend shape works by lerping the position of each vertex between the original and the morph target, the vertices move linearly. If you want to rotate a chuck of vertices around a pivot, you need to insert several in-between blend shape frames to make them move like rotating. If linear movement is all you need, one frame is enough.

 

Yeah, I tried to use several shape keys to simulate the rotation, and it seems to need more than 2 shapes in the sequence to rotate 135 degrees naturally, that might be many for longer rotation.

I wonder if there's other methods, maybe simulation of the deforming process?

 

Hey TMPxyz - thats a pretty extreme morph / blend shape.
Possibly a better option for that is a scaled non-uniformed bone chain which is listed as one of the improvements (nonuniform scale) in U5.
Or possibly a dummy/null node path constrained to a spline and animated along the curved spline. This dummy/null would be a skin weight control which makes the spiky part extrude like in the photo above.

Unless the image is just an example of an extreme case (testing to see how far we can push the morph/blend shape) api - in which case I think you may have identified the limit of linear blending.
Animation curves would be an awesome way to control morphs, but (in Max) I have never not been able to hit the morph timing needed with a similar process as Unity has created for morphing.

 

If you can completely forget about the normals and just consider the vertex position, then insert a dynamic bone at the rotation pivot would be the simplist way to go for the above example. And if this is not a generic case, the bone can even exist only inside a shader, by dynamically modifying vertex color to pass in as the bone weights, and normals can also be recalculated in shader. But you are obviously talking about a much more generic case, then blend shape would be the better choice, since it's less likely to conflict with whatever the users already have.

Of course if modifying the mesh in C# each and every frame is acceptable, then the options won't be so restricted.

 

Just thinking out loud.. To save on memory overhead would it be possible to compare the original mesh to the target blend shape mesh and only store the vertices that move to a 2D array that contains the vertex number and the vertex target position. This way you only need to iterate through the vertices that are moving and ignoring the rest.

The process would be something like.
1.Import blend shape.
2.Bake blend shape to new vertex array
3.Discard imported blend shape mesh.

 

@Mecanim.Dev - Is there any news on the release of the Blendshape API? It'd really help bridge the gap between runtime randomized character creation and lip-sync/emotions/muscle-deformation. I'm not suggesting rushing it out if it's not ready - I'm just curious about when it might be ready or if it has been put on hold.

 

@Mecanim.Dev Hi, talking about blendshapes and new API, will there be negative blandshape option? ..or is there a way now to allow this. Example is when we have a regular animation with negative blandshape values, they just get clamped to 0. I would presume that it is only one line check in the shader? is there a way that we can modify this ourselves now or some other way we could make this work?

Thank you very much!

 

I'm pretty sure that blendshapes aren't shader-based. From the API example above, I think it's safe to assume that negative values will be supported. For doing negative blendshape values right now, you can set the blendshapes to their minimum value in modeling software, and keep the blendshape values of the SkinnedMeshRenderer at their zeroed point (like 50%) when not in use.

 

Yeah - just to expand on IFL's comment or maybe give an alternative - in Max - set the morph range from -100 to 100. Set the morph to -100, clone the mesh as a single frame snapshot then add it to the morph modifier. It's a separate morph/blend shape, but it's there and will be available in Unity. Good for setting a frown morph from a smile morph.

 

Spoiler

@Mecanim.Dev

...bump... ... still wondering about the API.

 

Hi IFL,

No it not on hold. We are still working on it but some user did raise some serious issue with memory allocation and GC with the proposed API so we are currently looking at how we could change the API to remove some pressure on the GC

 

@Firewalker Negative weight is not supported. Create your equivalent negative blend shape in your authoring tools.

 

About the API without any allocation, do you need to add new blend shape or you simply want to modify existing blend shape?

 

@sunnydavis
So after some investigation here what we are proposing

To add, remove blend shape at runtime you still need to use Mesh.blendShapes; which does some allocation.
To modify existing blend shape frame we will add a new API function that doesn't do any allocation.

Would this match your needs?

 

My project could really use the ability to add blend shapes at runtime so I'm excited to see this thread!

Thanks

 

I do understand this. What I was wondering is there a possibility it will be enabled in future versions? and will it be a hard issue for you guys to implement? if lets say you decide it would be a good option.

..and I can give you very serious reasoning why this would be a great option for Unity team. (only I cannot disclose it publicly yet)

 

Hello!

I'm really excited to see that you are working on this--adding blendshapes at runtime is a great and useful feature!
(Especially since it's kinda hard to find a workaround for it...)
Can you make an estimate when adding blendshapes will be released?

- KW

 

There is no ETA at the moment, but the new API is currently reviewed so we are close to add this.

 

@Mecanim.Dev - Would it be horribly annoying if someone was curious about the possibility of the blendshape API being included in v5.2, even though it's obviously not in the 5.2 release notes...?
I'm just asking... for one of my friends... who is really excited about the possibilities of such a feature.

 

Unfortunately it won't be for 5.2, we just finish our work on blendshape API and 5.2 is already in RC, Your friend will have to wait a little bit more.

 

Oh by the way the interface did change a little bit from what we did propose
Our first implementation was using property accessor

Code (CSharp):

1. public BlendShape[] blendShapes;

But after a few iteration and based on your feedback we did change it for an allocation free API which give better performance result.

 

@Mecanim.Dev I am having some problems with the export/import process. From what I've seen, it's simple, and there's not much I can do wrong. However, somewhere along the line, the blend shapes have been lost.

I'm using a humanoid rig, and I've given it a big smile using a blend shape. I can see the big smile in the importer animation window, but I can't see it in game.

I've used skinnedMesh.blendShapeCount to output the number of blend shapes. I get 0. Any clues?

 

It turns out that we're using Morph-o-Matic to create our blendshapes, and that this is what is causing our problem. Does anyone have any experience with this? Is there a way around this without using Mega-Fiers?


Edit and Update: blendshapes were added to the mesh when using Max's Morph Targets, they were visible and modifiable. There were no keyframes attached to them, so they did not animate except when code manipulated them.

 

Hello there!

@Mecanim.Dev - How is the Blendshape-API going?
I'd really love to use this in my Character-Creation-System--do you have an idea when it's going to be released?

Best Regards,
Kristina Waldt

 

@KWaldt - It's in the 5.3 beta.

 

so how do i use the new BlendShape api?

can't see it in 5.3 or 5.4

 

I understand how to combine vector array results from multiple shapes to create new combined shapes, but is there no way to create new shapes from partial derivatives of other shapes? For example: 10% wide eyes, 20% smile, 15% brows lift to make a new smile blend shapes that scales the 10%, 20%, and 15% to a 0-100 scale?

Thanks,
Michael

 

@AVOlight : everything can be found in the Mesh class
http://docs.unity3d.com/ScriptReference/Mesh.html
take a look at those methods:
AddBlendShapeFrame Adds a new blend shape frame.
ClearBlendShapes Clears all blend shapes from Mesh.
GetBlendShapeFrameCount Returns the frame count for a blend shape.
GetBlendShapeFrameVertices Retreives deltaVertices, deltaNormals and deltaTangents of a blend shape frame.
GetBlendShapeFrameWeight Returns the weight of a blend shape frame.
GetBlendShapeIndex Returns index of BlendShape by given name.
GetBlendShapeName Returns name of BlendShape by given index.

 

@Crazy-Minnow-Studio: there is no built in function that does what you want, but you could write a script that do it, simply set all the blend shape at the right value: 10% wide eyes, 20% smile, 15% brows lift and then read the mesh vertices, normals and tangents and create a new blend shape with those values.

 

Thanks for the quick response. Can you elaborate on the method you're referring to to "read the mesh verticies, normals and tangents"? In my testing, I first tried to do this with Mesh.GetBlendShapeFrameVertices on single frame shapes, but this only returns the 100% position of a blend shape's vertices. Next I tried setting a blend shape to 20%, then passing the Mesh.vertices array directly to Mesh.AddBlendShapeFrame, but this creates a shape that scales the entire model larger.

On another project I was trying to parent to a vertex position in code and found the Mesh.vertices positions don't seem to be updated to account for blend shapes. I'm not sure if this is by design, a bug, or I simply missed something.

Thanks,
Michael