SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

That was one of the more effortless tutorials because SALSA is so easy to use. Highly recommended for dialogue.

 

Hi, I have another feature request to add to SALSA - basic jaw-hinge with volume.

I suddenly have a very minor figure (3D cat) with a simple hinge-mouth and I'm wasting a few days looking for a ready solution that works in Unity5 just to get it to meow and maybe mouth a few lines.... Would be great to still be able to use the SALSA workflow (the Playmaker actions for instance).

I realize that is exactly what SALSA is NOT doing and there were several alternatives (free and pay) to do this, but most seem to be abandoned.... There is an old script in the AdventureCreator package, which oddly does not reset the jaw to it's neutral state when the volume drops).

In theory, can't figures use both a jaw hinge AND blendshapes for phonemes? Sorry if I am off base or the solution is just too obvious to include as a feature.... Since many of the solutions I am finding haven't been updated in a long time, maybe it could be added to SALSA as an option? (or maybe there is a ridiculously obvious solution someone could suggest?)

 

Hi wetcircuit!

Right, currently we don't have any jaw-bone helpers, primarily because of what you stated...this isn't what SALSA's goal is. In fact, it's exactly what we strive *not* to be. A jaw hinge would not give a good representation of mouth/lip movement. However, if you wanted to do something simple for your 3D cat, there are a couple of options:

1) Best option: you could create a single blend shape for the model and animate the single blend shape in SALSA -- you don't have to use 3 different shapes. We have a very old blog post that describes how to do this.

2) If light 3D modeling isn't an option, the other option is to use the blend shape values (exposed as of version 1.3.0) to drive your bone with a small script.

 

For example, you could do something like this (NOTE: this is only an example and is not intended to be used as final production code):

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. public class JawHelper : MonoBehaviour {
5.  
6.     public GameObject jawBone; // link your jawbone to this slot
7.     public float range = .25f; // simple range controller for the jawbone
8.     private CrazyMinnow.SALSA.Salsa3D salsa;
9.     private Vector3 boneZero;
10.  
11.     void Awake() {
12.         salsa = GetComponent<CrazyMinnow.SALSA.Salsa3D>();
13.  
14.         // grab the jawbone's initial rotation to use as a reset point
15.         boneZero = jawBone.transform.localEulerAngles;
16.     }
17.  
18.     void Update () {
19.         // if SALSA isTalking, then animate the jawbone
20.         if (salsa.isTalking) {
21.             float amount = salsa.sayAmount.saySmall * range;
22.             jawBone.transform.localEulerAngles = new Vector3(boneZero.x + amount,
23.                                                         jawBone.transform.localEulerAngles.y,
24.                                                         jawBone.transform.localEulerAngles.z);
25.         } else {
26.             // ensuring the jaw goes back to original rotation
27.             jawBone.transform.localEulerAngles = boneZero;
28.         }
29.     }
30. }
31.  

Attach that code to the same object as SALSA and then setup Salsa3D similarly to the following:

 

Wow! Once again I have to thank you for going above and beyond! I see how it must seem too simplistic (but honestly, when you don't know where to start everything is a mystery.

Thank You!

 

I have a character without face rig and without blend shapes.

Is it useable with SALSA ?

 

How does the mouth work?

Salsa3D works with blendshapes. Salsa2D works with 2D sprites.

 

Wetcircuit is right about needing BlendShapes or Sprites. SALSA will still process audio and output value changes, but without a BlendShape or Sprite to target, it unfortunately won't be very interesting to watch. See our online manuals for an overview of how the 2D and 3D systems work.

SALSA manual
http://crazyminnowstudio.com/projects/salsa-with-randomeyes-lipsync/manuals/salsa-manual/

RandomEyes manual
http://crazyminnowstudio.com/projects/salsa-with-randomeyes-lipsync/manuals/randomeyes-manual/

 

The character has only a head end bone.

 

SALSA won't work if there's nothing for it to interact with. If the model has mouth interior geometry already, then adding Blendshapes is relatively simple if you have some experience with Blender or another 3D application that supports BlendShapes. BlendShapes are a preferred approach because they are much easier to deal with than armature and weight painting.

We have a three-part video learning series on creating BlendShapes using Blender on our youtube channel.
https://www.youtube.com/playlist?list=PLcVmXGedVLuahpG6bRniRryIc1QBz11Nk

 

How good works SALSA with the mixamo blend shapes from rigged fuse characters ?

 

Please check out our blog post concerning FUSE.

 

Hi there,

I just discovered your software, very nice indeed!
A question : do you think is possible to integrate your lipsync with the expressions obtained by the Mixamo Face Plus?
I think will have amazing results!
thanks in advance

 

Yes, add the possibility to add mixamo fuse facial expressions to the lipsync data, like smile, angry and any other ...

 

Hello micuccio & Frogger007,

Thanks for your support, we're glad your enjoying the product so far. There are currently a couple of ways to drive facial expression groups (combining multiple BlendShapes at varying values to create expressions). If you're using our API, you can easily create your own shape group functions by combining multiple calls to the exposed Custom Shapes functions. See below. You might also want to use our event system to help determine when these shape group functions are called. Alternatively, if you use Playmaker and our free Playmaker add-on, you can use our included RandomEyes3dExpression custom action that mimics the same behavior in a Playmaker action.

Free Playmaker add-on
http://crazyminnowstudio.com/posts/playmaker-actions-for-salsa-with-randomeyes/

RandomEyes functions
http://crazyminnowstudio.com/projec...randomeyes-manual/classes-methods-properties/

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using CrazyMinnow.SALSA;
4.  
5. public class ShapeGroup : MonoBehaviour
6. {
7.     public RandomEyes3D rEyes;
8.  
9.     void Start ()
10.     {
11.         if (!rEyes) rEyes = GetComponent<RandomEyes3D>();
12.  
13.         if (rEyes)
14.         {
15.             Smile();
16.         }
17.     }
18.  
19.     public void Smile()
20.     {
21.         rEyes.SetCustomShapeOverride("Facial_Blends.BrowsUp_Left", 1.5f, 60f, 1.5f);
22.         rEyes.SetCustomShapeOverride("Facial_Blends.BrowsUp_Right", 1.5f, 60f, 1.5f);
23.         rEyes.SetCustomShapeOverride("Facial_Blends.EyesWide_Left", 1.5f, 60f, 1.5f);
24.         rEyes.SetCustomShapeOverride("Facial_Blends.EyesWide_Right", 1.5f, 60f, 1.5f);
25.         rEyes.SetCustomShapeOverride("Facial_Blends.Smile_Left", 1.5f, 60f, 1.5f);
26.         rEyes.SetCustomShapeOverride("Facial_Blends.Smile_Right", 1.5f, 60f, 1.5f);
27.     }
28. }

 

Thanks a lot for the reply.
Just a clarification:
As far I understand Salsa will work in real time moving the lips, while the expressions made with Mixamo Face Plus are just animations that can be called via Mecanim.

My idea was, for example using Playmaker, to "run" salsa for the lips and "blend" on top via Mecanim thé animation generated with Faceplus.

Do you imagine this possible?

Thanks a lot,

 

Actually, you can use RandomEyes to facilitate the implementation of any other blendshapes on your models. RandomEyes can randomly call the shapes or you can programmatically access them via the API. And likewise, you can use Playmaker with RandomEyes as well. RandomEyes is included with SALSA.

RandomEyes Manual

Additionally, to your question, blendshape animations will mix along with Mecanim rig animations. So using SALSA and/or RandomEyes will not hinder Mecanim bone animations.

Hope this helps!

 

Hi..., I am having an issue with the FUSE Smile_left and Smile_right blendshapes in RandomEyes... Frown works, the brow morphs work, but not Smile.

On a hunch, I removed the two smile morphs from the SayMedium setting in the Fuse_Sync script, and they began to work again.... so if a blendshape is assigned by the fuse_sync it is locked out of RandomEyes even when that mouth shape is not being applied.... Is there a workaround? or hopefully the settings can be additive rather than exclusive?

 

Hello!
Yes, they are exclusionary to eliminate fighting against each other. Since it would be the same shape, there wouldn't be any additive blending, just canceling out or maxing out. The way we currently see it, there are a couple of options, neither are probably all that appealing to you, but here goes...

First, you could pick different shapes for the SayXYZ groups. This is probably the easiest way to get the end-result that you're looking for. Of course, assuming a good blend can be obtained with different shapes.

Second option, assuming you have a viable 3D editing solution, you could simply duplicate (easiest) the shape or create a new shape (harder). The new shape could be used for either the SayXYZ group or the RandomEyes CustomShapes slot and would be additive within the blends.

 

Well, if I wanted to go that route I wouldn't be needing the sync script. lol
Thanks for the info.... I will work around it.

err. I see those blendshapes are also blocked from Expression Actions.... hmmm, I guess I can enable the Fuse_Sync only when the character speaks... It makes sense. Blending an open mouth shape with an open mouth shape would never look right for speaking.... I will also simplify the mouth settings (looks like smile is the only shapes I'd want for expressions). It is still a great time saver!

Thank you!

 

Hehe...right! Well, I said you probably wouldn't find them appealing. Sorry 'bout all that. If we think of another solution, you'll be the first to know!

 

Your asset looks very useful but I'd like to know if the lipsync feature works with Master Audio asset and it's playlist feature which uses 2 audio sources. Any experience on that?
Thanks!

 

Hello ilesonen! Thanks for the inquiry!

The short answer is YES! -- MasterAudio's playlist feature can be used to drive SALSA 2D & 3D lip-sync input.

Disclaimer: Each of us minnows owns a copy of MasterAudio; however, we haven't much experience with using it yet, so this integration method may or may not be the preferred way to do this. I used a small script to link up the playlist's current clip to the SALSA input. (NOTE: I didn't find a master AudioClip to attach to, so I updated the clips with MA's event system, subscribing to the PlaylistController.SongChanged event and discarding the string clipName parameter).

There may be a better/easier/more-efficient way to do this, I'll get with Dark Tonic and see what they recommend.

I'm putting up a little video (shortly) that demonstrates how I put this together.

 

We've posted up a short video showing how we put the integration between SALSA and Master Audio together with a helper-script.

 

It's official!

Watch the video, download the helper files, and check out the blog post for our new SALSA / Master Audio workflow.
http://crazyminnowstudio.com/posts/using-salsa-lip-sync-with-master-audio-playlists/

Not familiar with Master Audio? Check it out here

 

Thats awesome! Going to buy it today
Thank you!

 

2 quick questions:

1 - Salsa & Unity 5 compatibility - I read that Salsa 1.3.0 is Unity 5 compatible (fantastic!). Does all the functionality that Salsa had until now also work in Unity 5 or are there advantages to staying with an earlier vers. of Unity and using Salsa?

2 - Is Salsa the same, better or different than using Face Plus animation from Mixamo?

thank you,
cn

 

Hello Cloud Ninja!

1. All of SALSA's feature set is the same in Unity 5 as it was in Unity 4. The new WebGL preview functionality in Unity 5 does not currently work with SALSA due to WebGL's limited audio API support. However, the legacy web-player still exists in Unity 5 and therefore so does SALSA's web functionality. There are no advantages that we can think of to using Unity 4 over 5 as far as SALSA is concerned.

2. While the products share some similarities, they are really quite different. SALSA with RandomEyes is less about live performance capture and more about providing a game-time enhancement of your characters, giving them a good amount of life without a lot of extra work. While we do have workflows for using FUSE models, the product can also be used with your own models (2D & 3D) and also with Autodesk Character Generator models and potentially with other model sources as well, like DAZ.

Whether it's better or worse or different is really a matter of opinion of who's using it and for what. There are quite a few user testimonials here and in our product reviews section that might provide you with that sort of opinion. Our product does work in Unity 5 and Face Plus currently does not (according to their site information). So that could be a determining factor in and of itself. We've got a blog post describing exactly what SALSA with RandomEyes goal is, check it out here: http://crazyminnowstudio.com/posts/history-of-salsa-with-randomeyes/ If you're looking for a solution to give your characters some life by driving easy, automated easy lip synching, eye movement, and even other facial expressions, SALSA with RandomEyes might be your cup of tea!

Please do check out our documentation and videos to see if SALSA with RandomEyes is right for your project and hit us up if you have any other questions. Hope we've answered the ones you've posted here!

Our product page has links to videos, manuals, and other information that might be of interest.
http://crazyminnowstudio.com/projects/salsa-with-randomeyes-lipsync/

 

Using Blendshapes with 2D Characters for Lip-sync



Our SALSA lip sync asset offers work-flows for 2D and 3D characters. Our normal 2D approach uses standard sprite swapping that looks great, but in certain scenarios, smooth blending between mouth shapes really brings a lot to the animation. Some 2D only solutions do this by applying a sprite to an underlying mesh, then moving verities in the underlying mesh to deform the sprite. This essentially works the same way as a BlendShape, shape key, or morph target in 3D characters. So what's stopping anyone from using Unity's SkinnedMeshRenderer component, that supports BlendShapes, on 2D characters? Nothing...it works great! We explain our work-flow and demonstrate side-by-side results in our latest blog post.

http://crazyminnowstudio.com/posts/using-3d-blendshape-assets-with-2d-characters-for-lip-sync/

 

Hello Crazy Minnows!

Your new integration scripts with Master Audio are great but I have some issues in my setup.

The trouble seems to be related of having several playlists. The script works fine in one playlist which have Salsa component but when I have other playlists playing at the same time it doesn't work properly anymore. It sounds like it plays MA audio source and Salsa audio source at the same time but they are out of sync.

My playlist's use Crossfade Time which is set to zero and playlists use synchronize mode.
Maybe I'm doing something wrong? Any idea what is causing this?

Thanks!

 

Hey ilesonen! Glad you're enjoying SALSA! Lemme see what I can dig up on that.

Please check your pm mailbox when you get a chance. Need to get a few details in the specifics of what you're trying to do. Thanks!!!

 

We're proud to announce another brand new workflow for SALSA...SALSA with RandomEyes for DAZ3D characters!

This all new work-flow radically simplifies the process of linking SALSA with RandomEyes to Daz3D Characters. There's no need to modify these beautiful models to add SALSA or RandomEyes specific BlendShapes. Instead, we leveraged new features in 1.3.0 to create our new CM_DazSync script, along with a custom inspector, that provides a unique shape group capability that creates a great default setup, while also extending full shape group customization.

Read about the details, check out a sample video, and download the helper files from our latest blog post.

http://crazyminnowstudio.com/posts/using-daz3d-characters-with-salsa-with-randomeyes/

 

Discovery: it appears that Master Audio's playlist synchronization functionality, when used with the workflow implementation we have provided, introduces timing problems. There are a couple of solutions for this.

First, use our new script (v1.0.1) available on our website. There is new functionality, allowing the Master Audio playlist to be muted. This will prevent the audible overlapping delay effect if the timing is out of sync. This is recommended at all times to prevent both AudioSource components used by Master Audio and SALSA from audibly playing the same clip.

Second, we recommend not using the playlist synchronization feature Master Audio implements on the playlist that is feeding the SALSA lip-sync component. Both of these caveats are posted on our devblog.

EDIT: playlist synchronization has been fixed and is available in the latest helper script (v1.0.2).

 

Master Audio Playlist Synchronization Mode followup: We've added new functionality to the Master Audio workflow script. In the previous post, we recommended against using Master Audio's playlist synchronization feature when using it with SALSA. We have now updated the workflow script (v1.0.2) to properly handle the synchronization functionality. The helper-script, by default, auto-detects synchronization and allows SALSA to work appropriately with the feature. The Master Audio playlist is also muted, by default, to ensure there is no playback ghosting between it and SALSA.



Please check out our updated blog post for more information and to download the files.

 

Hi, I just got Fuse recently and remembered picking up Salsa a while ago and tried it out. Super simple for a noob like me and amazing results! I saw the Master Audio support but your download link is dead. Salsa has me excited to try out those 3rd party assets you support that I already bought but haven't had the time to learn to use.

 

MIK3K,

It's great to hear you're having fun with SALSA. Thanks for pointing out the link, it's fixed now.

http://crazyminnowstudio.com/posts/using-salsa-lip-sync-with-master-audio-playlists/

Mike

 

I'm try to follow the video "Using SALSA with RandomEyes and Mixamo Fuse Character Creator" but i can't find the blender script. I donwloaded "SALSA_3rdPartySupport_FuseCharacterCreator.unitypackage" but the script is not in that asset i'm also looking for the script in http://crazyminnowstudio.com/posts/using-mixamo-fuse-character-creator-with-salsa-with-randomeyes/ but maybe i'm blind.

Someone please tell me the link???

 

Hi b4c5p4c3,

The python script, and that youtube video are part of our previous Fuse workflow, and are no longer required. The new workflow uses the BlendShapes already included in the Fuse characters, so no SALSA specific shapes are needed. Instead, the included CM_FuseSync.cs script builds SALSA BlendShapes by grouping multiple Fuse BlendShapes into shape groups that approximate the SALSA BlendShapes.

Please let us know if you have additional questions,

Mike

Here is the updated video tutorial.

 

Thanks! I was having a problem with the character's eyes not moving or blinking but the facial blends were working okay. I have a playmaker 3rd person controller from an animation pack on the asset store. I tried it on a different character without the controller and Salsa works perfectly. Just mentioning this in case anyone else runs into something similar. I think that controller is also making my Unity Cloth jitter like crazy when I press any controls but it's fine when just idling or quickly tapping a control, so will have to dig into the controller to figure out why.

Can't say enough great things about Salsa. It makes a huge difference in the characters, they really seem alive, and the 3rd person integration has made it so simple to use with other assets.

 

MIK3K,

Thanks for the kind words, and for the review!

One thing you can check if you have eye movement and blink issues with the Fuse workflow, and if you're using two instances of RandomEyes (one for eyes, and the other for custom shapes), make sure that the eye control instance of RandomEyes is linked to the CM_FuseSync.cs RandomEyes3D public property, and not the custom shapes only instance.

Mike

 

Was it really as simple as just unchecking "Sync Custom Shapes" on the CM_Fuse_Sync script? I deleted that component and then added it back but left that box unchecked and it seems to be working fine, but my level of scripting, game development is very low - just started learning this stuff about 6 months ago in my spare time.

 

If you're using one instance of RandomEyes for custom shapes, you should leave this checked. It sounds like the custom shapes instance was linked to the CM_FuseSync script, and when you removed that instance, the CM_FuseSync script auto-linked to the eye control instance (the correct instance). I believe this is what solved the eye movement, and I think you should now be able to re-enable "Sync Custom Shapes".

 

Thank you for the help and explanation. Just read today that Mixamo was acquired by Adobe. Hope this workflow stays similar because it's so easy and creates great results.

 

I was reading about the Adobe acquisition also, it will be interesting to see what they do with it.

 

SALSA real-time microphone lip sync (UPDATED)

If you're currently using our free real-time microphone add-on script for SALSA, or interested in reading about it, head over to our website to grab the latest version updated today.

Read the instructions and download the update
http://crazyminnowstudio.com/posts/real-time-microphone-input-for-salsa-lip-sync/

Check out the Video tutorial

 

First off - great lipsync solution!! I'm very impressed with the results and with the ease of audio-analyzing automated lipsync.

However I am having a very odd issue with SALSA in that it worked perfectly on the first two Fuse characters I tried it on (using the Fuse integration script and following video tutorial)... yet on the third character it is not working. The audio will play (using PlayMaker SetClip & Play actions for SALSA) but there is zero mouth movement. This same setup worked on the other two characters.

What is weird is that RandomEyes IS working on that third character. So it would appear that the Fuse integration script worked... somewhat, and I know the model has BlendShapes as I checked the MeshRenderer. I see no difference between it and the first two characters.

The only distinction I can think of is that the first two characters were male & female humans and the third I'm struggling with is an Orc which I believe is part of the Brute DLC pack. However as mentioned the BlendShapes seemed to be named identically.

I've tried adjusting the sayTrigger values, update delay, blend speed and such on the SALSA 3D component of the third character - but to no avail. I'm fairly stumped at this point... any suggestions?

 

Hi Dralger,

Thanks for your support, we're glad you like the product. I'd like to start by verifying a couple of settings on this Fuse character setup. Can you check the Salsa3D component and verify that the [Skinned Mesh Renderer] field is not populated? Also, please verify that the [Salsa3D] field on the CM_FuseSync script is populated with the Salsa3D component (you can drag the Salsa3D component into this field to be sure). Can you post screenshots of your CM_FuseSync and Salsa3D inspectors? If everything looks good and it's still not working, can you send us a copy of your model so we can test it? You can send it to assetsupport@crazyminnow.com.

Thanks,
Mike

 

Appreciate the surprisingly quick response there Mike. Everything looks as it should so I'm going to send you the model as suggested and a few screenshots of my settings panels.

 

Dralger,

Thanks for finding the bug we discussed via email, and for testing our 1.3.2 patch update. If your tests and ours demonstrate the issue being resolved, we'll push the update to the asset store.

Mike

 

We're the 24hr SALE today! 44% off!

https://www.assetstore.unity3d.com/en/#!/content/16944