SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

A quick demonstration of SALSA 1-click lipsync, eye movement, and facial animation on Autodesk Character Generator, Fuse Character Creator, Daz3D, and MCS (Morph 3D) characters, using recorded audio, live microphone input, and runtime text-to-speech using the RT-Voice asset from Crosstales.

http://crazyminnowstudio.com/posts/salsa-lipsync-simple-flexible-quality/

 

Experimental: SALSA & UMA 2

This is an experimental demonstration of SALSA lipsync on an UMA 2 character. This is our first pass to add support, but we hope to offer official support soon. We're excited to add UMA support to our growing list of supported character generation systems (Fuse Character Creator, Autodesk Character Generator, Daz3D, MCS by Morph 3D).

 

Hello,

Just bought it. Cool stuff

Is there a demo or a script for UMA 2 yet?

I have a working Scene with a UMA 2 troll in it, he should read some News.

 

Hi Firlefanz73,

Thanks for your support! If you'd like to participate in our UMA 2 beta add-on, please email your invoice number to assetsupport@crazyminnow.com, and include a gmail address if you have one. We'll use the gmail address to share the add-on over gdrive once it's ready. We're hoping to begin private beta testing in the next week or two.

Mike

 

Still got no mail. The promised UMA 2 Support would really be helpful
Please send a mail or a PM, thanks and have a nice sunday!

 

Hi Firelefanz73,

We haven't forgotten about you. Thanks for sending your email and invoice, and your willingness to participate in our closed beta. You will receive an email containing the beta add-on as soon as it's ready. We're finalizing documentation and our internal testing of the add-on. It will be ready soon.

Mike

 

This experiment demonstrates SALSA lipsync using our live microphone workflow, and our runtime text-to-speech workflow (using RT-Voice), with our upcoming support for UMA 2 characters.

 

Hello,

any idea when the new Version with UMA Support will be published or at least the beta will be released?

I bought it a few days ago because I thought it was already part of it, and since I am using UMA 2, now I cannot do anything with it beside watching the demo ;-)

Thanks a lot,
Firlefanz

 

Sent you an email.

 

We're excited to announce the public release of our UMA 2 workflow for SALSA. This is a brand new workflow, so we look forward to your feedback.

Check the link to read the details of our implementation or to download the helper package.
http://crazyminnowstudio.com/posts/uma-lipsync-using-salsa-with-randomeyes/

A quick video tutorial with annotations to reflect changes since the first private beta.

 

Hello,

is the latest Version now on the store newer than the preview beta?

Thanks a lot for the info

 

Hi Firlefanz73,

The beta 2 that you helped us improve became the 1.0 release version available in the downloads section on our site. From now on, point releases will be reflected in the file name as our other workflows are. The date stamp on the workflow page will be updated, along with release notes for future updates.

Workflow and download link
http://crazyminnowstudio.com/posts/uma-lipsync-using-salsa-with-randomeyes/

Date stamp example from the workflow page
10/18/2015 - Requires SALSA with RandomEyes 1.3.3 or above. Public release available now!

 

Hi!

Quick question. I got my (uma) troll and my goblin both talking now.

How do I load a second Speed SFX and make one of them say it from script (for a special timepoint I already have)?

Thanks and have a nice Weekend

PS: Found something in the manual: Salsa3D.Play(), Salsa3D.SetAudioClip(AudioClip)...
I'll try with those, thanks!

 

Hi,

after I added Salsa3D to my two UMA meshes they didn't move their mouth anymore.

I think I do not need Salsa3D here, the CM_UMASync Script already has all the mouse stuff.

I have two UMA with UMA_Sync and UMA_Basic in my Scene. Both working when I give them one speech sfx only.

I want to start speech from the script. What Namespace and class do I have to add, to use SetAudioClip and Play with my UMA beings?

Thanks a lot,

Firlefanz

 

Hi,

The CM_UmaSync script is already adding Salsa3D to your character and exposing a reference to it in the Salsa3D field. You can use this reference to set audio clips, play, or access any of the other public methods available. The easiest way to control when speech happens is using a dialogue system like Dialogue System for Unity, or our free Playmaker addon. But you could also do this from code using a Coroutine to add a delay before Play(), then use the Salsa broadcast system to detect the end of one clip and start the next. See the example scenes in the SALSA with RandomEyes asset for more information on this.

Code (CSharp):

1. Salsa3D.SetuAudioClip(AudioClip);
2. Salsa3D.Play();

For the list of public methods, see the [Salsa3D] section in the link below
http://crazyminnowstudio.com/projec...uals/salsa-manual/classes-methods-properties/

For code examples, see the [C# (Salsa3D Syntax example only)] section in the link below.
http://crazyminnowstudio.com/projec...s-lipsync/manuals/salsa-manual/code-examples/

 

Hi,
Your CM_AutodeskSync script is an interesting solution.
But we have found that it is very, tragically CPU-expensive. Even one instance of it consumes more cpu resources then all other scripts of a big project in conjunction. And these other scripts are not simple.
It is totally unacceptable.
When do you plan to carry out work on its optimization?
Thanks

 

Hello Alex,

We are always working to expand platform support and improve existing processes. One of the challenges with the Autodesk Character Generator models is their inherently inefficient design. They use a separate mesh for body and teeth, and each has it's own long list of non-matching BlendShapes. This requires a lot of expensive processing to sync shapes between the body and teeth when using our RandomEyes3D custom shapes features to generate random facial expressions and twitches. If you can live without the random facial movement you will see a massive performance boost by commenting out the for loop at lines 141-154.

If this is a potential solution for you, we're happy to include a checkbox in the inspector to enable/disable this sync in the next update.

Please send your invoice number to assetsupport@crazyminnow.com for further assistance.

Michael

 

Alex,

We're also experimenting with a method to handle this body/teeth sync with a lot less validation in the process that should provide a good speed boost over the current BlendShape lookup method.

 

I also wanted to clarify, since I didn't make it very clear above, disabling the for loop doesn't stop facial animation/twitches, it introduces shape changes that may not be synced to the teeth correctly. It's debatable whether or not the precise synchronization approach we use in this loop is worth it since you could also just exclude any problematic shapes from the RandomEyes3D random selection list.

 

Autodesk Character Generator

We've published version 2.2 of our Autodesk 1-Click setup workflow. This update adds a [High Sync Accuracy] option to the CM_AutodeskSync inspector that is disabled by default for efficiency. Enabling this option will produce better synchronization between the body and teeth, but at a high CPU cost.

Grab the latest version here
http://crazyminnowstudio.com/posts/...er-generator-with-salsa-with-randomeyes-v1.2/

 

Hi, Michael,
I'm really impressed by so fast update.
Your solution for Autodesk Character Generator is getting better and better!
Version 2.2 is that we need.
Thanks!

 

Hello,

I deleted my own script that was mentioned to Play multiple Speech sfx from a timeline for a uma mesh.

Then I added to CM_UMASync:

public AudioClip[] audioClips;
public float[] startTimes;

These two do not appear in the Editor? Any idea why? I have no Compiler Errors.

I also tried to remove public AudioClip salsaClip; but got Errors in CM_Setup then.
But I don't care about this one. I Need my two Arrays in the Editor to attach multiple Speed to that script.

In the Update method it should then Play each Clip one time for the selected timepoint:

void Update () {
for (int i=0; i < audioClips.Length; i++)
if (Globals.time_startelapsed > startTimes && !isAudioPlayed)
{
salsa3D.SetAudioClip (audioClips [ i ]);
isAudioPlayed = true;
salsa3D.Play ();
}
Globals.time_startelapsed += Time.deltaTime;
}

Thanks a lot!

 

Hi

I would recommend not editing the CM_UmaSync script, but instead use another script to subscribe to the CM_UmaSync.OnSalsaCompleted event. This event passes the Salsa3D reference. You can then use this reference to perform any actions you wish to perform.

CM_UmaSync is a custom inspector, this is why your variables don't show up.

Code (CSharp):

1. using UnityEngine;
2. using CrazyMinnow.SALSA;
3. using CrazyMinnow.SALSA.UMA;
4.  
5. public class GetSalsaFromUMA : MonoBehaviour
6. {
7.     public Salsa3D salsa; // Variable to hold the Salsa3D reference
8.  
9.     /// <summary>
10.     /// Subscribe to the CM_UmaSync.OnSalsaCompleted event
11.     /// </summary>
12.     void OnEnable()
13.     {
14.         CM_UmaSync.OnSalsaCompleted += CM_UmaSync_OnSalsaCompleted;
15.     }
16.  
17.     /// <summary>
18.     /// Unsubscribe from the CM_UmaSync.OnSalsaCompleted event
19.     /// </summary>
20.     void OnDisable()
21.     {
22.         CM_UmaSync.OnSalsaCompleted -= CM_UmaSync_OnSalsaCompleted;
23.     }
24.  
25.     /// <summary>
26.     /// Unsubscribe from the CM_UmaSync.OnSalsaCompleted event
27.     /// </summary>
28.     void OnDestroy()
29.     {
30.         CM_UmaSync.OnSalsaCompleted -= CM_UmaSync_OnSalsaCompleted;
31.     }
32.  
33.     /// <summary>
34.     /// Grab the Salsa3D reference from the CM_UmaSync.OnSalsaCompleted event.
35.     /// </summary>
36.     /// <param name="salsa3D"></param>
37.     private void CM_UmaSync_OnSalsaCompleted(Salsa3D salsa3D)
38.     {
39.         salsa = salsa3D;
40.  
41.         // Do one-time actions here
42.     }
43.  
44.     void Update()
45.     {
46.         if (salsa) // Do not perform actions until we have reference
47.         {
48.             // Do reoccuring actions here
49.         }
50.     }
51. }

 

Thanks for that Piece of code

I now added this as a new script to each uma char in the Scene.
I added my code above to it and added some speech sfx.

I can hear what they are saying, but they do not move their mouth.
Before I had a "SalsaClip" fixed to each and then they moved their mouth.
Now the new sounds can be heared, but no mouth moving... any idea?

I can also send you the complete script if that helps and you can add it later to the sample if you like to.
Then each uma char can speak more than just one file

Thanks a lot

 

Please paste your complete code here so I can see what you're doing. Use the (Insert -> Code) block in the message toolbar so the code has proper formatting and is easier to read.

 

I got a Scene just like your demo Scene but with two uma chars in it.

Each one got this script attached and speech sfx I can hear but not see mouth movement.

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using CrazyMinnow.SALSA;  
4. using CrazyMinnow.SALSA.UMA;
5.  
6. public class TalkingBehaviourScript : MonoBehaviour
7. {
8.     public Salsa3D salsa; // Variable to hold the Salsa3D reference
9.     public AudioClip[] audioClips;
10.     public float[] startTimes;
11.  
12.     private bool[] isAudioPlayed;
13.     private float timeElapsed = 0;
14.  
15.     // Use this for initialization
16.     void Start ()
17.     {
18.         isAudioPlayed = new bool[audioClips.Length];  
19.     }
20.  
21.     /// <summary>
22.     /// Subscribe to the CM_UmaSync.OnSalsaCompleted event
23.     /// </summary>
24.     void OnEnable ()
25.     {
26.         CM_UmaSync.OnSalsaCompleted += CM_UmaSync_OnSalsaCompleted;
27.     }
28.  
29.     /// <summary>
30.     /// Unsubscribe from the CM_UmaSync.OnSalsaCompleted event
31.     /// </summary>
32.     void OnDisable ()
33.     {
34.         CM_UmaSync.OnSalsaCompleted -= CM_UmaSync_OnSalsaCompleted;
35.     }
36.            
37.     /// <summary>
38. /// Unsubscribe from the CM_UmaSync.OnSalsaCompleted event
39.     /// </summary>
40.     void OnDestroy ()
41.     {
42.         CM_UmaSync.OnSalsaCompleted -= CM_UmaSync_OnSalsaCompleted;
43.     }
44.  
45.     /// <summary>
46. /// Grab the Salsa3D reference from the CM_UmaSync.OnSalsaCompleted event.
47. /// </summary>
48. /// <param name="salsa3D"></param>
49.     private void CM_UmaSync_OnSalsaCompleted (Salsa3D salsa3D)
50.     {
51.         salsa = salsa3D;
52.         // Do one-time actions here
53.     }
54.  
55.     void Update ()
56.     {
57.         if (salsa) // Do not perform actions until we have reference
58.         {
59.             for (int i=0; i<audioClips.Length; i++)
60.             {
61.                 if (timeElapsed>startTimes[i] && !isAudioPlayed[i])
62.                 {
63.                     salsa.SetAudioClip(audioClips[i]);
64.                     isAudioPlayed[i]=true;
65.                     salsa.Play();
66.                 }
67.                 timeElapsed += Time.deltaTime;
68.             }                  
69.         }
70.     }
71. }
72.  
73.  
74.  

 

I tried your code and it works, there might be something else going on in your project.

 

Can you tell me more about your scene, how many UMA characters are in the scene? Is this one character speaking dialogue or a two-way conversation between characters?

 

Two of them. each has this script attached with different timepoints and Speed files.
And if you have a line of code handy how I can make them animate Standing like in the demo but looking at a target transform / camera that would be great ;-)

If you do not I can figure it out by myself, too Thanks!

 

Here is an updated version of your script. I fixed a bug I introduced when I first posted it. For adding mecanim animations/controllers, please reference the CM_UmaBasic script.

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using CrazyMinnow.SALSA;
4. using CrazyMinnow.SALSA.UMA;
5. public class TalkingBehaviourScript : MonoBehaviour
6. {
7.     public CM_UmaSync umaSync; // Local CM_UmaSync instance
8.     public Salsa3D salsa; // Variable to hold the Salsa3D reference
9.     public AudioClip[] audioClips;
10.     public float[] startTimes;
11.     private bool[] isAudioPlayed;
12.     private float timeElapsed = 0;
13.     // Use this for initialization
14.     void Start ()
15.     {
16.         isAudioPlayed = new bool[audioClips.Length];
17.         if (!umaSync) umaSync = GetComponent<CM_UmaSync>();
18.     }
19.     /// <summary>
20.     /// Subscribe to the CM_UmaSync.OnSalsaCompleted event
21.     /// </summary>
22.     void OnEnable ()
23.     {
24.         CM_UmaSync.OnSalsaCompleted += CM_UmaSync_OnSalsaCompleted;
25.     }
26.     /// <summary>
27.     /// Unsubscribe from the CM_UmaSync.OnSalsaCompleted event
28.     /// </summary>
29.     void OnDisable ()
30.     {
31.         CM_UmaSync.OnSalsaCompleted -= CM_UmaSync_OnSalsaCompleted;
32.     }
33.          
34.     /// <summary>
35.     /// Unsubscribe from the CM_UmaSync.OnSalsaCompleted event
36.     /// </summary>
37.     void OnDestroy ()
38.     {
39.         CM_UmaSync.OnSalsaCompleted -= CM_UmaSync_OnSalsaCompleted;
40.     }
41.     /// <summary>
42.     /// Grab the Salsa3D reference from the CM_UmaSync.OnSalsaCompleted event.
43.     /// </summary>
44.     /// <param name="salsa3D"></param>
45.     private void CM_UmaSync_OnSalsaCompleted (Salsa3D salsa3D)
46.     {
47.         // Match the event sender name to the CM_UmaSync character name
48.         if (umaSync.characterName == salsa3D.gameObject.name)
49.         {
50.             salsa = salsa3D;
51.  
52.             // Do one-time actions here
53.         }
54.     }
55.     void Update ()
56.     {
57.         // Do not perform actions until we have reference
58.         if (salsa)
59.         {
60.             for (int i=0; i<audioClips.Length; i++)
61.             {
62.                 if (timeElapsed>startTimes[i] && !isAudioPlayed[i])
63.                 {
64.                     salsa.SetAudioClip(audioClips[i]);
65.                     isAudioPlayed[i]=true;
66.                     salsa.Play();
67.                 }
68.                 timeElapsed += Time.deltaTime;
69.             }                
70.         }
71.     }
72. }

 

Thanks a lot, I'll try it when I am home from work

 

Works very good. I f you want to see what we did with it, here is the first Episode of "Troll TV":

 

Awesome, glad to hear it's working well.

 

Hello all,

We have finished beta 1 of our free (for SALSA customers) SALSA and RandomEyes actions for Cinema Director, and we're looking for testers for a closed beta. If you own Cinema Director and SALSA, and are interested in participating, please email your SALSA invoice number to assetsupport@crazyminnow.com and let us know you'd like to participate in the Cinema Director add-on closed beta.

Check the link for more information:
http://crazyminnowstudio.com/posts/salsa-and-randomeyes-actions-for-cinema-director/

Thanks,
Michael

 

Hello, I follow this video "Unity3D Lip Sync - SALSA and Mixamo Fuse". I create a model in Mixamo Fuse, but I do not find "Fuse character Creator > CM_FuseSync" and in the Random Eyes 3D can't Auto-Link.




Can you help me?
And I want to add expression on my character, example : sad, angry, happy
What features are provided in additional licenses?

 

Hello Elang_Jawa,

To use the Fuse 1-click setup, please download the Fuse workflow add-on using your purchase email and the invoice number from your SALSA order, and follow the instructions in the link below. The download link is at the bottom of the page.

http://crazyminnowstudio.com/posts/using-mixamo-fuse-character-creator-with-salsa-with-randomeyes/

The 1-click setup will add one Salsa3D component, two RandomEyes3D components (one for eye control [no SkinnedMeshRenderer linked], and one for facial expressions [SkinnedMeshRenderer linked]), and the CM_FuseSync.cs component that serves as a bridge between SALSA's BlendShape requirements and the BlendShapes available on Fuse characters. You can use a reference to the RandomEyes3D component that has the SkinnedMeshRenderer linked to call Custom Shape functions included in our API to activate/deactivate facial expression. Please note that BlendShapes used in the CM_FuseSync script for lipsync cannot be used simultaneously for facial expressions. The smile left and smile right shapes are used for our SayMedium lipsync shape by default, so you can either remove these shapes from CM_FuseSync, or disable the salsa component while activating these shapes for facial expression through the RandomEyes component. All other shapes, not used by Salsa, require no special handling, just call the applicable Custom Shape Functions.

See the Public Custom Shape Function section of the manual below for details about which functions are available for controlling facial expressions.

http://crazyminnowstudio.com/projec...randomeyes-manual/classes-methods-properties/

Additional licenses do not include extra functionality, they are only required if you have more than one developer on your team working with SALSA.

Please let us know if you have additional questions,

Michael

 

thank you, your response :-D
my invoice number is invalid, i send email to assetsupport@crazyminnow.com, it is correct?

who the sender of the invoice number? from unity or bank? maybe I'm wrong get the invoice number.

 

Hello Elang_Jawa!
I believe we got your email (was a different name) and you should have a response from us. Additionally, if you can't find your invoice or didn't get the email from Unity, you can use the following method to retrieve your invoice number:

1. Login to the Asset Store.
2. Click the person icon on the top right side of the page.
3. Select Credit Card / PayPal.
4. The invoice number is in the Description column, just above the asset name.

Hope this helps!!!

 

Thank you very much for your help.

 

UMA 2 beta workflow 1.2 is now live. This update adds support for UMA asset files and facial expressions.

Come join the fun!
http://crazyminnowstudio.com/posts/uma-lipsync-using-salsa-with-randomeyes/

 

I just bought this and am using the fuse plugin and i noticed that one thing is missing that is in your YouTube tutorial. On the CM_Fuse Sync script the "Sync Custom shapes" is missing????

 

Hello gamegirl1984,

Thank you for your purchase. Custom Shape synchronization now occurs automatically. Please let us know if you have any other questions.

Michael

 

Question for the future... using the audio mixer routing you show ~1:20...

Will it be possible (in theory) to have multiple mics each route to a different SALSA character? tbh, I never got the mic script working on my Mac - which I'm sure is due to my overcomplicated studio audio hardware box and osX (gives me audio source issues in nearly every program), but has always been my goal to have live dialog with multiple speaking figures, like an interview or podcast.... Assuming I can route 2 or more mics in, do you know if it's possible to direct each mic to a different mixer group, then route each mixer group to a different figure? At the very least I could split my 2-channel input with 2 mics (left/right)....

 

Hey wetcicuit! Sorry that you weren't able to get the micInput script to work. Admittedly, our tests have been with simple mic-input scenarios, nothing quite as complicated as you probably have in your line of work. Potentially, a mic selector might help you out. I'll see if I can put something together.

In response to your multiple mic input channels, my gut reaction would be that it isn't possible -- but if it is, it would require mic selection anyway, that's for sure. So, before I say 'not possible' let me do some testing.

At a minimum, we'd like to get you working with one microphone. Would it be possible/amenable for you to disconnect your "overcomplicated" setup and just plug a single, basic microphone in and give it a whirl?

Stay tuned!
Darrin

 

Wow! I was afraid you would do this! Hahaha! You are amazing. Yeah, the potential for multiple audiostreams.... Even if it's just routing stereo, I need to look at all the new U5 stuff.... But if multiple headsets could be selected, like by script, well that would be ideal!

I'm sure I have a usb headset somewhere.... tbh I just assumed your mic script would work so I never went to the next step of troubleshooting (sorta down the road, so to speak...). I will update and report.

 

Hehehe...we like to think we're amazing too...but it's nice to have someone else agree with us!

OK, so I just performed a VERY crude test and was able to setup two separate microphone inputs, for two separate SALSA-enabled characters and was successful in routing one to one character and one to the other character. Now to clean up this code and make it a bit more user-friendly. I'm assuming you won't mind beta-testing for me when I'm done?

-D

 

Our latest SALSA workflow (beta) for UMA 2 supports UMA.asset files and includes our new expression controller. This controller allows us to extend expression control for UMA 2 to our Playmaker and Cinema Director add-ons. The video below demonstrates a quick setup of a SALSA enabled UMA with dialogue swaps and complex facial expressions being easily added in a Cinema Director timeline. The Cinema Director add-on is currently in private beta, but will be freely available soon to SALSA customers.

UMA 2 workflow
http://crazyminnowstudio.com/posts/uma-lipsync-using-salsa-with-randomeyes/

Cinema Director workflow
http://crazyminnowstudio.com/posts/salsa-and-randomeyes-actions-for-cinema-director/

 

Our SALSA Custom Actions for Cinema Director are now available. Over 40 Custom Actions, including dialogue switching, advanced facial expressions, look controls, and more!

http://crazyminnowstudio.com/posts/salsa-and-randomeyes-actions-for-cinema-director/

 

wohoo thats great, is it also working with Fuse Chars, or only with UMA2?

 

Just played the new Fallout 4, they should have bought Salsa, too