SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Thank you

 

Is there a way to bake the Salsa lip sync animation back to a model so that it can be exported out to FBX?
I want to author animation in Unity then bring it back to Maya for rendering.
Thanks,
Jeff

 

Hi Jeff,

There isn't a supported way to do this within SALSA. The project linked below is probably your best bet.
https://github.com/newyellow/Unity-Runtime-Animation-Recorder

Michael

 

Thanks, I'll check it out.
Jeff

 

Hi, I've created an editor script to change the lip sprites on a Salsa2D object. When I hit play and test it the fields on the sprites go to none. The sprites are there before I hit play. The object works when I drag the sprites into the field but not through code. How would I fix that?

 

Hi the rockswell,

Here is an example script that allows you to toggle between two sets of Salsa2D sprites. Link your Salsa2D instance, and two sets of four mouth sprites: rest, saySmall, sayMedium, sayLarge. On start the first set will be selected and linked, check the [swap] checkbox at runtime to toggle back and forth between the two sets. I used the two example sets that are included in the Examples/Emots folder of the asset.

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using CrazyMinnow.SALSA;
4.  
5. public class Salsa2DSpriteSwap : MonoBehaviour
6. {
7.     public Salsa2D salsa;
8.     public Sprite[] set1;
9.     public Sprite[] set2;
10.     public bool swap;
11.  
12.     private bool status;
13.     private bool showErrorOnce;
14.  
15.     void Start()
16.     {
17.         if (salsa != null && set1.Length == 4 && set2.Length == 4)
18.         {
19.             status = !status;
20.  
21.             salsa.sayRestSprite = set1[0];
22.             salsa.saySmallSprite = set1[1];
23.             salsa.sayMediumSprite = set1[2];
24.             salsa.sayLargeSprite = set1[3];
25.         }
26.         else
27.         {
28.             if (!showErrorOnce)
29.             {
30.                 showErrorOnce = true;
31.                 Debug.LogError("Salsa2DSpriteSwap.cs - Setup links to Salsa2D, and two sets of rest, saySmall, sayMedium, and sayLarge sprites.");
32.             }
33.         }
34.     }
35.  
36.     void Update()
37.     {
38.         if (salsa != null && set1.Length == 4 && set2.Length == 4)
39.         {
40.             if (swap)
41.             {
42.                 swap = false;
43.  
44.                 status = !status;
45.  
46.                 switch(status)
47.                 {
48.                     case true:
49.                         salsa.sayRestSprite = set1[0];
50.                         salsa.saySmallSprite = set1[1];
51.                         salsa.sayMediumSprite = set1[2];
52.                         salsa.sayLargeSprite = set1[3];
53.                         break;
54.                     case false:
55.                         salsa.sayRestSprite = set2[0];
56.                         salsa.saySmallSprite = set2[1];
57.                         salsa.sayMediumSprite = set2[2];
58.                         salsa.sayLargeSprite = set2[3];
59.                         break;
60.                 }
61.             }
62.         }
63.         else
64.         {
65.             if (!showErrorOnce)
66.             {
67.                 showErrorOnce = true;
68.                 Debug.LogError("Salsa2DSpriteSwap.cs - Setup links to Salsa2D, and two sets of rest, saySmall, sayMedium, and sayLarge sprites.");
69.             }
70.         }
71.     }
72. }
73.  

 

Thanks, I got it to work by setting the component of the Salsa2d .enabled to true and then to false in the script. That seemed to reset it somehow to keep the sprites during play.

 

We have posted MCS 1-click v2.2.0. It addresses an issue where BlendShapes were not consistently being removed from the [Remove BlendShapes] section of the CM_MCSSync inspector.

https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

 

I have just created a unity project using a Daz character, RT-Voice, Salsa with Random Eyes and Dialog System. It works great in Unity, I got the lip-sync and eye movements. But when I exported the project to run on an iPad, while the eye movements are still there, the lip-sync is gone. Any clue why that is happening? Thanks!

 

Hello eggtart,

On desktop, we use the RTVoice.Speaker.Speak method to generate the audio output and link it to the same AudioSource that SALSA is linked to, or the RTVoice.Speaker.SpeakNative and RTVoice.Speaker.OnSpeakCurrentViseme event for native mode. However, both of these options are unavailable on iOS. Our RT-voice add-on page provides information about the option available on the iOS platform.

SALSA RT-Voice add-on
https://crazyminnowstudio.com/posts/salsa-and-rt-voice-for-runtime-lipsynced-text-to-speech/

On the iOS platform, we leverage the RTVoice.Speaker.SpeakNative method and the RTVoice.OnSpeakCurrentWord event, combined with our TextSync text-to-lipsync add-on to provide a workflow for SALSA and RT-Voice.

TextSync (text-to-lipsync)
https://crazyminnowstudio.com/posts/salsa-text-to-lipsync-using-textsync/

 

Hi @eggtart - If you're using the Dialogue System for Unity, can you confirm that SALSA lipsync works correctly without using the Dialogue System? (For example, try a scene with an audio source that auto-plays.) If so, then the issue is probably that Unity is stripping out sequencer commands. The Dialogue System provides a link.xml file to address this issue. Depending on your version of Unity, you may need to move this file into your root Assets folder. Depending on your version of the Dialogue System, the link.xml file may be in Plugins/Pixel Crushers/Dialogue System/Templates/Link/link.xml (for version 2.x) or in Pixel Crushers/Dialogue System/Prefabs/Mobile/link.xml or Pixel Crushers/Dialogue System/Prefabs/iOS/link.xml (for version 1.x.).

Then again, if you're controlling the eyes through the Dialogue System's Eyes() sequencer command and it's working, the issue may be elsewhere.

 

Hi @TonyLi , the SALSA lipsync does work correctly with RT Voice and Dialog System. It is only when I compile to the IOS platform that lipsync stops working.

I am now trying this - Using Speaker.SpeakNative with the OnSpeakCurrentWord event for use on iOS as mentioned on this https://crazyminnowstudio.com/posts/salsa-and-rt-voice-for-runtime-lipsynced-text-to-speech/

Thanks!

 

If possible, please confirm that RT-Voice and SALSA work correctly in iOS without the Dialogue System. You don't have to remove the Dialogue System from your project; just run a test line of speech without using the Dialogue System's RTVoiceActor component. The issue may be specifically related to code-stripping for iOS builds. If it works in iOS without the Dialogue System, please use the link.xml file I described in my previous reply.

 

I have following added the TextSync and Native_IOS components, and have added the xml file to the Assets folder. (please see below). But lipsync still does not work in the iOS build.

 

One possibility is that the dialog is in Chinese. I see that the dialog text is correctly passed to Speaker_OnSpeakCurrentWord() and hence to textSync.Say(). But the text is something like "早晨，你好嗎？", there is no space between the characters, so maybe textSync just considers the whole sentence as one word and does not know how to handle it?

 

Hi eggtart,

When you pass in a sentence of Chinese text to RT-Voice, how many OnSpeekCurrentWord events are firing, just one or multiple?

Thanks,
Michael

 

Hi Michael,

The event was fired once, text like "早晨，你好嗎？" is passed as a whole sentence to TextSync. Since normally there is no " " between the Chinese characters, so TextSync may think there is only one word when it does a word split. I have tried putting a space between each character, but that will cause RT-Voice to fire multiple OnSpeakCurrentWord event - one for "早", one for "晨", one for "，", etc etc, which is also no good.

P.S.
I changed the code from
int wordCount = this.text.Split(' ').Length;
to
int wordCount = this.text.Length;
to get the correct word count for Chinese, and it seems to be working fine now.

Thanks

 

Thanks eggtart,

We'll add this as an option to the TextSync inspector in the next update.

Michael

 

Hi Michael,
I need to change some blenshapes of the sayRest position. (I need the character to rest with a smile, smile_L: 50% smile_R: 50%)
Can I define a sayRest position as I did with the other 3 positions?
Thanks,
Robert.

 

Hello Robert,

The sayRest position isn't represented by a BlendShape, but rather the lack of any BlendShape values being applied. That being said, one way to work around this would be to set the shapes you want to be active up in RandomEyes3D Custom Shapes or Groups and use Custom Shape override or SetGroup with the true parameter to make these shapes/groups permanently active. Keep in mind that if you are using the same shapes for speech and expression, speech will override the values as soon as speech is performed, you would need to duplicate the BlendShapes so you have dedicated shapes for speech and expression. The result of activating the duplicated shapes is additive. You can duplicate Blendshapes in your favorite 3D application, or using our MorphMixer asset, you can duplicate and mix and combine to create new BlendShapes within the Unity editor.

https://assetstore.unity.com/packages/tools/animation/morphmixer-111278

Michael

 

Thank you so much, that would be a great help!

 

OK. I will try that. Thanks for your help!

 

TextSync v1.3.0 has been uploaded. It adds a "No Spaces" property that splits the text by an average word length that can be specified in the inspector.

https://crazyminnowstudio.com/posts/salsa-text-to-lipsync-using-textsync/

 

That is fast!!!

 

I am using a Daz Genesis 8 character in Unity. After using the 1-click Daz setup. I got the following



Nothing is linked. Is it normal?

I tried linking in the female shape and got this.



The indexes are all linked to CheeksBalloonPucker...

Somehow the eye movement does not look very natural, the eyes will only rollover to the right and never to the left. I have tried using a tracking target and it is the same. Any thoughts? Thanks!

 

Hi eggtart,

The DAZ 1-click setup uses two instances of RandomEyes3D, one for eye control where the [Use Custom Shapes Only] property is unchecked, and the other for facial expression where the [Use Custom Shapes Only] is checked. The eye control instance should remain unlinked to any SkinnedMeshRenderer because the linkage is handled in the CM_DazSync script and the RandomEyes3D instance is only doing the logic processing. The facial expression instance can be used to access our Custom Shapes and Groups API. When controlling look target tracking, blinking, or other eye controls, be sure you are addressing eye control instance of RandomEyes3D.

Michael

 

TextSync v1.4.0 has been uploaded. We removed a missing test script from the example scene, and fixed a bug with audio clip checking.

https://crazyminnowstudio.com/posts/salsa-text-to-lipsync-using-textsync/

 

I see that there are 2 instances of RandomEyes3D. Just that I notice the eye movement is not that random, it is always to one side and not the other. I am wondering if this has something to do with my initial character setup.

 

Our CM_DazSync script applies the calculated eye movement BlendShape values from the eye control instance of RandomEyes3D directly to the eye bones in the DAZ rig. I just tested with a Genesis 8 female and the 1-click setup found and linked the left and right eye bones and drives them correctly at run-time.

You can test this by adding an empty target GameObject to the scene, place it a meter or so in front of the characters eyes, and link it to the look target field of the eye control instance. Run the scene in the editor and move the target around to verify the eyes are tracking correctly. You should turn the range of motion value up to 100 when tracking.

 

Hi, could you please update Salsa for the last Unity 5 version? I'm very happy that Unity just fixes the Unity 5 version, I don't want to upgrade. The feature would be needed: 04/13/2018 - MCS v1.6.4+ (2.1.0) Added a new function that automatically remaps

 

SALSA already works in v5.6.6. Are you seeing a problem/error?

 

Hi, you right, you provide the latest version for Unity 5.6.6. The problem is, that under Components->[company name]->Salsa-> are just 4 options available. Random Eyes2d/3d, Salsa 2d/3d. The configuration entries eg. for MCS are missing. I read your release notes and I thought that you provide the 1.4.3 version for Unity 5.6.6, because that does not have this feature. My fault.

However, I missing the configuration entries for automatic mappings.

Edit: Cannot track this issue down, because the editor code is not available, just a dll file. This is an example for why we game developer need the source code. Could be a #define, maybe could be my fault, could be something else.

 

Links to download our MCS 1-click setup, as well as all our other free add-on's, are available on the asset store page and on our website.
http://crazyminnowstudio.com/unity-3d/lip-sync-salsa/features/

EDIT....
Our programming API is documented here:
SALSA
http://crazyminnowstudio.com/unity-...uals/salsa-manual/classes-methods-properties/
...and here:
RandomEyes
http://crazyminnowstudio.com/unity-...randomeyes-manual/classes-methods-properties/

 

Thank you for the links! I just read the Readme.txt in the main folder and I watched some tutorial videos. Missed this information.

 

Besides the character creators do you have listed, does this works with any character that is rigged up for facial animations?

 

Hello! All rigs are different. SALSA is primarily a blendshape compatible system and should work with any character that has appropriate blendshapes. We have created add-ons to work with specific systems to make the process simpler. For more information, please check out our recommendations in our online manual (documentation linked from the landing page)

SALSA 2.0 will have much more flexible built-in support for varying character systems, but it will still vary from system to system. You can read about our progress with SALSA 2.0 in the devblog.

 

Any eta when 2.0 is out?

 

We will announce an ETA when we are confident we can achieve it. You can read about our progress with SALSA 2.0 in the devblog.

 

Awesome asset for prototyping with RT Voice. The dialogs will be spoken and so I have the first feeling how it will be later. I can make the levels without extensively preparing the actors. But I would not use it yet in released upcoming pc games. I hope that Salsa 2 will be more accurate and more customizable. But however, Salsa is a very valuable tool for my purposes!

 

Hi there,
I use the folllowing script which is attached to the MCS 3D character (alongside with Salsa LipSync and Random Eyes) :

Code (CSharp):

1.  
2. using System.Collections;
3. using System.Collections.Generic;
4. using UnityEngine;
5. using CrazyMinnow.SALSA;
6.  
7. public class SyncAnimWithSalsa : MonoBehaviour {
8.  
9.     public Animator anim;
10.     public GameObject player;
11.     public Salsa3D salsa;
12.  
13.  
14.     public AudioClip clip1;
15.  
16.     public string state1;
17.  
18.  
19.     private bool once1;
20.  
21.     // Use this for initialization
22.     void Start () {
23.        
24.         salsa = gameObject.GetComponent<Salsa3D>();
25.         anim = gameObject.GetComponent<Animator>();
26.  
27.         salsa.useGUILayout = false;
28.  
29.         salsa.saySmallTrigger = 0.001f; // Adjust the small trigger value
30.         salsa.sayMediumTrigger = 0.002f; // Adjust the medium trigger value
31.         salsa.sayLargeTrigger = 0.004f; // Adjust the large trigger value
32.  
33.         salsa.broadcast = true; // Set broadcasting to true
34.         salsa.broadcastReceiversCount = 1; // Setup one receiver slot
35.         salsa.broadcastReceivers = new GameObject[salsa.broadcastReceiversCount]; // Setup array length
36.         salsa.broadcastReceivers[0] = player; // Bind some game obejct to the receiver
37.         salsa.propagateToChildren = true; // Propagate broadcasts to SOMEGAMEOBJECT's children
38.  
39.         salsa.audioUpdateDelay = 0.08f; // The duration between audio sample updates
40.  
41.         salsa.blendSpeed = 10; // Shape key transition duration
42.         salsa.SetRangeOfMotion(85); // The percentage of total range of motion allowed
43.  
44.  
45.     }
46.    
47.     // Update is called once per frame
48.     void Update () {
49.  
50.  
51.         if (state1 != null && clip1 != null && anim.GetCurrentAnimatorStateInfo(0).IsName(state1) == true && once1 == false)
52.         {
53.             salsa.audioClip = clip1;
54.             salsa.useGUILayout = false;
55.             salsa.Play();
56.             once1 = true;
57.         }
58.  
59.         if (once1 == true && salsa.isTalking == false)
60.         {
61.             anim.SetBool("SarahTalk", false);
62.         }
63.     }
64. }

However, The Play(); function doesn't work. Why?

 

Sorted ...... The Play(); function requires the audioclip to be set and not just assigned. I used the salsa.setAudioClip(some audio) and then the salsa.Play();......Everything works fine.

 

Hello – we discovered Unity changed their invoice-check server address without notifying publishers. We have updated our system and have confirmed the downloads invoice confirmation is working. We apologize for the inconvenience.

 

I've been using Salsa successfully with Daz characters, but needed to implement LOD (Daz characters are quite high poly). I've developed a pipeline that involves importing the Daz character into Blender using Diffeomorphic's DAZ Importer, adding LODs, then exporting to Unity as an FBX.

Rather optimistically I applied Salsa's one click Daz set up and surprisingly, it mostly works. Eye blinking worked straight off. Lip syncing worked once I'd redirected Salsa to Blender's renamed shape keys. There is one thing not working and I've been unable to figure it out.

The eye bones are rotated (-89,180,0), whereas the direct Daz import is (0,0,0). So when playing the eyes roll back into the head. I've tried editing the avatar to no effect, and I tried various things with RandomEyes also unsuccessfully. The eye bones are (0,0,0) in Blender. They're getting rotated somehow during the transfer to Unity.

I watched some of the RandomEyes videos on Youtube on setting up from scratch, but they seem aimed at blendshape eyes rather than boned eyes.

Is there some simple fix I can apply here?

 

Hi Dorian-Dowse,

LOD on DAZ sounds great. These characters look so good but they are normally so heavy.

I would recommend using our free SalsaSync add-on instead of the 1-click setup. SalsaSync was designed to be a flexible mapper between SASLA's requirements and the many different configurations used in character models. It also provides some axis re-mapping for eye and jaw bones that aren't Unity axis aligned. It's a manual setup, but I believe it will give you better results.

https://crazyminnowstudio.com/posts/salsasync-flexible-salsa-map-for-custom-characters/

 

Awesome. That worked perfectly. For some reason the lip sync looks way better than the DazSync too. (bonus)

 

Thanks for providing the name of that DAZ/Blender add-on, it looks like a cool add-on.

 

@Crazy-Minnow-Studio A while back I reported an issue where when a character blinks that the hair would morph also. This is with MCS characters. You had a fix where it removes the blendshape for hair. However, now that I'm using this a bit more with characters that have various morphed heads and bodies, I'm seeing all kinds of issues. For example, for a young character that is morphed much shorter, the hair no longer fits to the head, but is floating above it. See below:



And with another character, the hair is getting sunk into the head:



For now I'm just removing those hair exclusions so the hair stays in place, but with the old blinking issue.

 

The MCS 1-click setup has been updated to v2.2.1. We added a blinkShape exclusion directly to the SyncShapes function in CM_MCSSync, and removed the default SMR's from the BlendShape removal list.

https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

 

Thanks. I didn't realize there was an update. I downloaded and it works fine now.

 

Really cool plug-in! I'm about to start working on a game where I'll be using NeuralNetworking for the AI to respond to player speech but all NPC's will be custom made in MAYA. Is there a way to use SALSA with custom-made characters and are there any tutorials on how to do that or any documentation on it? Because what I've seen so far all the characters used with SALSA are Software generated and I'll be making my own characters, blendshapes and animations.
Thanks!