SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hi Nathan,

I replied in email, but I'm also posting here for continuity.

Your UMA Expression Player sliders do not affect the bones of your character correctly. To confirm this, I removed all SALSA related components and just manually set two sliders on your UMA Expression Player [leftMouthSmile_Frown: 0.87, right MouthSmile_Frown: 0.87]. This is getting into the weeds on how UMA works and outside the scope of SALSA support, but if I were to guess it looks like your bone orientations are off. See the image below with a default UMA male and the same two sliders set to the same values [leftMouthSmile_Frown: 0.87, right MouthSmile_Frown: 0.87].



Michael

 

Thanks Michael,

Sorry I haven't responded to your email yet, yeah, you are right about the bone shapes. We're trying to figure out why the rotations of only a few bones are different, everything else in the body works.

When we figure it out I'll post back with the cause and solution.

Thank you for your time, I know this ended up not having anything to do with Salsa so I apologise for using your time.

Nathan

 

SALSA LipSync Suite v2.5.4.125
BugFix Release -- Available Now on the AssetStore

This is another small release to fix a couple of rare bugs/issues. If you are beginning a new project, this is the version of choice.

EXPECTATIONS:
If you are upgrading an existing project, read the release notes and make a backup of your project before you upgrade SALSA Suite!

If you find bugs or documentation issues, please let us know, we will knock them out as quickly as possible. Please ensure you've checked the Release Notes and latest documentation and ensure you have the correct (latest) Add-On/OneClick versions. It is best if you email us (assetsupport@crazyminnow.com), including as much detail as possible, screenshots and/or video, any errors received, versions of everything, and always include your Unity Invoice Number.

Thanks and enjoy!
D.

 

Hi,

I've recently bought your plugin and it's really useful!
Although, I currently have a little problem regarding the CC3 oneclick. After using it, the CC3 character has lipsync, but when the character speaks, its skin (on the whole body: arms, shoulders, ...) also seems to move a bit when the mouth moves. The CC3 character that I currently use is one of the default characters available in the CC3 demo. This seems a bit odd, so I'm curious if there's a fix for this. If necessary, I can probably upload a video showing the problem.

Regards,
Jelle

 

If what you are describing is a weird shadow effect, it sounds like you might need to switch Normals to 'calculate' mode on the Model tab of your model.

D.

 

Thanks, that was indeed the problem!

 

Hi,

After upgrading to LipSync Suite 2.5.4, and adding DAZ one-click 2.5.3, I'm seeing the following errors with my Genesis 8 character:

Array index (-1) is out of bounds (size=1)
UnityEngine.StackTraceUtility:ExtractStackTrace ()
CrazyMinnow.SALSA.ShapeController:SetLerp (single,bool)
CrazyMinnow.SALSA.QueueProcessorrocessQueue (System.Collections.Generic.List`1<CrazyMinnow.SALSA.QueueData>,int)
CrazyMinnow.SALSA.QueueProcessor:LateUpdate ()

The project is in Unity 2021.2.10f1.
But I had the same errors in 2020.3.27f1.

There are no errors when I uncheck "Enable Processing" for both Eye Configuration and Eyelid Configuration. But the character turns his head without blinking. And that zombie effect is not what I want.

I've used LipSync with DAZ characters since 2018. These errors are something new. And frustrating.

Any help would be greatly appreciated.

 

Can this work with Vivox voice chat?

 

Hello, Could not figure out the Mandible rotation in out UMA rig so are looking into Blendshapes.

I have most of it working excpet then the character is changed in runtime the reference to the SkinnedMeshRenderer is lost in each Viseme field on teh Salsa component, so I neet to reassign this through code. Except, I can not find the reference to it in the Salsa component (too many arrays for me .

Could you please tell me how I would reference this in code?:

Thank you!
Nathan

 

Hello,
Sorry you are experiencing problems with SALSA Suite. I am unable to reproduce your issue with DAZ8 and U2020.3.LTS. Is it possible you applied the DAZ8.1 OneClick on your DAZ8 model? That might create the issue you are experiencing. Even then, it should only result in an incomplete model configuration and should only generate a warning indicating so. I tried applying the 8.1 OneClick and did not receive errors other than the warning.

Please send us an email with your Invoice number and other relevant information such as OS and version. Also include screenshots of your complete configuration and/or a video demonstrating the issue.

Thanks,
D.

 

Hi Nathan,
Yes, the data structure can be pretty intimidating. Have you taken a look at our runtime setup documentation? I am still updating it for all of the controller types, but it has (hopefully) good basis information. You can also checkout the example runtime setup.

There are two things to consider with runtime setup. Do you need the Inspector to reflect the setup or not? I would recommend configuring for Inspector awareness. In that case, you will update your SMR in the following way:

Code (CSharp):

1. using CrazyMinnow.SALSA;
2. using UnityEngine;
3.  
4. namespace DemoCode
5. {
6.     public class UpdateSalsaSmrs : MonoBehaviour
7.     {
8.         public void UpdateSmrConfig(Salsa salsaInstance, SkinnedMeshRenderer newSMR)
9.         {
10.             foreach (var viseme in salsaInstance.visemes)
11.             {
12.                 for (int i = 0; i < viseme.expData.components.Count; i++)
13.                 {
14.                     var component = viseme.expData.components[i];
15.                     var componentHelperData = viseme.expData.controllerVars[i];
16.                     if (component.controlType == ExpressionComponent.ControlType.Shape)
17.                         componentHelperData.smr = newSMR;
18.                 }
19.             }
20.          
21.             salsaInstance.UpdateExpressionControllers(); // bake the Inspector helper data into the controllers.
22.         }
23.     }
24. }

Of course, you might need to modify this in order to integrate with your code base. Hope that helps,
D.

 

Hello,
We are not familiar with Vivox. However, if it provides access to the serialized audio data and you can analyze it or provide it to SALSA to analyze, it should work. We support other chat platforms like Dissonance and PhotonVoice so it is possible you could implement Vivox in a similar manner. However, it would likely require some code to implement.

D.

 

Thank you for this, my Skinned Mesh Renderer fields now populate!

I have added this to my code that checks the current recipe of the face wardrobe slot, if the desired recipe is loaded it instantiates a gameobject with Salsa and the Queue Processor. The Audio source is assigned on instantiaion and the visemes are populated as well.

Nothing happens as is, but if I click on the instantiaed game obejct and select Advanced Dynamics the lipsyncing works, but I get this error:

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. CrazyMinnow.SALSA.Salsa.FireVisemeTriggeredEvent () (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)
3. CrazyMinnow.SALSA.Salsa.LateUpdate () (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)

Obviously I can't select the game object when the game is actually playing, this is only a weird editor jump start. But at it is getting closer. Any idea how I can look to resolve this?

Thanks again for _ALL_ of your help with this.
Nathan

 

and the core of the code

Code (CSharp):

1. public void InstantiatePrefab()
2.     {
3.         string currentRecipe = avatar.GetWardrobeItemName("Face");
4.         if (salsaComponent != null)
5.         {
6.             DestroyImmediate(salsaComponent);
7.             salsaInstance = null;
8.         }
9.             switch (currentRecipe)
10.         {
11.  
12.             case "Radiohead_Recipe":
13.                 salsaComponent = Instantiate(Radiohead);
14.                 salsaComponent.transform.parent = gameObject.transform;
15.                 UpdateSalsa();
16.                 break;
17.             default:
18.                 break;
19.         }
20.      
21.     }
22.  
23.     private void UpdateSalsa()
24.     {
25.         salsaInstance = salsaComponent.GetComponent<Salsa>();
26.         salsaInstance.audioSrc = audioSource;
27.         newSMR = this.transform.Find("UMARenderer").GetComponent<SkinnedMeshRenderer>();
28.         foreach (var viseme in salsaInstance.visemes)
29.         {
30.             for (int i = 0; i < viseme.expData.components.Count; i++)
31.             {
32.                 var component = viseme.expData.components[i];
33.                 var componentHelperData = viseme.expData.controllerVars[i];
34.                 if (component.controlType == ExpressionComponent.ControlType.Shape)
35.                     componentHelperData.smr = newSMR;
36.             }
37.         }
38.         // apply api trigger distribution...
39.         salsaInstance.DistributeTriggers(LerpEasings.EasingType.SquaredIn);
40.         // at runtime: apply controller baking...
41.      
42.         salsaInstance.UpdateExpressionControllers(); // bake the Inspector helper data into the controllers.
43.  
44.     }

 

That bug/behavior-requirement should have gone away with the latest update. It happened because the event args were initialized and cached in the start-up of a valid SALSA configuration (with respect to the Inspector). So, if SALSA is simply added to a object and not configured, it would not initialize the event args on startup. The idea was a minimal configuration needs to be setup for SALSA to run, so it expects this. A runtime configuration after the fact wasn't expected. If using a runtime configuration, it was expected that SALSA would be added at runtime and this wouldn't be an issue. When you click on (select) the game object, it re-runs the configuration checks in the Inspector and gets everything fired up again.

This behavior *should* be modified in the new version and should initialize the event args whether SALSA is properly configured or not. If you do not wish to update, you can minimally configure your SALSA component, or just add it at runtime and then configure the settings in code.

If this description doesn't fit your situation, send us an email and reference this forum post. Hope that helps,
D.

 

Hello,

I missed that last update and can confirm that the errors are fixed now, what good timing!

However, I still need to manually click on the component before the lipsycn starts. Is there something I can call through my code to initialize the event args?

Many many thanks for your help

Nathan

 

Hello, I just started looking into Salsa Lipsyncing, have been through your documentation on their website.
Can someone over here shed some light on how does Salsa Detect which viseme to use based on audio?
They have mentioned that the names of visemes are not used anywhere so for sure salsa doesn't pick the viseme by names, so how does Salsa know what viseme to use based on the audio?

 

Hi Crazy-Minnow-Studio,

The Microsoft Rocketbox avatar library recently updated its repo to include blendshapes for its avatars (link). This is a collection of avatars that can be used for free in research and other projects. It might be interesting to add a one-click install for these avatars.

Regards

 

Ahhh, OK, well, assuming you still have a non-configured SALSA, the Inspector has flipped the

Salsa.configReady

flag to false. No worries. Instead of calling

Salsa.UpdateExpressionControllers()

do the following:

Code (CSharp):

1. // salsaInstance.UpdateExpressionControllers(); // bake the Inspector helper data into the controllers.
2. salsaInstance.configReady = true;
3. salsaInstance.Initialize(); // this performs the helper data baking.
4.  

Let me know if that works for you.
Darrin

 

Hi Darrin

That was it! Thank you!!!

A massive thanks for your continued help with this

Nathan

 

Hello Nishchhal,
I believe this information should answer your question. https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/overview/#salsa-lipsync

Hope that helps,
D.

 

Hello JelleBelly,
We are a very small team with limited resources and do monitor customer interest in model systems. When sufficient interest evolves and systems are supportable (having adequate, standardized facial animation controls), we may add new OneClicks to the list.

Thanks for your interest!
D.

 

Just a quick note:
Crazy Minnow is currently not receiving notifications for new forum posts. We have a ticket in with Unity, but have not heard back from them. As such, if your question is directed at Minnow, please send us an email to make sure we see it.

Thanks!
Darrin

 

hello there,

I'm having a issue with the eyes and head. it looks like they interfere with my animation. the moment i use my animation the head and eyes start flickering/glitching. (changing positions every millisecond.) i never had this problem before, but i dont know where i can find the problem.

 

Hello, it sounds like your animation is conflicting with SALSA Suite's animations. If you want to use your animation instead, simply disable head and eyes processing in their respective sections. Likewise, if you want to use SALSA Suite animations, then remove the head and eye bones from your animation.

D.

 

Hello! I know initially salsa only supported 4 blend shapes for 2d has this increased for salsa 2.0

 

SALSA LipSync Suite 2.x is a complete rewrite of the original product with loads of improvements across the entire suite. The shapes are unlimited, but you can view the recommendations section of our manual to see more details.

https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/recommendations/

Michael

 

Hi @Crazy-Minnow-Studio

Apologies if this has already been covered.

We have recently setup some character lip-sync tests using Salsa and FMOD audio and have got this to a place where we are happy with the effect. We have found an issue that only happens when some of our scene handling code disables and re-enables a character after this they no longer open their mouth (looks like they are mumbling). We found a solution to this was to tick the use persistence option. Given the warning that comes with that tick box I wanted to check if that is the best solution or if there is something else could/should be doing in this case?

Thanks in advance,
Alex

 

Hello, this is an interesting situation. I would not think that persistence mode would resolve your issue, but perhaps I need some more information on exactly what is happening when you disable and re-enable the character (and SALSA components). This may be the case but the situation doesn't quite sound like it. Please send us an email with more details on what is going on, your invoice number, perhaps some video and screenshots, versions, OS, platform, etc. Thanks! Persistence mode is generally only recommended when there are external influences that are conflicting with SALSA.

D.

 

Hello, sorry to bother you, did you found a solution for the Eyes component? I have the exact problem, it randomly appears and disappears.

Code (CSharp):

1. NullReferenceException
2. CrazyMinnow.SALSA.Eyes.DestroyProcTransOffsetGizmos (System.Collections.Generic.List`1[CrazyMinnow.SALSA.ProcTransOffset]& list) (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)
3. CrazyMinnow.SALSA.Eyes.Initialize () (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)
4. CrazyMinnow.SALSA.Eyes.Awake () (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)

 

Hello,
Looking back through our tickets, I believe Mike was trying to replicate the issue as a suspected bug. He is currently unavailable and will not be back until next week. I will check with him when he gets back to see if there was a resolution.

Thanks,
D.

 

Hello,

Does anybody have a list of DAZ morhps I should export for SALSA? I can't use "head" as Match world as it exports huge amount of morphs in my case, making fbx too big and import to Unity takes several hours. I wrote an Excel macro which creates csv with desired morhps and can be imported in DAZ Studio, so it's not an issue to list them by name.

And is it easy to set all up in SALSA to work with multiple Skinnedmeshrenderers? My DAZ figure is separated by Blender script into several objects. Relevant for SALSA would be Face, Eyes and also Eyelashes and Facial Hair as separate prefabs.

I didn't buy SALSA yet, just want to have my figures and props prepared for later.

 

I'm using Unity 2020.3.20f1 version.
After confirming that the character lip-sync is working, I try to create a track on the timeline, but I don't see an option to create a SALSA LipSync track.
So, we tested the Unity version by installing the assets in multiple projects.
The result is that the options are visible in some projects and not in others.
What causes this to happen?
Please tell me how to solve it.

 

Hello, you need to download the Timeline add-on from our downloads portal. According to your screenshot above, it is not installed.

Hope that helps,
D.

 

Hello,
Once you purchase SALSA, you will have access to the DAZ OneClick and can search through to see the blendshapes used. It depends on which generation you are using. But you can see which morphs are used in the OneClick.

D.

 

Hi @Crazy-Minnow-Studio, can you add an option for an emote to cancel Blinking during it's duration ?
If not, can you provide me a scripting example for me to do that, thank you.

This is abomination combining Smiling with Blinking.

 

Hello @Spikebor,
Yes, it looks like your model has a problem in that it includes closed eyes with the smile shape and therefore overdrives them when the model blinks.

You can turn off the Eyes module blinking when you turn on your emote by using:

Code (CSharp):

1. eyesReference.blinkRandom = false;

And then set it back to true when your smile is turned off.

D

 

Thank you for your reply, that will work if I set the emote in script myself, but I'm talking about the case where emote is play at random by your system. So I do not know when the emote is playing, and when it stop. Can do if I have those 2 events.

 

Gotcha. OK, send me your Invoice number and reference this thread and I will see if I can implement a feature to support this. I need to think on the best implementation. Ultimately, the EmoteR module doesn't know when an emote has left the QueueProcessor, so depending on how critical the timing is of such a notification, the event would need to come from the QueueProcessor itself. But, the QueueProcessor works on the component level, not the "emote" level. QueueProcessor is completely decoupled from EmoteR (and SALSA/Eyes).

From EmoteR, I can pretty easily put in an event when the emote is fired, containing info on how long it *should* last or if it was fired as a OneWay or not, etc. You would just need to calculate the timing on your own in that instance and deal with occasions where it is time extended by being fired again.

It would be more flexible to work at the component level for your particular issue. It would need to be conceptualized (in this instance) as 'which component do I need to care about', not 'which emote'. Ultimately, it might be a good idea to implement both. I can probably implement the EmoteR event in the current version, but will have to wait for the QueueProcessor events until the next version. Send me your info and I can see what I can do.

D.

 

I sent it, thanks.

 

Hello everyone,

Kindly let me know if anyone is using SALSA with RT-Voice for WebGL platform and able to display the lip-synch successfully in WebGL build.

Currently I'm facing the issue that Lip-synch is not working in WebGL build for me but it is working in unity editor.
I'm using Amplitude SALSA, WebGL Speech Synthesis, RT-Voice for run time text to speech conversion, and Dialog system from Pixel crushers. All the implementation is double checked and working fine in editor and I'm able to reproduce the lip-synch but when I build it for WebGL and run it in Chrome the lip-synch is not working.

I would really appreciate if anyone help me to resolve this issues.

Thank you.

 

@Lakshmangc - Try a small test using only Amplitude and SALSA to narrow down the issue. If you still don't get lipsync in WebGL, then it's probably not related to RT-Voice or the Dialogue System for Unity. This should shorten the list of things you need to look into.

 

Hi,
what about the One-Click preset for the new CC4 characters?
Massimo

 

@TonyLi We already tested with only Amplitude and Salsa with pre recorded audio files for WebGL build and it is working, but in our project we need Run time Text to speech conversion so using RT-Voice. As soon as we changed to RT-Voice we are able to hear the voice but not able to produce lip-sync in WebGL build.

 

@Crazy-Minnow-Studio or @Stefan-Laubenberger might be able to provide more insight. I can only address the Dialogue System, and it seems like that isn't part of the issue. Maybe RT-Voice is playing audio on a different GameObject than the one with the SALSA components.

 

Hello @Lakshmangc, I suspect you are experiencing an async issue with the WebGL RTVoice audio. I believe RT Voice has a demo package/scene available that shows how to work with async audio in WebGL for RTVoice and SALSA integration. I know @Stefan-Laubenberger worked hard to ensure that process worked with SALSA. Async audio can come back and play in Unity while not being in a Unity 'ready' state which prevents SALSA from being able to analyze it.

Hope that helps,
D

 

Hello @Crazy-Minnow-Studio and @TonyLi I suspect the issue was causing from the WebGL Speech Synthesis asset.
Because now we have changed the bridge from WebGL Speech Synthesis to Google cloud TTS as it provides wide range of voice over. And a small fix from @Stefan-Laubenberger on GC TTS everything is working as expected. Thank you for your time on assisting and referring us in the right direction.

--
Thank you,
Lakshman

 

Hello there,

I've been faced with an interesting issue regarding the head clamping rotation.

In our environment, it's possible for characters to lay down. However this is done via animation and not a 90 degree rotation. The result of which looks like this:


Since it still expect the head to be facing forward. For now I can just disable head movement, but we would eventually want this too.

Do you know of a workaround (aside from asking artists to redo a lot of work to have a rotation) or a possible method to override the direction the eyes are suppose to face?

I think it would add more to your plugin's flexibility, supporting non facing forward eyes.

Thank you.
JFV

 

Hi JFV,

You might be able to address this by enabling the [Parent Random Targets] checkbox, though this may not work if your character isn't actually rotating.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/eyes/using/
Search "parent random targets"

Alternatively, you may be able to use target tracking during the transition to lying down, then set the random generation clamp parameters to generate vertically.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/eyes/using/
The green and yellow frustum scene gizmo show the field of view where random targets are generated.

Michael

 

For some reason the the head and eye tracking on Eyes is causing my character to look some distance above the target rather than at it. Target is set to that cube floating in space.



Here is the orientation of the head bone. You can see that it's tilting back.



This character does have an Animator and an AnimationRigging component on it, but all of that has been disabled. When I turn off the head processing, the character's head goes back to normal position.

I've tried adjusting the target offset to about 0.5, but when I do the eyes still seem to be pointing too high.




Any idea what might be causing this offset to happen?