SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Thank you for replying.

Yes I am stop/change/playing. I'm also not starting a new clip unless the previous one has finished playing:

Code (CSharp):

1.            
2. if(!audioSource.isPlaying){
3.                 audioSource.Stop();
4.                 audioSource.clip = clip;
5.                 audioSource.Play();
6. }
7.  

I've recorded a video that I hope will help you diagnose the issue

 

From the information I've seen and going out on a limb, what is happening is your audio files are too short for the SALSA processing pulse timing. I would recommend shortening the audio delay option in SALSA Settings, maybe to about 0.4f. And probably increase your sample size to 1024. I can't tell if you are using single frame swapping or multiple frame animations. If single frame, ensure your animation timings are 0.0f for ON and OFF.


D.

 

Hello,

Here is a look at my viseme configuration, I have 3 sets (small, medium, large), so here is the one for small (attached). It's 7 sprites, and it works well for a normal (and long) audio clip of spoken words.

I followed your advice, setting the sample size and update delay, and also setting the on/off for my visemes to 0. It still does this

Is there anything else that can be done? I had hoped salsa would be able to lip-synch for this!

 

You will want your animation ON/OFF timings to be more than 0.0f for multiple frames.

Actually, one other thing, make sure the look-ahead (playhead bias) is zero. What are your audio sample frequencies and the sample length per file? For your very short files, set the playhead bias to 0.

If this doesn't work, send your project to us with your Invoice number and I'll see if what you are trying to do is viable.

Thanks,
D.

 

See attached for "a.wav" which is read out for character "a", just in case this information helps... I'm going to try the playhead bias as soon as I can - will get back to you! thank you

 

The playhead bias being set to 0 seems to be a trick to getting this working...

the Update Delay 0.04 seems to be important too

Sample Size, I can't tell the difference between 1024 and 512.

The PlayHead at 0 doesn't make it perfect, but close! Occasionally the mouth doesn't move, but it's moving more often.

I have two types of speaking that I lip-sync:

1) making a noise per character (that is where the lip-sync had an issue)
2) a normal and full audio file of speech (which was working fine)

Should I be switching the playhead bias, update delay, and sample size dynamically as the spoken audio switches between 1 and 2 above? Is there an easy way to do this?

 

FANTASTIC! If it looks fine to you with the manual settings, you can simply leave them intact. However, the auto-adjust is designed to eliminate animation lag.

If you want to flip settings back and forth, the following should suffice. Auto-adjust wasn't designed to re-configure based on bias change, it was designed to change based on sample frequency change between audio clips. I'll make this available in the next update, but for now, you can capture the bias value beforehand and then restore it. Something like the following should be sufficient to swap back and forth. This will toggle, but you could just as easily specify the switch:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class FastAudioConfig : MonoBehaviour
4. {
5.     private int prevBias;
6.     private float prevDelay;
7.     public CrazyMinnow.SALSA.Salsa salsaInstance;
8.    
9.     public void ToggleSettings()
10.     {
11.         if (salsaInstance.autoAdjustAnalysis)
12.         {
13.             prevBias = salsaInstance.playheadBias;
14.             prevDelay = salsaInstance.audioUpdateDelay;
15.             salsaInstance.autoAdjustAnalysis = false;
16.             salsaInstance.audioUpdateDelay = 0.04f;
17.             salsaInstance.playheadBias = 0;
18.             return;
19.         }
20.  
21.         salsaInstance.autoAdjustAnalysis = true;
22.         salsaInstance.playheadBias = prevBias;
23.         salsaInstance.audioUpdateDelay = prevDelay;
24.     }
25. }

Of course, you'll want to adjust how you setup or capture your SALSA instance. Hope that helps,
D.

 

SALSA LipSync Suite v2.5.3.113
BugFix Release -- Available Now on the AssetStore

This is a small release to fix the prefab instance bugs some of you have seen lately. If you are not experiencing this issue, you do not need to upgrade to this version. If you beginning a new project, this is the version of choice.

EXPECTATIONS:
If you are upgrading an existing project, make a backup of your project before you upgrade SALSA Suite!

If you find bugs or documentation issues, please let us know, we will knock them out as quickly as possible. Please ensure you've checked the Release Notes and latest documentation and ensure you have the correct (latest) Add-On/OneClick versions. It is best if you email us (assetsupport@crazyminnow.com), including as much detail as possible, screenshots and/or video, any errors received, versions of everything, and always include your Unity Invoice Number.

Thanks and enjoy!
D.

 

Hi We have purchased the Salsa lipsync suite for Unity, but we needed better accuracy for lipsync as that is the main feature we are trying to implement, We see you have a sample

with high accuracy, can you let us know if its part of the new release and how can we access this.
Thanks.

 

Hello, this was an experimental prototype that we decided to not go forward with. We do not have any plans at this point to continue with the project.

Thanks,
D.

 

Hello, please note, an incompatibility has been discovered with the EmoterControl module and Unity 2020.LTS and 2021.Tech. I am trying to resolve the issue now and will post back here with an update/resolution.

Thanks,
D.

 

Thank you for the prefab instance fixes! And so fast! We will try it out this week and let you know how it went.

 

TimelineSalsaCore v2.6.0 released...

Hello everyone! I've released an update to TimelineSalsaCore (v2.6.0) to fix the incompatibility issues between the EmoterControl module and Unity v2020.LTS+. It should now properly allow emote selection.

The EyesControl module also got a new feature that came about while I was making the tutorial video for the Eyes module. It is now possible to filter specific settings from being applied in the clip which makes the Eyes clip module much more flexible. I'll be releasing a video tutorial early next week for the EyesControl module. Stay tuned.

NOTE: I'm calling this release beta till I get more feedback on it, but it is the latest stable.

Thanks,
D.

 

So is there any other way using Salsa that we can have close to accurate lipsync? like manually mapping phonemes to visemes?

 

We have quite a bit of documentation covering SALSA's capabilities and usage as well as videos specific to tuning SALSA to a wide variety of design looks and feels. If you haven't checked out the tuning videos, I would recommend them first and then check the documentation to dig deeper into features or terminology. As per the documentation and marketing material, SALSA is not a viseme/phoneme baking or mapping tool and produces its results using several procedural technologies.

D.

 

TimelineSalsaCore EyesControl tutorial released!

The video tutorial for the Eyes module of TimelineSalsaCore is now available. The add-on also received an update, as mentioned in the previous post on the subject, to version 2.6.0. If you have v2.5.0, please update to v2.6.0. It fixes a bug that prevented emote selection in Unity v2020.LTS+. It also includes a huge enhancement to the EyesControl module, introducing setting filters. These filters make the module much more flexible and easier to get the desired results. Watch the video for demonstration and explanation on filter use.

Enjoy!
D.

 

Looks great. Thanks for this. Will come in handy.
Questions. Can Salsa overide a bone based or Blendshape based Eye rotation being triggered via the Unity Timeline?

 

Hello,
All of the Suite's main modules run in the LateUpdate cycle in an attempt to override other animations/processes. Of course, if a conflicting process also runs in LateUpdate, that may not be the case and it may be necessary to coordinate script execution in Unity.

D.

 

I'm getting a jiggling type of motion happening with my Salsa driven eyes as they compete with the Animation Clip in the Timeline. I tried messing around with the execution order but this isn't my forte and I couldn't get the eyes to stop jiggling. Where would the CrazyMinnow.SALSA.Eyes execution go in the execution order?

Can you offer any other tips to get the SALSA Eyes working with my animation clips via timeline? I love the effect of Salsa Eyes and the option of triggering them via the timeline is great. If I can get this to work it would be awesome. Thanks.

 

It would be highly recommended to remove the conflicting animation, either by editing the animation and removing the transform animation or masking it, or (depending on where you got it) exporting the animation without the animation of the eye bones. There are a lot of possibilities depending on what you have going on. It is also possible the eyes are being animated via different transforms, which would require you to make edits as well. In other words, there are a myriad of things that could be animating your eyes (outside of Eyes).

You would want the SALSA Suite QueueProcessor to execute last so it overwrites everything else. But to be honest, you'd be better off removing your conflicting animation.

D.

 

We upgraded just for this and our prefab settings are saving faithfully!
I notice now that Eyes seems to have a setting that won't stay saved and so it's always in the overrides list.
We aren't using that module so it doesn't affect us. just a heads up

Thanks so much for the quick fix!

 

Awesome, thanks for the follow-up! Which setting isn't saving in Eyes? I'll see if I can get it fixed up for the next release.

Thanks!
D.

 

It looks to be the status warning about 'Queue Processor is available and linked but is disabled' that is not saving.
In this image you'll see that the warning is there in the old version (on the left) and is fixed in the new version (on the right).
Though, it could also be the 'Look at Target' assignment just below that too.

 

OK, thanks. That should just be an Inspector trigger. It is not actually a stored value but a computed helper message. We will take a peek at it to make sure. Thanks for the update and let us know if there is anything else you find. Good luck on your project!

D.

 

Hey,
Sorry if this issue has already been answered but I couldn't find a solution and I watched the different emoter tutorials.
I am trying to create an emote that is "Smiling" and I'm triggering this emote as a OneWay ManualEmote. The issue I am facing is that when the character starts speaking, the emote stops. I would like the character to be constantly smiling even when he is speaking and to not having to set the smiling again when he is done speaking. Is there a way to do this?

Also I saw you mentioned previously that the one clicks were created by taking a snapshot of the Salsa and Emoter configuration, I would love to have the possibility to take those kind of snapshots to create customized snapshots and presets of my configuration to easily set up Salsa with new characters.

 

Does Salsa support the new CC3+ explus blendshapes?
I mean in the one-click setup.

 

Doing a quick check, it looks like they have done the same sort of thing DAZ did with their Gen8.1 characters, separating them out into tiny core segments. I don't have a model to check against, but if they keep the previous shapes, they will still work. If they have replaced all of the shapes, they won't work. If that is the case, it will have to go on the list of todo's.

D.

 

Hello, I loaded up a model and it appears they have left the previous blendshapes intact, which is great news. This means the existing CC3 OneClick works with the ExPlus exports. As we have previously advised, it may be necessary to tweak the tongue settings in some visemes (mainly the 'th' config) -- it seems to be exaggerated in some CC3 models.

Happy SALSAing!
D.

 

Reallusion CC3 OneClick v2.5.1
Release employing Darrin's tweaks as implemented in the SALSA Tuning videos.

Added some shape tweaking (hopefully this will work better with more of the CC3 models' tongue shapes and morphs). Also added my own tweaks for emote emphasis and SALSA timing overrides. Very similar to how I tuned up the models in the SALSA Tuning videos. This also was checked to work with a model with ExPlus blendshapes exported. The original blendshapes remain intact (unlike the DAZ gen8.1 models), which is a good thing as fewer shape components are required for shape implementation.

Enjoy!
D.

 

EventController.AnimationStarting triggered for all network player, using version 2.5.3.113
Hello, I’m using SALSA LipSync and Photon Voice for a network game，A EmoteR event was configured on player prefab(see pic-1) and a script was attach to it (see pic-2), the player will play the emphasize animation when triggered event.There is a problem when local player triggered the event, the remote player will also notified by EventController.AnimationStarting.So one player triggered event, Everyone will start play the animation.I want’s the event just notify the speak-one, how can I accomplish this.

 

FYI for anyone else reading this post (the customer also sent us an email through which we have been corresponding). Basically, the idea of the EventController was to implement unique event component naming to sort through them. However, it would be beneficial and more flexible to be able to identify the sender and that will be implemented in the next SALSA Suite update. Also, for what @chaobaji was trying to implement, the AnimatorController would also have worked well.

D.

 

Hi @alexis78963_unity sorry I missed this post earlier.

OK, so this is something that has come up with other customers and the issue you are seeing is somewhat resolvable simply by setting the emote to be persistent. When it is persistent, even if SALSA overrides it, it will return to shape once SALSA is done. The reason it happens is because the shape is overridden by SALSA, including being reset. But, if you make it persistent, it will return once SALSA has released.

HOWEVER...it does also create a somewhat weird look when SALSA animates the same shape that is being used by the emote. In certain instances, it can "fluctuate" with the SALSA animation and this is usually undesirable. The good news is, I have a big update coming for SALSA/EmoteR that will implement (among other things) a new "Mood" expression type with its own queue that is the highest priority queue (other than blinks). This will prevent any other lower-hierarchy shape registrations from jacking with the emote. There is also a new dynamics feature for emotes (particularly for the Mood emotes) that give a nice subtle movement to a normally static face and once you've seen it, you can't imagine your characters without it.

I don't have a release date for the update yet, but I am hoping very soon.

D.

 

That is correct, I wrote my own little tool for doing so. It basically creates the OneClick file, but then I have to go in and edit it to make it more universal for multiple mesh/blendshape names. So it's not a completely automatic "save" system, but it does help. It is available as an unsupported tool on our downloads portal. But hold of on grabbing it if you intend to do so since I just made some updates to the tool and will post up the new version tomorrow. There are some customers successfully using and enjoying the tool right now. It is only for the SALSA and EmoteR configurations.

I'm also implementing some base-level changes in the OneClick editor to potentially allow for an easier plugin of CustomOneClicks to the existing editor calls. It's not a big deal, but it might make it a bit nicer for those creating their own OneClicks. Oh, and the next update of the Suite will include a SALSA visemes copy system where you can copy a viseme configuration from one SALSA instance to another assuming the models have the same hierarchy and shape naming.

D.

 

Thanks for reply

 

Hi,
in using Eyes script I have some problems with my model. If I set a local target and set the Update Delay to 0.07 and the head's movement is like the video:

https://drive.google.com/file/d/12fCjfL2Pw4jlJD3dafG3C3Ka9R6YL7sx/view?usp=sharing

If I set Update Delay to 0 it stops... But SOMETIMES with Update Delay to 0 the head's model don't looks at the local target anymore!

Any help??

Massimo

 

Hello! It appears you have another process in contention with the SALSA Suite head animations. Please check the Eyes module documentation for disabling SALSA Suite processing of the head animations or remove the conflicting animations. Only one process can control the animations, it will be necessary for you to make a decision on which one you want to control your head movements.

Hope that helps,
D

[edit] Additionally, it is generally discouraged from setting the delay too low (<0.07f) since it does not give animations a sufficient amount of time to animate before being told to animate to another position. 0.07 should be your bare minimum delay. If you set it to 0, a new registration will occur every frame and the QueueProcessor will not have any time to calculate and animate in a direction.

 

Hi, thank you for your answer, but I find the right solution (I had no other animation on my model).
I had to order the components in the right way: first SALSA, then Eyes, then Queue processor. If I put the Queue processor as first component I have the head flickering.

Massimo

 

Interesting, that technically has no bearing on operation, so it is a curious issue. At any rate, it is good your problem is resolved.

D.

 

Hello, I'm bringing Ready Player Me .glb files successfully into a Game Creator Unity environment at runtime.
I'd read that applying SALSA profiles are possible at runtime. What might your advice be in this scenario, is it possible?
Thank you and Good Health!

 

Hello,
I don't really know anything about Ready Player Me, but here is a link to our requirements and recommendations:
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/requirements/
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/recommendations/

And here's the documentation for the Custom OneClicks (I'd also recommend reading about OneClicks in general):
https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/custom-one-clicks/

Hope that helps,
D.

 

DAZ3D OneClick v2.5.3
Release supporting DAZ Genesis 8.1 models and employing Darrin's tweaks for all DAZ generations as implemented in the SALSA Tuning videos.

Wow, I think I finally have this DAZ OneClick under control, get it on our downloads portal:

• Added support for Genesis 8.1 models.
• Fixed a lot of DAZ Legacy bugs.
• Tweaked the legacy Genesis OneClicks to implement new SALSA and EmoteR settings as demonstrated in the "Tuning SALSA" Youtube videos.

I would recommend this OneClick as the go-to for DAZ and update to it if you are using older versions. There were a lot of issues I found in the old version that produced lop-sided results due to shape search issues. I think I've wrangled them all in to a nice and orderly OneClick compilation. As with all other updates, backup your projects before using any new component or asset.

Enjoy and good luck on your projects!
D.

 

Hello!

So, in my previous workflow, I used the Adobe Fuse characters, with the blendshapes generated by Mixamo. Unfortunately, this doesn't seem to be supported anymore.

Is there a way around it, without changing the Fuse models? Some software to generate the belndshapes?

Thanks!

 

I'm using SALSA Version 2.5.3.113 with a Character Creator v3.32 and the blink processor seems to not work. I've tried a few different set ups with the eye lids and I'm able to get to blink in preview or running in the editor when I prese the Generate button in the Blink Properties section. I don't see any blinks in the queue either. Please advise.

 

Hi @chuckyluv869 please send us an email with more details about your configuration, i.e. screenshots or video (preferred) showing your config and the issue. Honestly can't think of why 'generate' would work but the blinks are not occurring on their own unless somehow 'random blink' is not enabled? Anyway, send us an email and we'll see if we can help you figure it out.

D.

 

Hello, im trying to use a Daz gen 8 model with Salsa and the Daz One click (latest version), but there is no ONE CLICK menu for Daz



note: i added the one click from the website with the invoice.

 

Hello, OneClicks are added from the GameObject menu, not the Component menu.
https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/

Good luck on your project!
D.

 

TimelineSalsaCore v2.6.2 released...
Specifically addresses the EyesControl LookTarget assignment.

Since the LookTarget reference is a scene object, there was an omission in the EyesControl module that prevented the LookTarget assignment from saving and any populated reference would be lost as a result of scene or Unity loads. This field needed to be converted to an ExposedReference lookup and is thereby managed by the Unity engine in the drawer display. This created a bit of an issue with associating other settings with the LookTarget and the Inspector had to be modified in an attempt to re-associate the behaviors. As such, the Affinity setting is now displayed by default instead of only when a LookTarget was populated. NOTE: Please remember the Affinity setting only affects processing when the underlying Eyes module has a non-null object assigned to the LookTarget slot.

Old EyesControl Inspector:


New EyesControl Inspector:


Also note, some descriptions and design of the Inspector have been modified in an attempt to clarify intent. For example, "filter" was changed to "exclusion" to better indicate a particular setting is included or excluded from processing. When the exclusion button is enabled (yellow), the associated setting is *NOT* processed in the Timeline clip.

Please replace version 2.6.0 with version 2.6.2 if you are using the EyesControl track/clip for Timeline. There are no other changes to this version, if you are not using EyesControl, you can ignore this update.

Hope this helps!
D.

 

hi I am using I clone CC3 characters and setting them up with on click and the text sync

however there lip flaps are extremely fast even with the word per minute reduced to 10 the character speeds trough the lip flaps.

how can I slow that down?

btw it speed trough the promo sound as well
and I am using cc3+

 

quick fallow up question before I go to bed

when I try making a character change from looking at some one to looking freely again.
x.view.lookTarget = null;
I get a null ref that brakes my script?
hew the hell do I clear it safely ?
view is a ref to CrazyMinnow.SALSA.Eyes

 

Hi CristalSonic,

Setting the lookTarget property to null is correct and shouldn't through a null under normal conditions. Please send an email to assetsupport@crazyminnow.com and include your invoice number, the error messages you're receiving, and more context about your setup. What are you using, have you made any settings changes, which add-on's, etc.

Thanks,
Michael