SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

will do,

any tips for my first question?

 

Hello,
We would need to see your configuration (screenshots) and a video of what you are seeing to make sure we are on the same page to offer suggestions. Please send that with your email. If you haven't already, we also recommend watching our Tuning SALSA video set.

Thanks,
D

 

I did not know there was a Tuning SALSA video set is that on YouTube?

 

Yes, there are quite a few video tutorials available for the Suite on our YouTube channel. Most of them are complementary to our online documentation.

 

hello, can I use salsa just for the lip-syncing? I don't need the eyes control nor the look at options, I'm using an iClone character with custom blend shapes.

 

You can use whichever combination of modules your project requires. Please see the online documentation for more details on recommendations and requirements.

Thanks,
D

 

Hello.
Does Salsa v2 work when rendering timeline out as video using Unity Recorder?

Thanks
James.

 

Hello,
SALSA's Timeline implementation triggers SALSA Suite functionality, therefore it would not work in a rendering fashion. You would have to record all of the resultant blendshape, bone, etc. values on your models during the Timeline playback.

Hope that helps,
D.

 

Thank you for the reply. If I understand you correctly then this sounds like a very good solution to my needs.

Kind regards
James.

 

I want to check the operation of Salsa Control in Timeline without running the game
I added [Execute Always] to the script, but it didn't work

unity2019.3
timeline 1.2.1.7
Salsa 2.5.3

 

The first emote of timeline emoteR control works, but the second emote does not work
If you put two one way emote Clips side by side as shown in the image, the first one will not work, but the second one will work.
I'm using SalsaControl EyesControl.


SalsaControl settings
Enable Modules
Blink
Lids

Enable Random
Blink

Reset on clip end ON


EyesControl settings
Override Dynamics OFF


unity2019.3
timeline 1.2.1.7
Salsa 2.5.3

 

Hello, the SALSA Suite is not currently designed to work in this mode. There is however, an editor-based configuration preview that is available for SALSA that works with Editor mode to preview lipsync at design-time. This preview will also drive Emphasizer emotes from SALSA.

Thanks,
D.

 

Hello,
I have created a round-trip emote followed by two one-way emotes (all using the same emote as per your setup). I cannot reproduce the behavior you are describing. The round-trip fires and returns (as intended). The first one-way fires (as intended) and since it is already fired, the second one-way does nothing (as intended). Please send us an email with your Invoice number and a short video demonstrating what you are seeing.

Thanks,
D.

 

I've been getting this error and I can't find what the error is referencing and I'm hoping to get some help. The error is:

NullReferenceException
CrazyMinnow.SALSA.Eyes.DestroyProcTransOffsetGizmos (System.Collections.Generic.List`1[CrazyMinnow.SALSA.ProcTransOffset]& list) (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)
CrazyMinnow.SALSA.Eyes.Initialize () (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)
CrazyMinnow.SALSA.Eyes.Awake () (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)

I've gone through all settings in all of the SALSA components several times and I can't find any missing references.

I'm also getting this error as well:

NullReferenceException: Object reference not set to an instance of an object
CrazyMinnow.SALSA.Salsa.SalsaLssGetAudioClipSampleCount () (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)
CrazyMinnow.SALSA.SalsaAdvancedDynamicsSilenceAnalyzer.ProcessAudioStream () (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)
CrazyMinnow.SALSA.SalsaAdvancedDynamicsSilenceAnalyzer.LateUpdate () (at <c6912b865df54414bfb4f0cf1b09ee9c>:0)

I don't see any missing references for this either but I am using the TextSync componenet in my project. I plan on using voice overs and muting the output (if possible) instead of using TextSync. I assumed this error was related to using the TextSync component.

Any insight or help getting rid of these errors is greatly appreciated.

Thanks!
Chuck

 

Will have to let Mike weigh in on the Eyes, go ahead and send all of your particulars to the support email for that one. Versions, invoice, etc.

For the SALSA issue, take Silence Analyzer off of your model object. If you are not using audio, you don't need it. It should disable itself if you are on 2.5.3, so that is odd -- although we did recently discover an issue in 2019.LTS where there is a single NULL ref thrown when there is no audio to process. The way Unity initializes audio allows a pass through the code for a frame when it should not. Perhaps that is the issue. I haven't tracked it down yet, but I should have that fixed with the next release. Regardless, you don't need it if you are using TextSync.

Hope that helps,
D.

 

That definitely helped. It got rid of the errors related to the Audio Clip Sample Count! Thank you!

I sent the other error and relevant info to the support email.

Thanks again!

-Chuck

 

I'm coming over from the Lip Synch Pro group since I heard about SALSA there by a couple of users here.
I've run into an issue in my first week of using SALSA. The eyes on my character model (blend shape) are jittery.
My work around right now is to flip off eye component and then back on, and things work just fine after that.
I sent @Crazy-Minnow-Studio a support email request. Not sure if you can offer a solution.

So I'm using Unity 2021.1.23f1, and Windows 10.

SALSA, EmoteR, Head config, and Eyelid Config are all working great. Its just the eyes that are shaking up and down that is the problem. As I mentioned my work around is using Playmaker FSM to flip off and back on the Eyes component. Then things work fine for the rest of the timeline sequence. It happens fast enough the player does not notice it. I think things are setup correctly with eyes. I'm fine with this workaround as there are no performance issues. Just think there might be a better way to do this to stop the eye jitteriness without flipping off and on the eye component.

Having been a long time SALSA users, I can say that my workflow is much faster with SALSA. But its not as good with the mouth shapes as Lip Synch Pro which is now deprecated but became open source. For me at this point in my project/game time is the most limiting factor. So I'm going with SALSA. It has a lot more capabilities with the head, and eye components and scrubbing through my timeline is very nice both in the editor and in play mode.
Well back to animating, and voiceovers which is what I'll be doing for the next three months! Any advice or tips on the jittery eyes on other solutions would be greatly appreciated.

Thank you.

 

Hi Chuck,

As mentioned in my email response, I believe this may be a bug. I'm working to confirm and correct it.

Michael

 

Hi P_Jong,

I responded via email with a few things to try and some additional information gathering to helps us identify the source of the jitter.

Michael

 

Dear Orbatrix,
I want to do exactly the same with UMA avatars in a multiplayer environment using PhotonVoice2, but since I lack experience coding I think you lost me in the last point. So, I want players to see my avatar lipsync when I speak. I'm using a networked prefab containing the UMA avatar, the Photon Voice View script, the Salsa script (one click-added), and the Upstream Audio Filter Processing script with the networked prefab as the Salsa instance. Salsa, automatically linked this same networked prefab as the audiosource. The Photon Recorder is in another game object in the scene using the Unity Microphone.
Of course, I am doing something wrong because it doesn't work. I saw that you enabled "external analysis" in the audio source: how did you do that? I understand that you made it work after realising that "the SALSA component reference inside the script wasn't correctly loaded in play mode" and then you fixed it. What Salsa component? what specific script and how did you fix it?
Sorry for all these roockie questions. I will greatly appreciate any orientation you can provide.

 

I'm on Unity 2020.3.22f1. I have only the EmoteR and Queue Processor. asset is last version.

I get a lot of Editor.EmoteR.OnInspetorGUI messages each time I select the object that have these scripts attached.

There are 5 emotes set correctly (works in game mode, all good)

I've opened and close the editor a couple of times and only once I didn't got this message showing up every time and the editor was smooth. So i don't know what could be. Everything the same but one time it worked the rest of the times no.

Same thing a couple of months back on a previous unity version and with 15+ emotes of the same complexity had no issue. and the editor was not hanging up.

It is very difficult, near impossible to change anything in the properties panel as every time one moves the mouse the message pops up for 2 3 seconds up to 10 seconds.

Do you know if maybe there is some bug from unity or something, can you replicate the issue on your side?

EDIT: so the curios thing is that if I enter the game mode I can select the object with the emoter and the interface is working as it should, there is no delay, I can collapse or show all various values in the emoter.

so in editor outside of gamemode doesn't work, while in game works perfectly. go figure.

 

Hello @dorusoftware,
The behavior you report is very odd. This is the first time we have heard of this sort of issue and I cannot reproduce this on my side. I have tried 2020.3.16 and 2020.3.22 and EmoteR is working without error and without issue. However, it is possible you have discovered a setting configuration that has surfaced an interface bug somewhere. Please send all of your details to our support email address and we will get to the bottom of this. Please include your invoice number, OS version, complete screenshots of your configuration and error notifications and video demonstrating what you are seeing at design and when switching to runtime.

Thanks,
D

 

I've found an explanation.

The character is a prefab and had the emoter component in the prefab with no emotes set up. In the scene the emoter prefab had the emotes set up.

So there was a difference between the prefab content which had no emotes and the prefab in the scene.

I've applied the changes from the scene to the prefab so the emotes had been transferred inside the prefab. Now the emoter component is the same inside the prefab and in the scene.

Everything works.

I've then deleted one of the emotes in the component in the scene. So the emoter is once again different. The error returned.

I've reverted the changes on the prefab.

works

there are 5 emotes with an average of 50 elements inside for reference if you want to test. I would be glad if you can confirm this, if you have time or want to do the test.

as long as the emoter have the same configuration in the prefab as in the scene everything should be fine.

 

Hello @dorusoftware,
I converted my object to a prefab and then made alterations to the instance and I cannot reproduce the issue you are describing. Please send all of your details to our support email address to further troubleshoot the issue. Please include your invoice number, OS version, complete screenshots of your configuration and error notifications and video demonstrating what you are seeing at design and when switching to runtime.

Thanks,
D

 

Thank You for verifying, I will do this when I will have some time, although I'm not saying this is related with the plugin, just an odd behavior I've observed when using the asset. Probably unity is struggling with complex UI's on the properties panel.

As long as I keep the setup so the asset match is fine by me.

 

Greetings, I put a Ready Player Me (RPM) .glb model through Blender to Reallusion 3dXchange/iClone to FBX to Unity (yes, I'm figuring it out haha) and used the iClone on-click to mostly set up the model.

I made my own visemes and emotes. However the viseme Trigger sliders seem locked, I can't change the values. The visemes' preview works ok.
I could slide them to zero during one session. I close, reopen Unity, then Target sliders and input fields freeze up (value changes for a split second then reverts).

Might there be something I'm missing about the sliders?

Unity 2019.4, Windows 10, latest SALSA and one-clicks.

 

Hello,
The triggers lock within their neighboring trigger values. I recommend using one of the auto setting options (curve or linear) to get them in a good starting position to manually adjust if necessary. See this page in the manual: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/using/#trigger-display-mode

D.

 

That worked! Thank you.

 

This looks interesting, but where is the asset's intro video? I can't see any on the asset store, and looking over the creators youtube videos, I see only tutorials, and have yet to see it playing after several minutes of searching and seeking tutorial videos...

 

Yes, you are correct, we don't yet have a demo video -- it is in the works. The tuning videos do have a lot of demonstration of the different looks achievable by the product.

This link starts with a DAZ model and goes through some tuning.
DAZ model

and here is a CC3 model:
CC3 model

Hope that helps!
D.

 

@justaddice83 - If it's any help, here's a video of SALSA in action with the UMA one-click setup. Jump to 09:00:

It's the default setup without any tuning. It was literally just one click and go. Setup couldn't be easier. A bit of tuning could make the lipsync even more accurate, but the purpose of this particular video was to explain how to import Celtx content into the Dialogue System for Unity, not as a SALSA demo per se.

 

Hi I use salsa with CC3. (one clic setup)
how can avoid the head move (neck move)? where i can configure this?

Thanks

 

Hi @claudius_I you can disable Head processing in the Eyes component. Please see the documentation for more details. https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/eyes/using/#head-configuration

Hope that helps.
D.

 

Hello,

Trying to set up an UMA character but I can't find the SalsaUmaSync.cs script. I've installed the Base OneClick add-on and the UMA addon but the script is not in my project.

Using the latest releases of Salsa and UMA.

Thank you,
Nathan

 

Hello @nathanjams SALSA Suite's UMA OneClick does not use a SalsaUmaSync script. The legacy version of SALSA did have a sync script.

Hope that helps,
D.

 

Just wanted to say WOW what a great time to be an indy.. Salsa is awesome.. ty!

 

A new update of Amplitude for WebGL (version 1.2.2) was just submitted to the Asset Store.
Unity 2021+ is now supported!

This update includes the following:

• Chg - Dual builds added to address WEBAudio framework changes. Legacy build designed for Unity 2019.4.34f1 through 2020.3.12f1. Modern build designed for Unity 2020.3.25f1 through 2021+.
  
• Fix - Replaced all deprecated references to Pointer_stringify with UTF8ToString.
  
• Fix - Fixed a bug in an analyzers index reference.

 

Hello I'm trying to use a OneClick for runtime configuration using a method like this:

Code (CSharp):

1. public void OneClickSpanishSetUp(GameObject gameObject, AudioClip clip)
2.     {
3.         OneClickNatiSpanish.Setup(gameObject, clip);
4.     }

It runs at the start of the application as it depends on the player language preference. But I got this error:

NullReferenceException: Object reference not set to an instance of an object

CrazyMinnow.SALSA.QueueProcessor.Register (CrazyMinnow.SALSA.ExpressionComponent expressionComponent, System.Boolean isActivating, System.Single frac) (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)

CrazyMinnow.SALSA.Salsa.RegisterComponentOn (CrazyMinnow.SALSA.ExpressionComponent component, System.Single fracAmount) (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)

CrazyMinnow.SALSA.Salsa.TurnVisemeComponentsOn (System.Int32 triggeredIndex, System.Single fracAmount) (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)

CrazyMinnow.SALSA.Salsa.TurnVisemeOn (System.Int32 triggeredIndex, System.Single dynamicsAmount) (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)

CrazyMinnow.SALSA.Salsa.LateUpdate () (at <185af4eefb1d4b8d94ad89e69aa201e4>:0)

It appears two times in a row. And the documentation doesn't say much.

 

Hello @Maikohen it looks like you did not bake the controller data as mentioned in the documentation a little further down the page.

Hope that helps,
D.

 

My bad! Thank you so much!

 

@Crazy-Minnow-Studio I'm working on a new project where I want to use UMA and Salsa. However my UMA character when edited will be able to swap between multiple different heads, each with very different topology, eyes and mouth. Ideally I'd like to be able to mix 2d, bone and blendshapes on different heads, but I could live with just blendshapes (assuming all the heads would have the same set).

Is this possible?

 

Hello @Dorian-Dowse
SALSA Suite works with multiple different controller types. Please see the list of supported controllers in our online requirements documentation.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/requirements/

SALSA and EmoteR are capable of mixing any type of controller within a single viseme/emote. Eyes has some restrictions due to the the template and calculation requirements for different controllers.

If you are swapping out heads, you may have some runtime reconfiguration to do remapping the resources. That all depends on your implementation. If you are actually putting blendshapes on your UMA characters, you would likely want to create your own OneClick or run-time-configuration for them. The UMA OneClick uses the ExpressionPlayer. You can read more about our UMA implementation in our online documentation.
https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-click-uma/

Good luck on your project!
D.

 

EDIT: Just found this: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/further-reading/runtime-setup-uma/

Beauty! Thanks

Is it possible to apply OneClickUma in runtime?

Say we change the head of our character and want to remap the objects to Salsa, can we just call OneClick through code and have it auto fill?

 

Yes...I do it for my UMA's

I found I had to make a minor change to the utility scripts and and the eye portion on the relinking

Code (CSharp):

1. using UMA.CharacterSystem;
2. using UMA;
3. using UMA.PoseTools;
4. using UnityEngine;
5. using CrazyMinnow.SALSA;
6. using CrazyMinnow.SALSA.OneClicks;
7.  
8. public class SalsaUMAUtil : MonoBehaviour
9. {
10.     private Salsa salsa;
11.     private Emoter emoter;
12.     private Eyes eyes;
13.     private UmaUepDriver driver;
14.     private QueueProcessor qp;
15.     private DynamicCharacterAvatar _umaDynamicAvatar;
16.     private UMAExpressionPlayer expPlayer;
17.  
18.     private void Awake()
19.     {
20.         salsa = GetComponent<Salsa>();
21.         emoter = GetComponent<Emoter>();
22.         eyes = GetComponent<Eyes>();
23.         driver = GetComponent<UmaUepDriver>();
24.         qp = GetComponent<QueueProcessor>();
25.         _umaDynamicAvatar = this.gameObject.GetComponent<DynamicCharacterAvatar>();
26.         expPlayer = this.gameObject.GetComponent<UMAExpressionPlayer>();
27.     }
28.  
29.     public void InitialStart()
30.     {
31.         driver.ManualStart(expPlayer);
32.     }
33.     // Called by some function or functionality in your application to 'change'
34.     // the UMA avatar.
35.     public void ReconfigureRunningUma()
36.     {
37.         Logging.Log("SuiteRelink: Initiated.");
38.         DisableSalsaSuite();
39.     }
40.  
41.     public void RestartSalsaForCharacter()
42.     {
43.         UMAData umaData = _umaDynamicAvatar.umaData;
44.         RelinkSalsaSuite(umaData);
45.     }
46.     // This method should be called by your UMA callback...i.e. CharacterUpdated, etc.
47.     // NOTE: It requires the UMAData structure to complete the re-setup of your avatar.
48.     private void RelinkSalsaSuite(UMAData umaData)
49.     {
50.         eyes = GetComponent<Eyes>();
51.  
52.         Logging.Log("SuiteRelink: Re-linking.");
53.         // re-enable SALSA Suite
54.         salsa.enabled = true;
55.  
56.         emoter.enabled = true;
57.  
58.         if (eyes != null)
59.             eyes.enabled = true;
60.  
61.         // OneClick runtime setup...
62.         AudioClip clip = GetComponent<AudioSource>().clip;
63.         OneClickUmaDcs.Setup(this.gameObject, null);
64.         OneClickUmaDcsEyes.Setup(this.gameObject);
65.  
66.         // reconfigure the UmaUepDriver...
67.         driver.CharacterCreated(umaData);
68.  
69.         // Bake Expression controllers for SALSA and EmoteR...
70.      
71.         salsa.UpdateExpressionControllers();
72.      
73.         emoter.UpdateExpressionControllers();
74.         emoter.UpdateEmoteLists();
75.  
76.         // Bake Expression controllers for Eyes...
77.         // NOTE: We don't call Eyes.Initialize() here...already applied
78.         //  in the UmaUepDriver call above.
79.  
80.         if (eyes != null)
81.         {
82.             eyes.UpdateRuntimeExpressionControllers(ref eyes.heads);
83.             eyes.UpdateRuntimeExpressionControllers(ref eyes.eyes);
84.             eyes.UpdateRuntimeExpressionControllers(ref eyes.blinklids);
85.             eyes.UpdateRuntimeExpressionControllers(ref eyes.tracklids);
86.         }
87.     }
88.  
89.     private void DisableSalsaSuite()
90.     {
91.         Logging.Log("SuiteRelink: Disable Suite components.");
92.  
93.        
94.         eyes = GetComponent<Eyes>();
95.  
96.         // disable Suite components to prevent writing to queue.
97.         salsa.enabled = false;
98.         emoter.enabled = false;
99.  
100.         if(eyes != null)
101.         {
102.             eyes.EnableAll(false);
103.             eyes.enabled = false;
104.         }
105.         // clear QueueProcessor
106.         qp.priorityQueueHeads.Clear();
107.         qp.priorityQueueEyes.Clear();
108.         qp.priorityQueueLipsync.Clear();
109.         qp.priorityQueueEmotes.Clear();
110.         qp.priorityQueueBlinks.Clear();
111.         qp.priorityQueueLids.Clear();
112.     }
113. }
114.  

 

Hello, I see you found the runtime UMA doc. Be aware, the document and our UMA OneClick in general are designed for a standard UMA ExpressionPlayer scenario and may not completely apply to your project. Regardless, runtime setup is possible in nearly any situation. If you are placing blendshapes on your UMA models, they won't necessarily require the UMA OneClick setup (at least not completely). But, you may need to create your own OneClick or runtime setup. UMA is a fairly complex system that can be used in many ways. Since it is not possible to know or predict how designers are using the UMA system, our OneClick was designed more as a starting point or example of what can be done.

Hope that helps,
D.

 

SALSA LipSync Suite v2.5.4.119
BugFix Release -- Available Now on the AssetStore

This is another small release to primarily fix an issue where runtime configuration of Eyes gizmos could cause a null-ref. There are a couple of other changes you might find interesting in the Release Notes. If you are not experiencing these issues, you do not need to upgrade to this version. If you are beginning a new project, this is the version of choice.

Additionally, this is the first release of SALSA LipSync Suite that requires Unity stable/release versions 2019.LTS+.

EXPECTATIONS:
If you are upgrading an existing project, make a backup of your project before you upgrade SALSA Suite!

If you find bugs or documentation issues, please let us know, we will knock them out as quickly as possible. Please ensure you've checked the Release Notes and latest documentation and ensure you have the correct (latest) Add-On/OneClick versions. It is best if you email us (assetsupport@crazyminnow.com), including as much detail as possible, screenshots and/or video, any errors received, versions of everything, and always include your Unity Invoice Number.

Thanks and enjoy!
D.

P.S. -- I have also added the previous version to our downloads portal if you need to revert back for any reason.

 

Hello,

Struggling to set up Salsa with our custom UMA rig using the bone option. We have made a custom UMA righ that is based closely on the default UMA female fbx (same transforms).

In the simple setup demo scene I can play the default avatars fine after running it through the One Click process to set up Salsa. However, when I select our model the mouth shapes look a bit weird. Any chance you get any idea of what could be wrong from this video?

I should add that the face is a wardrobe item. The base model's face looks like the rest of its body, the face you see is, again, a wardrobe item.

Thank you
Nathan

 

Hi,

It's really hard to guess what's going on, but it sort of looks like maybe you have the min and max captured values flipped on one side of the mouth versus the other. The mouth left and right sides seem to move in opposition of each other. The best way to troubleshoot this would probably be the editor preview, so that you can see that the transforms are moving as you expect from min to max.

Michael

 

Hi, I used TimelineSalsaCore V2.6.2, and I found that I could not only play a part of the audio clip, I could only play the whole audio in the timeline, which was very inconvenient for me to align the audio track, just like the picture. When the audio clip in Unity played, You can freely push the clip so that it only plays one part of the clip, and the SALSA control provided by SALSA can only play the full length. Could you please solve this problem?

 

Hello @MagicK47. If I understand what you are asking, you are correct, the TimelineSalsaCore add-on does not offer that feature. You cannot modify the start and end points of the selected AudioClip on the SalsaControl. It was created to allow developers the basic ability to use Timeline to swap out AudioClips in SALSA. SALSA cannot read the default Timeline Audio implementation due to the fact Timeline Audio bypasses the standard Unity AudioClip implementation where SALSA gets the data. If you need the functionality you describe above, you can use the normal Timeline Audio track/clip but you will need to implement an audio filter reader to supply the data to SALSA.

You can read about how to implement something like this here:
https://crazyminnowstudio.com/docs/...alsa/delegate-processing-custom-filter-chain/

It is also recommended to have a pretty good understanding of Unity's filter chain implementation to understand what is happening and what the requirements are for that sort of scenario.

Hope that helps!
D.