SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Thank you.
I will take a look to SalsaSync add-on.
I sent you the model at AssetSupport@crazyminnow.com
Thanks
Max

 

@Crazy-Minnow-Studio
Hi,
i am using your salsa lip Sync package and it was working fine until i updated my unity project to 5.6.2f1 and i also updated the RT Voice package to its newest release 2.8.5. Now lip syncing is not working when Character speak.
but when i am setting emotions manually then emotions works fine.
Can you please look into this matter...
Thanks

 

Hello AR_Rizvi,

Please send your SALSA invoice number to assetsupport@crazyminnow.com.

Which RT-Voice mode and API method are you using?
Are you using our RT-Voice add-on (which version)?

SALSA works with RT-Voice standard mode natively, and functionality is demonstrated in our RT-Voice add-on to work with the RT-Voice Speaker.OnSpeakCurrentViseme and Speaker.OnSpeakCurrentWord events.

As the RT-Voice API is updated, we also update our add-on to accommodate changes, please be sure you're using the latest version of each system.

Michael

 

Hi salsa team,

I have MCS female character which is working with RT voice. My problem is my character lip sync are not good and unrealistic please check this. and guide me what to do?

Looking for you reply
thanks.

 

Hello MFKJ,

Thanks for the video, that definitely demonstrates something isn't working right. Can you send your SALSA invoice number, as well as the following details to assetsupport@crazyminnow.com, and we'll look into it right away.

• SALSA version
• RT-Voice version
• Unity version
• Are you using RT-Voice in standard or native mode?
• Are you using our demonstration files add-on from here: https://crazyminnowstudio.com/posts/salsa-and-rt-voice-for-runtime-lipsynced-text-to-speech/?

Thanks,
Michael

 

Hi, Crazy-Minnow-Studio,
It seems that 1-click setup for MCS 1.6 is out-of-date. Blinking doesn't work at all, eye movement is unrealistic. All this is checked in an empty project with last available verstions of assets.

Unity 2017.1.2p2
Salsa pro 1.5.1
1 click setup 1.4.0 for MCS 1.6+
MCS 1.6.4

 

Hi Alex-3D,

It looks like MCS is still on version 1.6.4, which is the version the 1-click setup was checked against. Please save, close, and re-open your project, then select the button near the top of the CM_MCSSync inspector to remap the BlendShape indexes.

Does this correct the issue?

Thanks,
Michael

 

Yes, it helps! Thank you.
Do you plan to release an update of this MCS' 1-click-setup add-on?

Thanks,
Alex

 

Hi Alex,

We typically update our add-on's when SALSA updates, or the target system, require it. Is something still not working?

Thanks,
Michael

 

Hi, Michael,

Well, that's my opinion as assets's user:
1. It would be very nice to exclude <possibility of> this save-close-reopen step. Or document it in the Readme file.
2. It would be nice to automate LOD switching.

Also as the user I want to express my gratitude for the stable and fast support of your product for many years.

Thanks
Alex

 

Hi Alex,

I certainly understand and agree with the desire to not have to deal with the save, close, and re-open step. However, it's actually a symptom of how MCS saves and maintains it's Blendshapes. When you activate BlendShapes on a MCS character, the indexes are assigned incrementally in the order you add them, but when you close and re-open the project the BlendShapes are lost and added back to the character in a predetermined order specified by the MCS system. We added the remap BlendShape button to the CM_MCSSync component to address this reordering of Blendshape indexes.

Automatic LOD switching is already implemented in the current CM_MCSSync component. When you switch LOD's using the MCS tools, the MCS component fires an LOD switch event that the CM_MCSSync component subscribes and responds to, to remap SkinnedMeshRenderers.

Thanks,
Michael

 

Hello,

I am working on my In-Game Talking Scene. I is getting very cool and looking like this:



The left head is the Player, it is using UMA (just head Slots) and working perfect, the right is the NPC also UMA, both from a head prefab I made.

The Player talks and moves lips, the npc does not move, it Looks static.

I debugged, the salsa is there. For both it Looks like this (with different Audio Clips)::



When I press Play on Player, it moves lips, when I do on the npc it does not move.


Any idea? Thanks a lot!

 

Hi,

With the game running in the editor, select the character that isn't working, find the UMAExpressionPlayer component and move the Jaw Open_Close slider, does the mouth open and close? If not, there's probably an issue with your characters bone rig.

Michael

 

I found the slider, nothing happens. strange thing, error is somewhere else, thanks!

 

We're excited to announce MorphMixer, our upcoming new asset (Submitted to the Unity Asset Store).

MorphMixer is an editor extension for the Unity3D game engine. It allows you to create new BlendShapes by combining one or more existing BlendShapes within the Unity editor. Whether you're using one of the many character generations systems, contracting out your characters, or designing your own and simply want the convenience of mixing up new BlendShapes from existing ones within the editor, MorphMixer is a simple, non-destructive, in-editor tool designed specifically for the task.

Character generation systems such as Fuse, iClone, Autodesk Character Generator, Daz3D, and MakeHuman include many BlendShapes. Some of these system provide phoneme shapes for speech while others do not, some provide emotion BlendShapes while others provide basic facial movements. MorphMixer allows you to mix and match existing BlendShapes together to form the perfect smile, frown, or other expression and save your mix as a new BlendShape. This simplifies animation by allowing you to merge down complex multi-shape expressions into a single BlendShape, or you can add speech phonemes to a character that only includes basic facial movement BlendShapes.

MorphMixer is non-destructive because it duplicates your mesh and performs all BlendShape actions on the duplicate only, so you can always fall back to your original .fbx or .blend mesh file. It also includes a preset system so you can create BlendShape mixes, save them as presets, and easily apply them to multiple characters that share the same BlendShapes and BlendShape indexes. If you create a BlendShape, then change your mind about the way it should look, MorphMixer allows you to delete BlendShapes as well.

We don't want to clutter this thread, so if you're interested or have questions about MorphMixer, head over to the MorphMixer forum, or check our website for more information.

MorphMixer Unity forum
https://forum.unity.com/threads/morphmixer-create-new-blendshape-mixes-in-unity3d.504283/

MorphMixer website
https://crazyminnowstudio.com/unity-3d/morphmixer/

The Crazy Minnow Team

 

Hello again,

I did not find the error yet. Both look good and throw no Errors or something.
But the playerhead does move his eyes and mouth, but the talkerhead does not move anything it is totally static.



The only difference I see is the not-working Talkerhead has a sub-component RandomEyes3D_Eye_Position which was not in the prefab, it was added automatically. Any idea why?

In CM_UmaSync I found this line
randomEyes3D.FindOrCreateEyePositionGizmo (); // Add the RandomEyes
but I cannot debug into.

Maybe you can give me a hint what's wrong? Thanks a lot!

 

The RandomEyes3D_Eye_Position gizmo is used by RandomEyes as a point of reference for eye tracking calculations. RandomEyes instantiates this object and attempts to find the eye bones and parent the RandomEyes3D_Eye_Position gizmo to the head right between the left and right eye bones. I say "attempts" because there are so many armature variations from one model to the next that we cannot guarantee we'll find the eyes while traversing the armature. The FindOrCreateEyePositionGizmo() method looks for an existing RandomEyes3D_Eye_Position gizmo and creates one if it does not exist.

The UMAExpressionPlayer sliders drive facial bone animations on your UMA, if these sliders aren't working it likely means you have an issue with your characters rig. I would recommend working to resolve whatever is interfering with proper UMAExpressionPlayer functionality, then move on to SALSA since the CM_UMASync script is only acting to translate BlendShape values to UMAExpressionPlayer slider values.

Michael

 

MORPH 3D charter eye blinking issue.
Hi, I followed the tutorial here : http://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/
I'm using the free mcs female model.

I used the SALSA 1-Click MCS Setup, the lip syncing is functional. Only problem here is eye blinking. When blinking, Sometime the charters eye only half open (looks like the charter is sleepy), I noticed in the generated Random eye 3d(Script), all the Eye Shape Indexes values are eCTRLEyesClosed, does it have anything to do with the issue?


any help would be appreciated.

 

Hello todd85,

There should be one instance of Salsa3D, and two instances of RandomEyes3D after running the 1-click setup. Salsa3D and the RandomEyes3D instance with the [Use Custom Shapes Only] field disabled should both have no SkinnedMeshRenderer linked. The CM_MCSSync script reads the processed output from these two instances and applies it to the MCS model. The second instance of RandomEyes3D with the [Use Custom Shapes Only] fields enabled is used for facial expression animation. Please trying saving and closing your project, and after you re-open the project, select the button near the top of the CM_MCSSync inspector to re-map the BlendShapes.

If this does not resolve the issue, please send your invoice number to assetsupport@crazyminnow.com, along with details about the following version numbers:

Unity3D
SALSA
MCS 1-click setup

Thanks,
Michael

 

Hi, Michael,
When I'm using only one gender of MCS models, everything seems OK. But then a scene has both MCS model genders actors, properly works only Salsa on actors, which gender's actor was save-close-open-buttonpushed the last.Thus, if male actor was salsa-setuped the last, and if you open project <next day>, all female actors will be broken like Todd85 wrote ( the only exists shape is eCTRLEyesClosed)

And othervise (if a female actor was salsa-setuped the last).

 

Todd,

Thanks for providing additional details about the symptom you were experiencing with the MCS 1-click setup. We found and fixed the bug you identified and released MCS 1-click setup v1.5.0.

https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

Thanks,
Michael

 

Hello Alex,

Todd helped identify a bug in CM_MCSSync that handles the automatic LOD switching. Please download the new MCS 1-click setup v1.5.0.

https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

Thanks,
Michael

 

I removed previous salsa components and made setup 1.5, then save-close-open-pushbutton. Actors don't speak and don't blink, only move eyes.

 

Are you applying the 1-click setup to the character root?

I have setup a test scene with multiple male and female characters, applied the 1-click setup to the root of each character and all is working as expected.

Also, the save-close-open-pushbutton step is only required if you add or remove BlendShapes on the character. Once all the BlendShapes you plan to use are active and you've saved / closed your project once, it's not required again unless you alter BlendShape again.

Michael

 

Michael,

No, our characters have extra-root with game logics, and MCS character is enclosed into it. 1-click setup is applying to the MCS root (child of the real root). Also may be it's important that tested actors are instantinated from pre-made prefabs.

I found some partial decision experimentally. I checked necessary morphs in MCSCharacterManager manually. Salsa failures, described above, after this action are not noticed. In addition, all the LODs began to have the necessary morphs, and not just LODO. It would be nice if the SALSA setup did this checking automatically.

And one more problem. I can't add custom shapes although I checked the necessary morphs like eCTRLAfraid and other emotions in MCSCharacterManager.Morphs, and then save, close, open, push Remap Blendshape Indexes button, push RandomEyes3D.AutoLink button (many times in different orders). New emotion morphs don't appear in the list.


Thanks
Alex

 

Hello Alex,

The MCS 1-click setup uses the M3DCharacterManager API to activate the BlendShapes used for a default SALSA setup, the same shapes that are visible in your RandomEyes3D screenshot above. Upon initialization, RandomEyes3D retrieves a list of all BlendShapes available on the SkinnedMeshRenderer. This process only happens upon initialization to keep the process as light as possible, and MCS is the only character system that currently allows activation/deactivation of shapes. We will update the MCS 1-click inspector to also force a refresh of the Custom Shapes RandomEyes3D inspector. In the meantime, the easiest way to avoid issues is to activate all the shapes you plan to use on your MCS character, save, close, and reload your project before applying the SALSA 1-click setup.

Michael

 

Michael,

If I will delete all Salsa components from characters, activate necessary morphs, save and reload project, and then setup Salsa for this characters again - this will help to get the emotion shapes into current vertion of Random Eyes 3D?

Alex

 

Yes, that is correct

Also, we are working on another MCS 1-click update that will extend the remap button's effect to the RandomEyes3D component as well so that pressing this button will reacquire all BlendShapes for both the CM_MCSSync and RandomEyes3D components.

Michael

 

Our MCS 1-click setup has been updated to 1.6.0. The CM_MCSSync inspector [Remap BlendShape Indexes] button now updates the BlendShapes on the RandomEyes3D instance that is used for Custom Shapes and Groups as well.

https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

 

Another bug squashed in the MCS 1-click setup, eyelids were not blinking at run-time due to a failure in blink BlendShape index retrieval. The latest update v1.7.0 is available now.

https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

Sorry for the inconvenience.

Michael

 

Hi,
I use Salsa with Adventure Creator.
The problem I ran into is that setting a Look Target via the "Re3D_SetLookTarget" action doesn't set the target.
At least not for me in the latest Unity, Salsa and AC builds.
Target tracking works fine when I manually populate the Look Target field btw.

I have loaded the latest CM packages:
adventurecreator_0_3_0
daz3d_3_5_0
SALSA With RandomEyes
textsync_1_2_0
textsync-aclistener_1_2_0

Everything is working well, except changing the Look Target via an AC action.
Not sure what I am doing wrong..?

- Kay

 

It is possible yet to use Timeline to craft a dialog scene with Salsa, Crazyeyes, humanoid animations and blendshape animations? It seems an obvious way to make dialog scenes but I can't figure it out.

 

Hi Kay,

I found and fixed a bug in the Re3D_SetLookTarget action for Adventure Creator. Please download the latest SALSA Adventure Creator add-on v1.1.0.

https://crazyminnowstudio.com/posts/latest-version-of-adventure-creator-supports-salsa/

Michael

 

Hello Dorian-Dowse,

We are researching Timeline to see if it's a good fit, but nothing is released yet.

Michael

 

Hi Michael,
That did the trick. Beautiful!
Now the speaker can shift his focus from one person to the other during a conversation.

Have a good Thanksgiving,
Kay

 

Is there a way to turn my lip sync in to a mecanim animation? I am having issues with timescale. Whenever the game slows down, salsa stays the same. Is there a way to make Salsa follow unity timescale? I would be open to any solutions for this problem. Thanks so much!

 

Hi Phrygian,

SALSA does obey Unity's Time.timeScale, but the Unity AudioSource component playback does not. You can work around this by setting the AudioSource.pitch property to the same value as your Time.timeScale value.

Michael

Edit: The AudioSource I'm referring to is the one that is referenced by SALSA.

 

Im not sure how to do this, could you point me in the right direction? I can't seem to find a way to access the script for my audio source component.

 

Link in the AudioSource that's also referenced by Salsa3D, select a timeScale and check the set checkbox to activate the change. Obviously, replace the set with whatever logic you're using to set the timeScale.

Code (CSharp):

1. using UnityEngine;
2.  
3. public class TimeScale : MonoBehaviour
4. {
5.     public AudioSource audioSrc;
6.     [Range(0f, 2f)]
7.     public float timeScale = 1f;
8.     public bool set;
9.  
10.     void Update ()
11.     {
12.         if (set)
13.         {
14.             Time.timeScale = timeScale;
15.             audioSrc.pitch = timeScale;
16.             set = false;
17.         }      
18.     }
19. }

 

Hello,

I have made a new Scene from my prefabs. I have 3 uma salsa gameobjects.
Only one moves the lips when talking. If I remove it, another one talk. But not more than one.

The uma Basic and uma sync character names are unique, and for every gameobject the same.

Any idea? I am searching for an hour now... no Errors or warning in logs.

Thanks!

PS: I am having three uma with salsa. I can hear everybody talk, but only one moves it's lips. I remove this one (deactivate in inspector). Now one of the others moves it's lips! None has an error, but always only one moves ist lips...

 

Hi,

Is there a reason you're still using the old UMA 2 workflow instead of the newer DCS workflow? The DCS system is much simpler to use.

https://crazyminnowstudio.com/posts/salsa-add-on-for-uma-dcs/

Michael

 

Yes. But I have other Scenes where it is running very well, and it would be a huge effort to Change everything now (I have multiple races and different beings).

Even if I change the order in which I instantiate them dynamically, the first one wins, later ones cannot move their lips and eyes. I thought maybe you have an idea what could be the general Problem for this case...

Thanks.

 

If you take a look at the UMA 1-click run-time setup. Both the CM_UmaBasic and the CM_UmaSync scripts have a [Character Name] field. It is important that both of these names are the same.

CM_UmaBasic subscribes to the [UmaData_OnCharacterCreated] event and uses the event to fire the [OnUMACreated] event, which passes the UMAData object and the characterName string reference.

CM_UmaSync subscribes to the [OnUMACreated] event and checks the event parameter [characterName] against it's own [characterName] to ensure that it's only responding to it's own chain of events and not another characters.

If you've modified or created your own version of [CM_UmaBasic] to generate your character, the event subscription and generation described in the script notes must be maintained for the system to work.

I've posted v1.5.0, our final update for this branch. Since UMA has moved on to the DCS system, we have moved on to that system as well. Our small team simply does not have the resources to support systems that are no longer supported by their creators.

https://crazyminnowstudio.com/posts/uma-lipsync-using-salsa-with-randomeyes/

Michael

 

Hi Michael,

I installed exactly the updates you mentioned. Did Change nothing else.
Now it works! All are talking, even in my intro with 3 talkers and in my talking scene with two big uma heads.

Looks great and works perfect!

Thanks!

 

Hi, I am trying to use SALSA with RT-voice and Make Human. I found a few tutorials on the website but they seems to be outdated since I cannot find some elements and I just cannot manage to make things work.

In this one : http://crazyminnowstudio.com/posts/using-makehuman-characters-with-salsa-lipsync/
I cannot find the component CM_SalsaSync

In this one http://crazyminnowstudio.com/posts/salsa-and-rt-voice-for-runtime-lipsynced-text-to-speech/
I cannot find the package Salsa-RTVoice_1.0.0.unitypackage
and I cannot find the Salsa_RTVoice_Native component

I would like to have my MakeHuman character read a txtAsset file at a specific event(button pressed for example).

I managed to get RT-voice to work but need to figure out the SALSA part now. I imported all the possible packages from both SALSA and RT voice but cannot find the missing elements/component.

I would really appreciate if someone can redirect me to up to date tutorials / video or at least tell me what I may be missing.

Thanks

 

Hello faddou,

All of the tutorials should be up-to-date and relevant. Have you already downloaded and installed the add-ons from our site?

 

I downloaded SalsaSync. Which other addon should I download ?

 

If you're using/wanting the RT-Voice add-on, you should grab it as well; however, you are perfectly welcome to use your own RT-Voice implementation. The add-on we provide is mainly for demonstration purposes and provides similar functionality to that seen in our demo video using RT-Voice.

 

How can I use several Salsa2D objects on one scene?
I have two "SALSA mouth" on one scene and when I call Play() from script for one of them, both are talking together

 

For Salsa3D: Each must have a unique character Name.
Maybe it is the same Problem for 2D?