SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

I'm trying to get Salsa to work with the UMA Orc Race Pack from Dragansan studios. I've only been partially successful so far. I have humans working fine. But when I use the Orc Race the lip sync does not work (head and eyes do).

The Orc DNA is being loaded from a resource at startup.

If I manually switch the race to Human DCS at runtime the animation breaks (goes to T-Pose) but the Lip Sync starts working. If I switch back to Orc it continues like that (T-Pose, Lip Sync works).

Any pointers?

 

Hi fgardler,

If you are changing your characters hierarchy at runtime, you will also likely need to rebind SALSA, EmoterR, and Eyes to the new hierarchy at runtime. Given the limitless flexible nature of how UMA can be created, our one-click setup should be considered a demonstration of what is possible. Have a look at our API documentation and the UMA one-click setup to learn more about runtime configurations.

 

Ahhh... ok, not sure if the "one click is a demo" was clear in the docs. It does make sense. Thanks for the pointers I'll take a look and come back if I can't find my way.

 

Yeah UMA is very powerful and flexible, which means we simply can't anticipate every possible implementation, so instead we offer our UMA one-click as a guide/demonstration of how you can implement it for your system.

It's actually the first sentence of the first paragraph in our UMA one-click documentation.

"Since UMA is a run-time character system with many options, our setup scripts are slightly different than our other one-clicks and meant primarily to demonstrate what is possible when using the SALSA LipSync Suite with the UMA system."
https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/#uma

 

Well at least I read "some" of the docs A shade better than a full RTFM.

Ashamed thanks

 

No worries. If you come across something that doesn't make sense or could use further explanation, shoot us an email and let us know.

Thanks,
D

 

Greetings!

I've gone through the Photon Voice tutorial successfully at https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/photon-voice/ using my own environment and Character Creator 3 models. Yay!
In that written tutorial 'Wait for Audiosource' Is used with the Boxhead and three simple visemes.

The SALSA CC3 'one-click' appears to provide a different audio preference which doesn't include 'wait for audio'.

1. Should I rebuild visemes with CC3 blendshapes and this 'generic' SALSA to work with Photon voice, or use CC3 'one-click' and link Photon to it via other means?

I'd also done the Watson tutorial which shows how to link Text-to-speech audio files to SALSA running on a CC3 avatar but I don't know if Photon essentially needs the 'wait for audio' option to work.

 

Hi,

The one-click setups are just configuration scripts for the model. You are free to adjust and set any property changes you wish after the one-click has been applied. Alternatively, you can create your own one-click setup with whatever settings you like. See the CC3 one-click setup as an example, and check out our documentation on creating custom one-clicks. See our API documentation for information on accessing properties from code.

[EDIT] You definitely need to enable the Wait for AudioSource option in PhotonVoice. Basically, you need to set your CC3 character up just like you did the boxhead character. So, after you apply the OneClick, make all of the same adjustments you made to boxhead.

[EDIT 2] I just considered you may not know how to set "Wait for Audio" once and AudioSource is attached. Simply remove the AudioSource link and also the AudioSource created by the OneClick. "Wait for Audio" is an available option when an AudioSource has not been linked.

 

is there a way to set a "look at" target in the time line when using Salsa ?
I would just animate the head but I noticed you change the Hierarchy with the one click.

 

Hello! The Timeline events currently include SALSA audio and Emoter emote clips. Eyes is currently going through a considerable update to remove the fix-axis hierarchy, so perhaps once that is complete the Timeline add-on will get more options. Creating Timeline clip actions is not trivial, so it is likely some sort of generic implementation will be created that allows for more flexible feature access inside the Suite.

Thanks,
D

 

well if the fix-axis hierarchy goes away then I can at least use my clips thanks

 

This is an old post, I own the first version you made and it was great but it lacked this. aspect of it..
So in your newer version, have you added any sort of text to speech aspects?

I been making a game for my kids to handle home schooling, so I'm teaching them to spell, type, code etc. So I may just use google's TTS, but looking for something better if possible that I can leverage, so have you added anything like this?

It would be great to leverage having a character so the kids could see there lips moving that I know Salsa can do, but looking for the text aspect to Speech or something like it to leverage. Thanks!


I plan to update to the newer version, sooner or late anyways, but hoping this exist. Thanks.

 

Hello Recon03!
That was indeed a very old post, right after the launch of SALSA with RandomEyes (v1). Some time after that we began supporting RT-Voice for TTS processing. You can use any TTS you like as long as it funnels the audio stream back into an AudioClip to play on the AudioSource -- although, we highly recommend RT-Voice as a well-supported, feature-packed, valuable partner.

So, to answer your question, it existed as an option in v1 and is still there in v2.

Hope that helps,
D.

 

I didn't know if you actually added one your self. TTS . Ya not really interested in RT voice. anyways, I will still use this for ours ( something else we are working on)
but,
.. I was just curious if you added your own. Thanks anyways

( I been using googles.) it works fine, but was curious, before I bothered, re-writing anything, been a bit since I used Salsa. and not interested in spending anymore money...

This is just to home school the kids, not make money off of.... So and don't need anything to special. but since I owned this version 1, and planned to get V2, I figured I may be able to kill two birds with one stone....

Thanks anyways for that info.

 

Hi!

Got this error:

Array index (89) is out of bounds (size=0)
UnityEngine.SkinnedMeshRenderer:SetBlendShapeWeight(Int32, Single)
CrazyMinnow.SALSA.ShapeController:SetLerp(Single, Boolean)
CrazyMinnow.SALSA.ShapeController:MergeWithAnimator(Single, Boolean, Boolean, Boolean)
CrazyMinnow.SALSA.QueueProcessorrocessQueue(List`1, Int32)
CrazyMinnow.SALSA.QueueProcessor:LateUpdate()

Actually got this error for twenty different Array indexes (Array index (90),Array index (96),Array index (102)...).

It's something with blendshapes? I didn't change/add/remove any blendshape.

Can I fix it somehow please?

Thanks,
Jakub Pilař

 

Nope, sorry...but if you can get Google's into an AudioSource.AudioClip, you should be good to go!

D.

 

Hello Jakub, looks like part of your configuration has been messed up. It doesn't necessarily mean the model's blendshapes have changed, but if you are programmatically setting up the Suite, likely some configuration data was not baked into the ExpressionController(s) after character config. If that's the case, check out this document: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/further-reading/runtime-setup/

D.

 

Hey folks! Can anyone recommend an affordable character creation option that would work well with SALSA to make humanoid monster-type characters? FUSE is getting discontinued in a few months, and is the only One-Click option available to use with SALSA that isn't crazy-expensive. I tried Character Creator 3 free trial, and it would maybe be worth buying except that the actual customizing options (i.e. morphs) are so shallow and limited that it shocked me. You can't make humanoid monsters (orcs, goblins, ogres, etc.) which is absurdly limiting for game design. CC3 should be called "human-only" character creator...there isn't even an option for pointy ears or a long nose for a witch. I have not tried Autodesk, but it appears even more limited than CC3. Anyone have a "hidden gem" that I am overlooking?

 

Actually, CC3 can do orc, goblins, ogres, etc. I like CC3, but it has bigger problems than first meet the eye. Mainly, because Reallusion is in the business of nickel & diming you to death through add-ons and DLC. But instead of nickel & dime it's more like hundreds and thousands. You basically can't get the most out of it without buying thousands of dollars worth of add-ons and packs to get what you want. Take, for example, if I go to their user creation store and want to buy a single outfit from there. It doesn't look too bad at maybe $20-$30, but then you find out that they have a rule that no one can sell anything unless it derives from a Reallusion asset. So, in order to buy the $20 outfit, you first have to buy the $300 Reallusion pack that it uses as a base.

I've been searching for that perfect solution too, but so far there isn't one as far as I know. UMA serves a purpose in some cases, but it has very limited morph options as well and good outfits are expensive add on content as well. Unfortunately, the best option, Morph3D, was deprecated and then open source version is a mess.

 

Thanks for the answer.

So should I bake configuration data to the controllers according that document through this script?:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class SALSA_bake : MonoBehaviour
4. {
5.     void Start()
6.     {
7.         CrazyMinnow.SALSA.Salsa salsa; // reference to Salsa
8.         salsa.UpdateExpressionControllers(); // Bake configuration data into the controllers.
9.  
10.         CrazyMinnow.SALSA.Emoter emoter; // reference to Emoter
11.         emoter.UpdateExpressionControllers(); // Bake configuration data into the controllers.
12.         emoter.UpdateEmoteLists(); // Parse emotes into their associated emote lists.
13.  
14.         CrazyMinnow.SALSA.Eyes eyes;
15.         eyes.Initialize(); // configure calculation gizmos
16.         // Bake configuration data into controllers.
17.         eyes.UpdateRuntimeExpressionControllers(ref eyes.heads);
18.         eyes.UpdateRuntimeExpressionControllers(ref eyes.eyes);
19.         eyes.UpdateRuntimeExpressionControllers(ref eyes.blinklids);
20.         eyes.UpdateRuntimeExpressionControllers(ref eyes.tracklids);
21.     }
22.  
23. }

Is it safe?

Jakub

 

Only if you are doing it at runtime. Since your script is operating at Start(), it indicates maybe you aren't doing a runtime configuration???

To eliminate confusion for you and other customers, contact us in email to confirm what you are ultimately trying to do and why your configuration was lost and then we can follow-up here with a resolution. Please include details of what was working, then what happened when it stopped, and some detailed screenshots showing your configuration before and after running the scene.

Thanks,
D.

 

I looked a bit more into it and yeah, you ain't kidding about the price of CC3 extras. Literally thousands. And you're also right about Morph3d. The best looking humanoid-monsters I've been able to make were from Morph3D. Morph3D remains my all-time most despised product ever made on the asset store--I've never been lied to so frequently and with so little regard to basic decency as I have with the makers of Morph3D. I still quiver with anger at the thought. By comparison, SALSA in my #2 all-time FAVORITE Unity asset. I looked into the open source version of Morph3d, and there are two heroic souls toiling away to make it usable. I have a glimmer of hope there, but they are encountering a lot of struggles. They are also SALSA fans, and are talking about making a One-Click, so I'm crossing my fingers! If I could figure out how to morph a character with Morph3D, then export it to Blender with SALSA-appropriate shapekeys intact, then back into Unity--I might be able to use it. But it is too technical for me, and I have had no luck. If anyone has done this, please let me know how! Also, has anyone managed to make a One-Click for Makehuman? Seems like Makehuman would lend itself to a SALSA One-Click, but I am no good at that kind of set-up.

 

Upcoming release of SALSA Lipsync Suite v2.5.0
Hello everyone! Here is a quick update on the progress of SALSA LipSync Suite v2.5.0.

EYES: Mike has been putting a lot of work into the Eyes module to remove the fix-axis architecture. We know a lot of you have had issues with external animations and the fix-axis has been the root cause most of the time. We are almost complete with re-working the way Eyes handles wonky axes. The templates and component displays are getting an overhaul as well to make things a bit simpler both under the hood and in the Inspector. Due to all of this, we expect a need to modify the OneClicks (at a minimum) to work with the new Eyes requirements. As with v2.3.0, we will like-version any add-on with the Suite release. Stay tuned.

SALSA and EmoteR are getting some updates as well. I've been refactoring a LOT of code and have found a few small bugs in the process and have implemented some changes that hopefully will make life better.

New features slated for v2.5.0:

• Plugin options to further open and extend SALSA for devs that wish to incorporate advanced implementations. The analysis calculation module was opened up in v2.4.0, this new release opens up the data acquisition process.
• Ability to apply the Global Dynamics setting even if you are using Advanced Dynamics.
• Error trapping for null-refs when the component controllers are externally changed or deleted. @magique requested this in a previous post. I've been testing the new implementation and so far, it is rock solid.
• Enhancements, updates, and fixes for some extreme runtime configuration scenarios, such as an option to Flush the QueueProcessor.
  
• SALSA and EmoteR are getting some new settings defaults, not a big deal, but something to be aware of.

Mainly a lot of underpinning changes, but these will hopefully surface as a more solid and enhanced system for our customers. We are really close, but don't yet have an expected date for release.

See ya soon,
D.

 

You can do them on daz3d and check their store, works perfect with salsa and comes with many blendshapes

 

What is the current pricing model for Daz? The last time I checked (some time ago) it was similar to CC3--it cost hundreds of dollars to piece together even a few characters with a render-only license, and hundreds more to add a game-license to it. Have things gotten more reasonable?

 

I had the first version of Salsa, but didn't get around to trying it. I rely on Game Creator since I can't code at all, so was wondering if this will work with Game Creator (GC is like playmaker/ adventure creator type of game making asset for non coders and has it's own player and character controllers) Any comments appreciated.

 

Hi tork12,

We've never had an add-on for Game Creator that offered any specific support. Our current list of add-ons and integrations can be found on our website.
https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/features/

If your system can play audio on a specific AudioSource (referenced), you could at least activate lipsync since SALSA will perform lipsync to any audio playing through it's associated AudioSource. However, other functions would likely require some kind of linkage between the two systems. At this time, we do not have plans to build integrations for Game Creator.

Michael

 

Hello,

I'm using Eyes and Emoter scripts on the same character with the same Queue Processor. But there are times when eyelid blinks occurs over the emotion which also uses same eyelid blend shapes. So the caharcter blinks twice or blinks when his eyes are actually closed (over sleepy emotion). I use random emotions (random pool) and random blinks. Are there any special setup for such a case?

Best,
Nikita

 

Hello @OneCpublishing,

You are seeing two conditions, one is strictly an Eyes configuration and you can adjust the blink rates to increase or decrease how the blinks occur. Increase the min timer value for more guaranteed time between blinks. EmoteR and Eyes should work together quite well using the same blendshapes, but blinks do override emotes, but should also return to the emote setting. This was a big feature in the 2.3.0 release that enhanced the queue hierarchy. However, if your character is asleep and you don't want to see any blink interference, I would suggest programmatically suspending blinking during this time. Check out the API for information on accessing the Eyes settings.

https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/eyes/api/#eyelid-blink-configuration

Hope that helps,
D.

 

A quick update on the 2.5.0 release:
We are still churning through the Eyes module updates and testing -- there are a LOT of scenarios and templates to test and re-work and generally speaking, all add-ons that work with Eyes will need updates due to the improvements. Strong foreshadowing here: there are breaking changes in Eyes configuration requirements. In the process and thanks to a customer query, we identified a bug in lid tracking and that, in turn, uncovered a very rare animation bug in the QueueProcessor hierarchy overrides. I believe that bug is now squashed and the end-result is just better animation hierarch overriding in general.

In addition to the v2.5.0 updates, a few of the add-ons have received a little love and some new attention. As I mentioned above and in the previous post, all add-ons that work with the Eyes module will have to be updated. We will perform a similar versioning scheme on add-ons with requirements of specific Suite versions. As for new add-on updates (coming soon):

• SalsaPhotonVoice: provides local avatar lipsync. We worked with ExitGames to come up with a solution and it appears to work quite well. (Reqs: SALSA v2.5.0)
  
• TimelineSalsaCore: (runtime) scrubbing support accurately adjusts the audio playback. Audio track clip support for PlayableDirector.Pause()/Play() as well as support for setting graph speed to 0 to implement pause and 1 for play. (Coming soon)

With the new delegate options for SALSA processing (v2.5.0), a couple of other add-ons will get some efficiency updates:

• SalsaDissonanceLink will be updated to remove an Update() loop and operate on SALSA's tick cycle.
• SalsaMicInput will also be updated to remove an Update() loop and operate on SALSA's tick cycle for the record head pointer updates.

That's all for now!
D.

 

Hello,
The only way I can trigger a Salsa driven NPC to speak (using Dialogue System) is if the audio source is on the root of the game object. Previously, with Salsa 1 we were able to make the audio source a child component. This was idea because we could set the audio source to the same transform value as the NPCs mouth.
Is there any way to achieve this?

Thanks for this great asset!
Nathan

 

Hi Nathan - That's probably more of a Dialogue System question. The Dialogue System's SALSA() sequencer command expects the audio source to be on the root GameObject. In version 2.2.7 (pending on the Asset Store), the audio source can be on a child object. I'll PM you a patch that you can use until 2.2.7 is available.

 

Thanks @TonyLi

This is so helpful!

Nathan

 

Ah! I see it. Removing the Audiosource component then selecting 'Waiting for Audiosource' created all-blue component states. I have to mention it's phenomenal (pun intended?) how complete the auto-setup is CM has programmed into this.

 

Hey!

Regarding the Eyes module, is there a way to control how fast the head/eyes move from one point to another?

When two points are on opposite ends of the field of view the transition between points happens a little too fast, which can make it look unnatural. Perhaps you can base that tween on the magnitude of the transition?

 

Hi,

SALSA LipSync Suite uses time-based animation. You can read more about it in our documentation. Head and eye tracking are persistent and only use the On timer.

https://crazyminnowstudio.com/docs/...ession-components/#notes-on-animation-timings

 

Not sure I follow. I'm talking about having the 'use affinity' option active in the system properties of the Eyes module. From what I understand it switches between looking at a target (if one is assigned) and looking around. If the target it is looking at is on one edge of the heads field of view and then a random point inside the FOV on the other edge is chosen, the transition from looking at one point to the other seems to happen a little too fast. Im not sure how animation timings affect this logic?

Thanks for the quick reply! Much appreciated

 

Since animation is time-based, the On value is the amount of time it will take to move the head or eyes from their current position/rotation to a new position/rotation, regardless of how the new position/rotation is determined. If you set the On duration to a longer value, then all moves will be complete within this duration of time.

 

Hi I have a question.
Someone would like to load the music library into SALSA with IOS. Do you know a good way? You can use paid or free assets. If you know it, please let me know! Thank you.

 

Ok that makes sense Thank you again for your help!

 

My animator contains animations that controls the jaw of my model,
how can I make Salsa override the transform rotation of the jaw?.

 

I think you'd need to set up an animation mask in your animator (not sure if you need a mask for just 1 joint, but that's where I'd start).

 

My character is well rigged but with facial bones not Blendshapes. Is it better than blendshapes ? Do you have some tutorial on how to animate it with Salsa ?

Thx in advance for your help

 

https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/overview/#requirements

The requirements for bones are similar to the requirements for blendshapes, you will need to have "sufficient capability to individually manipulate the mouth (for lipsync), the face (for EmoteR), and the head/eyes for Eyes". Our documentation and video tutorials are a good starting point.

 

Hi there,
I'm trying to get Salsa to work with MakeHuman on HL2 with MRTK v2. Before I can even try to deploy, I'm getting following error in Unity-Editor (2019.2.3f1):

KeyNotFoundException: The given key was not present in the dictionary.
System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) (at <599589bf4ce248909b8a14cbe4a2034e>:0)
CrazyMinnow.SALSA.QueueProcessor.ProcessQueue (System.Collections.Generic.List`1[T] queue, System.Int32 hierarchyLevel) (at <9171104005e141b28996a0490f756eca>:0)
CrazyMinnow.SALSA.QueueProcessor.LateUpdate () (at <9171104005e141b28996a0490f756eca>:0)

I don't set up Salsa on runtime or via scripts at all. Trying to configure everything from scratch again didn't help

EDIT: What I forgot to mention: the error is not consistent, every few trys it simply vanishes for a few tests

 

It appears UMA 2.10 includes some breaking changes with our current integration. We're looking into it.

Michael

 

Hi MATERNATMT,

Please send the following details to assetsupport@crazyminnow.com:

• SALSA version number
• SALSA invoice number
• Unity version number
• Screen shots of your entire SALSA configuration (Salsa, EmoteR, Eyes) with all collapsible sections expanded to show all configuration details.
• Reference your Unity forum name so we know who the email is coming from.

Thanks,
Michael

 

I have SALSA working with IBM Watson Speech to Text and it's very cool. Latency is long depending on how much text is being transcribed to speech. Would the group recommend a way to measure latency of downloading the speech wav file or otherwise describe your experience?

 

I can't seem to get Salsa to work with the Asset Store version of UMA. It works perfectly with UMA 2.5, but anything above that it doesn't seem to work. I think you guys need to update your UMA-Salsa integration to work with UMA 3. I used the One Click and the lips don't move. When I edited the UepProxyScript myself, I got the eyebrows and eyes to blink but no mouth movements with the Salsa Mic Input script on the gameobject. I hear the microphone too and it says I am getting lip sync data. I saw other people are having the same issue, please look into fixing this. Thank you.

 

Hi troach06,

We're aware of breaking changes with newer versions of UMA and our current integration, we're looking into the issue.

Thanks,
Michael