SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Gotcha, maybe both options would be nice. One for triggering an Animator directly and another for more generic use.

 

This. I use Animancer for the bulk of our character animations, as well as some custom scripted additions, so a more generic event trigger would be fantastic to add even more finesse to the emotion setup.

 

Does Playmaker have an action for AudioSource.isPlaying? If so, that would be the same thing. If the AudioSource is playing, SALSA is playing -- unless, of course, SALSA has been disabled or unlinked from the AudioSource.

 

Would that work with the Mic, or Dissonance, or RT-whatever-I-forget-its-name….?
edit: "RT-Voice (sorry about that)

 

Dissonance would not since it uses SALSA's external analysis option to 'feed' SALSA. RTVoice would work in all but the iOS Native Mode option since it does a similar thing. micInput would work as long as the start and stop options are used to control the input.

The big question is, how detailed should SALSA.isSalsaing be? Technically it is always listening (salsaing) to whatever source is (AudioSource or external analysis). However, that could be silence. We technically could simply monitor the analysis value (valid regardless of whether SALSA is calculating the value or it is provided externally) for non-zero or the last triggered index for >=0. However, this will be on/off constantly with the nuance of speech. The next stage would be to provide a silence threshold of sorts to wait for a specific amount of silence before switching to false.

Thoughts? This could be built-in or easily created in an external script.

 

sorry… It's been a while, maybe I was using a threshold on the [analysis?] value… I think that is maybe best?

 

OK, this implementation for isSALSAing seems to work quite well. It leverages the timing already occurring in SALSA so the overhead is pretty much nothing. The silence tick threshold applies to the Update Delay value (the cycle SALSA operates on naturally).

This will be available in the next release unless some show-stopper comes up before then.

We also plan to release the following new features:

1. Opt in/out of EmoteR hold-timing-variation calculations, thanks to @wetcircuit's suggestion.
2. Global overrides for SALSA timings (with added option to apply overrides to visemes). A great way to experiment with timing changes. And also a new mode for timing experimentation (hinted at in the above video).
3. Global overrides for SALSA easing settings (with added option to apply override to visemes).
4. A new delay timing for EmoteR which allows for more complex emote setups, delaying the start (ON) animation phase on a per component basis. i.e. open mouth, then stick out tongue.

Darrin

 

Wow, that video looks horrible. I'm assuming Youtube is still processing it. If it still looks horrible in a bit, I'll upload it again.

 

Haha I thought it was me…

I was going to say "thank you" and then trust that you knew what you're doing. LOL!

 

LOL -- OK, I transcoded it, reposted, and changed the link above. Try it again. I think maybe it didn't like the nvidia codec.

 

oh NO! hahaha you used "is SALSAing"

That is great!. It's really responsive!

 

Gotta give credit where credit is due! SALSA.IsSALSAing is perfect!

 

I've bought your product and tried to use One-Click Setup (iClone Character Creator) on a few characters.
One of these characters: https://s3-eu-west-1.amazonaws.com/wip.krej.net/Erin.unitypackage
Unfortunately, after selecting the character base and launching setup, I see an error:

It does some configuration, but it is far less cool than results on the tutorial video.
Could you please fix the issue (or tell if I am doing something wrong?)

(I've tried to mail support at assetsupport@crazyminnowstudio.com, but mail is not working (timed out))

 

Hello Crazy-Minnow-Studio.
I'm a buyer of the first version of Salsa, but I never used it. I was surprised, that the asset was gone. On the other hand, I understand that the further development has to be financed somehow. In order not to have this experience again, I would like to clarify a few things before I buy the upgrade.

1. On Youtube you can find tutorials of SALSA LipSync Suite (v2) tutorial series. Is this another (paid) upgrade coming, or is this already the version I purchase and download from the Asset Store?
2. In my current project I want to make a UMA character speak. I also got a recommendation from the UMA2 forum - however I see here that it doesn't seem to work that easy after all?!
https://forum.unity.com/threads/sal...id-control-system.242135/page-30#post-4894106
https://forum.unity.com/threads/sal...id-control-system.242135/page-30#post-4894208
https://forum.unity.com/threads/sal...id-control-system.242135/page-30#post-4894328
Question - is UMA2 now supported?

3. I found the following implementation steps in this forum.

Steps to implement:

Same as previous, apply the OneClick to your avatar root.

On the UmaUepDriver, disable (uncheck) the "UMA Character is Dynamic" option.

Configure an UMAExpressionPlayer on your avatar and you will need to configure it with these parameters:

UMAExpressionPlayer.expressionSet = yourExpressionSet;

UMAExpressionPlayer.umaData = yourUmaData;

UMAExpressionPlayer.Initialize();

Then, call our UEP driver manual start function, passing a link to the UMAExpressionPlayer you just configured:

UmaUepDriver.ManualStart(yourUMAExpressionPlayer);

Same as previous?! I do not understand the context. Is this related to the documentation https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/#uma
(Unfortunately there is no tutorial for UMA 2 and LipSyncSuite v2 on youtube – but for the "old" one (probably the SALSA LipSync Suite (v1)) version I found a tutorial for the integration of UMA2

4. Is it a good idea - to do this tutorial with V1 and then upgrade? Salsa Suite to v2?

I watched the first two videos of the (v2 Version) In the dropdown of the Viseme - controller type - there is UMA…

5a. This is UMA 2 support?!
5b. And where can I find the explanation / the setup for this point? Or is this done with the OneClick?!


Thanks for your answers - sorry for asking all the questions... but I wanted to clarify before I buy the upgrade... ;-)

Have a nice day

 

Hi FrozenDistrict,

Please use our CC3 one-click setup. Our support email is assetsupport@crazyminnow.com.

 

Hi Boegi,

SALSA with RandomEyes v1 isn't gone for those who already purchased it.

1. SALSA LipSync Suite (v2) is a complete rewrite of the entire system and replaces v1 for new customers and those who want to upgrade. Anything labeled v2 is referring to SALSA LipSync Suite.
   
2. The release version of UMA DCS is supported. See our features page for a list of all supported systems.
3. Our one-click documentation details the setup steps. As with any complex system, UMA requires some studying to use it to it's full capabilities. https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/#uma
   
4. SALSA LipSync Suite is a complete rewrite and is not backwards compatible with v1. The concepts are familiar, but it's vastly more flexible and capable.
   
5. Our UMA support is for the current release of UMA DCS. The one-click gets you a fully functional setup and is designed to work out of the box with male and female base meshes.

 

Thanks for your answers.

 

Whenever @wetcircuit is participating to a thread, I know it's in good hand with alazing suggestion, she align perfectly with what I want to achieve lol, so that I can safely focus on uncertain untested thing that receive little care lol, thank you @wetcircuit

 

@Crazy-Minnow-Studio I'm trying the latest CC3 One-Click, but the download page says that OneClick Reallusion CC3 v2.1.9 requires OneClickBase 2.1.4, but the only version on the downloads is 2.1.6. When I try that version it gives me all kinds of errors.

 

Hello!
The CC3 requirements indicate v2.1.4+ of Base and v2.1.6 is the latest, so you have the correct Base. Please elaborate on the errors you are receiving. Send us an email with details and screenshots of:

• Your model's hierarchy (mesh names)
• The errors/warnings you are receiving
• The Inspector for the body mesh, with blendshapes exposed

 

I can't give exact errors this moment, but it's compiler errors regarding uwp or uep missing or something like that.

 

Ah, that sounds like maybe you are running on an older SALSA Suite version? Version 2.1.0+ is required for OneClickBase 2.1.5+. I don't believe it states that on the downloads page...only in the Base release notes. I'll update the site to reflect that requirement. If you do not want to update the Suite, you can roll back your Base and CC3 to an earlier version without issue.

Thanks!
Darrin

 

Ah, yes. I needed an update and now all is well. Thanks.

 

@Crazy-Minnow-Studio Although the CC3 OneClick no longer has errors, I'm seeing some weird results with the eyes. Watch the eyes, especially the right eye, during the following video:

I'm using Unity 2017.1.5f1. If you need the character for reproducing this let me know and I can send in a PM.

 

It looks like maybe your animation is trying to drive the eyes, possibly creating a conflict. Can you send us your model and the animation to assetsupport@crazyminnow.com?

Do you see this when the animation isn't linked?

 

Yes, it is there even with no animation linked.

I Sent the e-mail.

 

For any others following the issue @magique is experiencing, it appears to be a bug in the Reallusion export, which has created a bad rig on his model. He is following up with Reallusion for resolution.

 

2018.3.8fl
2.1.2 Sep 6, 2019
Have an issues where Salsa and Eyes components run super fast whenever the animator component is enabled. If I disable it in the inspector when running, Salsa and Eyes run at the proper speed again.

If it's helpful I can make a video, but could really do with some help as it look really good when it's working
Happy to send a package too.

 

Hello,
Likely what you are seeing are conflicts between your animator and SALSA Suite. If your animator is controlling blendshapes and bones that you are trying to animate with SALSA Suite, the two will fight. You could edit your animation file and remove the blendshapes from it (recommended). Even if they are 0, the Animator will write the 0 value every frame, throwing off the calculations. Is this a Reallusion model?

 

No it's a custom Rig, will check the animator. I believe I keep the Salsa shape channels un keyed.

SOLUTION: Ah thanks, yes it's the 0 keys on the shape channels than came through with the animation baking.

Thanks for the speedy reply - keep up the great work.

 

I'm looking at SALSA to add lip syncing to our project. But I'm concerned about how it might interact with the other blendshapes we're already using to show facial expressions. These include smiling, frowning, concentration, etc. It's very important that these expressions continue to show even when the character is talking.

Can you tell me how SALSA deals with this? (This is an Adobe Fuse character, if it matters.)

 

Hi Joe,

Regardless of the controller, blendshapes can only be set to one value at a time. The only way to prevent blendshape conflicts with any system is to have enough blendshapes to prevent dual usage. This is why we're starting to see character systems implementing emote blendshapes in addition to visemes. SALSA's queue deals with conflict with the following prioritization rules:

1. LipSync override all
2. Emotes override eyes
3. Eyes overrides none, except blink (which is sent to the queue as an emote).

Fuse characters barely have enough blendshapes for speech to begin with, unless you're adding additional blendshapes (you could do this with our MorphMixer asset), or using a smaller less detailed set than our one-click setup uses. There simply aren't enough blendshapes to do comprehensive lipsync and emoting at the same time using Fuse character default blendshapes. If you use a shared blendshape for speech and an emote, the emote will fire but as soon as lipsync needs that blendshape it will override the emote.

 

I use Fuse with SALSA.
One option is to use CM's other product MorphMixer to create new visemes out of the existing blendshapes. You can mix the dedicated visemes with the original blendshapes.

smiling with mouth open won't work, but mouth corners up will be fine.

 

Hello everyone!
SALSA LipSync Suite v2.2.2 is now available on the AssetStore. This is a feature release which contains several goodies requested by our awesome customers.

SALSA:

1. Check SALSA's processing status via an active check to Salsa.IsSALSAing or subscribe to the C# events for notification.
2. Timing and Easing overrides - set the global values for all viseme components.
3. Easy mode for animation timings - 3 sliders interlinked to help you find the perfect look-and-feel for your SALSA-based project.

EmoteR:

1. Exclude specific emotes from the Hold-Variation option.
2. Add a start delay to individual emote components, allowing total freedom on emote timings.
3. Repeater emotes now have an initial animation timing option to fine tune how repeaters startup.
4. New AnimatorController -- use EmoteR to manipulate Animator parameters (bool, int, float, trigger).
5. New EventController -- EmoteR can now fire events based on animation phases.

And a few small bug fixes and Inspector enhancements.

Be sure to check out the release notes for more details.

 

I believe most of the above new features are now included in the online documentation. The new controller options are only for EmoteR at this point. If you all have some use cases, I can see about at adding them to SALSA.

Documentation for the controllers is in the further-reading section for ExpressionComponents.
https://crazyminnowstudio.com/docs/...n-components/#specialty-component-controllers

I hope to get a couple of videos out early next week covering some of the new stuff.

Thanks and enjoy!
Darrin

 

Hey everyone! Just released the next video in the tutorial series (#9 Events). Enjoy!

 

Yay!!!

EmojiGuy has the most lip action of any 3D character I've ever seen…

 

Another video tutorial was published just now, 2D Components in SALSA. This one is actually two videos since it ran a little long. There's a lot of information here about 2D Components and how to configure the settings for SALSA. While this is primarily geared towards a SALSA configuration, each of the two parts has information that is beneficial for working with 2D in EmoteR and Eyes as well.

part 1

part 2

 

Hello all,

As most of you already know, our complete SALSA with RandomEyes (v1.x) rewrite to SALSA LipSync Suite (v2+) was released in June 2019. This all new product has been a big hit so far and we're thrilled that most of you seem to love the new features and flexibility. SALSA with RandomEyes (v1.x) was deprecated when we released SALSA LipSync Suite (v2+). To ensure that our small team can maintain the highest level of focus and support for the SALSA LipSync Suite, we will be discontinuing support for SALSA with RandomEyes (v1.x) on 2020-01-01.

What does this mean for existing v1.x customers who do not upgrade to v2+? As with any deprecated Unity Asset Store asset, if you purchased v1.x before it was deprecated, you can continue to access it from your Unity Asset Store account. Additionally, you can continue to access the deprecated v1.x add-ons from our website using the legacy downloads page. However, after 2020-01-01, we can no longer provide support for v1.x or any of the v1.x add-ons.

SALSA with RandomEyes v1.x and all v1.x add-ons stopped receiving updates when it was deprecated in June 2019. We encourage all v1.x customers to upgrade to our new v2+ product...it's way better!
https://assetstore.unity.com/packages/tools/animation/salsa-lipsync-suite-148442

See the blog post.
https://crazyminnowstudio.com/posts/salsa-with-randomeyes-end-of-life/

Thanks,
Crazy Minnow Studio Team

 

Hey everyone!
Two more tutorial videos, covering EmoteR Overview and EmoteR Random Emotes have been published.

There are two more in the EmoteR series (EmoteR Repeater Emotes and EmoteR Manual Emotes) coming soon, hopefully tomorrow. They are already recorded, just need to be processed.

 

Also, I found a bug affecting EmoteR's delay timing and fixed that in SALSA LipSync Suite v2.2.3 (already published and available). That was the only change in v2.2.3 -- I discuss it in tutorial 11a at the end of the tutorial.

D.

 

The Repeater Emotes tutorial has been published. That leaves one more to finish up -- Manual Emotes.

Enjoy!

 

Hi @Crazy-Minnow-Studio !
I'm (finally) integrating Salsa into my game and your new videos couldn't be more welcomed!
I'm watching the whole series and they're great!
I have a doubt though:
Should I manually disable the Salsa components when no audio is playing (ie: normal gameplay) or is Salsa smart enough to not eat my CPU when it's not needed?
I mean, I have my Player and several NPCs with each their own Salsa components, but they will only be used for dialogues. Will Salsa still execute each frame (or each analysis interval) and eat CPU even when there is nothing playing on it's audiosource?
Should I place Salsa components in a separated gameobject inside the characters and enable and disable it only when I know Salsa will be used? (I can do this, but if Salsa already manages it internally it would be overkill)
I'm targeting the Nintendo Switch, so performance and optimization is very important.

 

Hello!
The question to disable SALSA is purely your call. The loop will obviously run, just as any component with a loop will run. However, that said, SALSA will not "process audio" every frame. As you saw in the Audio Analysis video, it operates on the update delay cycle and this is the only cycle it actually analyzes audio (which occurs very quickly when it does analyze). Outside of analysis, it is a quick conditional to see if it should run analysis and if not, it bails. The best thing to do would be to run a deep profile (which will capture the calls to the QueueProcessor for a total view) and see if the process is something you want or need to eliminate. Registering ExpressionComponents is the part that takes the most processing and if you are sitting silent, there are no components to register.

When I deep profile my system (old i7 2700K) on Unity 2017.4, I never see anything more than .01ms used by SALSA every so often when the character is silent.

Hope that helps,
D.

 

Ok, I'll just let it run then.

On another topic, I'm surprised there is no way in Salsa (or EmotR) to set an emote permanently from script:
I'm parsing the text and dialogue I feed to Salsa from a custom script and I set emotions based on instructions inside the dialogue text in order to set the according blendshape emotions on the character's face.
Right now I can fire EmotR animations manually from script, but they have a start and an end. I would want to set an emotion permanently until I fire a new one.
I can manage this from my script, but since Salsa is already so sophisticated, it would make sense to support this, don't you think?

-EDIT: Again, I can do this myself, it's just that I think it would be a great addition to Salsa.

 

Hello, Im wondering how to save the salsa settings for each game object, since are preparing a demo and importing a new fbx version of the character a couple of times a week, I want to know if its possible to setup Salsa once and keep all the settings.

I will set it up on a Root game object, but what about the actual setup of the blendshapes?

 

I have serious problem, because the sauce only works when I click on the prefab that contains the script?
and in the mobile tests it doesn't work, in the editor it doesn't work either, it only works when I click on the object that has the salsa script.

i used unity 2019.1.4f1

this only happens in runtime integration

 

Hi,

This is already possible, see the documentation on manually triggering an emote using the OneWay ExpressionHandler.

 

Hi,

Probably the best way to handle this is to write a custom one-click setup for your specific character design. This is what we do for all of our existing one-click setups for Fuse, Daz, CC3, etc. They aren't doing anything except creating a shortcut to a standardized setup.

 

How did I miss this?
Thanks!