SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hello tiago_carvalhais,

If I had to guess, it sounds like a Unity subsystem isn't being exported to your Android Studio project. SALSA relies on Unity's integrated version of the FMOD sound library to process audio. That's my best guess, but since we've never had a need to built a SALSA-based project using Android Studio directly, we have no experience with your solution approach.

Michael

 

where did you end up finding CM_SalsaSync to download? I've been trying to figure it out for hours. I can't find it on unity when I click import or on the Crazy Minnow Studio website. Is it still supported?

 

Hi milopez,

You can use the add-on links on the Asset Store page, the search box on our website, go straight to the download link on our SALSA product page, or just google "SalsaSync."

 

Hi,

Is it possible to get Salsa working with Reallusion Character Creator?

Thanks

 

Hi Caeru,

Our iClone 1-click setup add-on v1.3.6 was updated to work Character Creator 3.
https://crazyminnowstudio.com/posts/1-click-lipsync-for-iclone/

 

Question about Random Eyes with UMA 2.0

Do you have to use blendshapes for the eye rotation or can you use Joints? I watched the
UMA 2 Lipsync using SALSA with RandomEyes video
and it seems like it was still using Blendhshapes to drive UMA's joint system. SO again, just want to confirm, if Im using UMA can I use joints to drive the eye or do I still need to make blendshapes for Random Eyes.

Thank you.

 

Hi MarkFitzpatrick,

The currently supported add-on is our SALSA for UMA DCS found at the link below.
https://crazyminnowstudio.com/posts/salsa-add-on-for-uma-dcs/

It uses the UMAExpressionPlayer to indirectly drive facial bones for lip sync and emotions, and RandomEyes eye movement amounts are converted and applied to eye bones within the SalsaUmaSync add-on.

 

So I dont have to make one single blendshape, all joint drive and Random eyes would still work? I think I understand.

Just installed the UMA one Click Solution and running into a problem. Crazy Minnow is not poping up under Game Objects. I also posted an image to show everything I have installed. Currently using 2018.3.4f1

Thank you.





Also running into this error

Assets\Crazy Minnow Studio\SALSA with RandomEyes\Third Party Support\UMA 2\Editor\CM_UmaSyncEditor.cs(39,147): error CS0246: The type or namespace name 'UMADynamicAvatar' could not be found (are you missing a using directive or an assembly reference?)

Assets\Crazy Minnow Studio\SALSA with RandomEyes\Third Party Support\UMA 2\Editor\CM_UmaSyncEditor.cs(39,49): error CS0246: The type or namespace name 'UMADynamicAvatar' could not be found (are you missing a using directive or an assembly reference?)

 

Are you sure you imported the correct file? I just imported the Salsa/UMA integration and had no errors. I don't even see that file below in the current package - all of the current files in that folder start with "salsa", not CM.

The file you need is the 1.7.0 version: UMA DCS 1-Click v1.7.0

 

Jaimi, that did it! Thank you!

 

@Crazy-Minnow-Studio @Jaimi I am trying the UMA setup per the video UMA 2 Lipsync using SALSA with RandomEyes , but see different results. When I select the UMA setup from the menu it looks as follows:



This looks different from what is in the video so I hope I have the right integration package. I downloaded uma-dcs_1_7_0

After the avatar is created in the scene the video doesn't say to do anything else, but shows that the SALSA_UMA2 object will have just 2 scripts attached: CM_Uma Basic and CM_Uma Sync. However, mine has a bunch more:



And is missing the CM_Uma Basic.

Then the video says to play the scene. And magically, the character is clothed, has hair, and is speaking with an audio file, but the audio file was never even added in the video. It seemed to already be there with the one-click setup.

In any case, when I run mine at this point, I just get a static character without clothing and not eye/lip sync.



If I analyze the Salsa components, you can see it is having issues:




So, what could I have possibly done wrong?

 

Hi magique,

Sorry for the confusion. There are two UMA add-ons, one current and the other deprecated:

• UMA 2 (Deprecated and is associated with the video you mentioned)
  • We posted a notice about our move to the DCS system on the add-on page. "11/26/2017 - v1.5 Final update. Since UMA has moved on to the DCS system, so have we."
• UMA-DCS (Current, but text only tutorial)
  • https://crazyminnowstudio.com/posts/salsa-add-on-for-uma-dcs/

Please follow the written tutorial for the current UMA-DCS add-on and you shouldn't have any issues.

 

OK, so that part works. Now I have another issue, which is not specifically SALSA related, but maybe you have the answer. If I save the resulting avatar as a prefab using UMA Power Tools, the SALSA eyes and lip-syncing no longer work at all. Do you know what has to be done for SALSA to work with prefabs created using Power Tools?

 

Unfortunately I don't, I've never used UMA Power Tools. However, the run-time setup subscribes to the UMA CharacterCreated event to add and configure all the necessary components. I assume if your character already exists at design-time, then you should be able to perform all the steps that are normally done in the CharacterCreated event within the SalsaUmaSync.cs script.

 

Hmmm. I'll take a look and see.

 

Do salsa characters need blend shapes for the animation to work at runtime (from streamed audio)?

 

Only if using 3D facial rig the 2D one doesn't need blend shapes and uses sprite key frame animation.

 

Hi,

Im using daz3d characters but I dont have the head morphs for each letter and I cant export my characters again because Ive already set them up in my game. Is there a way to get this working without even having any blendshapes on my daz characters?

 

I'll make some assumptions about your work flow here, so forgive if I'm way off...

Assuming you have the character set up with morphs on it in another 3D program, so you can add the extra SALSA morphs to it, then re-export the FBX, then re-import it in Unity. Depending on what scripts you've added to the characters in the scene, you can re-import the new mesh with the extra blend shapes and it will add them to the character without disturbing anything else. Make backups first, of course (of the original FBX, etc.) before trying this.

 

@Crazy-Minnow-Studio 2.0 done yet?!

 

Actually...YES! We're rolling into our second round of private beta testing this weekend. The first round excluded the Eyes module because it wasn't ready, this round includes everything. If no major issue are discovered, we'll likely submit to the asset store in the next couple weeks.

 

What if I dont have them can I use bones for the lipsync animation? And is it going to look good with bones for daz characters

 

I don't think SALSA works with bones, so you'd have to roll your own bone-based solution (I did in the past). You might want to load your DAZ FBX into Blender and add some simple morph shapes, if you want to use SALSA with it.

 

You can also use SALSA as the engine, but create your own mapping to bones or whatever you like using available SALSA API. We do this in the SalsaSync add-on to map computed blendshape output to eye and jaw bones. Our upcoming SALSA 2.0 allows native mapping to a wide array of things such as bone, blendshapes, sprites, textures, materials, and can be extended to support others. The snippet below from the SALSA 1.x manual describes the API you can hook into.

https://crazyminnowstudio.com/unity...uals/salsa-manual/classes-methods-properties/

Public properties

Salsa3D.say (string)

String value that is always updated with the current shape (Rest, Small, Medium, Large). This is useful for asset integrators that want to use Salsa3D as a lip sync engine with their own integration.

Salsa3D.sayIndex (int)

Int value that is always updated with the current shape (0=Rest, 1=Small, 2=Medium, 3=Large). This is useful for asset integrators that want to use Salsa3D as a lip sync engine with their own integration.

Salsa3D.sayAmount (SalsaBlendAmounts)

• Salsa3D.sayAmount.saySmall (float)
• Salsa3D.sayAmount.sayMedium (float)
• Salsa3D.sayAmount.sayLarge (float)

A custom type that contains the say BlendShape amounts. It's updated in realtime and provides direct access to the values even when no SkinnedMeshRenderer is linked. This is useful for asset integrators that want to use Salsa3D as a lip sync engine with their own integration.

Salsa3D.isTalking (bool)

Check this boolean value to determine when the audio file has stopped playing

 

Timeline Audio

Great asset, particular with the Unity timeline support.

Question: I would like to use a regular audio track on timeline and route its audio to the Salsa Lipsync component. Can't get this to work.
The custom "Salsa audio track" from the Salsa timeline addon is a workaround, but feels a bit buggy (trimming does not work very well) and there is no waveform to work with.
How can I route audio from a regular Unity timeline audio track to salsa lipsync?

 

Hello,
The SALSA Audio Track/Clip solution was created because the Unity Timeline Audio Track does not route. The SALSA Timeline clips auto-adjust to the audio length by design, they are auto-trimming to the AudioClip. You are correct, there is no waveform display. I'm not sure it is possible to do so with the public API framework at this point.

Please let us know if you have found bugs in the add-on and we will certainly take a look to see if they can be corrected. Additionally, the full source code is available for the SALSA timeline add-on, please feel free to modify it to better meet your needs if you wish.

Thanks,
Darrin

 

Thanks

 

Inspector Window scroll issue with Random Eyes 3D Component
I'm stuck with the Inspector window not showing all UI elements of the Random Eyes 3D component.
I just upgraded to 2019, in 2018 I had a horizontal scroll bar after adding custom shapes and groups to the component.
Now I can't get to the add groups button, it is cut off by the window. If I stretch the inspector, those UI elements are still cut off.
Any idea?

 

Hi Volkerku,

We are probably two weeks or so away from submitting SALSA 2.0 to the asset store. 2.0 is a complete rewrite of the entire system, and improves quality and flexibility across the board. We recommend moving to 2.0 once it's available. For a very limited time, existing customers will be able to obtain 2.0 for the $4 different in price between the existing price ($35) and the initial new price ($39). See our SALSA 2.0 dev blog for more information.

https://crazyminnowstudio.com/posts/salsa-v2-devblog/

Michael

 

Wanna have
Will the UMA Integration still work or will there be a new one?
Will it be compatible or changes to do?

thanks!

 

SALSA v2 is a complete re-write and, while the underlying methodology and goals are the same, it is not a drop-in replacement for SALSA v1. SALSA v2 is vastly more flexible and powerful than SALSA v1 and with all of that flexibility, of course comes some complexity. So, requirements and setup are going to be somewhat different.

We are still planning on UMA DCS integration and there will definitely be changes.

Hope that helps!
Darrin

 

With UMA DCS, will it have the ability to work with avatars ported to prefabs from UMA PowerTools? @UnLogick @Jaimi

 

SALSA 2.0 supports bones, blendshapes, sprites, textures, materials, and can be extended to support others types. I suspect a design-time UMA, created with PowerTools, should work like any other design-time character. However, we have not ticked the UMA box on our todo list yet so I'm speculating based on how other design-time characters work.

 

Waiting for Salsa V2

 

Hi
I’m using Salsa with lipsync and I want to emphasize talking more with additional blenshapes such as eye squinting or eyebrow lifting as the player speaks into the microphone.

I need a way to detect when the player is speaking and when the player has stopped speaking,
is there some way I can find that out from Salsa?

Kind Regards

Roy

 

@Crazy-Minnow-Studio Its been 2 weeks+

 

Yes it has!

 

SALSA v1 has a flag you can test (SALSA.isTalking). This will simply tell you if the AudioSource is currently playing, which could be playing and silent. You could also test the SALSA.average value to be greater than some arbitrary value to determine the level of audio. Or you could roll your own on the AudioSource itself. Or you could test to see if/which trigger is active. Etc. There are a lot of ways to achieve this.

In SALSA v2, SALSA can link with EmoteR and has its own trigger threshold to fire emphasis emotes. In other words, this is built into SALSA v2. SALSA v2 is currently being wrapped up for submission to the Asset Store. We will notify everyone when that has happened, but we expect this week.

 

https://crazyminnowstudio.com/posts/salsa-v2-devblog/

They've been very candid about their development schedule which is way more than most asset developers out there.

 

Hey,

Excited for Salsa V2, have been waiting for a long time but I know it's going to be worth to wait. So I'll continue to wait as long as it takes because we loved V1 and V2 should be no exception. Meanwhile, I got a quick question about two of your videos. First one is this (

) and the second one is this (

)

Now I am a newbie so pardon me if this is a silly question to ask but I want to know in both of your videos, the characters move slightly. The character in the first video, the entire body moves slightly like this: https://d.pr/free/i/Bv2ytc

And the character in the second video, it actually turns (YAW) like this: https://d.pr/free/i/GNSEIr

So I wanted to ask, is this randomized and a feature of Salsa? Or you did this using a third party app to add these effects to your chracter (move body + yaw turn)? If third party, then please let me know which tool you used because I have spent a lot of time just trying to achieve this same effect but no luck (I am a newbie as I said).

I'll greatly appreciate your help for this,
Thank you!

 

Thanks for the kind words!!!

OK, so for the Morphmixer video, Mike did that one and I believe it was just a simple idle animation (probably from the Unity Ethan model). The second video (the SALSA v2 teaser) was actually head tracking/movement built into the product. So you'll get that as part of v2 (for head movement -- that's as far down the bone chain as we go! ).

We really appreciate everyone being patient for the v2 release. It's a much bigger beast than SALSA v1 was and as such we're trying to make sure it's as rock solid as possible. With all of the new flexibility comes a lot of complexity and we're trying to keep the system as simple as possible to maintain its friendly nature.

Hope that helps!
Darrin

 

Hey @Crazy-Minnow-Studio , can you tell me exactly which phonemes did you use for the second video? (the tech preview with the talking head)
I have to move forward and finish the characters, so it would help me A LOT if you could tell me exactly the amount and WHICH phonemes did you use for this result. I know Salsa doesn't use phoneme recognition, but you obviously used different phoneme poses for that video, not just open-closed mouths.

 

The shapes/amounts used will vary by model type and that was an MCS model (which we no longer support and is no longer in my project). Generally speaking and subject to change, at this point, I'm configuring the following (approximate) visemes in the trigger groups: (f, w, uw, t, ow, ee, th, ih).

Hope that helps,
Darrin

 

That was very helpful, thanks

 

We are thrilled to announce that SALSA 2.0 has been submitted to the Asset Store for approval.
We're working hard to get the rest of the manual, the add-ons, and new tutorials posted as soon as possible. Just a reminder, as soon as v2 is posted v1 will be deprecated. Don't want to upgrade to 2.0, no worries, v1 will still be available to existing customers but will not receive any new updates. If you want to upgrade, the upgrade price will be $4 for a limited time.

 

Nice to hear! Will the upgrade still be $4 until the addons (for me personally UMA) are published or should I buy it as soon as possible?

Thanks

 

The $4 upgrade offer isn't linked to add-on availability, it's just our way of thanking our existing customers who want to get the upgrade right away. It will only be offered at $4 for a limited time since we have to eat as well.

 

What's the full price of Salsa v2 by the way?

 

The full purchase price will initially be $39.