SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

UMA OneClick update:
We are nearly wrapped up with the OneClick. As Mike mentioned, we have switched from an all-bone implementation back to using the ExpressionPlayer as we did in v1. While SALSA v2 is certainly capable of using an all-bone implementation, version 1 used the ExpressionPlayer and we figured it was best to continue that methodology. Using a bone implementation would technically be a bit more of a hard-coded setup and would be more difficult to adjust or create new emotes or visemes. Using the ExpressionPlayer is very similar to using blendshapes on other models.

Also, we were able to integrate a live-preview-update at design-time using the ExpressionPlayer and leveraging the same avatars the UMA Core Tool - Expression Clip Creator uses. This is a great way to get a good idea of how the configuration changes look for visemes/emotes.

I should have the SALSA/EmoteR OneClick finished today and will hand-off to Mike to finish up the Eyes portion. This will require an update to the core SALSA LipSync Suite files, so that will have to be submitted before we can publish the OneClick.

 

Salsa 2D keeps coming out null material, please help!

I have set sayRest, saySmall, sayMedium and sayLarge; however, during playback, it kept coming out null (though sayRest, saySmall, sayMedium and sayLarge are presented), please help, thanks a lot!!thanks a lot!! The following images show my setup

 

Cute lovely character, Sphoenix!

 

Hey Sphoenix!
You have "null rest" enabled. Null rest will blank out the rest state (no image; or in this instance, no material). This is an option for flexibility but in normal usage, you'll want to include your null.png image as the rest sprite (on each of the visemes). Also, you should choose sprite (or texture) as your controller. The material switcher controller is a special use case that probably won't get much use -- but it is there if people want/need it.

The errors look like they're coming from another system/script (ImmediateStylePainter.cs?) probably because your rest position is being nulled.

Hope that helps!
Darrin

 

Hello all!
We have submitted SALSA LipSync Suite v2.1.0 to the Asset Store. This update paves the way for the UMA OneClick. It also adds some other nice updates for usability and of course, a few small bug-fixes. It is a required update for UMA and also for OneClickBase v2.1.5+.

For anyone wanting the ability to target a Texture controller type to a specific material in multi-material renderers, this update brings that capability as well.

​

Once the update is posted on the Asset Store, we will update the OneClicks and upload the UMA OneClick. This will signify the last of the addon/one-clicks outstanding on the v2 deployment. Yea!!!

 

Does that potentially mean wrinkle maps?

 

OOOOOooooh... that's a REALLY good idea...

 

Potentially! Is a wrinkle map a texture or a normal map? The texture controller applies strictly to texture. Are you considering a separate texture applied for something like a squint? Do you have an example?

 

Wrinkle maps are normal and texture maps that swap gradually the default textures of a model.
In order to implement them, we would need 2 things:
1- a shader that swaps textures gradually using a 0 to 1 float parameter
2- for SALSA to control that value based on emotions or visemes

That way, you can assign a shader parameter value to each emotion and let SALSA rise or lower it in realtime based on emotions or visemes.

Example: the user creates a custom parameter in SALSA named "WRINKLES"
The name of the custom parameter must match the name of the parameter you want to change in the characters material.
Once created, that parameter value can be set in every viseme or emotion inside SALSA
On runtime, SALSA linearly morphs this value depending on the current viseme or emotion and set it to the character's material using Material.SetFloat("WRINKLES",value)

Then a custom shader can use this float value to mix wrinkle maps or to animate any other property.
That would be very useful.

 

Actually, the parameter value should go from -1 to +1 since there are 3 wrinkle stages:
-1: full extension (surprise face)
0: relaxed (idle face)
+1: full contraction (angry face)

(by default, the value should be set to zero)

 

I guess here's an example in video that's just the normal maps….

As @atomicjoe says, it's basically matching each emote and viseme with another face texture that's triggered by the blendshapes (or in this case by SALSA)…. I think the more realistic ones use texture and normal maps...

 

Hello @wetcircuit and @atomicjoe!
The Texture controller type is a "switcher" type that switches out textures based on a [0..1] interpolation of texture frames. This is not a blend, but a swap. It was designed for 2D animators using any number of cell frames to create animation. While this could be used in a rudimentary fashion for what you are describing, it isn't quite the same thing. The Texture, Sprite, and Material controllers operate on the associated renderer and not the shader so cannot produce infinitely "blended" effects and requires a sufficient number of images to create a smooth animation.

However, all of the underpinnings are present in SALSA to make this work like a champ. The ability to mix and match any controller types together makes this trivial if the appropriate controller type is implemented. I'll look into this some more and see what the requirements are. Unity did some work on this with the Blacksmith and their package is available on the asset store. Perhaps this can be leveraged as a required starting point.

Thanks a bunch!
Darrin

 

Hi,

What does my character need to have in order to have lipsync? Do I need bones or blendshapes? Is there a way to automatically generate the blendshapes in another software?

 

The problem with [-1 .. 1] blending is the opposite poles cannot exist together. This may or may not be desired, but using counterbalance blending certainly prevents it. We can implement this sort of blending (the capability was added in v2.1.0 for UMA), but I would probably recommend against it for flexibility.

 

If it's a 3D character, you would need either pre-made blendshapes or pre-made bones in the face to animate it.
Blendshape generation is part of the actual character modelling process and facial bones part of the character rigging process. SALSA expects both of this things to be done in advance and doesn't help you with it.
Normally, those processes are done on 3D modelling and animation applications like Blender, Maya or 3DStudioMax MANUALLY.
There is however a plugin for Unity in the assetstore that could be useful for you if you don't master those other aplications: PUPPET3D
With this plugin you can rig any 3D model inside Unity using bones. I have never used it to rig a face, bout it could be done.
However, keep in mind rigging is a very complex technical process. If you don't want to dig this deep into technicalities, you should use third-party 3D models ready to use, like UMA or Adobe Fuse.

 

Hi skinwalker,

Yes, what @atomicjoe said. Additionally, SALSA LipSync Suite currently supports the following:

• Blendshapes
  
• Bones (position, rotation, scale)
• Sprite swap (single or mult-frame)
• Texture swap (single or mult-frame)
• Material swap (single or mult-frame)
• UMA (Expression Player [as of SALSA Core v2.1.0 && OneClickBase v2.1.5]

We are currently using these phonemes (w, t, f, th, ow, ee, oo), but you are free to experiment and use what works best for your style.

 

SALSA v2.1.0 is live on the Asset Store!

I have uploaded the UMA DCS OneClick (v2.0.0) and a new OneClickBase v2.1.5. The UMA OneClick ONLY works with SALSA core v2.1.0 and OneClickBase v2.1.5.

All OneClick documentation can be found here:
https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/

I've also freshened up the OneClick packages for DAZ, Fuse, and Reallusion. Minor updates on some. A couple didn't have the prefab fix code in them and now do.

 

Is there a PlayMaker add-on for SALSA2? I need EmoteR control.

I checked the website, didn't see it….

 

Hi wetcircuit,

It's available on the downloads page. The documentation is current but hasn't been moved to the new documentation system yet, it's still on the blog page.
https://crazyminnowstudio.com/posts/playmaker-actions-for-salsa-with-randomeyes/

 

Hello.
Tell me, can I customize Salsa to work with the Russian language?

 

SALSA works with audio volume. It can be any language (it could be beeping and whistling).

 

Im having trouble changing facial expressions with Salsa with Randomeyes shape groups.
I can create a group with a new expression and preview it and I can see the change in the expression in the model:



When I run the game, if the PREVIEW is active in the Salsa properties window, the character retains the expression, but if I turn the preview off, he loses the expression.





I try to turn the preview off and call the expression with the third party support action re3D_SetGroup, but this doesn't seem to do anything.

I have also tried to leave the preview setting on and use re3D_SetGroup to change the expression and that doesn't work either.



Can someone help?

 

Hi PinheadGames,

I gather from your screen shots you're using SALSA 1.x with Adventure Creator. The Re3D_SetGroup action appears to be setup correctly. Is this action being triggered? Is the RandomEyes Target linked? Please email us at assetsupport@crazyminnow.com and send your SALSA invoice number.

Michael

 

Hello.

We have been using V1 for a long time and excited to get V2 working. However, we hit a snag. The 3d model outputted from AvatarSDK used to work seamlessly with Salsa V1.

But after the upgrade to Salsa V2, the lip-sync etc does not work.

So my question is, if we are transitioning from V1 to V2, what is the easiest way to get our models working with Salsa v2 because we really like the features of V2 and have already bought it.

Any help would be much appreciated.

Thank you!

 

Can you explain "does not work"? Do the demo scenes work?

 

Hi MindSparkInteractive,

As @wetcircuit mentioned, we will need more information about what you're doing. I don't know much about AvatarSDK, but I believe this is a run-time system? If that's the case, we recommend looking at our newly published SALSA API.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/api/

Our EmoteR and Eyes API's will be published soon. You may also want to have a look at the code-based setup scripts included in our one-click scripts as they demonstrate how to setup various character attributes from code.
https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/

SALSA v2 is a complete rewrite and is not backwards compatible with v1, so any setup scripts used for v1 will need to be rewritten for v2.

 

API information for EmoteR has been published.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/emoter/api/

 

Hello everyone!

Unity v2019.2 has been released into the stable channel. We are testing now to confirm SALSA LipSync Suite v2 compatibility. So far, so good -- no code errors. A warning is issued for the boxHead model -- based on the new smoothing groups Unity is looking for on the blendshapes. The model still works fine, just has this warning on import.

boxHead test scene works fine in editor and also with a PC build!

2019.2.0f1 looks good so far folks! Let us know if you see any issues.

 

Hello,

I want to upgrade my project to latest Salsa and Uma, then test new Salsa-Uma Integration.
Did you test it with UMA on 2019.2 already?

Thanks

 

One more Question:
Did anybody try the UMA-compatible Avatar maker over here:
https://assetstore.unity.com/packages/tools/modeling/avatar-maker-uma-150931

Can somebody confirm the heads Build with it support Salsa for lip and eye movement?

Thanks

 

Nope. Just SALSA. Let us know how it goes. Always backup before you upgrade a project.

 

API information for Eyes has been published.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/eyes/api/

 

Hello,

In addition to this matter, i have this scenario: i have two avatars (from MCS Morph), a male and a female one. I can't have the lipsync working on both of them, but when i made a change on a variable of the male avatar (for example, when i check/uncheck the option "Reset Blendshapes on Start" from the script "M3D Character Manager"), save the scene and build again, the lipsync on the male avatar starts to work, however on the female avatar it still doesn't work. If i do the same, but for the female one, it's the opposite, the lipsync works on the female avatar but doesn't work on the male one. If i do another build, without any change, the lipsync fails in both male and female avatars. I have both avatars on the same scene, by the way. To remember, i export the Unity project to an Android Studio project and build the app from there. What could be the problem?

 

Okay, I have just upgraded to Salsa V2! I still have v 1.x running in my game. What do I need to know about removing the old components and bringing in the new Salsa components? How do I properly upgrade my project?

 

Hello tiago_carvalhais,

Morph3D has dropped their support for MCS characters, so we no longer support MCS characters as well. Of course you are free to download our 1-click and continue to use them without support. We're happy to offer SALSA specific assistance within the Unity environment but can't offer much help in an Android Studio project.

Michael

 

Hello PinheadGames,

Always backup your project before making any big changes.

SALSA v2 is a complete rewrite of the entire system and is not backwards compatible, or v1 add-on compatible. We recommend removing the entire [Crazy Minnow Studio\SALSA with RandomEyes] directory before installing SALSA v2. If you do not have any of our other assets, you can remove the entire [Crazy Minnow Studio] directory.

After SALSA v2 is installed, head over to our Adventure Creator add-on page, and read and download the new add-on. It has been updated to reflect the new v2 compatible Adventure Creator add-on rewrite.
https://crazyminnowstudio.com/posts/latest-version-of-adventure-creator-supports-salsa/

Michael

 

I'm mostly getting things back up and running with v2. Are there some tutorials available for using the new system?

Additionally, I have a character in my game that I outsourced from a freelancer so I have no idea how it was created. The lipsync ALMOST works, but only the lips and jaw moves, not the teeth. The character worked fine in Salsa v1.x. Any help would be appreciated.

 

For the visemes that use teeth – you might be able to copy the jaw setting and then replace the "body" mesh with the "teeth" mesh. (how ever they are named on your figure)

 

As @wetcircuit said, you can potentially just link the teeth in, it just depends on what your character has available for animating. We have quite a bit of documentation published already, but no new video tutorials yet. We're still working on documentation and new video tutorials. There's some good information in the overview that discusses what we use in our character system one-click setups.

https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/overview/

 

Hello,

I just removed all the older Salsa stuff from my project, then Imported Salsa LipSync from asset store, downloaded and installed the OneClick UMA asset package.

Is there a update documentation for UMA? I know the workflow has changed, but to Point me how to update my exisiting classes? I used CM_UmaSync and cannot find some similar...

I am creating every UMA in Code from a basic DCS prefab.

Thanks

 

Hi Firlefanz73,

Once your UMA is configured with the UMA one-click, you can control SALSA, EmoteR, and Eyes respectively with their standard API.

https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/api/
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/emoter/api/
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/eyes/api/

 

Hi,

I made a new character in zbrush from scratch now I want to add blendshapes for lipsync, so what kind of blendshapes I need to make the character able to say any word? Also I have human character and another humanoid but with lion head so I guess it will be harder

 

Hi skinwalker,

We use a seven expression configuration in our one-click setup scripts for the various character generations systems we support. This isn't required, but we found it produces good results. It really depends on the look you're going for and your desired workflow since you are creating everything yourself. SALSA v2's advanced dynamics feature produces much more dynamic result with fewer shapes as well. We've tested characters with the v1 shape requirements and they look much better with advanced dynamics enabled.

Please see our v2 documentation for more details.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/overview/

 

Hello,

I removed the old Salsa. Then I imported Salsa Lypsync from the asset store and I imported the UMA OneClick asset package.

I get this error message:
Assets\Crazy Minnow Studio\Addons\OneClickUMA\OneClickUmaDcs.cs(5,32): error CS0246: The type or namespace name 'OneClickBase' could not be found (are you missing a using directive or an assembly reference?)

What am I missing? I am using Unity 2019.2.0.

Thanks!

 

Hi Firlefanz73,

Please see our one-click documentation.

https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/

 

Thanks!

 

Hi guys, I'm attempting to trigger the audio from a State Behaviour, which appears to run inside Unity fine. However, when I attempt to Build an Android package. I receive error CS0246:

error CS0246: The type or namespace name 'Salsa3D' could not be found (are you missing a using directive or an assembly reference?)

I'm fairly new to the Unity scripting scene and suspect it's something simple but I'm missing a link somewhere. Any help would be greatly appreciated.

Aaron.

From my State Behaviour script:

Code (CSharp):

1. [IMG]http://X:\OneDrive\Being_VR\Preview_Images\Salsa_Build_Error.png[/IMG] using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using CrazyMinnow.SALSA; // Import SALSA from the CrazyMinnow namespace
5.  
6.     public class LucieState_Behaviour : StateMachineBehaviour
7.     {
8.         public Salsa3D salsa;
9.  
10.         // OnStateEnter is called when a transition starts and the state machine starts to evaluate this state
11.         public void OnStateEnter(Animator animator, AnimatorStateInfo stateInfo, int layerIndex)
12.         {
13.             salsa = (Salsa3D)FindObjectOfType(typeof(Salsa3D));
14.             salsa.Play();
15.             Debug.Log("Playing Audio");
16.         }
17.  
18.     }
19.  

 

Hello Aaron!

It looks like you are using the legacy version of SALSA (version 1.x). This error typically is because the asset is not imported into the project. Can you confirm you have SALSA with RandomEyes imported into your project?

 

Ok. I've imported the SALSA LipSync v2 Suite. Which is what I believed to be the latest one. Is that right?

I'm just removing all the Salsa packages and re-installing from scratch. If I need a different asset package, can you link me to the correct location?

Kind regards

Aaron Munro

 

Yes, SALSA LipSync Suite v2 is the latest version. That being said, you are referencing the version 1 classes in your above code -- Salsa3D. The new class is `Salsa` regardless of 2D or 3D. With what you are trying to do, start the AudioSource playing, you can do that directly on the AudioSource, you don't need to do it from Salsa.

AudioSource audSrc;
audSrc.Play();

You can also do it via Salsa, if you prefer, but via its reference to the AudioSource. For example:

Salsa salsa;
salsa.audioSrc.Play();

Check out the API docs for more info.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/api/#references