SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

If they have a hinge jaw, you might be able to use SALSA Sync to work the mouth open and closed, but for mouth shapes (visemes) yes you will need blendshapes.

 

Sweet! 2.0 out yet?

 

That is correct. As wetcircuit stated, you can also get some movement using only a jaw bone, but the result will typically look better with blendshapes.

We have a 3-part tutorial series on creating blendshapes in Blender.

 

The Eyes module is still in the oven, you can follow our progress here...
https://crazyminnowstudio.com/posts/salsa-v2-devblog/

 

Hello,

the Monster I'd like to add blendshapes has no Format I can import. It is a fbx I cannot Import with Blender or Carrara...

Could you please give me a hint how I can try with ther jaw and other sub-transforms, how can I use them instead of blendshapes? I have only used it with UMA until now…

Thanks!

 

I don't see a way to do this without spending a little money.

I recommend SKINN:VM in the Unity asset store. The idea is that you bring the FBX into Unity, right-click on the skinned mesh render component and select “Bake to Mesh Filter”. This creates a copy of the figure without the rig, which can be exported as OBJ (any number of free OBJ exporters for Unity). Edit the mesh to create your various blendshapes, and export each as separate OBJ, import these new meshes into the Unity scene, and merge them back onto the rigged figure with SKINN:VM as blendshapes. The new figure can be saved to Unity as a prefab, and SALSA will work with the custom blendshapes.

 

Just a note about Carrara: As much as I like it myself, FBX import and export in Carrara is corrupted.
Carrara export to FBX will break Unity long term. It seems to work, as it imports in Unity just fine, but you WILL have big issues in the long run: inverted rotation axis, wrong vertex information, broken skeletons and even runtime crashes! (try using a cloth component on a Carrara FBX and the cloth will behave erratically)
This seems related to Carrara's skeleton information so, as long as you DON'T use skeletons in your FBX, it should be fine. But since FBX is primarily used for characters with skeletons, there is no much use for it anyway.
What DOES work ok (as long as I know) is using FBX to export objects without skeletons but with blendshapes (morphable objects). Exporting to OBJ meshes from Carrara is ok too. (keep in mind the import scale for Carrara models in Unity is 0.0254 if your mesh doesn't import at the correct scale)
This will not help you with your current issue I'm affraid, but will improve your sanity in the future

It seems Blender has some compatibility issues with Unity too, but at least you can find tons of tutorials about Blender with Unity on youtube.

EDIT:
It should be noted that FBX export only works consistently on Autodesk software (3dsmax, maya...)
Other 3D software can have FBX weirdness.
Daz Studio is working perfectly though, so maybe you should try making your blendshapes with Daz Studio and Hexagon and then exporting FBX to Unity.
This DOES work.

 

Thanks for your hints!
We are using Carrara for static gfx only. But we are moving to blender more and more…

And until now all animated stuff like animals and monsters where bought from the asset store, and we are using UMA for humanoids. This was perfect for us until now. But now we want some funny cutscenes with some Monsters talking, and since we are hobby game developers I am not sure if we want to spent another over 30$ just for that. I hoped this would be possible with Salsa using transforms / sub-meshes. I Need to think about it, thaks for the suggestions.

We are using Salsa already with UMA that works very well.

 

When I disable `Show Gizmo` on the `RandomEyes3D_Eye_Position` game object (to remove the googly eyes thing!) it keeps coming back as soon as I run my app. What does this happen? I do save the scene obviously! But it keeps re-enabling...

 

Hi all, recently purchased last Friday and so far I love it. However, I don't know if this is because I used the 1 click addon for Fuse characters, but I cannot change the look sensitivity on random eyes 3D of our character models as the slider is simply missing. I can change the look range of motion which helps, but the eyes really don't "lock on" as they should. This is used in VR, so if the character model isn't looking right at the player, it looks kinda funny similar to you being able to easily tell if someone has a lazy eye.

With my script I used in the past, which was just a simple Quaternion.Slerp() on each eye, the character model really locked on to the user in the headset and really brought it alive as far as eye contact goes. If this addon can do so (which I was under the impression it was), then this addon is perfect for what I need. Is there any other way to either:

1. Show the slider?

or in general

2. Control/have more accurate "Looking at" abilities with the eyes?

Any help is appreciated. Thanks!

 

Are we there yet for v2.0 ?

 

Hi vibedev,

I have tried to reproduce the symptom you described with the RandomEyes3D_Eye_Position gizmo [Show Gizmo] field automatically re-enabling at run-time, but have been unable to reproduce it. Maybe there is a rogue instance somewhere in the scene? If you use the tiny search drop down in the Unity Hierarchy, select "Type", then search for RandomEyes3DGizmo, how many instances do you see, and does this match the number of characters you have setup in the scene?

Michael

 

Hi TJD269,

SALSA 1.x performs eye targeting calculations from a single point (the eye position gizmo). It calculates natively for blendshapes, and uses a conversion to bone translations in our 1-click setups scripts. In our upcoming SALSA 2.0 (a complete rewrite) all calculations are bone-based, then conversions are made to accommodate a wide variety of other eye configurations, and all eyes are processed independently to produce much more accurate and realistic eye tracking and movement.

Since we are still applying finishing touches before the release of SALSA 2.0, we have posted our Fuse 1-click v3.8.1 update that includes an eye sensitivity slider and defaults to a more accurate value.

SALSA 2.0 Progress
https://crazyminnowstudio.com/posts/salsa-v2-devblog/

Fuse 1-click
https://crazyminnowstudio.com/posts/using-mixamo-fuse-character-creator-with-salsa-with-randomeyes/

 

Hi atomicjoe,

You can follow our SALSA 2.0 progress here.
https://crazyminnowstudio.com/posts/salsa-v2-devblog/

 

Wow just outstanding. Thank you so much for the update, I reimplemented and it is vastly better! Thank you for updating that for me!

I cannot wait for 2.0 as individual eye tracking will still be better, but this is miles ahead of what I had going yesterday.

This saved me a bunch of time, thank you so much! I will gladly go leave a nice review on the asset store now!

 

I know.

 

2.0, 2.0, 2.0!!!!

 

Really looking forward to 2.0 and being able to use this with master audio.

 

Hello I'm using Salsa for the first time with a DAZ character. I'm following this tutorial https://crazyminnowstudio.com/posts/using-daz3d-characters-with-salsa-with-randomeyes/

I have imported the Genesis 8 character from Daz and am stuck at the #2 in the second section. It says Open the [{FILENAME}_base.fbm] folder and select the eyelashes texture, my file is called [V5BreeLashes1.jpg]. Check the boxes next to [Alpha from Grayscale] and [Alpha Is Transparency], then click [Apply].

First I don't see a folder called __base.fbm I can find eyelash asset in teh All Materials section, but when I click on that it does not look like the image they show in the tutorial at all. It was an importer so I did import everything. I don't see any texture.jpg or alpha settings etc. Here is a screen shot and I clicked on "6_Eyelash" in All Materials. Any ideas?

 

Recent versions of Unity don't extract textures and materials from the model anymore by default (it uses them from inside the FBX now)
You have to explicitly tell Unity to extract the textures and materials so you can access them in the assets folder.
Check out the manual HERE.
(use the "Use External Materials (Legacy)" option at the bottom of the page)

 

Thanks AtaomcJoe! Okay I'm past that issue, now here is where I am. Any help is appreciated!!!

 

Hi. I want to retrofit some lip sync to an existing fbx model in a Unity project. There are various arbitrary blendshapes from which I can make the 4 shapes, but they don't conform to what seems to be expected with SalsaSync. It would be useful if SalsaSync could (1) accept a setting for rest, as in my case the rest (zero) position of the relevant blendshape has the mouth open and (2) handle inverted scales for when a target position is before the rest position (that is, the shape increases in the negative direction). It doesn't look like we can currently do this sort of mapping, is that correct?

 

@Crazy-Minnow-Studio I'm using SALSA on some MCS characters and seeing an issue. I like to modify the MCS hierarchy by removing the higher detailed LODs and going to just a single LOD such as LOD2 or LOD3. However, the CM_MCSSync script gives the following error:

UnassignedReferenceException: The variable skinnedMeshRenderer of CM_MCSSync has not been assigned.
You probably need to assign the skinnedMeshRenderer variable of the CM_MCSSync script in the inspector.
CrazyMinnow.SALSA.MCS.CM_MCSSync.VerifyIndexes () (at Assets/Crazy Minnow Studio/SALSA with RandomEyes/Third Party Support/MCS/CM_MCSSync.cs:320)
CrazyMinnow.SALSA.MCS.CM_MCSSyncEditor.OnEnable () (at Assets/Crazy Minnow Studio/SALSA with RandomEyes/Third Party Support/MCS/Editor/CM_MCSSyncEditor.cs:39)

And I can see that it is unassigned in the inspector. But if I drag my SkinnedMeshRenderer of LOD2 or LOD3 into the slot to assign it, it refuses to let it set that. Is there some reason this has to be LOD0 only?

 

Hi MarkHenryC,

Our SalsaSync add-on tries to be as flexible as possible for mapping SALSA and RandomEyes requirements to a wide range of character designs. However, we have leveraged all of the major character generation systems as a pseudo standard of character design primarily because it's what many of our customers use, and most people building custom characters can relatively easily add the simplified blendshape requirements our system has.

SalsaSync cannot handle your two bullet points requests directly. However, this is possible if your willing to do a bit of coding and leveraging of our API. The fully processed SALSA and RandomEyes output values are exposed, as is some useful state information. See the following manual pages for details:

https://crazyminnowstudio.com/unity...uals/salsa-manual/classes-methods-properties/
In the section Salsa3D > Public properties > Salsa3D.say, Salsa3D.sayIndex, Salsa3D.sayAmount, and Salsa3D.isTalking

Rest as a shape
There are a number of approaches you could take, but the basic idea for dealing with a rest that isn't the inactivity of all other shapes would be to use the Salsa3D.isTalking = false state to trigger your rest blendshape. You could stick your rest blendshape in RandomEyes3D custom shapes or a shape group and activate it with the following API accordingly:

• RandomEyes3D.SetCustomShapeOverride(string shapeName, bool overrideOn)

Activate a custom shape by name, with override (true), this will activate it regardless of the status of other custom shapes or the [Current Custom Shape]. Deactivate with override (false).

• RandomEyes3D.SetGroup(string groupName, bool status)

Activate or deactivate a group of BlendShapes.

Inverted scales
Inverting a scale is easy once you have access to our processed output values, see Salsa3D.sayAmount above.

I hope that helps, good luck!

 

Hello magique,

Our MCS 1-click was designed to work with the existing LOD switching built into the M3DCharacterManager. Adding our 1-click to a default MCS character activates all of the blendshapes that we use in the MCSSync inspector on the LOD0 mesh...more about this in a second. MCSSync subscribes to a MCS API LOD change event that triggers the SkinnedMeshRenderer remapping within our system. The blendshapes that we activate are initially only activated on LOD0 since the other SkinnedMeshRenderers are all disabled. Only upon switching to another LOD level are the blendshapes also activated on the other SkinnedMeshRenderer components. This is a function of the MCS system, not SALSA.

I recommend setting up your characters without modifications to the LOD structure, then use the LOD slider in the M3DCharacterManager inspector to set the LOD you want (which has the added benefit of activating those blendshapes ahead of time).

FYI, due to lack of support for the MCS system from the vendor, Crazy Minnow Studio no longer provides support for these models. There will be no further support or modifications for this add-on. You are free to continue using it as long as it works for your needs, but we cannot spend time on any system where the author isn't wiling to spend time.

Good luck!

 

Thanks for the response. That all makes sense. Just wanted to be sure I wasn't missing anything before starting to code custom behaviour.

 

Salsa works great with an mp3 and microphone input, but would it be possible for Salsa to access audio that we have streaming for lip sync? I could use this asset Audio Streaming but am not sure how to connect Salsa to see the stream as it's audio source. Any suggestions? https://assetstore.unity.com/packages/tools/audio/audiostream-65411

 

As long as the 'streaming' package provides an AudioClip to an AudioSource, it should work in SALSA 1.x and 2.x. Since streaming is typically an asynchronous process, you will need to handle the timing of the file's preparedness. The AudioSource will need to have sufficient data before the data is available for further processing. SALSA v2.x requires uncompressed audio since it analyzes audio earlier in the pipeline to avoid issues like spatial falloff, muted AudioSources, etc.

NOTE: SALSA's audio processing does not support the use of WebGL out-of-the-box. WebGL usage requires Amplitude for WebGL (available on the Asset Store https://www.assetstore.unity3d.com/en/?stay#!/content/111277).

 

Any updates coming up soon on the dev blog for 2.0?

 

Will 2.0 allow position changes of eyelid bones and not just rotation?
2.0 info says, "...It can manage transform rotations, clamped position changes ...", which makes be think so.

I am working with 3DMax models, and the eyelids have bones for translation (not rotation), so I want to use SalsaSync, but I can't move the upper eyelid because I need to change the position of the eyelid bone along the local normal of the y-axis of that bone, and only rotation is provided.

1) Is there a current solution you provide that I am not aware of?
2) Will 2.0 handle eyelid bone translation?

 

1) We don't offer a solution, but you could modify SalsaSync since it's already translating blendshape values to rotation values.

2)Eyelid control is being written now. The idea is to work similar to eye control by offering templates that adapt the math behind the scenes to be applicable to the template selected. For example, eye control offers the following templates:

• Bone_Rotation (typical in 3D characters)
  
• Bone_Position (typical for sprite transforms)
  
• BlendShapes (SkinnedMeshRenderers)
  
• Sprite_Sectors (e.g. forward, up, up right, right, low right, down, low left, left, up left)
  
• Material_Sectors (e.g. forward, up, up right, right, low right, down, low left, left, up left)
• Texture_Sectors (e.g. forward, up, up right, right, low right, down, low left, left, up left)

However in the case above, Bone_Position is for orbiting an eye sprite around a center point with an adjustable offset or range of motion. Eyelids are a bit different since there are two distinct operations we're evaluating: blink, and tracking. Blinking is relatively simple across a wide array of character designs since it's really just a state change, but tracking is a bit more involved when evaluated across the same wide array of character designs from 2D to 3D. The short answer is that we can't provide a definitive answer yet.

 

Hi,

I was trying to get Amplitude installed to work with Salsa (both purchased). When I tried to enter my invoice details to get the plugin, it refused my invoice. Then the website seems to have crashed. Not sure what I can do.

thanks

Richard

 

Hi Richard,
Not seeing any problems with the site. Send us an email (with your invoice number) if you're having issues and we'll get them sorted out.

Thanks,
Darrin

 

Actually, I did just find a couple of small issues that might have caused the issue you experienced. I believe I have corrected them. Let us know (email) if you have any further issues.

Thanks!
D

 

Tried again today. Site is working and my error was putting in the wrong invoice number (The Salsa one and not the WekGL). All good now. Many thanks

Richatrd

 

I use SalsaSync on a custom character.
The character has the saySmall, medium and large blendshapes and these work fine with he salsa 3D component. But once I add salsaSync and specify the eyeLid and eye bones, the character mesh starts to behave strangely. Almost as parts of the body move together with the mouth blendshapes.
Any idea?

 

It sounds like you might have bones linked in SalsaSync that aren't eye or eyelid bones. Hard to say for sure without seeing your full inspectors for Salsa3D, RandomEyes3D, and SalsaSync.

 

Hey @Crazy-Minnow-Studio !
I'm making the final character for my game and I would need to know what phonemes blendshapes do Salsa v2.0 needs for the best quality.
Also, I'm using Daz3D character Genesis 1, which uses both blendshapes AND bones for moving the face and the jaw (the blend shapes control the skin deformation but the internal lower mouth, tongue and teeth must be moved rotating the jaw bone)
Will this be a problem?

Also, you know what would be awesome for Salsa V2?
An audioless mode.
That would be a mode were Salsa makes a character move it's lips without audio for a specified time.
You see, it's very common for games to not do voice-overs of ALL dialogues but only the important ones, since voice-over is expensive.
That way Salsa would still animate the mouth of the character (even if it's just random movements)

 

I also really could benefit from a feature like this.

 

Is there anyway of doing something like this with the current version of salsa?

 

Yes, SALSA 1 has an add-on script called TextSync:
https://crazyminnow.com/posts/salsa-text-to-lipsync-using-textsync/

 

The "full" list of possible visemes (mouth shapes) is the 10 Preston Blair phonemes (mouth sounds):
(actually 9 and a rest shape)

• AI

• E

• U

• O

• CDGKNRSThYZ

• FV

• L

• MBP

• WQ

• Rest

This is overkill for SALSA since SALSA doesn't differentiate the phonemes, just assign visemes to various volume gates – cross a volume threshold and SALSA triggers a mouthshape…. I think videos of SALSA 2.0 show 7(?) and theoretically unlimited(?), but it might really depend on the figure's lips and how much mouth movement looks realistic for the genre –

imo, too much movement looks weird on realistic figures as people in real life mumble and barely use their lips when speaking, but a toon figure with a large articulate mouth might look better with more….

But to future proof your figure, the 10 Preston Blair phonemes supposedly cover everything we can say. I usually combine W U and Q, so that's one less. The rest shape is usually the unmorphed mouth…

Since SALSA is volume-based, it's maybe a good idea to add some mouth shapes for shouting or singing (something bigger, rounder than a normal viseme) at the high end.

 

Wetcircuit (as usual) offers some awesome information about SALSA and the animation/cut-scene industry. To expand on her excellent SALSA info:

Yes, SALSA v2 is theoretically unlimited in the number of visemes you can add. Of course, there are practical limits and technical reasons why you may want to limit that count. If there are too many viseme expressions, the trigger ranges are so tight there will likely always be a new trigger fired. This seems like a desirable outcome; however, the reality is the shapes are never afforded the opportunity to reach their full potential. This isn't necessarily bad and does smooth out the movement to a more realistic mumble (as wetcircuit mentioned). Giving animations time to complete (without simply popping them in place) and also sampling audio frequently enough to get a good sense of the amplitude dynamics is definitely a balancing act. Sample too frequently and animations don't have time to complete. Sample too infrequently and movement takes on a slow-motion look and dynamic nuances are missed.

With v2's new Advanced Dynamics options, stagnant shapes (when the same shape is fired repeatedly - due to similar analysis outcomes - resulting in an animation that appears stuck or stagnant) are eliminated and shape variation is nearly infinite. SALSA processes the amplitude analysis and then variably applies the configured visemes along the trigger range with some bias and jitter options. Additionally, there is a Secondary Mix option that works in conjunction with this to blend a percentage of an adjacent viseme configuration with the current triggered viseme for even more dynamics.

SALSA's real-time lip-sync approximation is all about perceived dynamics and accuracy. It is *not* a phoneme mapping solution so all of your added shapes are simply adding dynamics to lip movement. This is rarely an accurate phoneme representation; however, there are two main conditions that work together to suspend disbelief. Condition one is natural movement. SALSA v1's 3-shape limit is a bit shy of optimal and frequently stagnates, drawing attention to the unnatural movement and halting the immersion. However, with Advanced Dynamics, 3-shape configurations can look dramatically better. Five-shape viseme expressions may be even better and so-on, but it is certainly up to the designer. This allows workflows to remain light-weight and quality to increase significantly. The v2 3D demo video used 8 viseme expressions, but to be honest, I've been testing with lower counts and it's really a toss-up with the new dynamics options. The great news is, you're free to experiment with just about any configuration you might want.

The second condition for realism immersion is timing. Lag can be jarring -- think about how the audio track in a movie that is just slightly off affects you. SALSA v1 was at the mercy of Unity's FMOD audio system and by the time the data was returned to SALSA, it was already late and contributed greatly to the lag. SALSA v1 responded to audio that had already played. In reality, lips move before the sound emits. SALSA v2 looks at the data differently, allowing it to know what is currently playing and stay one step ahead of the audio process to time movement just in...time. This feature contributes most to the perceived accuracy and immersion. If lips are moving in-time with the vocalizations, we tend to (moreso) believe they are correct and start watching the rest of the scene or character and not just the lips.

We have also recently added a Global Dynamics option to SALSA v2 that offers an easy way to limit the overall movement of configured viseme expressions. This would allow the designer to configure bigger, more expressive visemes and globally decrease them from yelling to talking to whispering (for example) using a single variable. As wetcircuit so sagely advised, realistic characters and lip-movement is more subdued and cartoon characters are more energetic. But again, it is totally up to your design vision. NOTE: due to the way Advanced Dynamics processes visemes, Global Dynamics does not work with Advanced Dynamics, but a similar effect can be accomplished with Advanced Dynamics by adjusting the input bias up or down.

To sum up the above information, we have taken away most of the v1 restrictions; however, we have also added new technology that provides vastly better results even when applied to limited viseme configurations. As always, keep an eye on the devBlog for more info. Mike recently posted a video of the new Eyes (formerly RandomEyes) lid tracking, which is way cool. And prior to that, for the 2D folks, I posted a video demonstration of some new options for 2D that are equally cool.

And as a reminder, v2 is a complete re-write (none of the old code is used) and it is *not* a drop-in replacement for SALSA v1 configurations/projects. The plan also remains for v2 to get a price increase. And it *is* a paid upgrade. On initial release of v2 (for a short time) the upgrade will simply be the cost difference between today's price and the release price. In other words, we plan on releasing v2 at $39US, meaning upgrades will be $4US. Soon after, the new purchase cost will most likely increase again (TBD) and upgrade costs will definitely increase beyond the cost-difference level.

Hope that helps!
Darrin

 

Now THAT is a detailed answer!
THANK YOU !

You didn't answer the blendshape + bone question though:
I'm using Daz3D character Genesis 1, which uses both blendshapes AND bones for moving the face and the jaw (the blend shapes control the skin deformation but the internal lower mouth, tongue and teeth must be moved rotating the jaw bone)
Will this be a problem?

 

Some people are never happy...

SALSA v2 implements a *very* flexible viseme/emote expression design system. You can add as many components to an expression as desired. You are limited only by your imagination and the possible performance effects such animations would have on your destination system. Components may be anything SALSA supports; currently: blendshapes, bones/transforms, 2D [sprites, textures, materials]. So you can definitely have blendshapes and bones mixed together, defining a viseme expression. The screenshot shows a viseme configured on an iClone character with 2 blendshapes and 3 bones:

 

How is 2.0 going?

 

Great!

 

I CAN'T WAAAAIIIIT !!!

 

Hi!

I'm trying to run an Unity app on Android, but since I want that app to have a smaller window size and allow the user to move it on the screen, I need to export that to an Android Studio project, made the changes I want and generate an aar file to be read by another app, also built in Android Studio.

The app consists on a simple avatar which speaks and moves its lips, with the help of the Salsa3D plugin. When I build the app to Android directly from Unity it works perfectly. However, when I export to Android Studio and generate the plugin, most of the times the avatar doesn't move its lips. It worked only once and I verified that the generated .aar file was 3MB heavier in the working version.

I would like to know what's happening! Thanks!