SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hello. I'm really impressed by Salsa and can't wait for Salsa 2.0! I've been looking for a marketplace with a large selection of animation-ready 2D characters that would include mouth expressions. Something like DAZ for 2D Do you have any suggestions?

 

Hi username132323232,

I'm not aware of any Daz, Fuse, etc type character generator systems for 2D, I wish I was.

 

Thanks! How about 3D? What I'm looking for is a "talking head" avatar for giving instructions during the game. I know it would be cool to create everything from scratch in Blender, but that would be very time-consuming.

I have a good conceptual understanding of how character creation and rigging work, but not much in terms of practical experience

I checked your website and it seems that there are 6 choices:

• DAZ3D
• Mixamo Fuse
• Freedom's Gate
• iClone
• MCS (Morph3D)
• UMA 2

Which one would you recommend?

 

The 7th option is Autodesk Character Generator...

We can't recommend the best system for your project because it completely depends on your requirements, the look you're going for, the accessories you need and their availability, customization options, model efficiency (poly count, object count, BlendShape count), price, etc. Your best bet is to define your own requirement for your project, then research each system to see which one comes close to fulfilling that criteria.

 

Thanks Michael! Sorry, I missed Autodesk Character Generator (it's probably way over my budget anyway). I think I'll go with Fuze as it's a part of Adobe CC now

 

Here is the blog post and download link for our Fuse 1-click setup.

http://www.crazyminnowstudio.com/po...character-creator-with-salsa-with-randomeyes/

Good luck with your project, and thanks for being a SALSA customer!

 

This is awesome. Thanks again!

 

I downloaded this character ("Aj") from Mixamo as .fbx for Unity file and tried SALSA 1-Click Fuse Setup.





That didn't go very well. I suspect the model is missing something. Most fields in the SALSA components remain unassigned. Here is the error that I get from Unity:

Code (CSharp):

1. UnassignedReferenceException: The variable body of CM_FuseSync has not been assigned.
2. You probably need to assign the body variable of the CM_FuseSync script in the inspector.
3. CrazyMinnow.SALSA.Fuse.CM_FuseSync.LateUpdate () (at Assets/Crazy Minnow Studio/SALSA with RandomEyes/Third Party Support/Fuse Character Creator/CM_FuseSync.cs:188)

 

The Fuse 1-click setup is for Fuse Character Creator models. After selecting and stylizing a character in the Fuse desktop application, you upload the character to the Mixamo auto-rigger, select an armature and check the box to add BlendShapes to the character.

http://www.adobe.com/products/fuse.html

The free downloadable models on Mixamo are not Fuse models and I don't believe any of them include BlendShapes, which are required for SALSA to work. You can check if your character has BlendShapes by selecting the SkinnedMeshRenderer component in the character hierarchy. If you don't see a section on this component called BlendShapes, then your character doesn't have any and won't work with SALSA.

 

Thanks Michael for the explanation! Your instructions really helped. The Fuse 1-click setup is unbelievable, almost magic

 

Loving the expression groups, btw. Calling them by name through Playmaker is working great for me.



RandomEyes isn't so random anymore. LOL

(Hoping the transition to 2.0 isn't too painful.)

 

Hi Michael. I have a question about using SALSA with MasterAudio. I did look at the relevant Integration page, but it deals with Playlists and I'm using Group Variations.

My use case is simple. I have a Group with several audio files and play one of them like this:

Code (CSharp):

1. MasterAudio.PlaySound("Group 1", 1, pitch, delay, "Hello");

So my question is: how do I set SALSA audio clip to the clip that MasterAudio is currently playing and also make sure that the same clip is not played by both SALSA and MasterAudio?

 

Hello! This is pretty simple actually. You will need to have reference to the SALSA object (Salsa2D or Salsa3D) that will be 'speaking' the MasterAudio Group variations. Next, when you call the MasterAudio.PlaySound() method, you will capture the returned PlaySoundResult and then use that to assign the current AudioSource to SALSA. Something like the following:

Code (csharp):

1. var playSoundResult = MasterAudio.PlaySound("MissMuffet"); // "MissMuffet" = group name
2. salsa.audioSrc = playSoundResult.ActingVariation.VarAudio;

This can also be written a little more succinctly, albeit maybe not as readable, as:

Code (csharp):

1. salsa.audioSrc = MasterAudio.PlaySound("MissMuffet").ActingVariation.VarAudio;

And it also doesn't work as well if you want to do other things with the PlaySoundResult object. But, if your only need is to link SALSA to the AudioSource, it works just fine.

Hope this helps!
Darrin

 

Thanks Darrin! This is exactly what I was looking for.

 

Excellent! Glad it helped out.

For anyone else interested in this functionality, I've created a simple addon script (download from our site) as an example, allowing you to configure a MasterAudio Group and trigger it (from code or a UI event (button, etc.) or however you wish). It includes a sample scene with a button linked to the public PlayDialog() method of a 2D and 3D character. Clicking the button will play a random variation of the selected group on both characters.

This script works on either SALSA 2D or 3D and can be added from the Component menu. NOTE: this component must be added to the same object as your Salsa2D/3D component (it will automatically look for a SALSA component on the same object). And, of course, this addon requires SALSA Lip-Sync and Dark Tonic's Master Audio.

FYI: Repeated calls to the PlayDialog() method will stop a currently playing variation and select/play a new variation. You can test this in the example scene by pressing the "Play A Line Of Dialog" button before the current variation has finished playing.

 

What are the release notes for version 1.5.1?

 

The release notes for all versions can be found here.
http://www.crazyminnowstudio.com/unity-3d/lip-sync-salsa/release-notes/

 

Hi there,

I'm trying to make my robot talking by blinking his (mono) eye whenever he talks, instead of mapping a phoneme/facial movement to speech files.

So, was thinking to solve this problem maybe there is a way to hook a callback into the lip sync system to activate a custom action instead of a phoneme. That way you could perform any action you like while a character is talking.

Does anyone know of a way to do this natively?

This is the effect I'm talking about (just the mouth):

Any help would be much appreciated.

Cheers,
Simon

 

Hi Simon,

If I understand what your trying to do, you just want to drive a simple single BlendShape or bone. You can access the fully process output values from SALSA using the options below, then use these values to switch whatever mechanism you want to.

For Salsa2D

Salsa2D.say (string)

String value that is always updated with the current shape (Rest, Small, Medium, Large). This is useful for asset integrators that want to use Salsa2D as a lip sync engine with their own integration.

Salsa2D.sayIndex (int)

Int value that is always updated with the current shape (0=Rest, 1=Small, 2=Medium, 3=Large). This is useful for asset integrators that want to use Salsa2D as a lip sync engine with their own integration.​

For Salsa3D

Salsa3D.sayAmount (SalsaBlendAmounts)​

• Salsa3D.sayAmount.saySmall (float)
• Salsa3D.sayAmount.sayMedium (float)
• Salsa3D.sayAmount.sayLarge (float)

For a single BlendShape or bone, you could just use one of the values and set the trigger for that value to get the look you're after.

If you have additional question, please send your SALSA invoice to assetsupport@crazyminnow.com.

Thanks,
Michael

 

Hi,
I need a lip sync code for web gl please can you tell when is it possible to have Salsa for webgl.

 

Hi developerbhansalideepika,

SALSA does not support WebGL. The issue is that SALSA relies on API options provided by Unity's underlying audio system (FMOD), and though Unity uses this system for all other export types, WebGL uses a completely different audio system (Web Audio API). Though we have done experimentation with custom options outside of the Unity API, we don't currently, and don't know if we'll ever, have a viable WebGL option.

Michael

 

Just started using SALSA last week along with RTVoice and MCS. I had no trouble integrating everything together and there was no need to code anything to get it to work, which is pretty neat.

However, as I've been testing this setup with the same voice synth engine in order to finetune the Speech Properties fields (i.e. triggers, blend speed, audio update delay), I started to notice two unexpected behaviours:

1. When testing the same audio sample over and over again (Loop property set to true in the AudioSource), I noticed that the resulting lip sync animation varies a lot each time it plays, even though it's the exact same sample looped. I then tested with other samples multiple times and I got the same behaviour, sometimes the end result seems almost random. Tried this with audio files instead of runtime generated audio stream and I could detect some inconsistenciy in the resulting animation every time the audio loops. Moments of silence do seem to be properly respected with a mouth closed animation, though.

2. In the CM_MCS Syncscript, featured shapes for SaySmall/SayMedium/SayLarge occasionally get mangled (tested in Unity Editor). This behaviour happens everytime I load the scene with the parent object of my character deactivated (refer to shot1.png, shapes seem to disappear); if I activate my character in the meantime, the shape listing is incorrectly updated (refer to shot2.png). I need to press the Remap Blendshape Indexes button to revert shapes to the original settings (which are the ones I was using, but not necessarily). Is this a known issue? As a workaround I believe I could store the values myself and update the shapes listing when activating the character, but I guess it would be cool to have this fixed in the source. If you need more info on how to replicate this feel free to get in touch.

Thanks!

 

Hi pedro-meleiro,

We're happy to help, please send your invoice number to assetsupport@crazyminnow.com.

Thanks,
Michael

 

Hi, I'm using Helios to record Salsa lip sync animation. Salsa clips are invoke from SLATE (I don't think this matters). Helios records frame by frame, slowing down the framerate.

However I notice the animation doesn't follow the framerate. It simply plays the audio at normal speed and follow this normal speed of the audio. As a result the recorded video of the lipsync animation is much faster than the rest of the animations. Is there any way to make it follow framerate?

 

I think I've solved it, I didn't use SLATE plugin, use audio source instead. It works better with no bugs.

 

Hello! Admittedly, I'm not completely familiar with Helios' capabilities, but have used some similar screen recorders in the past and audio can be a trouble-point (or at least it was). Since SALSA analyzes the audio at run-time, it's animations are geared toward run-time audio. Again, in the past, this was the case and the symptoms you are/were seeing are exactly what we saw -- assuming audio even played. It doesn't really seem like a correctable scenario in SALSA since it simply analyzes the sound as it's played.

If you've reached an acceptable performance solution with Helios, that's awesome!

Darrin

 

Yes it was quite good. I use SLATE. If I were to use the extension provided from your site, it has a couple of problems:

1. Clicking or scrubbing on the time line will invoke multiple clips to play in full, it can't be stopped. It becomes very noisy and slightly unbearable
   
2. The symptom mentioned earlier; audio doesn't follow framerate

The solution was that I didn't use the extension. I simply use the native track in SLATE to play audio from the audio source component on the character (do this by checking "Use Audio Source On Actor" for the property of the audio track).

So just play an audio clip in the audiosource component on the character.

I think this will work even without SLATE, just don't call the playclip function from the SALSA class, but rather just play audioclip from the audiosource component

1. The scrubbing now works well, the multiple audio tracks doesn't play in full
2. The audio follow the framerate.

I'm not sure if this is bad in some way, but it works perfectly for me.

 

Hello!
It's perfectly fine to start/stop your AudioSource in any manner you wish for normal SALSA operation (via SLATE, script, SALSA, etc.). All SALSA needs is a reference to the AudioSource you want it to monitor.

The SLATE extension add-on wasn't really created with Helios in mind, so if you experience better performance by not using it, that is perfectly fine as well. Thanks for the follow-up and can't wait to see what you've created!

 

Any chance you have this integrated with Timeline? I want to do the same thing as the coverpage, only I would like to use Timeline.

 

Hi DGordon,

We are very excited about Unity's new integrated timeline editor, and are certainly interested in a smooth workflow between Timeline and SALSA. However, since it was only recently released, we need time to learn the system to best understand the integration requirements. I suspect basic lip-sync works inherently since the system only needs to be able to play an AudioClip on an AudioSource component, but we will likely need to write up a few custom actions for non-speech functions like expressions.

Thanks,
Michael

 

Glad to hear it! I did actually just pick up Slate after looking into it ... the above comment got me curious .

 

Also, I noticed you seem to automatically work with Daz3D blendshapes (haven't tested, but I plan on following your directions . If I have a beard on the model which has its own blendshapes, is there anyway to get this in sync using Salsa, or do I have to figure this out myself?

[Edit -- to be clear, the beard comes from Daz3d. It has its own separate blendshapes that need to be exported though ... my understanding is it uses the same blendshape ids as the model, judging from the tutorial for getting this working with iClone. If you don't do have a way to handle this, I'll probably write a script that syncs the beard's blendshapes with the body's every frame which will be good enough for recording purposes.]

 

Hi DGordon,

I don't think I've ever seen a DAZ character with a beard, so we haven't been able to test this. Is the beard an additional asset you bought? If it's a base character option, we would be interested in adding support in the 1-click add-on. However, even if it's an additional asset, you can likely use our SalsaSync add-on to do custom mappings. It's not as simple as our 1-click setups, but more flexible for custom setups since it allows you to map each shape to a unique SkinnedMeshRenderer.

http://www.crazyminnowstudio.com/posts/salsasync-flexible-salsa-map-for-custom-characters/

Michael

 

Hey,

It was a beard for Genesis 3 purchased in the Daz3d store and added in Daz. Obviously don't do a lot of work if only I need it (as I'm still investigating how I'm handling characters ... but I do already own your product ) ... but if you can figure something out for beards in Daz, it would definitely be a bonus for this product.

https://www.daz3d.com/santa-beard-for-genesis-3-male-s

 

With the SalsaSync add-on, you can link to as many SkinnedMeshRenderers as you like. It should work really well for this setup.

 

Hi, your Salsa is perfect for my app. However, I wonder why it's not working when I build my app in my andriod mobile?

 

Hello Nastomeya,

We can try to help you diagnose the issue. Please send your SALSA invoice number to assetsupport@crazyminnow.com.

Thanks,
Michael

 

Hi, I have just solved the problem by add salsa component to the parent main 3dModel instead of it's root of 3d head.But thanks anyway.

 

Hi,

I am interested to buy this plugin.
However, Can this plugin support below scenario :
"record user's speech then change it's pitch, then playback, similiar to 'talking tomcat' "

please let me know...thanks...

 

Hi Zukinet,

SALSA will perform lip-sync approximation on any Unity compatible AudioClip, played through a Unity AudioSource component. However, it is not an audio recording system, so that aspect would be up to you.

Thanks,
Michael

 

Any update on when Salsa 2.0 will be coming out?

 

Hi KyleWilcocks,

Sorry, no ETA yet. The 3D rewrite is mostly feature complete but still has bugs that need squashing, and the 2D rewrite is still in the planning stages. We certainly hope to have it out before the end of the year.

Michael

 

Thanks for the reply now I have a programming question.

I am building an inspector extension that lets you set an audio file and then set expressions for certain timestamps of the audio file. I saw in the RandomEyes3D inspector that it can preview shape groups. I was wondering how would I do that so I could test expressions without having to play the game. I tried looking for the RandomEyes3D editor script but it's a DLL.

~Kyle

 

Does anyone know of a way to record the SALSA blendshapes as a animation in real-time, so I can play it back while offline rendering?

 

Hey there! I don't know of anything with native capability, but you might be able to rig something up using the experimental GameObjectRecorder found here: https://forum.unity3d.com/threads/f...tal-feature-gameobjectrecorder-2017-1.467852/

It appears that you can bind to any object/component/property/etc. I'm not sure if you can record the blendshapes as SALSA animates them with this, but it might be worth a look.

 

Hi KyleWilcocks,

I think you're asking about previewing shape groups by scrubbing a custom timeline tool you're building, when you're in the editor and the game isn't running? If that's what you're asking, it's a bit beyond the scope of SALSA, but you'd have to make BlendShape value changes in an editor script that can be updated while your scrubbing your timeline. The SetGroup method, included in the RandomEye API, is dependent on the game running since it's prioritizing mecanim conflicts and interpolating over time.

Michael

 

Any way to catch events like when it changes to small, medium, large, or rest? I have a different kind of system using Anima 2D that has sprite meshes instead of sprite renderer so it makes it not work, but if I can catch the events I can manually update the sprite mesh to be the right sprite (by setting active to true or false to the proper faces.) Thanks for any help?

 

Hi nosajtevol,

We don't current have events for each mouse shape change, but we do expose the current shape that you can monitor. You could monitor this in a switch statement and use it to set the correct shape.

http://crazyminnowstudio.com/unity-...uals/salsa-manual/classes-methods-properties/

Public properties

Salsa2D.say (string)

String value that is always updated with the current shape (Rest, Small, Medium, Large). This is useful for asset integrators that want to use Salsa2D as a lip sync engine with their own integration.​

Salsa2D.sayIndex (int)

Int value that is always updated with the current shape (0=Rest, 1=Small, 2=Medium, 3=Large). This is useful for asset integrators that want to use Salsa2D as a lip sync engine with their own integration.​

Salsa2D.isTalking (bool)

Check this boolean value to determine when the audio file has stopped playing​

Michael

 

That sounds like a plan! Thanks so much for the quick reply

 

Hi, I'm getting a console filled with this error, and I'm having trouble tracking down the cause…. It involves the Other Skinnedmesh Renderers section of Salsa Sync… (I am using Eyelashes and Teeth. Either one, or both together trigger the error.)

Code (CSharp):

1. IndexOutOfRangeException: Array index is out of range.
2. CrazyMinnow.SALSA.Sync.CM_SalsaSync.LateUpdate () (at Assets/Crazy Minnow Studio/SALSA with RandomEyes/Addons/SalsaSync/CM_SalsaSync.cs:322)

FWIW, I am using a new Adobe Fuse figure (1 week?) and they seem to have changed some things under the hood, like new shorter names for all the blendshapes… Can send you my model but I assume this is from the Auto-Rigger and will effect all Fuse figures….