SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hi tgekerrion,

Thanks for your interest in SALSA. You can absolutely use SALSA with custom characters, and there are many different configuration options depending on the look you're going for. You can use Salsa2D for sprite swapping animation (like a cartoon), Salsa3D for smooth lerped animation using blendshapes, or access the processed output values of either and write custom linkage to other methods (texture / material swapping, etc.).

Our online manuals will tell you about the requirements and capabilities of each component included in the package.

https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/manuals/

 

I'm seeing a few companies support iPhoneX facial scan for blendshape control.
Any chance you might consider an iPhoneX add-on for Salsa 2.0?

I realize it's outside your normal approach for lipsync, but Salsa is already controlling all my face expressions.
(well, not *my* face expressions, but you know what I mean.)

 

Hello! It seems like very cool tech. As you said, it is outside of our current scope and goals for SALSA but we know better than to say never. We will probably take a wait-and-see approach on this one.

 

Hi. Sorry for my ignorance but I read here you are working with AI features in Unity, What is AI SDK for Unity that you are using? Thanks for your time

 

We code our own AI scripts for the generic stuff NPC walking etc. And we use the Neural network for the dialog NPCs.
I'm an artist myself so don't really know all the tech/programming details.

 

Hello,

where can I set the volume to set for every Clip? I cannot find it...
I am using Salsa3D:

Salsa3D salsa = UmaHero.GetComponent<CM_UmaSync>().salsa3D;
…

salsa.SetAudioClip(audio);
salsa.Play();

Thanks

 

PS: Found it on the audiosource at that prefab I guess….thanks.

 

Hi, how do we manage hand gestures in salsa with random eyes?
Also, comparing lip sync pro and salsa with random eyes, which one is better?

 

Hello mgmegag,

SALSA is a lipsync approximation system, I'm not sure what you mean by hand gestures.

Determining which assets best fits your projects objectives is very subjective depending on your priorities and available company resources. SALSA is focused on keeping the implementation process as simple as possible, and providing flexibility to adapt to many different scenarios. It is an approximation system and not a phoneme mapping system.

 

Hey guys,

Thanks for this awesome asset. It works good, but i have a question. Is there a way to use idle animations and once salsa can hear the voice (or any other audio sources) it starts animation with speech?

Thanks

 

@Crazy-Minnow-Studio I'm considering buying the new iClone Character Creator 3 tool from Reallusion because all the other character options are falling short of expectations for one reason or another. iClone seems to be the best option for variety or characters and huge library of outfits, hair and accessories. I know SALSA works with iClone, but can you confirm that the exported models from the latest CC3 will work with the SALSA 1-click setup?

 

Yes indeed, the CC3 chars are supported from v1.3.6+ (iclone one-click). We recently worked with another customer and confirmed these new models.

 

SALSA does not control mecanim animations directly (if that is the question). You can have any animation you like on the character and when the AudioSource linked to SALSA plays audio, SALSA will animate the configured lipsync shapes. One gotcha with Mecanim is to ensure the jaw-bone is un-linked in the avatar setup. This will ensure there are no strange rest positions or animations if SALSA is not "talking".

 

Hi daibutsu
If you need to move characters by Voice command maybe you could use Watson IBM SDK for Unity or Lexicon unitypackage to do it. more info here
https://assetstore.unity.com/packages/tools/ai/ibm-watson-sdk-for-unity-108831
https://assetstore.unity.com/packages/tools/ai/lexicon-113459
Hope this help !!!

 

Hi!
I have problems with Salsa + micInput and iOS build. Using Unity 2018.2.7, and v1.7 experimental micInput. iOS12.1 beta5.Xcode 10.1. It works fine in editor, but not on iOS build; seems like the mic are nor processing any audio. Have tried 1.6 also, same result. Side note: On application start in the iOS device, I do not ask for permission to use mic (I have entered permission in the player settings)....I also use camera, and it doen not as for permission for that either, som maybe it is a Unity/Apple issue?

 

Hello! It certainly sounds like a setup issue. Try checking the permission is granted:

Code (CSharp):

1. Application.HasUserAuthorization(UserAuthorization.Microphone)

If you don't have permission, make sure the Microphone is starting. You can call the micInput StartMicrophone(string micName) method directly -- perhaps link it up to a UI button.

 

Hi!
It seems like it is activating the MacBookPro mic when running on iOS, witch is not possible.
I run StartMicrophone method and fetch the string of selectedMic in the script, and print it to the canvas on every frame, it says "MacBook Pro Microphone".

What is going on?

 

Hello! Sounds like you have a specific microphone selected in the editor instead of "Default" and that is carrying over into your build. Try selecting "Default" in the editor and then build.

 

Ehem, thanks - solved the issue!

Another question - are there plans to give the controller more phonemes in the future on 2.0?

 

Hi, SALSA 2.0 is not limited by controller, phoneme, or trigger count. See our 2.0 blog for more info.

http://www.crazyminnowstudio.com/posts/salsa-v2-devblog/

 

Great! And micInput will work in v2.0 as well?

Any estimate on release and pricing model for existing customers?

Best

 

Yup, micInput works well in v2...have been doing a lot of testing with it the past few days, in fact. Watch the devblog for release info (there isn't estimate yet but we're getting much closer). Pricing model is in there as well -- always subject to change, but we will let everyone know as we get closer.

 

@Crazy-Minnow-Studio I have a scene in my game where I want a dragon to talk. I saw your video with the yellow character and the dragon talking so I'm wondering just how you went about setting up the dragon for lip-syncing. Is this done in Blender? If so, is there some sort of tutorial video or instructions on how this can be accomplished? I have a dragon character where the mouth is rigged to open and close in order to do a roaring and fire breathing animation, but I'm sure it does not have any sort of blendshapes. And I have no idea how I would make them. Any help would be appreciated.

[EDIT]
I found this video:

Would this be what I need to make it all work?

 

Hello! Yes, there are 3 videos in that Blender series and, while they were created in a slightly older version of Blender, they should give you the info you need to work with Blender's shape keys to create additional mouth shapes. The dragon in the video you are referring to is a DAZ character and already had some usable shapes on the model. Creating three shapes for your dragon should be pretty easy, especially if you are somewhat familiar with Blender. Be sure to watch the "Gotchas" video (pt 3) as well -- to avoid time-wasting problems.

 

Thanks. I'm not a modeler or Blender user so it will all be new to me unfortunately. I hope it won't be too difficult.

 

Gotcha, well the 3 videos we made assume some Blender experience IMO. The good news is, if our videos assume too much, there are uncountable free tutorials covering Blender basics out there. Andrew Price (Blender Guru) has a good tutorial set for Blender beginners --

 

Salsa 2.0?

 

Still cooking.
https://crazyminnowstudio.com/posts/salsa-v2-devblog/

 

Hi!
Good to see Salsa 2.0 is getting closer to release!
I do have a problem integrating Salsa and a MCS character.

I use then Salsa 1-click setup. Eye movement works fine, also lipsync.
But, it can not access the other blend shapes in the RandomEyes for custom shapes.
I can not add any shapes, Ive tried to select all the available LOD groups with skinned mesh renderer,
and nothing. I know all the facial blend shapes is available, it works fine in the M3D-Charater Manager script.

Any clues what I'm doing wrong?
Best

 

Hi JoMaHo,

It sounds like you might be using the wrong RandomEyes instance. There are two instances used for MCS character, one for eye control, and the other for custom shapes and groups. The custom shapes instance can be identified by the check box in the "Use Custom Shapes Only" field. This is the instance where you should add custom shapes and groups, and be sure to target this instance from code when using the API.

 

Greetings!
I´m aware that there are two instances of RandomEyes; one for the eyes and one for custom shapes only.
The skinned mesh renderer have 4 LOD groups (MCS Female free Unity asset from Morph 3D). None of these groups expose the facial blend shapes when I try to add a custom shape...

 

Hi JoMaHo,

If you activate a BlendShape using the M3D, you must then cycle the M3D LOD to each level for the shape to be added to each LOD mesh. MCS doesn't add the BlendShape to each LOD until the shape has been activated and the applicable LOD set to active.

I hope that helps.

 

Hi!
I'm sorry, I do not understand - could you please elaborate a bit on how to
add (potentially) all facial blenshapes, that are embedded in the M3D character.

Best
Johnny

 

Hi Johnny,

You have to have a good understanding of how MCS works to be able to use it trouble free with SALSA.

When you enable a morph / blendshape using the check boxes in the M3D Character Manager, the blendshape is only added to the currently active LOD. If you slide the M3D Character Manager LOD slider all the way back and forth, it will also adds that blendshape to the other LOD's when they are active.

If you activate all the blendshapes you want to use first, then slide the LOD slider back and forth to apply the shapes to each LOD, then save your project, exit, and relaunch, you can then perform the SALSA MCS 1-click setup, add whatever custom shapes and groups you wish to use, and when the LOD is changed, CM_MCSSync automatically remaps to the selected LOD.

It is important to save your project, exit, and relaunch after activating all of your shapes, because MCS does not maintain the blendshape indexes in the order that you activate them, but rather in a predetermined order. If you skip the project reset step, your indexes will likely be out of sync the next time you launch the project and you won't get the results you want.

If you are doing all of this at run-time it's a different story since the indexes will be activated and maintained throughout the run instance.

 

Ah - It worked like a charm; not very intuitive though ;-) Maybe add a note on this one in the documentation (or maybe its already there?)

Great tool anyway - I'm looking forward to even more goodies in v2.0

Thank you very much for you support!

best

 

Hi again!
I have a question. I´m using the micInput controller to control realtime lipsync from the microphone on one character. It works just fine for one character. Now I have two characters I want to swith between who is talking.

I have setup the 1-click lipsync controller on both characters, they work just great solo, but when I try to switch between them in real time, I get into trouble. I use a UI button for each of the characters, turning on and off the respective audio sources and micInput script. It works for the first switch, but then it breaks (no sound from the mic and no lipsync). Do I need to re initialize the controller or something on each switch?

 

If you are just controlling the component enable/disable with a single button set of actions, they don't necessarily fire in a controlled order. You would probably be better served with a simple script that switches/toggles the actions. And I would probably recommend simply controlling which SALSA instance is active, leaving the micInput and AudioSource instances alone.

So, something like the following:

• For organizational purposes, add an empty to the scene and add the AudioSource to it, with the micInput component. So, one AudioSource in the scene and one micInput. (remove the micInputs and AudioSources from the SALSA-enabled objects).
  
• Create a script that has two public SALSA references. Create a public method (that the button calls), that toggles which SALSA instance is active, call it something like ToggleSalsa(). You can add this to the same empty object above and link the SALSAs to the public slots.
  • Your class/method can look something like the following:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class ToggleSalsaInstances : MonoBehaviour
4. {
5.     public CrazyMinnow.SALSA.Salsa3D salsa1;
6.     public CrazyMinnow.SALSA.Salsa3D salsa2;
7.  
8.     public void ToggleSalsa()
9.     {
10.         salsa1.enabled = !salsa1.enabled;
11.         salsa2.enabled = !salsa2.enabled;
12.     }
13. }

• Link both SALSAs to the single AudioSource and initially, disable one of them (the SALSA component).
• Wire your button to point to the new script's ToggleSalsa() method.

Doing it in this manner eliminates the need to disable/enable the micInputs and AudioSources.

 

Sorry, just noticed you are using two buttons. Either way, you can turn this into a single button operation using the above technique.

 

Brilliant - thank you - made my day!
Great support, I must say!

best

 

@Crazy-Minnow-Studio At long last I've started to use SALSA in my game project. I am using the Wii U platform and iClone character. Everything appeared to be working fine on the Wii U. The eyes move and they blink fine in both editor mode and Wii U. However, I added a voice track today and am playing it using Playmaker. In editor mode on the PC it is working perfectly. However, on the Wii U it doesn't move the mouth at all. I thought there would be some errors written to the console, but I see nothing. There's no indication of any problem. Do you have any idea where I should look to debug this?

 

Hello magique,

Sorry your experiencing issues on Wii U, we're looking into it to see if the Unity API we leverage for SALSA is also available on your target platform. We'll get back to you soon.

While we research, can you test a simple scene on Wii U with nothing but the example boxhead character and Salsa3D, and let us know the results?

Thanks,
Michael

 

I tried the quick start scene with just boxhead and I get the same results. No mouth movement.

 

Thanks for confirming! We'll get back to you as soon as we have answers on Wii U API availability.

 

Hi! I'm eagerly waiting for Salsa 2.0
Any news?

 

Hi Atomic! Thanks for your enthusiasm! Our most recent, public information is located on the devblog site.
https://crazyminnowstudio.com/posts/salsa-v2-devblog/

 

Yes, I know! I have already read it.
I thought you would probably release it for Black Friday or something, since development seems complete

 

That would be awesome, but unfortunately, that won't be the case. We are super close. SALSA and EmoteR are (for all intents and purposes) complete. The eye module is nearly done.

 

CM,

I am not getting a response on Twitter or FB - could someone respond please?

thanks,
Phil.

 

Hi Phil,

We provide support on this forum and via direct email, both are listed on the Asset Store page. You also should have received an auto-response from FB providing you with our email, did you not receive it? How can we help you?

 

CM,

I did get a canned response on FB but no email address just a link back to your web site and I got the impression that that email address was for customers who had an existing invoice number . . not a new customer . .

Anyway, thanks for getting back to me! I have a job that I need to do relatively quickly - a month or so max (preferably in a week) - so although I usually like to do things myself, because of the shortage of time and because I don't want to be forced back into building a Windows machine after changing to Linux 20 years ago, I need to pay someone to do what I want for the first project and, depending how that goes, move on to a situation where I can set up seamless production stream where I can quickly create further short video clips.

Basically we can conveniently use the "Adam" character because it closely resembles my existing cyber face - see here:

http://philiprhoades.org

So the first job is an Adam talking head clip of about 3-4 minutes with reasonable lip-syncing and you guys / your tech looks like just what I need - do you take on these sorts of small discrete jobs yourselves or can you refer me to someone using your tech who can do what I want quickly and nicely?

thanks,
Phil.