Search Unity

  1. Calling all beginners! Join the FPS Beginners Mods Challenge until December 13.
    Dismiss Notice
  2. It's Cyber Week at the Asset Store!
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hi tgekerrion,

    Thanks for your interest in SALSA. You can absolutely use SALSA with custom characters, and there are many different configuration options depending on the look you're going for. You can use Salsa2D for sprite swapping animation (like a cartoon), Salsa3D for smooth lerped animation using blendshapes, or access the processed output values of either and write custom linkage to other methods (texture / material swapping, etc.).

    Our online manuals will tell you about the requirements and capabilities of each component included in the package.

    https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/manuals/
     
  2. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,092
    I'm seeing a few companies support iPhoneX facial scan for blendshape control.
    Any chance you might consider an iPhoneX add-on for Salsa 2.0?

    I realize it's outside your normal approach for lipsync, but Salsa is already controlling all my face expressions.
    (well, not *my* face expressions, but you know what I mean.)
     
    Crazy-Minnow-Studio likes this.
  3. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hello! It seems like very cool tech. As you said, it is outside of our current scope and goals for SALSA but we know better than to say never. We will probably take a wait-and-see approach on this one.
     
    wetcircuit likes this.
  4. castana1962

    castana1962

    Joined:
    Apr 10, 2013
    Posts:
    362
    Hi. Sorry for my ignorance but I read here you are working with AI features in Unity, What is AI SDK for Unity that you are using? Thanks for your time
     
  5. tgekerrion

    tgekerrion

    Joined:
    Sep 11, 2018
    Posts:
    3
    We code our own AI scripts for the generic stuff NPC walking etc. And we use the Neural network for the dialog NPCs.
    I'm an artist myself so don't really know all the tech/programming details.
     
    castana1962 likes this.
  6. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,102
    Hello,

    where can I set the volume to set for every Clip? I cannot find it...
    I am using Salsa3D:

    Salsa3D salsa = UmaHero.GetComponent<CM_UmaSync>().salsa3D;


    salsa.SetAudioClip(audio);
    salsa.Play();

    Thanks :)
     
  7. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,102
    PS: Found it on the audiosource at that prefab I guess….thanks.
     
    Crazy-Minnow-Studio likes this.
  8. mgmegag

    mgmegag

    Joined:
    Aug 7, 2018
    Posts:
    3
    Hi, how do we manage hand gestures in salsa with random eyes?
    Also, comparing lip sync pro and salsa with random eyes, which one is better?
     
  9. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hello mgmegag,

    SALSA is a lipsync approximation system, I'm not sure what you mean by hand gestures.

    Determining which assets best fits your projects objectives is very subjective depending on your priorities and available company resources. SALSA is focused on keeping the implementation process as simple as possible, and providing flexibility to adapt to many different scenarios. It is an approximation system and not a phoneme mapping system.
     
  10. daibutsu

    daibutsu

    Joined:
    Nov 28, 2017
    Posts:
    18
    Hey guys,

    Thanks for this awesome asset. It works good, but i have a question. Is there a way to use idle animations and once salsa can hear the voice (or any other audio sources) it starts animation with speech?

    Thanks
     
    castana1962 likes this.
  11. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,671
    @Crazy-Minnow-Studio I'm considering buying the new iClone Character Creator 3 tool from Reallusion because all the other character options are falling short of expectations for one reason or another. iClone seems to be the best option for variety or characters and huge library of outfits, hair and accessories. I know SALSA works with iClone, but can you confirm that the exported models from the latest CC3 will work with the SALSA 1-click setup?
     
  12. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Yes indeed, the CC3 chars are supported from v1.3.6+ (iclone one-click). We recently worked with another customer and confirmed these new models.
     
    cygnusprojects and magique like this.
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    SALSA does not control mecanim animations directly (if that is the question). You can have any animation you like on the character and when the AudioSource linked to SALSA plays audio, SALSA will animate the configured lipsync shapes. One gotcha with Mecanim is to ensure the jaw-bone is un-linked in the avatar setup. This will ensure there are no strange rest positions or animations if SALSA is not "talking".
     
  14. castana1962

    castana1962

    Joined:
    Apr 10, 2013
    Posts:
    362
    Hi daibutsu
    If you need to move characters by Voice command maybe you could use Watson IBM SDK for Unity or Lexicon unitypackage to do it. more info here
    https://assetstore.unity.com/packages/tools/ai/ibm-watson-sdk-for-unity-108831
    https://assetstore.unity.com/packages/tools/ai/lexicon-113459
    Hope this help !!!
     
  15. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Hi!
    I have problems with Salsa + micInput and iOS build. Using Unity 2018.2.7, and v1.7 experimental micInput. iOS12.1 beta5.Xcode 10.1. It works fine in editor, but not on iOS build; seems like the mic are nor processing any audio. Have tried 1.6 also, same result. Side note: On application start in the iOS device, I do not ask for permission to use mic (I have entered permission in the player settings)....I also use camera, and it doen not as for permission for that either, som maybe it is a Unity/Apple issue?
     
  16. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hello! It certainly sounds like a setup issue. Try checking the permission is granted:
    Code (CSharp):
    1. Application.HasUserAuthorization(UserAuthorization.Microphone)
    If you don't have permission, make sure the Microphone is starting. You can call the micInput StartMicrophone(string micName) method directly -- perhaps link it up to a UI button.
     
  17. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Hi!
    It seems like it is activating the MacBookPro mic when running on iOS, witch is not possible.
    I run StartMicrophone method and fetch the string of selectedMic in the script, and print it to the canvas on every frame, it says "MacBook Pro Microphone".

    What is going on?
     
  18. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hello! Sounds like you have a specific microphone selected in the editor instead of "Default" and that is carrying over into your build. Try selecting "Default" in the editor and then build.
     
  19. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Ehem, thanks - solved the issue!

    Another question - are there plans to give the controller more phonemes in the future on 2.0?
     
  20. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hi, SALSA 2.0 is not limited by controller, phoneme, or trigger count. See our 2.0 blog for more info.

    http://www.crazyminnowstudio.com/posts/salsa-v2-devblog/
     
  21. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Great! And micInput will work in v2.0 as well?

    Any estimate on release and pricing model for existing customers?

    Best
     
  22. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Yup, micInput works well in v2...have been doing a lot of testing with it the past few days, in fact. Watch the devblog for release info (there isn't estimate yet but we're getting much closer). Pricing model is in there as well -- always subject to change, but we will let everyone know as we get closer.
     
  23. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,671
    @Crazy-Minnow-Studio I have a scene in my game where I want a dragon to talk. I saw your video with the yellow character and the dragon talking so I'm wondering just how you went about setting up the dragon for lip-syncing. Is this done in Blender? If so, is there some sort of tutorial video or instructions on how this can be accomplished? I have a dragon character where the mouth is rigged to open and close in order to do a roaring and fire breathing animation, but I'm sure it does not have any sort of blendshapes. And I have no idea how I would make them. Any help would be appreciated.

    [EDIT]
    I found this video:



    Would this be what I need to make it all work?
     
  24. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hello! Yes, there are 3 videos in that Blender series and, while they were created in a slightly older version of Blender, they should give you the info you need to work with Blender's shape keys to create additional mouth shapes. The dragon in the video you are referring to is a DAZ character and already had some usable shapes on the model. Creating three shapes for your dragon should be pretty easy, especially if you are somewhat familiar with Blender. Be sure to watch the "Gotchas" video (pt 3) as well -- to avoid time-wasting problems.
     
  25. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,671
    Thanks. I'm not a modeler or Blender user so it will all be new to me unfortunately. I hope it won't be too difficult.
     
  26. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Gotcha, well the 3 videos we made assume some Blender experience IMO. The good news is, if our videos assume too much, there are uncountable free tutorials covering Blender basics out there. Andrew Price (Blender Guru) has a good tutorial set for Blender beginners --
     
    magique likes this.
  27. SickaGamer

    SickaGamer

    Joined:
    Jan 15, 2018
    Posts:
    822
    Salsa 2.0?
     
    atomicjoe likes this.
  28. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
  29. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Hi!
    Good to see Salsa 2.0 is getting closer to release!
    I do have a problem integrating Salsa and a MCS character.

    I use then Salsa 1-click setup. Eye movement works fine, also lipsync.
    But, it can not access the other blend shapes in the RandomEyes for custom shapes.
    I can not add any shapes, Ive tried to select all the available LOD groups with skinned mesh renderer,
    and nothing. I know all the facial blend shapes is available, it works fine in the M3D-Charater Manager script.

    Any clues what I'm doing wrong?
    Best
     
  30. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hi JoMaHo,

    It sounds like you might be using the wrong RandomEyes instance. There are two instances used for MCS character, one for eye control, and the other for custom shapes and groups. The custom shapes instance can be identified by the check box in the "Use Custom Shapes Only" field. This is the instance where you should add custom shapes and groups, and be sure to target this instance from code when using the API.
     
  31. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Greetings!
    I´m aware that there are two instances of RandomEyes; one for the eyes and one for custom shapes only.
    The skinned mesh renderer have 4 LOD groups (MCS Female free Unity asset from Morph 3D). None of these groups expose the facial blend shapes when I try to add a custom shape...
     
  32. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hi JoMaHo,

    If you activate a BlendShape using the M3D, you must then cycle the M3D LOD to each level for the shape to be added to each LOD mesh. MCS doesn't add the BlendShape to each LOD until the shape has been activated and the applicable LOD set to active.

    I hope that helps.
     
  33. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Hi!
    I'm sorry, I do not understand - could you please elaborate a bit on how to
    add (potentially) all facial blenshapes, that are embedded in the M3D character.

    Best
    Johnny
     
  34. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hi Johnny,

    You have to have a good understanding of how MCS works to be able to use it trouble free with SALSA.

    When you enable a morph / blendshape using the check boxes in the M3D Character Manager, the blendshape is only added to the currently active LOD. If you slide the M3D Character Manager LOD slider all the way back and forth, it will also adds that blendshape to the other LOD's when they are active.

    If you activate all the blendshapes you want to use first, then slide the LOD slider back and forth to apply the shapes to each LOD, then save your project, exit, and relaunch, you can then perform the SALSA MCS 1-click setup, add whatever custom shapes and groups you wish to use, and when the LOD is changed, CM_MCSSync automatically remaps to the selected LOD.

    It is important to save your project, exit, and relaunch after activating all of your shapes, because MCS does not maintain the blendshape indexes in the order that you activate them, but rather in a predetermined order. If you skip the project reset step, your indexes will likely be out of sync the next time you launch the project and you won't get the results you want.

    If you are doing all of this at run-time it's a different story since the indexes will be activated and maintained throughout the run instance.
     
  35. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Ah - It worked like a charm; not very intuitive though ;-) Maybe add a note on this one in the documentation (or maybe its already there?)

    Great tool anyway - I'm looking forward to even more goodies in v2.0

    Thank you very much for you support!

    best
     
  36. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Hi again!
    I have a question. I´m using the micInput controller to control realtime lipsync from the microphone on one character. It works just fine for one character. Now I have two characters I want to swith between who is talking.

    I have setup the 1-click lipsync controller on both characters, they work just great solo, but when I try to switch between them in real time, I get into trouble. I use a UI button for each of the characters, turning on and off the respective audio sources and micInput script. It works for the first switch, but then it breaks (no sound from the mic and no lipsync). Do I need to re initialize the controller or something on each switch?
     
    Last edited: Nov 13, 2018
  37. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    If you are just controlling the component enable/disable with a single button set of actions, they don't necessarily fire in a controlled order. You would probably be better served with a simple script that switches/toggles the actions. And I would probably recommend simply controlling which SALSA instance is active, leaving the micInput and AudioSource instances alone.

    So, something like the following:
    • For organizational purposes, add an empty to the scene and add the AudioSource to it, with the micInput component. So, one AudioSource in the scene and one micInput. (remove the micInputs and AudioSources from the SALSA-enabled objects).
    • Create a script that has two public SALSA references. Create a public method (that the button calls), that toggles which SALSA instance is active, call it something like ToggleSalsa(). You can add this to the same empty object above and link the SALSAs to the public slots.
      • Your class/method can look something like the following:
    Code (CSharp):
    1. using UnityEngine;
    2.  
    3. public class ToggleSalsaInstances : MonoBehaviour
    4. {
    5.     public CrazyMinnow.SALSA.Salsa3D salsa1;
    6.     public CrazyMinnow.SALSA.Salsa3D salsa2;
    7.  
    8.     public void ToggleSalsa()
    9.     {
    10.         salsa1.enabled = !salsa1.enabled;
    11.         salsa2.enabled = !salsa2.enabled;
    12.     }
    13. }
    • Link both SALSAs to the single AudioSource and initially, disable one of them (the SALSA component).
    • Wire your button to point to the new script's ToggleSalsa() method.
    Doing it in this manner eliminates the need to disable/enable the micInputs and AudioSources.
     
  38. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Sorry, just noticed you are using two buttons. Either way, you can turn this into a single button operation using the above technique.
     
  39. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Brilliant - thank you - made my day!
    Great support, I must say!

    best
     
    Crazy-Minnow-Studio likes this.
  40. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,671
    @Crazy-Minnow-Studio At long last I've started to use SALSA in my game project. I am using the Wii U platform and iClone character. Everything appeared to be working fine on the Wii U. The eyes move and they blink fine in both editor mode and Wii U. However, I added a voice track today and am playing it using Playmaker. In editor mode on the PC it is working perfectly. However, on the Wii U it doesn't move the mouth at all. I thought there would be some errors written to the console, but I see nothing. There's no indication of any problem. Do you have any idea where I should look to debug this?
     
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hello magique,

    Sorry your experiencing issues on Wii U, we're looking into it to see if the Unity API we leverage for SALSA is also available on your target platform. We'll get back to you soon.

    While we research, can you test a simple scene on Wii U with nothing but the example boxhead character and Salsa3D, and let us know the results?

    Thanks,
    Michael
     
    Last edited: Nov 18, 2018
  42. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,671
    I tried the quick start scene with just boxhead and I get the same results. No mouth movement.
     
  43. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Thanks for confirming! We'll get back to you as soon as we have answers on Wii U API availability.
     
    magique likes this.
  44. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    508
    Hi! I'm eagerly waiting for Salsa 2.0
    Any news?
     
  45. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
  46. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    508
    Yes, I know! I have already read it.
    I thought you would probably release it for Black Friday or something, since development seems complete :p
     
  47. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    That would be awesome, but unfortunately, that won't be the case. We are super close. SALSA and EmoteR are (for all intents and purposes) complete. The eye module is nearly done.
     
    atomicjoe likes this.
  48. phil_unity464

    phil_unity464

    Joined:
    Nov 28, 2018
    Posts:
    14
    CM,

    I am not getting a response on Twitter or FB - could someone respond please?

    thanks,
    Phil.
     
  49. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    919
    Hi Phil,

    We provide support on this forum and via direct email, both are listed on the Asset Store page. You also should have received an auto-response from FB providing you with our email, did you not receive it? How can we help you?
     
  50. phil_unity464

    phil_unity464

    Joined:
    Nov 28, 2018
    Posts:
    14
    CM,

    I did get a canned response on FB but no email address just a link back to your web site and I got the impression that that email address was for customers who had an existing invoice number . . not a new customer . .

    Anyway, thanks for getting back to me! I have a job that I need to do relatively quickly - a month or so max (preferably in a week) - so although I usually like to do things myself, because of the shortage of time and because I don't want to be forced back into building a Windows machine after changing to Linux 20 years ago, I need to pay someone to do what I want for the first project and, depending how that goes, move on to a situation where I can set up seamless production stream where I can quickly create further short video clips.

    Basically we can conveniently use the "Adam" character because it closely resembles my existing cyber face - see here:

    http://philiprhoades.org

    So the first job is an Adam talking head clip of about 3-4 minutes with reasonable lip-syncing and you guys / your tech looks like just what I need - do you take on these sorts of small discrete jobs yourselves or can you refer me to someone using your tech who can do what I want quickly and nicely?

    thanks,
    Phil.
     
    Last edited: Nov 29, 2018