Search Unity

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    I'm not familiar with that asset but after taking a quick look at it, it seems to modify the audio after the AudioSource has processed it, then output to the speakers. This means the lip-sync would not be in sync since the Voice Changer Filter asset add's roughly a half second delay, while SALSA operates in real-time. Also the asset is compiled, so this is an assumption of what's happening. You'd have to talk to the developer of that asset to know for sure.

    Michael
     
  2. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,143
    Could you at least add pitch as an optional Parameter to your Play method? Or maybe even more? Something that changes the voice but not the Speed and delay?

    That would be great! If not, it will be okay, too, I like my lypsynch UMA characters :)
     
  3. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    You can already adjust pitch using an Audio Mixer and a Pitch Shifter effect, these are built into Unity.
    • Window -> Audio Mixer
    • Click the + to add a new Mixer
    • Select Master Attenuation in the new Mixer to view the Master inspector.
    • Click Add Effect -> Pitch Shifter
    • Move the Pitch slider to your preference
    • On your Audio Source where you want to adjust pitch, select your new Mixer Master as the output.
    • There are several other effects available in the Master inspector.
    https://docs.unity3d.com/Manual/AudioMixer.html

    Michael
     
    Last edited: Feb 5, 2017
    Firlefanz73 likes this.
  4. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,143
    Thanks a lot Michael! If you ever run out of new ideas for salsa, some voice-changing stuff would be great and even fun for others, too I guess. Thanks a lot for your hint, I will see if this works out for me.
     
  5. grahag11

    grahag11

    Joined:
    Jan 1, 2017
    Posts:
    5
    Hello,
    Couple of questions... Is SALSA compatible with dearVR?

    And next, is it possible to use it in realtime with multiplayer voice chat as the sound source? such as photon voice or other implementations?
    Thanks,
    Garrett
     
  6. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hello grahag11!
    Unfamiliar with dearVR, however, if you can grab an AudioSource from it, it should work with SALSA. Keep in mind, 3D audio tends to reduce the dynamic nature of the audio (diminishing the amplitude) the further the 'source' is from the 'listener' -- in turn, reducing the perceived lip-sync (IOW - the lips don't move as much).

    Yes, it works in a photon voice configuration. We have a blog post and video demonstration of it on our website.

    Hope that helps!
     
  7. bulisor

    bulisor

    Joined:
    Feb 12, 2016
    Posts:
    7
    Hello. Wonderfull package. But I need a litle help. I didn't find a workflow with lod characters. We use Autodesk characters which have 3 lods. I added Salsa on lod 0, work excelent, but if the quality is low default is loading lod 1 which haven't salsa. I have to put salsa on all lods, or on lod groups, it's not inefficient? Thank you.
     
  8. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi Bulisor,

    Thanks for your purchase, please send your invoice number to assetsupport@crazyminnow.com and we'll be happy to help.

    Thanks,
    Michael
     
  9. Deckard_89

    Deckard_89

    Joined:
    Feb 4, 2016
    Posts:
    253
    Last update was January 2016. Does Salsa work with Unity 5.5?
     
  10. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi Deckard,

    SALSA is tested and compatible with Unity versions from 4.6 to 5.5, and on the following platforms: Windows, Mac, Linux, Android, IOS, Windows Store (8 & 10 [.NET Core]), WP8, and WP10. It is not compatible with WebGL due to limitations in the Web Audio API that Unity uses for their WebGL implementation. A minor upgrade to 1.5 is nearing completion, and a major upgrade to 2x is in the works. We'll be discussing the upcoming features of the 2x path soon.

    Thanks,
    Michael
     
    wetcircuit likes this.
  11. Deckard_89

    Deckard_89

    Joined:
    Feb 4, 2016
    Posts:
    253
    Hi, thanks for the information.
     
  12. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi folks,

    It was brought to our attention that Morph3D released a 1.6 update the includes some pretty major changes. We have identified that they are no longer using Unity's built-in mesh-based BlendShape system, but rather a custom component system. This completely breaks our MCS 1-click add-on for these characters since it mapped to standard BlendShapes. We are evaluating the feasibility of a revised add-on to support the new system, and will keep everyone posted here.

    Sorry for the inconvenience, and thanks for your patience as we dig our way through the revised system.

    Michael
     
    wetcircuit and Hormic like this.
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Morph3D 1.6 does indeed still use standard mesh-based BlendShapes, they just stream in only the shapes you wish to use to help optimize the heavy load of so many shape options...a very smart option. We'll tweak the current 1-click setup to adapt to the changes and get it posted to the downloads soon, and the upcoming SALSA 2x branch should be able to handle the changes natively with loads of new options and without any add-on's.
     
    wetcircuit and Gregorik like this.
  14. Gregorik

    Gregorik

    Joined:
    Dec 11, 2014
    Posts:
    396
    That made me salivate. :D
     
    Crazy-Minnow-Studio likes this.
  15. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    LOL...good, we're pretty excited about it to.
     
  16. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    magique likes this.
  17. S1mon

    S1mon

    Joined:
    May 31, 2016
    Posts:
    12
    Hi,

    I downloaded Uma 2 package and i want to use my microphone for lip synchronisation. I saw this video
    But i didn't found "CM_UmaLiveMic" script in any unitypackage.
    Someone knows in which unitypackage i can find it?

    Thanks
     
  18. S1mon

    S1mon

    Joined:
    May 31, 2016
    Posts:
    12
  19. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    micInput-lite is available in the downloads section of our website. See the minInput-lite blog post for details and a download link at the bottom of the page.

    http://www.crazyminnowstudio.com/posts/real-time-microphone-input-for-salsa-lip-sync/

    Michael
     
  20. S1mon

    S1mon

    Joined:
    May 31, 2016
    Posts:
    12
  21. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    We have discovered a bug/behavior in MCS’s 1.6+ update that reorganizes BlendShapes when new shapes are added to a Morph 3D character. The shape order is not maintained upon Unity Editor reload. This behavior breaks our 1-click SALSA setup. We have released an updated 1-click (v1.2) for MCS 1.6+ which addresses the issue by adding a “Remap Blendshape Indexes” button above the shapes in McsSync.

    If you add new morphs to your character, after you close Unity and open it again, the shapes will likely not be mapped correctly. Simply click the button to remap the shapes if the order has changed.

    https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

    Hope this helps you out!
     
  22. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    250
    Hello Michael,

    I have problem with a blendshape, as you can see on the picture when the mouse is open the skin is not separated properly. It is approx. 1 year old Fuse 1.3 Character with blendshapes from the mixamo pipeline.
    Do you have a suggestion how i can fix this issue easily? :)
     

    Attached Files:

  23. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi Hormic,

    You could just exclude the problematic shape from CM_FuseSync, otherwise you'd likely need to import the model into your favorites 3D editor to actually fix the bad BlendShape.

    Michael
     
  24. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    250
    Ok, thank you,
    Unfortunatly i have no really clues about blendshapes on how to locate and exclude them. I guess exclude something in CM_FuseSync won't help me that much, cause excluding the body makes no sense and opening the mouth is crucial so cant exclude that anyway. So the solution i'm capable of is try to reimport the model and hope for the best.
     
  25. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    I meant you could remove the problematic shape from the CM_FuseSync inspector, by clicking the [X] button next to a given shape. This won't fix the shape, it simply makes it so that SALSA doesn't use the shape. MouthDown is used in the SayMedium shape group and MouthOpen is used for SayLarge. Your screen shot didn't specify which of those two shapes broken so I'm including info on both. If it's MouthOpen, you will likely need to select another shape that still gives you a lip-sync look you like since it's the only shapes used in the SayLarge group.

    Fuse_MouthDown_MouthOpen.jpg
     
    Hormic likes this.
  26. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    250
    Ok, thank you very much, yes mouth open is the problem, but i think its hard to find a satisfying workaround.
    Cause you could see every time the mouth opens a bit that there is something wrong. so i guess i will evaluate the model and reimport it.
     
  27. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    250
    Ok, just to let you know and maybe others with the same issue.
    I did a model optimization with the fuse decimator -> https://www.mixamo.com/decimator
    I guess this was the reason that the blendshapes screwed up, all is fine with the original model.
     
    Crazy-Minnow-Studio likes this.
  28. sunseeker1988

    sunseeker1988

    Joined:
    Jul 10, 2014
    Posts:
    440
    Hello,

    I just bought salsa and maybe your videos on how to use it with character creator assume I know more than I do. I imported my character from Autodesk character creator and dropped it in my scene and then ran the one click setup. I dropped my audio file into the correct location like you did on the video but I do not have a play button..... I attached a photo so you can see what I am talking about.
     

    Attached Files:

  29. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hello sunseeker1988,

    Thanks for your purchase. The play button appears when you run the scene.

    Michael
     
    sunseeker1988 likes this.
  30. sunseeker1988

    sunseeker1988

    Joined:
    Jul 10, 2014
    Posts:
    440
    Thanks for the quick response. I am very new to character mouth animation. I have the option to select the type of facial expressions in Character creator as, Facial blendshapes or facial Bone Rig. Which one should I choose to be generated with my character to work with salsa? after I drop this character into my scene, what do I have to set up on him as far as animation settings ect to get him to talk with salsa?
     
  31. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hello! You would want to select blendshapes. For lipsync, you simply need to export your character with blendshapes, add it to your scene, then add SALSA via the 1-click option just as you saw in the video. There are no other settings to configure.

    Hope that helps!
    Darrin
     
    sunseeker1988 likes this.
  32. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
  33. sunseeker1988

    sunseeker1988

    Joined:
    Jul 10, 2014
    Posts:
    440
    Super cool stuff. I ts working now. Amazing how this software works. I am having an issue trying to get my characters mouth to animate a little better. hes barely opening his mouth and seems to have no to poor saymedium, saylarge blend shapes. I played with every property and I cannot get him to open his mouth wider or with the other blend shapes. Your video for a CG character looks much better than mine. Will the audio file im playing affect the blend shape look depending on the audio quality? not sure what im doing wrong. I want to show this software off at a unity meetup im going too this weekend.
     
  34. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hello sunseeker1988!
    Yes, the audio file will certainly have an affect on lip synchronization. The more dynamic your audio, the better the lip-sync effect. You can compensate for this to a certain extent by adjusting the triggers for saySmall/Medium/Large. If your audio lacks good amplitude dynamics, adjust the triggers (medium & large) to lower values. Also ensure your range of motion is set to an appropriate number, typically 100% -- if this is too low, it will impact the extents of the blendshapes used for animation.

    You can try using one of the included audio files (military moves is a good choice) to see if the character responds better. This will aid determination of whether it is a settings or audio issue. Even with poor audio, the available settings options can generally be used to great effect.

    Hope this helps!
    Darrin
     
    sunseeker1988 likes this.
  35. Mythcons

    Mythcons

    Joined:
    Mar 16, 2017
    Posts:
    4
    Greetings,

    Excellent product. However, I'm trying to get Salsa3D to initiate a 'play audio' (thus triggering the lipsyncing) with a compatible character using Slate's cutscene creator. Do you know how I might go about doing this? I've created an actor properties track, Salsa3D shows up, but none of the properties seem to fire the audio.
     
  36. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hello Mythcons,

    We're glad you're enjoying SALSA. We have created a custom Action Clips add-on for Slate that has not been released yet. If you'd like to participate in a closed beta, please send an email to assetsupport@crazyminnow.com requesting access and include your SALSA invoice number.

    Thanks,
    Michael
     
  37. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    SALSA 2.0 isn't released yet, but here is a quick comparison of the quality improvements to our real-time automatic lip-sync processing in 2.0 prerelease.

     
    wetcircuit, magique, TonyLi and 2 others like this.
  38. RonanSB

    RonanSB

    Joined:
    Oct 29, 2015
    Posts:
    3
    Hey, we recently bought SALSA and RandomEyes for a project, and I'm very impressed! I had thrown something simple together previously based on amplitude analysis and a linear blend between two animations, but this does a much better job.

    I have one major blocking issue though - I can't seem to get it to work alongside Mecanim on our 3d character at all. If the animator is on, it completely overrides SALSA. I've tried masking parts of the mesh (from the head up), but it won't work unless the animator is disabled entirely.

    Am I missing something? I'd love to be able to additively blend SALSA on top of our hand-crafted animations.
     
  39. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi SoapboxCloudBuild,

    This issue, along with a few others, and some usability improvements are included in our 1.4.1 patch release. I'm working to try to get it submitted to the Asset Store this weekend. If you need it sooner, please send your invoice number to assetsupport@crazyminnow.com.

    Thanks for being a SALSA customer!
    Michael
     
    Last edited: Mar 21, 2017
  40. RonanSB

    RonanSB

    Joined:
    Oct 29, 2015
    Posts:
    3
    Great, and thanks for an incredibly speedy response! Look forward to the update!
     
    Crazy-Minnow-Studio likes this.
  41. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,783
    Looks great. Is 2.0 a regular free upgrade? When can we expect a release?
     
  42. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    We believe it’s important to do our best to minimize the impact to existing SALSA implementations when rolling out new capabilities, and we have worked hard with the 1.x line to preserve backwards compatibility throughout its development, while adding new features, add-on's, and workflows. We understand the importance of a stable base to build upon, and SALSA 1.x has provided that for three years.

    However, this hard line approach can also be an obstacle to rolling out product enhancements. We believe the design improvements and feature and flexibility gains in SALSA 2.0 are significant, and warrant our first departure from this hard line stance on maintaining backwards compatibility. It will not be possible to upgrade to SALSA 2.0 without breaking existing project implementations, and we definitely do not want people accidentally upgrading to 2.0 without being fully aware of the new implementation requirements. For this reason, and because of all of the work going into 2.0, our plan is to increase the base price from $35 to $39 for new customers, and offer upgrades to existing customers for the $4 difference. We consider this a safety mechanism to prevent accidental upgrades, and a modest and justifiable price increase for all of the work being done.

    Thanks,

    Team Crazy Minnow
     
    Lurking-Ninja likes this.
  43. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,783
    No problem. I will consider the $4 well spent.
     
    Crazy-Minnow-Studio likes this.
  44. jdscogin

    jdscogin

    Joined:
    Oct 26, 2014
    Posts:
    63
    I have an issue. As you may know, Adobe now owns Mixamo. They also have an new version of Fuze. I created a character, and when I try to use RandomEyes, when I click the Auto-Link on the second RandomEyes, nothing happens.
    I also tried the 1-Click Fuse workflow, and again there is nothing linked. Thanks Jd
     
  45. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,187
    I use Adobe Fuse. I think maybe the problem is that one of the Fuse base bodies has renamed the eyes "Default" and the others still use "Eyes". I noticed the difference in some of my male/female figures…. I don't know if that is your specific issue, with the 1-click.
     
    Crazy-Minnow-Studio likes this.
  46. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi jdscogin,

    When using any Fuse character, the Auto-Link button on the RandomEyes3D inspector is not applicable since RandomEyes3D uses BlendShapes for eye control and Fuse uses eye bones. The eye bones are instead linked in CM_FuseSync and the RandomEyes3D BlendShape values are retrieved and translated to bone transform translations in CM_FuseSync. As wetcircuit said, if Adobe renamed the eye bones, that would certainly make the CM_FuseSync script fail to find them since it's looking for bone names ending in "LeftEye" and "RightEye". If eye bone naming is the issue, please let us know which Fuse base model you're using so we can investigate and make applicable changes to the 1-click add-on.

    Thanks,
    Michael
     
  47. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi wetcircuit,

    Do you happen to remember which base model you were using when you discovered the "Default" eye bone name change?

    Thanks,
    Michael
     
  48. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,187
    Ahh, I do not mean to add misinformation. The eyeball mesh itself is renamed "default", but the bone name is still the same (as far as I know). From your description, it should not matter to RandomEyes.
     
  49. jdscogin

    jdscogin

    Joined:
    Oct 26, 2014
    Posts:
    63
    Still having issues. Should the One-Click version still have shapes? Because I see none. An when I press the Play button on the Salsa 3D (Script), it plays the audio, but no movement at all on the character. I tried changing the Default to Eyes and changed the Mesh to Eyes, and the character's eyes bugged out. Kinda of funny though.
     
  50. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,008
    Hi jdscogin,

    SALSA does not add BlendShapes to your character, they must already exist on the character. Since you're using Fuse, you need to upload the character to the Auto-Rigger and select the option to include BlendShapes on the Adobe/Fuse website.

    Thanks,
    Michael
     
unityunity