Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. SickaGamer

    SickaGamer

    Joined:
    Jan 15, 2018
    Posts:
    785
    dialogue system and UMA integration?
     
    Crazy-Minnow-Studio likes this.
  2. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    9,764
    I'll be working on Dialogue System integration this week.
     
  3. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    UMA-DCS integration is planned and in our queue of to-do's.
     
    Firlefanz73 likes this.
  4. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    508
    Awesome. BTW are you familar with CharacterCreator3? I'm using CharacterCreator3 and UMA. Is it possible to use Salsa for CC3 UMA characters? UMA traditionally use bones for facials but CC3 uses blendshapes. I guess it's possible to use blendshapes but setting them up properly will be painful. I'm wondering if you will provide some sort of blendshape templates for CC3 to be used for UMA. Thanks.
     
  5. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    We are familiar with CC3, our iClone 1-click for SALSA v1 was adapted to also work with CC3 back in October 2018. CC3 integration is also planned and in our queue of to-do's for SALSA v2. SALSA v2 works with bones, blendshapes, sprites, materials and textures.
     
    chrisk likes this.
  6. therewillbebrad

    therewillbebrad

    Joined:
    Mar 2, 2018
    Posts:
    126
    Will there be a new box head?
     
    Crazy-Minnow-Studio likes this.
  7. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    508
    Awesome! I'm really looking forward to it. I hope supporting UMA version of CC3 is not much different from vanilla CC3. Cheers!
     
  8. Hexer_

    Hexer_

    Joined:
    Aug 25, 2016
    Posts:
    93
    Hi, is there an easy way for one object to mirror blendshapes of another?
    I'm thinking of using Broadcasting to target the other object and then a script to update the blendshape status of it to match the original.
     
  9. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Hi,

    The following property can be used to read the processed blendshape output values for saySmall, sayMedium, and sayLarge.

    Salsa3D.sayAmount (SalsaBlendAmounts)
    • Salsa3D.sayAmount.saySmall (float)
    • Salsa3D.sayAmount.sayMedium (float)
    • Salsa3D.sayAmount.sayLarge (float)
    A custom type that contains the say BlendShape amounts. It's updated in realtime and provides direct access to the values even when no SkinnedMeshRenderer is linked. This is useful for asset integrators that want to use Salsa3D as a lip sync engine with their own integration.
     
  10. SickaGamer

    SickaGamer

    Joined:
    Jan 15, 2018
    Posts:
    785
    The long drawn out process of getting Salsa 2 live......
     
    Crazy-Minnow-Studio likes this.
  11. Hexer_

    Hexer_

    Joined:
    Aug 25, 2016
    Posts:
    93
    Thanks, but in my case I'm trying to capture Custom Shapes
     
  12. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    In SALSA v2, Custom Shapes will be handled in EmoteR. EmoteR has the same sort of flexible configuration as SALSA with its emote expressions. You could configure any number of SkinnedMeshRenderers on an emote and effectively fire the shapes off from one instance. This would work pretty easily in small configurations, but may become cumbersome depending on the number of shape components used. Additionally, it does increase the queue length, which means more processing during shape registrations. So, all of that said, it is possible to implement in v2, but may not be practical, it just depends on your situation.
     
    Hexer_ likes this.
  13. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    494
    My characters have A LOT of blendshapes and I would like to make some custom combined ones and delete all the rest.
    MorphMixer seems perfect for the task, however:
    - Does it support Daz3D Genesis 1 characters?
    - MorphMixer supports making presets for combining blendshapes, which is AWESOME, but does it support DELETING blendshapes once the combined ones are done? (this would be really cool in case I would need to reimport the character from scratch in Unity because of a change in design)

    I know it's off topic, but since this is related to Salsa, I thought no one would mind :p
     
  14. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    MorphMixer works by creating an asset copy of your original mesh, and then remapping the copy to your Renderer automatically. It does not modify the original mesh. It should work on any character that imports into Unity with a SkinnedMeshRenderer and has blendshapes. On the asset copy you can create new blendshapes by combining one or more existing blendshapes of varying amounts, or delete any blendshape. The only issue we're aware of is on CC3 characters, the size of the created asset mesh file is about twice the size of the original, we aren't sure yet what causes this since it doesn't happen on other characters, possibly something within the CC3 system.

    FYI MorphMixer and Amplitude also have their own forums found at the links below.
     
  15. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,097
    Where can I get the update purchase? I just saw 2.0 in the store…

    Thanks :)
     
    Crazy-Minnow-Studio likes this.
  16. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Hello Everyone!!!

    Looks like SALSA LipSync v2 Suite is now live...WOW that was fast!

    To get the upgrade, you'll need to be logged in with the account you purchase SALSA v1 with and you should see update pricing. We're seeing quite a few $4 purchases, so this appears to be working.

    We thought we were going to have 2 or 3 weeks to get everything lined up here and that turned into 3 days, so please be patient and bear with us. Some things of note:
    1. v1.x addons do not work with v2! I'll be getting the downloads site operational and we'll get the packages uploaded that are ready. Please consider everything beta at this point and let us know via assetsupport@crazyminnow.com if you have problems.
    2. I will be linking the documentation we have ready on the site shortly. In the meantime, you can get to it directly from this link: https://crazyminnowstudio.com/docs/salsa-lip-sync/
    3. We will be updating the site to reflect SALSA LipSync v2 Suite as we move along.
    4. Addon development will continue as well.
    5. We will also get some video tutorials out there.
    We honestly thought we'd have at least a couple of weeks so please be gentle.

    Thanks a bunch!
    Darrin
     
    magique, Hexer_, Firlefanz73 and 2 others like this.
  17. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    494
    mmm... now I don't know if I'll buy it.

    Just kidding, I just did it. :p
     
  18. SickaGamer

    SickaGamer

    Joined:
    Jan 15, 2018
    Posts:
    785
    Ditto! I bought it as well! Good things do come to those who wait! @TonyLi Now Tony has to work his magic for his wonderful assets!
     
    Crazy-Minnow-Studio likes this.
  19. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    9,764
    @SickaGamer - I'll post here when it's done. Should be a few days more.
     
    Crazy-Minnow-Studio likes this.
  20. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Making some progress:
    • The downloads site is live. Currently ONLY the TimelineSalsaCore package is uploaded.
      • We will continue to upload packages and expect to have several available by tomorrow.
      • Most packages will leverage a core EXAMPLES package for demo scene usage. I will try to have this package up shortly (and we may have to iterate this process a couple of times to get it right but the hope is a unified package will keep the core product light and nimble as well as the add-ons).
    • We are updating the site as we go, deprecating the old information and adding new links and images reflecting the new version.
    • For SALSA v1 users that don't wish to upgrade, the legacy downloads and documentation are still available, links at the bottom of the side menu (see the image below). As a reminder, SALSA v1 is now deprecated. It will receive no further updates. You are free to continue using it for as long as you wish, but we recommend upgrading to the new version for the new shiny (currently only $4).
    usingSalsa_026.png

    Thanks so much for your patience and support!
    Darrin
     
    wetcircuit likes this.
  21. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,073
    Forget unlimited visemes…

    Screen Shot 2019-06-20 at 6.51.39 PM.png

    WE GET UNLIMITED HEADS???
     
    Crazy-Minnow-Studio likes this.
  22. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    :) We aim to please!
     
    wetcircuit likes this.
  23. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Crazy Minnow...delivering all of your Hydra animation needs. :)
     
    wetcircuit and SickaGamer like this.
  24. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    All SALSA LipSync Suite 2.0 compatible add-ons will have the initial release version of 2.0.0. The following compatible add-ons have been posted to the downloads page.
    https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/downloads/
    • Playmaker 2.0.0
    • Cinema Director 2.0.0
    • Adventure Creator 2.0.0
      • SalsaTextSync Adventure Creator Listener 2.0.0 (SalsaTextSync will be uploaded soon).
    • NodeCanvas 2.0.0
    • SLATE Cinematic Sequencer 2.0.0
    We're working very hard to get everything updated as quickly as possible. We hope to have several of the 1-click add-ons available in the next couple of days. Thanks for a successful launch day turn-out, and your patience as work to finish the remaining add-ons, documentation, and website updates.
     
    Last edited: Jun 21, 2019
  25. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,097
    Now counting the days for the UMA Plugin :)
     
    Jaimi and Crazy-Minnow-Studio like this.
  26. Volkerku

    Volkerku

    Joined:
    Nov 23, 2016
    Posts:
    50
    Is unity timeline supported in salsa 2.0?
    The Basic timeline support in v1 was a great start, and really useful to time emotes and play audio.
    V.
     
    Crazy-Minnow-Studio likes this.
  27. philc_uk

    philc_uk

    Joined:
    Jun 17, 2015
    Posts:
    81
  28. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Hello! We do not support this at this time. It is currently unclear if/how we would leverage this into our system, but we are always open to new options. And to clarify, SALSA is not a phoneme mapped solution. We analyze audio in real-time and approximate lip-synchronization based on a few different factors.
     
  29. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Yes, Timeline is supported and the core add-on is available on the downloads page. Although, I'm going to update it one more time very shortly, so wait for v2.0.2. Updating the Examples pack and will need to update the Timeline pack to the new GUIDs for the example scene. If you don't need the example scene -- go grab it now! ;)
     
  30. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    We have posted up a few more add-ons for v2:
    • SalsaTextSync v2.0.0
    • TimelineSalsaTextSync v2.0.0
    • SalsaRtVoice v2.0.0
    • TimelineSalsaCore v2.0.2
    • Examples-Pack (has some audio files and 2D assets used in sample scenes for some add-ons).
    The blog posts for these have received a very fast update for SALSA LipSync v2, but proper documentation will eventually be completed and moved to our docs repository.

    Hopefully coming by early tomorrow will be:
    • Dissonance
    • micInput
    • MasterAudio
    These are completed, just need a final test on the new base code and a proper export. And yes, we are hoping to get the OneClicks out as well. ;)
     
    Hexer_ and wetcircuit like this.
  31. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    494
    Faster!


    :p
     
  32. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    The following add-ons should now be available:
    A quick pass has been completed on the original blog posts for info/instructions. We will be moving documentation to the /docs repository in the near future.

    NOTE: SalsaMicInput no longer needs to be configured to use an attenuated mixer group. With the new analysis engine in SALSA LipSync v2, you can mute the microphone at the AudioSource and lip-sync will get along just fine!
     
    Hexer_ likes this.
  33. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,073
    OK! This should solve an issue I had with the MicInput… I always got a late echo like a weird sampling loop that came back 20sec later, but pitched slower… I could never solve it. I also route my mic through an audio app to delay it a fraction of a second to match the SALSA delay, but that weird sampling loop was a spoiler…. This new way should let me just mix my live Mic with Unity audio output (externally) and avoid the sampling issue.
     
    Crazy-Minnow-Studio likes this.
  34. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,073
    Oh dang! That totally works!
    Screen Shot 2019-06-22 at 4.19.55 PM.png

    BUY THIS THING, Y'ALL! :D MUTE IS WORTH ANY PRICE! :p
     
    Crazy-Minnow-Studio likes this.
  35. therewillbebrad

    therewillbebrad

    Joined:
    Mar 2, 2018
    Posts:
    126
    Will the old toon box head work with salsa 2? I've grown quite fond of him.
     
  36. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
  37. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Yep, he should work just fine. ;)
     
  38. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Unity's Microphone input is a touchy little beastie. Input data is stored in a circular AudioClip of configurable length and if the AudioSource play-head gets ahead of the Microphone record-head, you get that weird delay of how ever many seconds the clip buffer is set to. I believe the old version of micInput was set to 10 seconds. I can't explain the slower/lower pitch - that's weird. It is also pretty easy for the heads (pointers) to get out of sync. One of the biggest culprits is application focus and pausing -- at least from what I've seen. If the application is paused for whatever reason, it's best to simply restart the micInput recording using StopMicrophone() & StartMicrophone().

    Regardless of where playback thinks it is, SALSA LipSync v2's analysis is continually calibrated to the microphone's record position. That doesn't solve the audible playback issues, but it does make the analysis as close to real-time as it can be. So, if you don't need the microphone playback to be audible, it works about as good as it can. For those that need to output to loudspeakers, it may work best to split microphone output to an amp (preferably with a small or adjustable delay to really dial things in) and avoid using the AudioSource output from Unity's microphone input.

    Thanks for the feedback wetcircuit!
    Darrin
     
    Last edited: Jun 23, 2019
    wetcircuit likes this.
  39. angusmf

    angusmf

    Joined:
    Jan 19, 2015
    Posts:
    255
    Is there an equivalent to the properties for in v1? I want to trigger an event and none of the component types look particularly suited for that.
     
  40. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Hello,

    Which v1 properties are you referring to? Can you tell us more about what you're trying to do?
     
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
  42. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    TonyLi and wetcircuit like this.
  43. angusmf

    angusmf

    Joined:
    Jan 19, 2015
    Posts:
    255
    Um, not really, I can't find the original documentation and don't see what I'm looking for in v1. I had a foggy memory of there being a public property or properties set to the current "state" (saySmall, sayMedium, and so on.) I had planned to use that to drive properties on materials and VFX for animation.
     
  44. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    The v1 documentation is still available.
    https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/manuals/

    In v2, you can drive textures and materials directly from the UI by changing the animation controller type.
     
  45. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    9,764
    Dialogue System integration for SALSA LipSync Suite is now available. Please see this post for details.
     
  46. metafunnel

    metafunnel

    Joined:
    Jun 18, 2015
    Posts:
    2
    Since I upgraded to V2 I got this:
    Assembly 'Library/ScriptAssemblies/Assembly-CSharp-Editor-firstpass.dll' will not be loaded due to errors:
    Reference has errors 'SALSA-LipSync'.
    Before I was OK. Please help.
    Thank you,
    CJ
     
  47. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Hi metafunnel,

    It sounds like you might be using the unsupported Unity beta version 2019.2? Is this in a new project? SALSA v2 is not backwards compatible with v1 since it is a complete rewrite and require a new setup on all SALSA enabled character.
     
  48. innovationlover

    innovationlover

    Joined:
    Dec 25, 2018
    Posts:
    2
    Congrats on the V2 release, I just bought it. Very excited to get started. I wanted to know if you have a demo video showcasing the new head and eye animations? Curious to see how they look. Great job btw on the new version.
     
  49. angusmf

    angusmf

    Joined:
    Jan 19, 2015
    Posts:
    255
    Yep. that'll be good for a wholesale change to a from one material to another, but won't let me change their exposed properties as I understand it. Neither is any help with VFX graphs.

    --- Edited above to be slightly more clear about what I DO want to change using an expression component or whatever it would take. Also sent mail to support. Thanks.
     
    Last edited: Jun 23, 2019
  50. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    907
    Send us all of your pertinent information (in an email to assetsupport@crazyminnow.com), including your invoice number and a reference to your ID in this thread. It will, of course, be necessary to have sufficient detail about what you want to accomplish. SALSA v2 is very flexible in its underlying design, so we'll see what we can figure out for you.
     
    angusmf likes this.