Search Unity

  1. Unity 2019.1 is now released.
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, expression, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738


    Only 9.5 hours left to get our highly rated SALSA Lip-Sync solution for 44% off. Want to add lip-sync to your game or art project without spending hours, days, or more mapping phoneme shapes? SALSA is completely automated, requires no audio pre-processing, and works with the following systems:

    Real-time microphone lip-sync
    Fuse Character Creator
    Autodesk Character Generator
    DAZ3D
    Dialogue System for Unity
    Adventure Creator
    Master Audio Playlists
    Playmaker
    Custom character models

    https://www.assetstore.unity3d.com/en/#!/content/16944
     
    Last edited: Jul 15, 2015
  2. JeffG

    JeffG

    Joined:
    Oct 21, 2013
    Posts:
    11
    I am having an issue when selecting the Auto-Link option for the Custom Shape RandomEyes3D Script

    It doesn't seem to Auto Link.

    Using the latest version of Fuse 1.3 Unity 5.1.1f1

    am I missing something simple?
     
  3. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Hello JeffG,

    Thanks for your support!

    For a fully setup Fuse character, you should have two RandomEyes3D components, one for eye control where you leave the Skinned Mesh Renderer field empty and another for facial animations. To auto-link the facial animation BlendShapes, link the Fuse [Body] child object to the Skinned Mesh Renderer and check the box for [Use Custom Shapes Only].

    I demonstrate this at 2:38 seconds in the following video.


    Michael
     
  4. JeffG

    JeffG

    Joined:
    Oct 21, 2013
    Posts:
    11
    and I get nothing. Click the button and nothing happens. The list of custom shapes doesn't get populated. Nothing in the console either.

    Now, I am on Windows 10. Not sure why that would make a difference, but I can try it on my laptop (Windows 7) tomorrow morning
     
  5. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Can you look at the Skinned Mesh Renderer on your [Body] object, do you see a list of BlendShapes?

    We've tested up to Windows 8.1.
     
  6. JeffG

    JeffG

    Joined:
    Oct 21, 2013
    Posts:
    11
    Stupid User trick. Blend shapes for facial animations are disable by default when doing the auto rig. Thanks for the info though. Gave me a the right place to look.
     
  7. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Glad it's working Jeff, let us know if you have additional questions.
     
  8. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
  9. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Result from our 24:HOUR SALE



    WOW, we sold 251 copies of SALSA with RandomEyes, to 29 different countries, in 24 hours!

    We are a very small team and put in tremendous effort to provide a great product at a fair price. We want to thank all of our customers for supporting our product, offering great suggestions to help us improve SALSA, and spreading the word about our lip-sync solution. Finally, we’d like to welcome all of our new customers into the fold. If you love SALSA, please spread the word, and rate or review if you haven't already. Your word-of-mouth praise and great reviews gives others the confidence to give our solution a try.

    Click below to read our entire blog post about our 24:hr sale experience.
    http://crazyminnowstudio.com/posts/results-from-our-first-24-hour-sale-on-the-unity-asset-store/
     
    wetcircuit likes this.
  10. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738


    I'm attending the SIGGRAPH 2015 conference in LA on Tuesday, August 11th. I'll be walking the exhibit floor in a white polo with the SALSA logo on the front and the Crazy Minnow Studio logo on the back.

    If you see me, say hello or stop for a chat.

    Michael

    Registration
    Exhibitors
     
    Last edited: Jul 24, 2015
  11. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    248
    hi Michael, is it possible to use Salsa with a textfile instead of an audiofile. i would need this very much and in the tutorial videos i only saw audiofiles so far. thank you!
     
  12. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    940
    I bought EasyVoice asset to create text-to-speech audioclips inside Unity. It works well with SALSA, but does not make TTS voices at runtime "on the fly" - that would be very dependent on the end user's installed voices (there are a few TTS assets but I have not used them)....
    https://www.assetstore.unity3d.com/en/#!/content/22328
     
    Hormic and Crazy-Minnow-Studio like this.
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Hello Hormic,

    SALSA cannot read text files directly, it uses real-time waveform analysis to drive BlendShape value transitions. Without the data from an audio clip, SALSA has nothing to process.

    Depending on what you're trying to do, it might be possible to use a 3rd party text-to-speech software, like the one below or another, to generate the audio files SALSA needs ahead of time. However, this still results it SALSA processing audio clips instead of text files.

    Though not tested, maybe something like this? It seems to leverage the Windows Speech API to generate spoken audio clip files using Microsoft's text-to-speech engine. The caveats with an approach similar to this include Windows dependency, and a lack of rapid real-time processing needed in most game scenarios.
    https://www.assetstore.unity3d.com/en/#!/content/22328

    Michael
     
  14. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Jinx! Thanks wetcircuit. :)
     
  15. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    940
    LOL ;)

    While you are here, I have a quick question. In your Playmaker actions I'm a little confused over Duration and Blend.

    They are both time(?) settings but Duration is "x seconds" and Blend is what exactly? Fade In/Out time? A proportional relationship of the overall duration? a "speed" setting?

    Screen Shot 2015-07-26 at 3.45.46 PM.png

    I feel like I am using a Ouija board to control my face, LOL. (It's not just your actions, I think this is a convention of Unity or maybe Playmaker that I just don't know when we are talking "seconds" or "speed" or what...
     
  16. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    @wetcircuit

    Duration is the total number in seconds, that the expression will be enabled. The Blend fields represent a step increment towards the max Range. In your screenshot the expression will be enabled for 5 seconds, the Smile_Left and Smile_Right shapes will blend towards 100 at increments of 5, and the Squint_Left and Squint_Right shapes will blend towards 100 at increments of 20. You won't see actual stepping because the values are blended using linear interpolation.

    So stepping a smile shape from a value of 0-100, at a blend increment of 5, will occur over 20 iteration. A squint from 0-100, at a blend increment of 20, will occur over 5 iteration.

    If the Duration were set very short (1 second) and the Blend increment very small (0.001 or whatever), the shape will never reach it's max Range, instead it will move towards it's max range for 1 second, then use the same Blend (step size) to decrement back to 0.

    I hope this helps,

    Michael
     
    wetcircuit likes this.
  17. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    940
    Lemme see... If one train leaves Pittsburgh traveling 45mph....
    cat_with_calculator-600x450.jpg

    j/k the iteration steps never would have occurred to me....
    But now some of my previous tweening issues might need reviewing... lolol
     
    Crazy-Minnow-Studio likes this.
  18. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    And another train leaves New York at 2:30pm...LOL
     
    wetcircuit likes this.
  19. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    248
    Hmmm, thank you for your suggestions (also @wetcuircuit i will look into EasyVoice), but is it so complicated to integrate an uncomplicated behavior for textfiles?
    Not every game in the world needs and wants to use spoken audio files.
    So, for instance when a line of speech is visible on screen, create a random motion for the mouth if no audio file is available. So you have mouthmotion during reading a line of text on the screen. I can imagine that this is not soooo hard to implement. With this behavior Salsa would be the perfect solution for every cicumstance.
     
  20. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    940
    You just need the mouth moving silently? I think I would probably just create a short animation clip keyframing the mouth blendshapes in a few spots and then loop it. There is no reason to use SALSA (or any component) for that. It's overkill if there is no audio... That would be my suggestion. It is actually simpler than setting up SALSA.

    Although I would still use RandomEyes for the facial expressions and blinks, etc
     
    Hormic likes this.
  21. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    248
    Yes thank you, that is possible and is of course a good suggestion, but when i already have a software for it with salsa, i would reccomend that it can also do this.
    Or at least has the functionality to manually control start and stop mouthmotions and blendings so that i can use it in one flow with Random Eyes in my project, when i don't have or want audio speech for every single line of text.
     
  22. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Ahh, I thought you were asking for accurate text lip sync which would be a lot of unnecessary processing. As wetcircuit said, there are definitely other ways to handle this but I can see the advantage of having an option in our API so that if you're already using SALSA for your spoken dialogue workflow you wouldn't need another workflow to handle a simple "jabber mouth" style animation. It's an interesting idea, it's still common for games to have only some of the in-game dialogue with audio, and some text only.

    We'll look into it, thanks for the suggestion!
     
    Hormic likes this.
  23. Hormic

    Hormic

    Joined:
    Aug 12, 2014
    Posts:
    248
    Thank you, i'm looking forward to it! :)
     
  24. adamz

    adamz

    Joined:
    Jul 18, 2007
    Posts:
    891
    Im interested in lip sync using real-time audio from text-to-speech. Is it possible? I noticed that you have microphone support so I'm assuming it is. I bought RT-Voice and I would love to hear your thoughts on getting your asset to work with theirs. If it is, I would definitely invest in SALSA!
     
  25. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Hi adamz,

    That looks like a very cool solution. We are looking into it to see if it will work with SALSA. We'll post a reply here as soon as we have an answer.
     
  26. adamz

    adamz

    Joined:
    Jul 18, 2007
    Posts:
    891
    Fantastic!
     
  27. adamz

    adamz

    Joined:
    Jul 18, 2007
    Posts:
    891
    I mentioned your asset to the creators of RT-Voice, so perhaps you'll both be able to work together to integrate it!

    Are you planning on integrating phoneme-based lipsync matching, to get a more accurate animation to the audio?
     
  28. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Hello adamz!

    We purchased RT-Voice yesterday to take a look at it and also reached out to crosstales. As it turns out, RT-Voice uses the computer system's TTS engine to create audio (outside of Unity), so at this point, the two won't work together. crosstales is looking into it to see if there's a solution available.

    BTW, microphone input works in SALSA because the Microphone system links up with Unity's AudioSource.clip and that's what SALSA attaches to.

    At this point, we don't have any plans for phoneme mapping features. There are already some very good solutions available for that sort of workflow. SALSA fits a niche that provides very nice lip-sync with very little effort (no need to create dozens of phoneme shapes, nor manually map or pre-process audio files) and it works in real-time and on all platforms. Of course, we are always looking for ways to keep to SALSA's goals and improve upon its tried and tested technology. If you have not yet seen our latest promo video, please check it out. It uses DAZ3D models and demonstrates the power of SALSA with RandomEyes. Our blog post gives a high-level overview of how we set it up.

    If working with RT-Voice and phoneme mapping are your requirements, it certainly looks like SALSA may not be the asset you're looking for. Hope this answers all of your questions.
     
  29. adamz

    adamz

    Joined:
    Jul 18, 2007
    Posts:
    891
    Thanks for looking into this for me. Im trying to come up with a system that will allow me to take dynamic text created at run-time, convert it to audio, which then will allow me to use SALSA. Would something like Easy Voice work? I'm not sure how long it will take to create a audio file at run-time, but that could be another solution I could use...
     
  30. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Ahhh yes, Easy Voice. We took a peek at this a while back. If I recall, they create audio files in batch. So, I don't believe it's a run-time solution, but it's certainly a quick way to produce some TTS audio. This would certainly work with SALSA since the end result is an audio file.

    In script, you'd either attach to your AudioSource and set the clip or do it directly in the SALSA API. [NOTE: if you attach to the AudioSource, in v1.3.2 and earlier of SALSA, you'll need to collapse the SALSA inspector or SALSA will override the AudioSource -- this will be addressed in the next release of SALSA.] Again, it's not a run-time solution, but this option should work. It may be possible for crosstales to add a similar feature to RT-Voice at runtime -- if you ask them nicely. :)

    Your mileage will vary on how good the audio files will look in SALSA based on the voice quality used. We have tested some nice TTS output (Acapela Box) with SALSA in the early days and it looked good.
     
    Last edited: Aug 5, 2015
  31. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Our newest promo video is out, featuring DAZ3D characters EmotiGuy and Dragon. This video demonstrates real-time lip-synchronization using SALSA technology in a cinematic scene. It also fully leverages the RandomEyes asset (included with SALSA) for eye-movement, blink, and custom expression triggers. All lip-sync is completely controlled by SALSA, in real-time, simply by swapping audio clips for each model in a dialogue style. As mentioned, we used DAZ3D models in the promo along with our DAZ3D workflow script. The whole process is super easy to setup and achieves great results.

    Our blog post provides a little more detail on how we put it together. We're discussing whether or not we want to put out a more detailed "how-we-made-it" video/article.

    For those that are interested, we used Simple LOD by Orbcreation to create the flying dragon LODs. Simple LOD is pretty awesome! We also used Camera Path Animator by Jasper Stocker to a great extent, controlling the flight paths of the dragons and all of the camera movements/settings.
     
  32. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Experiment: SALSA real-time lip-sync with RT-Voice runtime text-to-speech

    This is a quick experiment to test SALSA real-time lipsync with the RT-Voice runtime text-to-speech system. This test was performed using pre-release beta versions for each system.

    SALSA version 1.3.3 (beta)
    RT-Voice 1.0.10 (beta)


     
    Stefan-Laubenberger, Hormic and adamz like this.
  33. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738

    SALSA v1.3.3 has been submitted to the Asset Store!


    We're happy to announce our 8th SALSA release. This is a minor release containing a few new features to enhance usability, and provide greater flexibility across implementations. If you want to read the highlights or check out the full release notes, follow the link below.

    http://crazyminnowstudio.com/posts/salsa-with-randomeyes-1.3.3/
     
  34. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738

    SALSA and RT-Voice...a winning pair!

    It's official, with the release of RT-Voice v1.1.0 from Crosstails and the recently submitted SALSA v1.3.3, Unity developers can now have lip-synced text-to-speech at run-time with no pre-processing. Follow the link below for all the details.

    ** This workflow will work using SALSA 1.3.2, but you will have to collapse the SALSA inspector. This caveat is removed in v1.3.3, which has been submitted to the Asset Store for approval.

    http://crazyminnowstudio.com/posts/salsa-and-rt-voice-for-runtime-lipsynced-text-to-speech/
     
    Stefan-Laubenberger likes this.
  35. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Stefan-Laubenberger likes this.
  36. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    9,334
    Dialogue System users: Some of the example assets have new metadata in SALSA v1.3.3. The Dialogue System's SALSA Support example scene needed to be updated to correctly reference the example assets. If you're using SALSA 1.3.3+ and want to play the example scene, you can download an updated support package here:

    SALSA_Support_2015-08-19.unitypackage

    The next release of the Dialogue System will also include a SALSA + RT-Voice integration package!
     
    Stefan-Laubenberger likes this.
  37. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Last edited: Aug 25, 2015
    Stefan-Laubenberger and TonyLi like this.
  38. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    940
    Ahh poor Ogre! hahaha

    Works great on Firefox :D
    (not so much on Safari)
     
    Crazy-Minnow-Studio likes this.
  39. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    9,334
    The WebGL version worked just fine in Chrome on Windows. What a fun little endless runner! Great job!
     
    Crazy-Minnow-Studio likes this.
  40. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Thanks, we're seeing issues on Safari as well and trying to figure it out. We also just posted a Mac desktop version that is working well for us. If you're interested in trying it, we'd love to hear if it's working on more than just our one test Mac.

    (sorry, the previous file link had changed - this link is to the page with the files)
    http://crazyminnowstudio.com/posts/run-ogre-run/
     
    Last edited: Aug 25, 2015
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Thanks! We're pretty beat after the 72 hour marathon, but it was fun, and nice to give our full tool chain a workout.
     
  42. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    940
    Crazy-Minnow-Studio likes this.
  43. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Sorry, we re-uploaded the file and the name changed. I've put a link to the page with the downloads instead to make sure that doesn't happen any more.

    http://crazyminnowstudio.com/posts/run-ogre-run/
     
  44. SheridaH

    SheridaH

    Joined:
    Jun 2, 2013
    Posts:
    7
    Hiya :)

    I purchased this asset last week ( it's really neat!) and am trying to set it up with Autodesk Character Generator. I downloaded the scripts and it all works neat except for one issue. The character has different blendshapes for the right and left part of the face, e.g. a smile is divided in 2 files, right and left. As a result instead of a complete smile my character does a smirk with only one side of her face. Is there a way to set multiple expressions as 1 custom shape?

    Thanks!
     
  45. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Hello SheridaH and thank you for your purchase. There are a couple of options for this scenario. Please send your invoice number to assetsupport@crazyminnow.com and we'll see if we can find a good solution for you. Thanks!
     
  46. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    9,334
    The Dialogue System for Unity version 1.5.5 is now available on the Asset Store with an integration package and example scene that uses SALSA and RT-Voice. SALSA made it really easy to set up: Just drop an RT Voice Actor component on your SALSA-configured speakers, and they'll automatically speak Dialogue System conversation lines and barks with full text-to-speech lipsync!
     
    Crazy-Minnow-Studio likes this.
  47. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Unity 5.2 AudioSource.GetOutputData Bug Work-Around:
    There is currently a bug in the AudioSource.GetOutputData component in Unity 5.2 that fails to produce data output when the AudioSource is muted. This becomes problematic when using something like SALSA's addon product micInput that relies on a muted AudioSource when collecting microphone input data to prevent echo from your voice. Luckily, there is a work-around until Unity can change the operation back to what it previously was. By the way, the documented bug can be found here. This does not appear to be intentional behavior (on Unity's behalf), so we are assuming it will be fixed at some point.

    This does not appear to be a microphone hardware or operating system issue and is strictly limited to Unity 5.2. MicInput and SALSA operate correctly on Unity 5.1 and lower. Read more in our blog post detailing how to work around this Unity bug.
     
  48. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    [CORRECTION] Unity 5.2 AudioSource.GetOutputData behavior change (bug)

    It appears as though this is the intended behavior that Unity has implemented. The 5.2 release notes do not mention this change, but the issue-tracker is pretty clear that they are playing catch-up on correcting something they feel is illogical in implementation.

    For micInput (and any other operation where you need to process audio and not hear the audio itself), use the Mixer Group method to pipe audio into a group with -80dB attenuation.
     
  49. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738


    SALSA with RandomEyes now supports MCS models from Morph 3D.

    If you haven't heard, Morph 3D has released their new Morph Character System (MCS). This awesome system starts with a male or female base mesh, and includes 350 individually configurable and combinable morphs allowing you to customize your characters to just about any configuration you could want. You can read more about it using the link below.

    https://www.morph3d.com/mcs-morph-character-system.php

    Since MCS models are based on DAZ3D Genesis 2 models, we added MCS support to the DAZ3D 1-Click workflow in our latest 3.0 add-on release.

    **SALSA with RandomEyes will not work with the Lite versions of the MCS models because the Lite version does not include any of the mouth or blink shapes required.

    Check the link below for more information.

    http://crazyminnowstudio.com/posts/using-daz3d-characters-with-salsa-with-randomeyes/
     
    Last edited: Sep 18, 2015
  50. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    738
    Hormic likes this.