Search Unity

  1. We would like to hear your feedback about Unity and our products. Click here for more information.
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Hello tiago_carvalhais,

    If I had to guess, it sounds like a Unity subsystem isn't being exported to your Android Studio project. SALSA relies on Unity's integrated version of the FMOD sound library to process audio. That's my best guess, but since we've never had a need to built a SALSA-based project using Android Studio directly, we have no experience with your solution approach.

    Michael
     
  2. milopez

    milopez

    Joined:
    Sep 14, 2018
    Posts:
    1
    where did you end up finding CM_SalsaSync to download? I've been trying to figure it out for hours. I can't find it on unity when I click import or on the Crazy Minnow Studio website. Is it still supported?
     
  3. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Hi milopez,

    You can use the add-on links on the Asset Store page, the search box on our website, go straight to the download link on our SALSA product page, or just google "SalsaSync."
     
  4. RonanBurgess

    RonanBurgess

    Joined:
    Apr 20, 2013
    Posts:
    97
    Hi,

    Is it possible to get Salsa working with Reallusion Character Creator?

    Thanks
     
  5. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
  6. MarkFitzpatrick

    MarkFitzpatrick

    Joined:
    Feb 8, 2019
    Posts:
    21
    Question about Random Eyes with UMA 2.0

    Do you have to use blendshapes for the eye rotation or can you use Joints? I watched the
    UMA 2 Lipsync using SALSA with RandomEyes video
    and it seems like it was still using Blendhshapes to drive UMA's joint system. SO again, just want to confirm, if Im using UMA can I use joints to drive the eye or do I still need to make blendshapes for Random Eyes.

    Thank you.
     
  7. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Hi MarkFitzpatrick,

    The currently supported add-on is our SALSA for UMA DCS found at the link below.
    https://crazyminnowstudio.com/posts/salsa-add-on-for-uma-dcs/

    It uses the UMAExpressionPlayer to indirectly drive facial bones for lip sync and emotions, and RandomEyes eye movement amounts are converted and applied to eye bones within the SalsaUmaSync add-on.
     
    Jaimi likes this.
  8. MarkFitzpatrick

    MarkFitzpatrick

    Joined:
    Feb 8, 2019
    Posts:
    21
    So I dont have to make one single blendshape, all joint drive and Random eyes would still work? I think I understand.

    Just installed the UMA one Click Solution and running into a problem. Crazy Minnow is not poping up under Game Objects. I also posted an image to show everything I have installed. Currently using 2018.3.4f1

    Thank you.

    upload_2019-4-22_14-5-58.png

    upload_2019-4-22_14-7-30.png

    Also running into this error

    Assets\Crazy Minnow Studio\SALSA with RandomEyes\Third Party Support\UMA 2\Editor\CM_UmaSyncEditor.cs(39,147): error CS0246: The type or namespace name 'UMADynamicAvatar' could not be found (are you missing a using directive or an assembly reference?)

    Assets\Crazy Minnow Studio\SALSA with RandomEyes\Third Party Support\UMA 2\Editor\CM_UmaSyncEditor.cs(39,49): error CS0246: The type or namespace name 'UMADynamicAvatar' could not be found (are you missing a using directive or an assembly reference?)
     
  9. Jaimi

    Jaimi

    Joined:
    Jan 10, 2009
    Posts:
    4,594
    Are you sure you imported the correct file? I just imported the Salsa/UMA integration and had no errors. I don't even see that file below in the current package - all of the current files in that folder start with "salsa", not CM.

    The file you need is the 1.7.0 version: UMA DCS 1-Click v1.7.0

    salsa.jpg

     
    Crazy-Minnow-Studio likes this.
  10. MarkFitzpatrick

    MarkFitzpatrick

    Joined:
    Feb 8, 2019
    Posts:
    21
    Jaimi, that did it! Thank you!
     
    Jaimi likes this.
  11. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,432
    @Crazy-Minnow-Studio @Jaimi I am trying the UMA setup per the video UMA 2 Lipsync using SALSA with RandomEyes , but see different results. When I select the UMA setup from the menu it looks as follows:

    upload_2019-4-27_12-24-10.png

    This looks different from what is in the video so I hope I have the right integration package. I downloaded uma-dcs_1_7_0

    After the avatar is created in the scene the video doesn't say to do anything else, but shows that the SALSA_UMA2 object will have just 2 scripts attached: CM_Uma Basic and CM_Uma Sync. However, mine has a bunch more:

    upload_2019-4-27_12-29-21.png

    And is missing the CM_Uma Basic.

    Then the video says to play the scene. And magically, the character is clothed, has hair, and is speaking with an audio file, but the audio file was never even added in the video. It seemed to already be there with the one-click setup.

    In any case, when I run mine at this point, I just get a static character without clothing and not eye/lip sync.

    upload_2019-4-27_12-32-35.png

    If I analyze the Salsa components, you can see it is having issues:

    upload_2019-4-27_12-33-41.png
    upload_2019-4-27_12-34-25.png

    So, what could I have possibly done wrong?
     
  12. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Hi magique,

    Sorry for the confusion. There are two UMA add-ons, one current and the other deprecated:
    • UMA 2 (Deprecated and is associated with the video you mentioned)
      • We posted a notice about our move to the DCS system on the add-on page. "11/26/2017 - v1.5 Final update. Since UMA has moved on to the DCS system, so have we."
    • UMA-DCS (Current, but text only tutorial)
    Please follow the written tutorial for the current UMA-DCS add-on and you shouldn't have any issues.
     
    Last edited: Apr 27, 2019
  13. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,432
    OK, so that part works. Now I have another issue, which is not specifically SALSA related, but maybe you have the answer. If I save the resulting avatar as a prefab using UMA Power Tools, the SALSA eyes and lip-syncing no longer work at all. Do you know what has to be done for SALSA to work with prefabs created using Power Tools?
     
    SickaGamer likes this.
  14. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Unfortunately I don't, I've never used UMA Power Tools. However, the run-time setup subscribes to the UMA CharacterCreated event to add and configure all the necessary components. I assume if your character already exists at design-time, then you should be able to perform all the steps that are normally done in the CharacterCreated event within the SalsaUmaSync.cs script.
     
  15. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,432
    Hmmm. I'll take a look and see.
     
  16. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    783
    Do salsa characters need blend shapes for the animation to work at runtime (from streamed audio)?
     
  17. therewillbebrad

    therewillbebrad

    Joined:
    Mar 2, 2018
    Posts:
    117
    Only if using 3D facial rig the 2D one doesn't need blend shapes and uses sprite key frame animation.
     
    Crazy-Minnow-Studio likes this.
  18. skinwalker

    skinwalker

    Joined:
    Apr 10, 2015
    Posts:
    254
    Hi,

    Im using daz3d characters but I dont have the head morphs for each letter and I cant export my characters again because Ive already set them up in my game. Is there a way to get this working without even having any blendshapes on my daz characters?
     
  19. Ony

    Ony

    Joined:
    Apr 26, 2009
    Posts:
    1,738
    I'll make some assumptions about your work flow here, so forgive if I'm way off...

    Assuming you have the character set up with morphs on it in another 3D program, so you can add the extra SALSA morphs to it, then re-export the FBX, then re-import it in Unity. Depending on what scripts you've added to the characters in the scene, you can re-import the new mesh with the extra blend shapes and it will add them to the character without disturbing anything else. Make backups first, of course (of the original FBX, etc.) before trying this.
     
    Crazy-Minnow-Studio likes this.
  20. SickaGamer

    SickaGamer

    Joined:
    Jan 15, 2018
    Posts:
    637
  21. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Actually...YES! We're rolling into our second round of private beta testing this weekend. The first round excluded the Eyes module because it wasn't ready, this round includes everything. If no major issue are discovered, we'll likely submit to the asset store in the next couple weeks.
     
    Ony, wetcircuit and SickaGamer like this.
  22. skinwalker

    skinwalker

    Joined:
    Apr 10, 2015
    Posts:
    254
    What if I dont have them can I use bones for the lipsync animation? And is it going to look good with bones for daz characters
     
  23. Ony

    Ony

    Joined:
    Apr 26, 2009
    Posts:
    1,738
    I don't think SALSA works with bones, so you'd have to roll your own bone-based solution (I did in the past). You might want to load your DAZ FBX into Blender and add some simple morph shapes, if you want to use SALSA with it.
     
    Last edited: May 18, 2019
    Crazy-Minnow-Studio likes this.
  24. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    You can also use SALSA as the engine, but create your own mapping to bones or whatever you like using available SALSA API. We do this in the SalsaSync add-on to map computed blendshape output to eye and jaw bones. Our upcoming SALSA 2.0 allows native mapping to a wide array of things such as bone, blendshapes, sprites, textures, materials, and can be extended to support others. The snippet below from the SALSA 1.x manual describes the API you can hook into.

    https://crazyminnowstudio.com/unity...uals/salsa-manual/classes-methods-properties/

    Public properties

    Salsa3D.say (string)

    String value that is always updated with the current shape (Rest, Small, Medium, Large). This is useful for asset integrators that want to use Salsa3D as a lip sync engine with their own integration.

    Salsa3D.sayIndex (int)

    Int value that is always updated with the current shape (0=Rest, 1=Small, 2=Medium, 3=Large). This is useful for asset integrators that want to use Salsa3D as a lip sync engine with their own integration.

    Salsa3D.sayAmount (SalsaBlendAmounts)

    • Salsa3D.sayAmount.saySmall (float)
    • Salsa3D.sayAmount.sayMedium (float)
    • Salsa3D.sayAmount.sayLarge (float)
    A custom type that contains the say BlendShape amounts. It's updated in realtime and provides direct access to the values even when no SkinnedMeshRenderer is linked. This is useful for asset integrators that want to use Salsa3D as a lip sync engine with their own integration.

    Salsa3D.isTalking (bool)

    Check this boolean value to determine when the audio file has stopped playing
     
    Last edited: May 18, 2019
    Ony likes this.
  25. Volkerku

    Volkerku

    Joined:
    Nov 23, 2016
    Posts:
    46
    Timeline Audio

    Great asset, particular with the Unity timeline support.

    Question: I would like to use a regular audio track on timeline and route its audio to the Salsa Lipsync component. Can't get this to work.
    The custom "Salsa audio track" from the Salsa timeline addon is a workaround, but feels a bit buggy (trimming does not work very well) and there is no waveform to work with.
    How can I route audio from a regular Unity timeline audio track to salsa lipsync?
     
  26. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Hello,
    The SALSA Audio Track/Clip solution was created because the Unity Timeline Audio Track does not route. The SALSA Timeline clips auto-adjust to the audio length by design, they are auto-trimming to the AudioClip. You are correct, there is no waveform display. I'm not sure it is possible to do so with the public API framework at this point.

    Please let us know if you have found bugs in the add-on and we will certainly take a look to see if they can be corrected. Additionally, the full source code is available for the SALSA timeline add-on, please feel free to modify it to better meet your needs if you wish.

    Thanks,
    Darrin
     
  27. Volkerku

    Volkerku

    Joined:
    Nov 23, 2016
    Posts:
    46
    Thanks
     
  28. Volkerku

    Volkerku

    Joined:
    Nov 23, 2016
    Posts:
    46
    Inspector Window scroll issue with Random Eyes 3D Component
    I'm stuck with the Inspector window not showing all UI elements of the Random Eyes 3D component.
    I just upgraded to 2019, in 2018 I had a horizontal scroll bar after adding custom shapes and groups to the component.
    Now I can't get to the add groups button, it is cut off by the window. If I stretch the inspector, those UI elements are still cut off.
    Any idea?
     
  29. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Hi Volkerku,

    We are probably two weeks or so away from submitting SALSA 2.0 to the asset store. 2.0 is a complete rewrite of the entire system, and improves quality and flexibility across the board. We recommend moving to 2.0 once it's available. For a very limited time, existing customers will be able to obtain 2.0 for the $4 different in price between the existing price ($35) and the initial new price ($39). See our SALSA 2.0 dev blog for more information.

    https://crazyminnowstudio.com/posts/salsa-v2-devblog/

    Michael
     
    TonyLi, wetcircuit and Firlefanz73 like this.
  30. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,024
    Wanna have :)
    Will the UMA Integration still work or will there be a new one?
    Will it be compatible or changes to do?

    thanks!
     
  31. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    SALSA v2 is a complete re-write and, while the underlying methodology and goals are the same, it is not a drop-in replacement for SALSA v1. SALSA v2 is vastly more flexible and powerful than SALSA v1 and with all of that flexibility, of course comes some complexity. So, requirements and setup are going to be somewhat different.

    We are still planning on UMA DCS integration and there will definitely be changes.

    Hope that helps!
    Darrin
     
  32. SickaGamer

    SickaGamer

    Joined:
    Jan 15, 2018
    Posts:
    637
    With UMA DCS, will it have the ability to work with avatars ported to prefabs from UMA PowerTools? @UnLogick @Jaimi
     
  33. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    SALSA 2.0 supports bones, blendshapes, sprites, textures, materials, and can be extended to support others types. I suspect a design-time UMA, created with PowerTools, should work like any other design-time character. However, we have not ticked the UMA box on our todo list yet so I'm speculating based on how other design-time characters work.
     
  34. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    361
    Waiting for Salsa V2
     
  35. nomax5

    nomax5

    Joined:
    Jan 27, 2011
    Posts:
    329
    Hi
    I’m using Salsa with lipsync and I want to emphasize talking more with additional blenshapes such as eye squinting or eyebrow lifting as the player speaks into the microphone.

    I need a way to detect when the player is speaking and when the player has stopped speaking,
    is there some way I can find that out from Salsa?

    Kind Regards

    Roy
     
  36. SickaGamer

    SickaGamer

    Joined:
    Jan 15, 2018
    Posts:
    637
  37. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
  38. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    SALSA v1 has a flag you can test (SALSA.isTalking). This will simply tell you if the AudioSource is currently playing, which could be playing and silent. You could also test the SALSA.average value to be greater than some arbitrary value to determine the level of audio. Or you could roll your own on the AudioSource itself. Or you could test to see if/which trigger is active. Etc. There are a lot of ways to achieve this.

    In SALSA v2, SALSA can link with EmoteR and has its own trigger threshold to fire emphasis emotes. In other words, this is built into SALSA v2. SALSA v2 is currently being wrapped up for submission to the Asset Store. We will notify everyone when that has happened, but we expect this week.
     
  39. therewillbebrad

    therewillbebrad

    Joined:
    Mar 2, 2018
    Posts:
    117
  40. innovationlover

    innovationlover

    Joined:
    Dec 25, 2018
    Posts:
    2
    Hey,

    Excited for Salsa V2, have been waiting for a long time but I know it's going to be worth to wait. So I'll continue to wait as long as it takes because we loved V1 and V2 should be no exception. Meanwhile, I got a quick question about two of your videos. First one is this (
    ) and the second one is this (
    )

    Now I am a newbie so pardon me if this is a silly question to ask but I want to know in both of your videos, the characters move slightly. The character in the first video, the entire body moves slightly like this: https://d.pr/free/i/Bv2ytc

    And the character in the second video, it actually turns (YAW) like this: https://d.pr/free/i/GNSEIr

    So I wanted to ask, is this randomized and a feature of Salsa? Or you did this using a third party app to add these effects to your chracter (move body + yaw turn)? If third party, then please let me know which tool you used because I have spent a lot of time just trying to achieve this same effect but no luck (I am a newbie as I said). :)

    I'll greatly appreciate your help for this,
    Thank you!
     
    Crazy-Minnow-Studio likes this.
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    Thanks for the kind words!!!

    OK, so for the Morphmixer video, Mike did that one and I believe it was just a simple idle animation (probably from the Unity Ethan model). The second video (the SALSA v2 teaser) was actually head tracking/movement built into the product. So you'll get that as part of v2 (for head movement -- that's as far down the bone chain as we go! ;) ).

    We really appreciate everyone being patient for the v2 release. It's a much bigger beast than SALSA v1 was and as such we're trying to make sure it's as rock solid as possible. With all of the new flexibility comes a lot of complexity and we're trying to keep the system as simple as possible to maintain its friendly nature.

    Hope that helps!
    Darrin
     
    atomicjoe and innovationlover like this.
  42. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    361
    Hey @Crazy-Minnow-Studio , can you tell me exactly which phonemes did you use for the second video? (the tech preview with the talking head)
    I have to move forward and finish the characters, so it would help me A LOT if you could tell me exactly the amount and WHICH phonemes did you use for this result. I know Salsa doesn't use phoneme recognition, but you obviously used different phoneme poses for that video, not just open-closed mouths.
     
  43. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    The shapes/amounts used will vary by model type and that was an MCS model (which we no longer support and is no longer in my project). Generally speaking and subject to change, at this point, I'm configuring the following (approximate) visemes in the trigger groups: (f, w, uw, t, ow, ee, th, ih).

    Hope that helps,
    Darrin
     
    atomicjoe and wetcircuit like this.
  44. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    361
    That was very helpful, thanks :)
     
    Crazy-Minnow-Studio likes this.
  45. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    store.main.png

    We are thrilled to announce that SALSA 2.0 has been submitted to the Asset Store for approval.
    We're working hard to get the rest of the manual, the add-ons, and new tutorials posted as soon as possible. Just a reminder, as soon as v2 is posted v1 will be deprecated. Don't want to upgrade to 2.0, no worries, v1 will still be available to existing customers but will not receive any new updates. If you want to upgrade, the upgrade price will be $4 for a limited time.
     
    Last edited: Jun 16, 2019
    atomicjoe, Deckard_89, TonyLi and 4 others like this.
  46. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,024
    Nice to hear! Will the upgrade still be $4 until the addons (for me personally UMA) are published or should I buy it as soon as possible?

    Thanks :)
     
    Crazy-Minnow-Studio likes this.
  47. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    361
  48. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    The $4 upgrade offer isn't linked to add-on availability, it's just our way of thanking our existing customers who want to get the upgrade right away. It will only be offered at $4 for a limited time since we have to eat as well. :)
     
    SickaGamer likes this.
  49. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    361
    What's the full price of Salsa v2 by the way?
     
  50. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    778
    The full purchase price will initially be $39.
     
    atomicjoe likes this.