Search Unity

  1. Looking for a job or to hire someone for a project? Check out the re-opened job forums.
    Dismiss Notice
  2. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice
  3. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. shelvin_g

    shelvin_g

    Joined:
    Dec 13, 2013
    Posts:
    5
    Good day, thank you for a great Asset. I've run into an issue. When I import the Adventure Creator v2.5.0 Add-on using Salsa v 2.5.2 (current in Asset Store) I get these errors (just going to post a few of them):

    Assets\Crazy Minnow Studio\Addons\AdventureCreator\Actions\Eyes_SetRandom.cs(108,8): error CS0103: The name 'eyes' does not exist in the current context

    Assets\Crazy Minnow Studio\Addons\AdventureCreator\Actions\Eyes_SetRandom.cs(130,21): error CS0103: The name 'eyes' does not exist in the current context

    Assets\Crazy Minnow Studio\Addons\AdventureCreator\Actions\Eyes_SetAffinity.cs(79,8): error CS0103: The name 'eyes' does not exist in the current context

    Assets\Crazy Minnow Studio\Addons\AdventureCreator\Actions\Emoter_Emote.cs(96,15): error CS0246: The type or namespace name 'ExpressionComponent' could not be found (are you missing a using directive or an assembly reference?)

    How can I fix this?
    Thanks in advance.
     
  2. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hi shelvin_g,

    It sounds like maybe you didn't add the define symbol as described in our AC add-on documentation.
    https://crazyminnowstudio.com/posts/latest-version-of-adventure-creator-supports-salsa/
     
    shelvin_g likes this.
  3. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    958
    I'm still not fully understanding why a universal oneclick cannot be created here. The ARKit 52 blendshapelocations involve mesh morphs that are universally defined to specific blend shape keys, similar to how a Daz or iClone character has its own specific set of blendshapes (?) ... is there something more standardized about the latter set of characters?
     
  4. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Again, it isn't that simple. The blendshapes are only part of the issue. The Eyes module would not be able to be configured since the bone names would be unknown to apply to the head and eyes sub-modules. So the Eyes module would have to be left out of the mix. The mesh names are also unknown, although this could possibly be overkill-mitigated and possibly error-prone with a search and configuration for all meshes with SkinnedMeshRenderers. And it would also be subject to everyone's interpretation of the "standardized" shape. As you might have noticed when you looked through our existing OneClicks when you were researching all of this, we don't always use full blendshape weight values. Many are partials to get a certain look. And this would be wildly variable based on the designer's interpretation of the shape. As it is, we suggest our OneClicks to be a starting point and not a final result depending on the model and voice characteristics. All of this completely ignores the idea of the designer possibly making mistakes with spelling and/or forgetting to put shapes on required meshes (like eyelashes, beards, etc.) and generating support requests that take additional time to resolve to ultimately not be an offical OneClick's issue.

    So the end result would be a partially-working OneClick that will require loads of adjustment based on the designer's interpretation of the blendshape names. Which ultimately will be a template that needs to be edited on a per-designer basis. Which leads us back to the original suggestion for the designer to create a relatively simple custom OneClick, for which we have already created documentation to achieve and there are several 'templates' (i.e. existing OneClicks) already available for editing.

    We would love nothing more than the ability to create a OneClick that worked for everything. At this point, that just isn't conceivable -- perhaps it will be at some point. We love the idea, we just cannot envision how this works at this point. Once you have created your shapes, and have configured a character with the Suite, and have attempted to make a custom OneClick, we are more than happy to assist with any issues you have with creating a custom OneClick for your custom character model. Let me know when you get to that point and I will be happy to assist if necessary.

    D.
     
  5. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    Hello! I have Salsa and RT-Voice set up on a Character Creator 3 import. I've added components Speaker, LiveSpeaker and Salsa RTVoice_Native to my Salsa-empowered CC3 model.

    If I test using the 'Speak' radio button in Salsa_RTVoice_Native it works! Audio and lipsync both work.

    If I send a string of text from Game Creator's CallMethod/CC3Model/.LiveSpeaker.SpeakNativeLive the string works in audio (I hear it) but the lips do not sync.

    Is it recommended to use Speech Text in this case?
     

    Attached Files:

  6. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hi XyrisKenn,

    I'm not familiar with Game Creator, but my first suggestion based on how I've seen some of these types of systems work would be to make sure that Game Creator isn't instantiating a one-shot audio source. If this is happening then SALSA would not be linked to it and would not be able to analyze the output audio.
     
    XyrisKenn likes this.
  7. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    Thank you, I'll start looking there. Great tech by the way. SALSA let's me work up a proof of concept in a very short time, and use it as the main attraction for a Watson-enabled demo. There's a lot of flexibility in setting up expressions and phonemes.
     
    Crazy-Minnow-Studio likes this.
  8. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hello!
    I'm pretty sure the problem is there is no id linkage to the returned audio when you are calling it from Game Creator. The RTV Uid needs to be set in the Salsa_RTVoice_Native script when calling "SpeakNative". If Salsa_RTVoice_Native doesn't have the Uid, it will bail on the the callback. This guard clause is to prevent the model from responding to all other model "speak" actions. The Salsa_RTVoice_Native script is meant to be an example for you to modify for your solution needs. The easiest solution would be to create a method in Salsa_RTVoice_Native that takes a string and makes the Speaker.Instance.SpeakNative call on it's own, similar to the way the LateUpdate bool trigger does it. Then remove the LateUpdate callback. Something like the following:

    Code (CSharp):
    1.         public void Speak(string speakString)
    2.         {
    3.             uid = Speaker.Instance.SpeakNative(speakString, Speaker.Instance.VoiceForGender(Crosstales.RTVoice.Model.Enum.Gender.MALE, "en"), 1.0f);
    4.         }
    Then you would call this function in your Game Creator action instead of calling the Speaker instance directly.

    Hope this helps!
    D.
     
  9. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    Hi!

    I compared the LiveSpeaker script with Salsa_RTVoice_Native and tried adapting the args from LiveSpeaker with the UID example - essentially to try and expose the test string input of Salsa_RTVoice_Native to external callmethods. It wasn't quite working; I'll start again with your example. Thank you!
     
  10. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    That did it! Thank you very much. :)
     
    Crazy-Minnow-Studio likes this.
  11. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    SALSA RT-Voice Integration
    v2.5.1 has been released and is available on the SALSA downloads portal.

    This version has a few changes to the RT-Voice Native Voice implementation example:
    • Clarified many of the comment areas to better explain the functionality.
    • Added a public Speak(string) method to increase the functionality and demonstrate posibilities for implementation questions such as the above few comments.
    • Fixed an index bug that caused a visual "pop" to occur at the end of sentences/paragraphs as if audio had a noise pop.
    • Added a 7-viseme configuration setup and selector to coincide with our standard OneClick viseme layout.
    2021-02-02_130839.png
    • Adjusted recommended SALSA configuration settings.
    Hopefully this script example is more useful to anyone implementing SALSA RT-Voice integration. Remember, please leverage this integration as an example of how to implement RT-Voice integration with SALSA. It is not intended to be a flexible API for all situations and will likely require modifications to meet your specific requirements.

    Enjoy!
    D.
     
    Last edited: Feb 2, 2021
    XyrisKenn likes this.
  12. shelvin_g

    shelvin_g

    Joined:
    Dec 13, 2013
    Posts:
    5
    Thanks! Yes I did do this however I didn't realise that my setting for the define symbol did not stick. For anyone else out there, once you add it in, you have to press ENTER -> Thanks to this post :) https://forum.unity.com/threads/cannot-set-scripting-define-symbols.501147/

    I'm sorted now, thank you.
     
    Crazy-Minnow-Studio likes this.
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    SALSA Dissonance Integration
    SalsaDissonanceLink v2.5.1 has been released and is available on the SALSA downloads portal.

    This release fixes a null-ref bug when combining SALSA, Dissonance, and Mirror. This is the only change in this release and if you are not experiencing this issue, you can bypass this update if you wish.

    D.
     
  14. chrisabranch

    chrisabranch

    Joined:
    Aug 15, 2014
    Posts:
    130
    Will it support Facial Action Coding System (FACS)? Daz3d Genesis 8.1 no Longer has Visemes, only FACS.
     
  15. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Likely, but we need to take a look at it first to determine how we do it.

    D.
     
  16. Sapien_

    Sapien_

    Joined:
    Mar 5, 2018
    Posts:
    59
    Hello again @Crazy-Minnow-Studio
    The asset has been very useful so far and has significantly cut dev time. However, I have come across a massive issue. Since I have applied the animator to the character for it to have animations, it mashes up the visemes. When I turn the animator component off, it's fine but obviously, that is not a solution. I am using the bone function for the visemes (due to the nature of the character). How do I fix this?
     
  17. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Not quite sure what you're describing. Please send us an email with screenshots, version numbers, invoice number, etc. and a video demonstrating what you are experiencing.

    Thanks,
    D.
     
  18. Sapien_

    Sapien_

    Joined:
    Mar 5, 2018
    Posts:
    59
    Hello I sent you an email to assetsupport@crazyminnow. The video should show what the issue is. The suite is version 2.3.2.112
     
  19. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    Hello, is it possible to assign a percentage of an emote to a 'rest' expression, and perhaps assign an affinity value?
    Use case are to have the model smile slightly in-between sentences rather than having a completely neutral expression.
     
    Crazy-Minnow-Studio likes this.
  20. Bomanden

    Bomanden

    Joined:
    Sep 7, 2014
    Posts:
    2
    Trying to stream audio from a socket to Salsa , using NAudio

    How do I get this to Salsa do i need to use the 'Use External Analysis' option, which params to send ?

    Code (CSharp):
    1. using (var wo = new WaveOutEvent())
    2. {
    3.  
    4.     WaveStream blockAlignedStream = new BlockAlignReductionStream(WaveFormatConversionStream.CreatePcmStream(new RawSourceWaveStream(ms, new WaveFormat(22050, 16, 1))));
    5.     wo.Init(blockAlignedStream);
    6.     wo.Play();
    7.  
    8.     while (wo.PlaybackState == PlaybackState.Playing)
    9.     {
    10.  
    11.         Debug.Log($"WaveOutEvent playing");
    12.         Thread.Sleep(50);
    13.     }
    14. }
     
  21. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hello, I have plans to implement the inverse of emphasizer emotes at some point. I am still tossing that in my head as to how to go about it. But, I have recently implemented a feature in the test branch, which is not exactly what you are looking for I don't believe, but it may help. All of the Suite modules, by design, reset all of their configured components to the OFF position at start, to ensure nothing was left on during the configuration process. Additionally, over time, each component resets to its min/OFF position during the course of operation. This can wipe out any pre-configured expression, or at least the parts of it that are common to any of the module configurations. The new feature (optionally) takes a snapshot of existing blendshapes and stores them persistently in the lowest-level QueueProcessor hierarchy, where they are maintained throughout operation or until programmatically turned off. There are API calls to capture, load, and unload this startup static expression.

    D.
     
    XyrisKenn likes this.
  22. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    If you have access to your stream data, you can override SALSA's internal data analysis mechanisms, providing it a place to grab your analysis data when it needs it. Please see the Delegate Processing documentation for details: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/delegate-processing/

    D.
     
  23. awsm1123

    awsm1123

    Joined:
    Mar 29, 2020
    Posts:
    6
    Hi, i am using salsa with random eyes but my model comes with visemes for vrchat so it has letter sounds like a,e,i,o, u and tt, ss but salsa is asking for small, medium and large mouth shapes, how do i configure that.
    upload_2021-2-18_23-19-21.png
     
  24. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    Greetings, I'm using RT-Voice.speaknative successfully with SALSA. In Unity editor RT-Voice is driving mouth shapes correctly.
    In the PC-standalone-development build, RT-Voice is heard and SALSA eye blinks are working but mouth shapes aren't activating.
    Might there be a compile command required for this sort of Windows standalone build?
    Thank you!
     
  25. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hello! Looks like you are using SALSA with RandomEyes v1.x. This is the legacy version and is no longer supported, but you are free to continue using it as long as you like. The legacy documentation is also available on our site. Generally speaking, you will want to select shapes that create the look and feel you desire, or you can use shapes resembling the visemes we recommended in our docs.

    D.
     
  26. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hello!
    There are not any special compile commands required on the SALSA side. I just created a test build for PC and it worked as expected. We would definitely need more information to assist if you cannot get it to work. If so, please send your pertinent details, *all* version numbers, Unity Invoice Number, details of your project setup, any errors your are receiving, etc.

    Thanks,
    D.
     
  27. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    I made a promotional video for my company, 1:54 you can see SALSA providing a great result for lip-sync and expression from text-RTVoice-SALA.

     
    Last edited: Mar 5, 2021
    Crazy-Minnow-Studio likes this.
  28. jnbbender

    jnbbender

    Joined:
    May 25, 2017
    Posts:
    402
    Before I go ahead and make a purchase I just want to ask one straightforward question. My model has a jaw bone, then simply transforms for lips, eyes, etc. Will SALSA work for me? (I know it won't have the quality of Blend Shapes)
     
  29. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    SALSA Suite works with bones/transforms. You can define your viseme components using the available bones to whatever degree of flexibility is built into your model. If you have sufficient transforms for lips, I don't see any reason you cannot achieve good quality SALSA lipsync.

    Hope that helps,
    D.
     
  30. Dajokn

    Dajokn

    Joined:
    Dec 24, 2018
    Posts:
    6
    I'm getting the following error using the one click for CC3.


    NullReferenceException: Object reference not set to an instance of an object
    CrazyMinnow.SALSA.Salsa.SalsaLssGetAudioClipSampleCount () (at <1382bb7a4ecf454e80eaf639bc8e4ca6>:0)
    CrazyMinnow.SALSA.SalsaAdvancedDynamicsSilenceAnalyzer.ProcessAudioStream () (at <1382bb7a4ecf454e80eaf639bc8e4ca6>:0)
    CrazyMinnow.SALSA.SalsaAdvancedDynamicsSilenceAnalyzer.LateUpdate () (at <1382bb7a4ecf454e80eaf639bc8e4ca6>:0)

    It has a valid audio source which is given a clip by a script, I can hear it playing but the lip sync doesn't do anything.

    The issue was using PlayOneShot() in the script, the audio clip has to be explicitly assigned to the audio source then played.
     
    Last edited: Mar 16, 2021
  31. Thordath

    Thordath

    Joined:
    Sep 7, 2017
    Posts:
    1
    I had an issue that, when I type my correct email and invoice number to verify and download addons - it shows me the error.
     
  32. skinwalker

    skinwalker

    Joined:
    Apr 10, 2015
    Posts:
    468
    The default genesis 8 female doesnt work with the one clicks for daz I get 14 warnings

    Removed expression w This may be OK, but may indicate a change in the model generator. If this is a supported model, contact Crazy Minnow Studio via assetsupport@crazyminnow.com.

    for different expressions and finally one null pointer error

    NullReferenceException: Object reference not set to an instance of an object
    CrazyMinnow.SALSA.OneClicks.OneClickDAZEyes.Setup (UnityEngine.GameObject go) (at Assets/Plugins/Crazy Minnow Studio/SALSA LipSync/Plugins/OneClickRuntimes/OneClickDAZEyes.cs:210)
    CrazyMinnow.SALSA.OneClicks.OneClickDazEditor.ApplyOneClick (UnityEngine.GameObject go) (at Assets/Plugins/Crazy Minnow Studio/SALSA LipSync/Editor/OneClicks/OneClickDazEditor.cs:76)
    CrazyMinnow.SALSA.OneClicks.OneClickDazEditor.OneClickSetup () (at Assets/Plugins/Crazy Minnow Studio/SALSA LipSync/Editor/OneClicks/OneClickDazEditor.cs:66)
     
  33. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Send us the error in an email.

    Thanks,
    D.
     
  34. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hello,
    Sounds like one of two things have happened. 1) Ensure your Gen 8 (not Gen 8.1) model has blendshapes on the appropriate mesh objects. If you don't have blendshapes, make sure you export the model with the appropriate morphs. Or, 2) You might be using a Gen 8.1 model which is completely different and not yet supported in the OneClick. DAZ changed the way they do morphs in the Gen 8.1 models and the OneClick doesn't find the blendshapes it is looking for.

    Thanks,
    D.
     
  35. skinwalker

    skinwalker

    Joined:
    Apr 10, 2015
    Posts:
    468
    Im using genesis 8 female not 8.1

     
  36. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Looks like you are using the DAZ to Unity Bridge? If so, make sure you are using the latest DAZ OneClick (currently 2.5.2). Bridge started changing names on the blendshapes.

    D.
     
    skinwalker likes this.
  37. masai2k

    masai2k

    Joined:
    Mar 2, 2013
    Posts:
    20
    Hi, what about the support of Genesis 8.1 version from Daz 3D?
    Do you think to implement it? And when?

    Thanks
    Massimo
     
  38. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    It is currently in the works along with several other things. DAZ completely changed their morph sets as well as morph and mesh naming. We don't have a timeframe for completion at the moment, but it will be supported.

    D.
     
  39. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    958
    I think it's easy enough to have the One-Click come with directions on what to name these standardized 52 parts and the 2 eyes? Or, just have them named from the Apple default names?
     
  40. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    Hi! I'm trying to trace down an error loop in Unity 2019.4.21f1 with Salsa Current version and CC3 one-click. At runtime this error is appearing several times a second. I've looked at the Salsa setup on my character and nothing has changed.
    Might you recognize this error?

    NullReferenceException: Object reference not set to an instance of an object
    CrazyMinnow.SALSA.SalsaAdvancedDynamicsSilenceAnalyzer.LateUpdate () (at <1382bb7a4ecf454e80eaf639bc8e4ca6>:0)
     
    Last edited: Apr 9, 2021
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    Hello! Without knowing more about your usage scenario, that error would generally indicate the SilenceAnalyzer is trying to reference a SALSA instance that no longer exists. It could also mean Use External Analysis was enabled after the startup mechanisms have run.

    D.
     
  42. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    38
    Hi, Thank you, I'll start there. This is a Photon-enabled scenario so avatars are instantiated after login. There are RPC commands as well for standing and sitting.
     
  43. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,126
    If you are using the lastest version of SALSA, SilenceAnalyzer should disable itself; however, it serves no purpose when using external analysis, so you can just remove it from your prefab configuration. Bear in mind, if you reapply a OneClick, SilenceAnalyzer will be reapplied and will need to be removed again. Or you can simply edit your OneClick to *not* apply the SilenceAnalyzer.

    Hope that helps,
    D.
     
unityunity