Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Object-Null

    Object-Null

    Joined:
    Feb 13, 2014
    Posts:
    70
    thanks again!
    ill dive into it and ill send you then the details over email.
     
  2. EugenC

    EugenC

    Joined:
    Feb 5, 2013
    Posts:
    1
    Hello,

    I'm trying to play an MP4 video file via the Unity VideoPlayer (version 2019.4.19 on PC) with the AudioOutputMode set to an AudioSource. That AudioSource is set as input AudioSource for Salsa. Seems like a simple setup, but I can't make it work. The same setup works fine with input from the microphone. I did search the forum, but I didn't see anyone else having this issue. Can you please provide some help on how to setup Salsa to lipsync from a VideoPlayer.

    Thank you in advance!
     
  3. sacb0y

    sacb0y

    Joined:
    May 9, 2016
    Posts:
    867
    If you accidentally reduce the index of the animator parameter selected to outside the index of the possible animators, it breaks the whole UI

    upload_2021-4-25_10-39-7.png
     
  4. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    I would suspect this is due to the manner in which the VideoPlayer uses the AudioSource. In my tests there is no available AudioClip on the source while the video is playing, meaning the Unity VideoPlayer is injecting the audio data further up the FMOD pipeline and avoiding the AudioClip buffer. SALSA uses the AudioClip buffer on an AudioSource to perform audio analysis. This all suggests you have a couple of choices. 1) extract the audio from your video and programmatically time and play it along with your video. 2) analyze the audio yourself and make the analysis available to SALSA's analysis delegate (this is preferred over manually force-feeding analysis to SALSA). The easiest would be to extract the audio and play it alongside your video.

    Hope that helps,
    D.
     
  5. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hi @sacb0y,
    Please send us an email with your Invoice number and steps to produce this issue. I have a new Suite build I'm working on and would like to get a fix into it if possible.

    Thanks,
    D.
     
  6. sacb0y

    sacb0y

    Joined:
    May 9, 2016
    Posts:
    867
    While I'm working on my own solutions, I would love better animator support for variances.

    For the moment, one implementation I've been trying is making each node a blend tree controlled by the processed value of viseme. It looks like this...

    upload_2021-4-26_12-1-50.png

    First part is standard triggers.

    Then each node has a blend tree with an adjusted threshold so the pose isn't too subtle.
    upload_2021-4-26_12-4-19.png

    here "w" the lowest value has a threshold of .2, because the processed value is so low. These mouth expressions were made previously not specifically for salsa but I'm intending to switch to this for most cases.

    The result plays out pretty well!

    EDIT: NOW with sound lol

    It basically helps it look more like speaking instead of switching between static poses with no consideration to how things are said.

    I plan on working on other methods, like adding a second animation layer, and maybe messing with layer weight. Only thing i'm not sure about is if "jitter" or dynamic values are accessible.

    The Code:
    Code (CSharp):
    1. using UnityEngine;
    2. using CrazyMinnow.SALSA;
    3.  
    4. public class VisemeIntensity : MonoBehaviour
    5. {
    6.     // Link the required SALSA instance for identification in the event callback.
    7.     [SerializeField] private Salsa salsaInstance;
    8.     public Animator NPCanimator;
    9.     public string intensityFloat = "vIntensity";
    10.     private void OnEnable()
    11.     {
    12.         // subscribe to the event
    13.         Salsa.VisemeTriggered += SalsaOnVisemeTriggered;
    14.     }
    15.  
    16.     private void OnDisable()
    17.     {
    18.         // unsubscribe from the event
    19.         Salsa.VisemeTriggered -= SalsaOnVisemeTriggered;
    20.     }
    21.  
    22.     private void SalsaOnVisemeTriggered(object sender, Salsa.SalsaNotificationArgs e)
    23.     {
    24.         NPCanimator.SetFloat( intensityFloat, e.salsaInstance.CachedAnalysisValue);
    25.     }
    26. }
    I'll send an email later today if i can but the process is simple.

    Add an animator component.
    Choose a parameter at the bottom of the list, delete any parameter above it (which reduces the index range of the list) and see the component break and throw an error.

    It simply needs a proper catch for the index being out of range.
     
    Last edited: Apr 26, 2021
    Crazy-Minnow-Studio likes this.
  7. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    OK, I figured that was what you were experiencing and I've located and fixed it for the next build. The existing checks were confirming the parameter list could be produced, but not confirming the chosen parameter was valid frame over frame beyond the initial selection of a valid parameter index. I've implemented an out of bounds check and set the animator to null to warn the dev the component is no longer valid.

    Your demo video looks pretty fantastic! Great job.

    OK, so if I'm following you, you are trying to read the level of animation fired. The audio analysis value is irrelevant to you at this point. Have you considered using a float parm in your animator to drive it? SALSA/EmoteR animate the float parm directly based on the calculations. The dynamics are calculated on the fly and are currently not publicly visible. Doesn't mean the value can't be made visible, it just is not currently. Effectively, if you used the Animator float parm, you would get the live calculated value over time, not the peak value, but the current normalized value based on QueueProcessor control.

    Additionally, I am working on a generic FloatController which would provide normalized float animation unrelated to the Animator and that may also work for your implementation. Send me an email with more details on what you are needing and we will see what we can do or if there is existing capabilities that you could leverage.

    Thanks,
    D.
     
  8. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    Hello, everyone! I'm facing the following problem: I'm building a virtual meeting that allow two or more users to join a room and talk to each other with microphone. Each user in the meeting is an UMA avatar. Now I wonder: is there a way to take the voice of a user and make sure that the his UMA will be "synchronized" with that voice through SALSA LipSync?
     
  9. AxonGenesis

    AxonGenesis

    Joined:
    Jun 24, 2016
    Posts:
    84
    I am assisting @EugenC with this. I followed your recommendations and have it working, however the quality of the lip sync is not as good as the microphone input. Basically, the mouth is remaining partially open and the lip sync is happening in short bursts. I tried reducing the Update Delay to 0.01 and the Event Silence Threshold to 1 with only slight improvement.

    Could you advise on how to get better results?

    The following script is what I'm using to capture the audio from the VideoPlayer output. Salsa is configured 'Using External Analysis' with the microphone input off.

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using Unity.Collections;
    5. using CrazyMinnow.SALSA;
    6. using UnityEngine.Experimental.Video;
    7. using UnityEngine.Experimental.Audio;
    8. using UnityEngine.Video;
    9.  
    10. public class SalsaAudioFromVideo : MonoBehaviour
    11. {
    12.     public AudioSource audioSource = null;
    13.     public Salsa salsaObject = null;
    14.  
    15.     private float audioAnalysis = 0f;
    16.  
    17.     void Awake()
    18.     {
    19.         salsaObject = gameObject.GetComponent<Salsa>();
    20.         audioSource = gameObject.GetComponent<AudioSource>();
    21.  
    22.         var samplesLength = AudioSettings.outputSampleRate;
    23.         audioSource.clip = AudioClip.Create("", samplesLength, 1, AudioSettings.outputSampleRate, false, null);
    24.  
    25.         if (salsaObject != null) {
    26.             salsaObject.getExternalAnalysis += GetExternalAnalysis;
    27.         }
    28.     }
    29.  
    30.     private void OnDestroy()
    31.     {
    32.         if (salsaObject != null) {
    33.             salsaObject.getExternalAnalysis -= GetExternalAnalysis;
    34.         }
    35.     }
    36.    
    37.     private void OnAudioFilterRead(float[] data, int channels)
    38.     {
    39.         if (salsaObject != null) {
    40.             if (salsaObject.audioAnalyzer == null) {
    41.                 return;
    42.             }
    43.             audioAnalysis = salsaObject.audioAnalyzer(channels, data);
    44.             if (audioAnalysis > 0f) {
    45.                 Debug.Log("OnAudioFilterRead:" + audioAnalysis);
    46.             }
    47.         }
    48.     }
    49.  
    50.     public float GetExternalAnalysis()
    51.     {
    52.         return audioAnalysis;
    53.     }
    54. }
    55.  
    Thanks for your assistance.
     
  10. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    You would need to use something like Dissonance or PhotonVoice to serialize the data to a source.

    D.
     
  11. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    Hello and thanks for the reply! Actually I already did the entire system: the users can correctly create and join a room and they can speak to each other with no problems. But since I used UMA to make sure that users have their own avatar in the meeting, I want to also have the lip sync in the program.

    How can I accomplish that? Note that I didn't use PUN for networking (since you suggested to use PhotonVoice).
     
  12. sacb0y

    sacb0y

    Joined:
    May 9, 2016
    Posts:
    867
    upload_2021-4-26_16-32-22.png

    you mean like this?

    EDIT: This method makes the mouth jitter a lot, looking at the blend tree it's constantly jittering back and forth. It's either too fast processing, or constantly jumping between zero?

    Also! Another bug.

    If you have the component on a prefab, then add a component to viseme it seems like it just dumps the information not in the prefab when you hit play. When i added the intensity above it kept erasing the information for it on play, only after when i updated the prefab did it work.

    EDIT: One more bug.

    It seems like when in a prefab, disabling a viseme component does not work. it must be removed entirely.
     
    Last edited: Apr 26, 2021
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, you are close, but there are a couple of things of note and you'll need to make a decision on how to handle the scenario. What you are experiencing is SALSA looking for external analysis which is trying to return a value based on SALSA analyzing variable data chunks when it thinks it should be getting specific chunk sizes. Additionally, SALSA's polling cycle is likely not matching up with the filter read callback.

    You typically want SALSA to pulse about every 0.08-0.12 seconds for the best animation runs. Check out the video tutorials on Tuning SALSA for more information. In this situation, OnAudioFilterRead is likely being called much more frequently and potentially with smaller (or larger) data sizes than are expected. SALSA does not need to analyze every sample. It typically will grab a chunk of data near the time frame of its pulse (the audio analysis delay) and summarize the data into a peak value, a trigger is calculated and the QueueProcessor receives a registration from SALSA defining the viseme it wants to fire.

    So, what you might want to do instead, let your OnAudioFilterRead callback continuously fill your own circular buffer of size Salsa.sampleSize (or whatever size you want). When SALSA polls the GetExternalAnalysis delegate, return your own analysis of the circular buffer. Also, don't add your GetExternalAnalysis method to the SALSA delegate, just assign SALSA's delegate to your method (i.e. Salsa.getExternalAnalysis = GetExternalAnalysis). If multiple methods are called it will accept the value from the last method called in the chain which could be ambiguous. As a side note, we've found that the peak value of the sample set works better than averaging the data since wave forms oscillate and averaging samples (usually) produces a much more muted expression. And as well as the expressive benefits, it is more performant to simply find the peak. SALSA expects normalized data, so simply negating the negative values and checking for the highest peak should be sufficient analysis to let SALSA handle the rest.

    In this manner your audio analysis process will be waiting for SALSA to get it when it wants it and timing shouldn't be an issue. Also, ensure you've configured SALSA for external analysis by removing the linked AudioSource and enabling External Analysis.

    Oh, and the Silence Threshold has no bearing on SALSA analysis values, it is only used to adjust the detection of "silence" to determine if SALSA is SALSAing or not.

    Hope that helps,
    D.
     
  14. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Yes, there is a known issue with prefab instance changes that we recently became aware of and believe we have addressed for the next build.

    As far as the float parameter, it is not going to be a static value, it will be animating based on the dynamics calculation as the peak. So its use would be more of a driver for the animation. We have had a customer using this method to fully control the peak and timing of the animation, reading the float parameter and converting it to the animation. It may not work in your situation or likely would require some updates to work.

    D.
     
  15. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Have you already checked out our PhotonVoice2 demo documentation? https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/photon-voice/

    It should hopefully get you along the right path.

    D.
     
    Last edited: Apr 27, 2021
  16. AxonGenesis

    AxonGenesis

    Joined:
    Jun 24, 2016
    Posts:
    84
    Thanks for the additional help! I implemented the changes suggested and set the Update Delay back to 0.08. I also added a multiplier to boost or decrease the audio level. Results are looking much better. And no need to store a buffer since it's just a matter of getting the peak value in the audio data between calls to GetExternalAnalysis.

    In case it helps anyone, here is the updated code:

    Code (CSharp):
    1.  
    2. using UnityEngine;
    3. using CrazyMinnow.SALSA;
    4.  
    5. public class SalsaAudioFromVideo : MonoBehaviour
    6. {
    7.     public AudioSource audioSource = null;
    8.     public Salsa salsaObject = null;
    9.  
    10.     [Tooltip("Collects the peak value sampled over pulse time")]
    11.     public bool useCustomAnalysis = true;
    12.  
    13.     [Tooltip("Minimum audio value required to register a response")]
    14.     public float threshold = 0.01f;
    15.  
    16.     [Tooltip("Multiplier to increase/decrease amplitude of audio. Default = 1")]
    17.     public float intensity = 1f;
    18.  
    19.     private float audioAnalysis = 0f;
    20.  
    21.     void Awake()
    22.     {
    23.         salsaObject = gameObject.GetComponent<Salsa>();
    24.         audioSource = gameObject.GetComponent<AudioSource>();
    25.  
    26.         var samplesLength = AudioSettings.outputSampleRate;
    27.         audioSource.clip = AudioClip.Create("", samplesLength, 1, AudioSettings.outputSampleRate, false, null);
    28.  
    29.         if (salsaObject != null) {
    30.             salsaObject.getExternalAnalysis = GetExternalAnalysis;
    31.         }
    32.     }
    33.  
    34.     private void OnDestroy()
    35.     {
    36.         if (salsaObject != null) {
    37.             salsaObject.getExternalAnalysis = null;
    38.         }
    39.     }
    40.  
    41.     private void OnAudioFilterRead(float[] data, int channels)
    42.     {
    43.         if (salsaObject != null) {
    44.             if (salsaObject.audioAnalyzer == null) {
    45.                 return;
    46.             }
    47.             if (useCustomAnalysis) {
    48.                 // Collect the peak value of the data
    49.                 foreach (float v in data) {
    50.                     float val = Mathf.Abs(v);
    51.                     if (val > threshold) {
    52.                         val *= intensity;
    53.                         if (audioAnalysis < val) {
    54.                             audioAnalysis = val;
    55.                         }
    56.                     }
    57.                 }
    58.             }
    59.             else {
    60.                 audioAnalysis = salsaObject.audioAnalyzer(channels, data);
    61.             }
    62.  
    63.             if (Mathf.Abs(audioAnalysis) < threshold) {
    64.                 // Force to 0 if below the threshold
    65.                 audioAnalysis = 0f;
    66.             }
    67.             //Debug.Log("OnAudioFilterRead:" + audioAnalysis);
    68.         }
    69.     }
    70.  
    71.     public float GetExternalAnalysis()
    72.     {
    73.         float val = Mathf.Min(1f, audioAnalysis);
    74.  
    75.         // Reset to collect data until next get
    76.         audioAnalysis = float.MinValue;
    77.  
    78.         return val;
    79.     }
    80. }
    81.  
     
    Crazy-Minnow-Studio likes this.
  17. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Awesome, glad you got results you are liking. There are some things that you might consider in your implementation. It is likely not beneficial to attach to the AudioSource or to create an AudioClip. Creating an AudioClip (potentially) can alter the processing/creation of data in the filter pipeline since it is considered the first data source. Since you are creating a clip and not doing anything with it, it possibly isn't an issue.

    Also FYI, when attaching to upstream audio filter data, it is already process by the AudioSource and will be subject to the effects of 3D positioning and audio volume. This is why we switched to AudioClip analysis, it is raw data unaffected by processing. But it definitely can be used if you prefer not to extract the audio from the video and play it along side the video. You may need to additionally process the data, adjust the AudioSource settings (such as the volume), or you can also use the built in scaling and noise floor adjustments within SALSA on externally provided analysis data.

    Here is an example I just put together of the implementation I discussed in the previous post:
    • Place this on the same object as the AudioSource mapped on the VideoPlayer.
    • Ensure the AudioSource is *not* set to Play on Awake.
    • You will need to manually link SALSA if it is not on the same object.
    • This includes a simple analyzer, but you could also leverage SALSA's analyzer in the GetExternalAnalysis() method.
    Code (CSharp):
    1. using UnityEngine;
    2. using CrazyMinnow.SALSA;
    3.  
    4. namespace DemoCode
    5. {
    6.    public class UpstreamAudioFilterProcessing : MonoBehaviour
    7.    {
    8.       public Salsa salsaInstance;
    9.       private float[] analysisBuffer = new float[1024];
    10.       private int bufferPointer = 0;
    11.       private int interleave = 1;
    12.  
    13.       private void Awake()
    14.       {
    15.          salsaInstance = GetComponent<Salsa>();
    16.          if (salsaInstance)
    17.             salsaInstance.getExternalAnalysis = GetAnalysisValue;
    18.          else
    19.             Debug.Log("SALSA not found...");
    20.       }
    21.  
    22.       private void OnAudioFilterRead(float[] data, int channels)
    23.       {
    24.          // simply fill our buffer and keep it updated for ad-hoc analysis processing.
    25.          interleave = channels;
    26.          for (int i = 0; i < data.Length; i+=channels)
    27.          {
    28.             analysisBuffer[bufferPointer] = data[i];
    29.             bufferPointer++;
    30.             bufferPointer %= analysisBuffer.Length; // wrap the pointer if necessary
    31.          }
    32.       }
    33.  
    34.       float GetAnalysisValue()
    35.       {
    36.          var analysisValue = 0.0f;
    37.          for (int i = 0; i < analysisBuffer.Length; i += interleave)
    38.          {
    39.             var bufferDataAbsVal = analysisBuffer[i] < 0.0f ? analysisBuffer[i] * -1 : analysisBuffer[i];
    40.             if (bufferDataAbsVal > analysisValue)
    41.                analysisValue = bufferDataAbsVal;
    42.          }
    43.  
    44.          return analysisValue;
    45.       }
    46.    }
    47. }
    Hope that helps explain what I was discussing earlier. An additional benefit is analysis is only performed when SALSA requests it and the data analysis window size is controlled.
    D.
     
    Last edited: Apr 28, 2021
  18. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    Yes, I checked it out, but since I used WebRTC and SocketIO for all the networking (voice chats with microphones included), I wonder if is there another way to "syncronize" the UMA avatars with SALSA.
    I don't know if PhotonVoice2 is strictly necessary to accomplish that or not.

    Thanks a lot for your precious help!
     
  19. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    If you already have your network sync implemented and working, you are more likely to find answers in the threads for those technologies (unless another customer has information that can help). Sorry, you are outside of our wheelhouse for assistance.

    D.
     
  20. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    Actually I still have to sync avatars talks with SALSA. Maybe I wasn't clear and I apologize for that. Let me explain the situation that I have:

    So, I have the prefab PlayerAvatar as shown in the picture I attached to this message. It is just an UMA avatar (the one that represents the single user in the chat). Now, this prefab has a child game object called AudioReceiver that is used to transmit the voice that the user (identified by that UMA) sends from the other side.
    Now, what I want is to do make SALSA synchronize the UMA using AudioReceiver as the UMA Audio Source.

    What I've done so far is to import SALSA suite, OneClick UMA and OneClick Base, then I added UMA DCS component to PlayerAvatar, via GameObject -> CrazyMinnowStudio -> Salsa LipSync -> One-Clicks -> UMA DCS.

    Note that AudioReceiver is an empty gameobject that has three child components, including an AudioSource component that make other users hear what that user say.

    Thanks a lot for your patience!
     
  21. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    Here's the screenshot. For some reason, I couldn't attach it in the previous message.
     

    Attached Files:

  22. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Ah, OK, well the process will be very similar to the PhotonVoice and Dissonance setups. You'll want to connect that AudioSource to the SALSA AudioSource reference slot. Just as with PhotonVoice and Dissonance, you will likely only get remote avatar lip-sync and if you want local avatar lipsync, you will need to implement a method to intercept the local audio in a manner that SALSA can process. Usually this is done by detecting the avatar as local, re-configuring SALSA for external analysis, and delgating SALSA to your external analysis processor for local lipsync. This prevents the need to pipe local audio data to an AudioSource for SALSA to process and subsequently creating a need for AudioSource mixer shenanigans.

    The documentation examples for Dissonance and PhotonVoice setup are still very much valid for your situation. And the add-ons for those products may help as examples as well.

    Hope that helps!
    D.
     
  23. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    I only need to get remote avatar lip-sync. Following the documentation you provided, after adding SALSA LipSync -> UMA DCS to PlayerAvatar, in SALSA component -> References -> Audio Source, I dragged the AudioReceiver game object to it, but I didn't get the remote avatar lip-sync.
    What am I doing wrong? :(
     
  24. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    First, does your UMA character work with a normal AudioSource with an AudioClip (like the promo clip included with SALSA)?

    If the answer to the above is 'yes', the next question: is there AudioClip data present on the AudioSource?

    If the answer is 'no', you will likely need to implement an external analysis processor similar to the above conversation using OnAudioFilterRead(). Or if you have access to the raw data prior to it entering the DSP chain, you can process that data.

    D.
     
  25. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    Yes, it does! The UMA works fine with a clip (I use promo-male for testing purposes). Regarding the second question, there are no clips, neither in the UMA, neither in the child component (See the images attached).
    And how can I implement an external analysis processor? I'm not sure if I've access to raw data.
    But still is there any change my configuration is wrong? Because it's very strange that it doesn't work...
     

    Attached Files:

  26. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    OK good, your UMA appears to be configured appropriately. As for the AudioClip question, let me clarify, when you are playing your scene, do you see an AudioClip reference in the AudioSource or does it still say 'none'. If it says 'none', the AudioReceiver must be inserting data via a filter, into the AudioSource instead of into an AudioClip container on the AudioSource. That's OK if it is, but it will require more work on your part to implement into SALSA. You will need to add an external analysis processor into the chain similar to the discussion above to see if you can get to the data, process it, and give it to SALSA.

    D.
     
  27. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    OK, the Audio Clip slot (in play mode) in the Audio Source says "None", both for the UMA and the Audio Receiver slots! So this means that I need to implement that filter, right? If so, can you explain me how to do that? Because I'm very new to SALSA and also to Unity in general.

    Thanks a lot for your help! I really appreciate it!

    UPDATE:

    I've read a little bit the previous posts and this is what I've done: I've attached the script you sent (UpstreamAudioFilterProcessing) to AudioReceiver and then I've dragged the SALSA component in PlayerAvatar on the one in the script (public Salsa salsaInstance, visible in the inspector). Then I've removed the linked AudioSource in the SALSA component in PlayerAvatar and next I've enabled "External Analysis", but still the lip sync doesn't work.
     
    Last edited: Apr 28, 2021
  28. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Great! Sounds like you've done it all correctly. One last thing I can thing to check. Ensure your AudioSource is *not* set to Play on Awake. This seems to throw off the filter. If the filter is properly inserted into the DSP chain, when you play, there will be a VU meter of sorts on the component. If there is a VU meter, but it's not displaying anything, it's not seeing data. If that's the case, it will be necessary to move over to your Network provider's support forum and troubleshoot/research there.

    D.
     
  29. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    At long last, it works! The problem was that the SALSA component reference inside the script wasn't correctly loaded in play mode and after fixing it, now the problem is solved!

    Thanks a lot again, @Crazy-Minnow-Studio! Your help was precious (just like your Amazing SALSA!).

    Cheers!
     
    Crazy-Minnow-Studio likes this.
  30. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,105
    This is a great tool, I love it, brings life to my characters.

    But I have one issue

    I'm using this only for the head, eyes, blink and emotes.

    One issue I have is that random emotes that control eyelids bones used for the blink setup mess up the blink cycle. The eyelids sometimes remain closed till next blink cycle happens.

    None of the random emotes force the eyes on the close position, they are setup to give a bit of movement to the eyelids, but not close them

    I was thinking what could be and I think somehow, the blink out animation doesn't fire sometimes and the eyes remain closed till the next close, keep, open cycle starts which may be a few seconds.

    What could be?
     
  31. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hi @dorusoftware please send us an email with more detail on your project, including your invoice number and a video of the issue you are experiencing. In the video, please show us your configurations expanded and include any pertinent narration.

    Thanks,
    Darrin
     
  32. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,105
    mail sent, thank you, would be great to get to the bottom of this
     
  33. Orbatrix

    Orbatrix

    Joined:
    Mar 16, 2021
    Posts:
    26
    Hello, it's me again! In the past days I've done some tests and I have noticed that the quality of the lip sync is not as good as the microphone input: in fact the words I say are not correctly "spelled" with lips by the UMA. What can I do to improve the lip sync? Maybe change some settings?
     
  34. TapCrush

    TapCrush

    Joined:
    Dec 15, 2020
    Posts:
    4
    Hey Guys
    We are unable to apply our Viseme settings to the prefab characters we are working on. When we do; the bone links are reset to 'None' and everything in the Viseme section goes yellow. So all that data is gone.

    We are able, however, to save the viseme settings to the scene file. Which works but this is bad practice.

    We tried modifying the Viseme settings in Prefab-mode but in Adjust-mode; the Transforms are grayed out. Unable to edit Visemes in Prefab-mode.

    *We had only one seat and were getting the results above.
    *We have 2 seats licenced now (only two of us working on it) and we still get this result.

    Are we missing something? Have you encountered this before? Any help is appreciated!

    p.s. using Undo (ctrl+z) will also remove all references to bones in the current Viseme that is being edited.

    Edit: Nevermind, Ya'll don't support Unity 2020/2021 yet! :3
     
    Last edited: May 10, 2021
  35. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,314
    Hello,

    I would like to use salsa with a photo. I already have a canvas and panel with the desired photo on it.
    I tried to follow the workflow like in the 2d additional sample, but I cannot see my sprite renderer...

    Is it because it is child of canvas? So can I get it working for on a canvas / panel?
    Using Unity 2021.1.6...

    Thanks a lot!

    PS: Got it working with an image and ugui sprites. Solved!
     
    Last edited: May 11, 2021
  36. jrbourne

    jrbourne

    Joined:
    May 8, 2015
    Posts:
    8
    Hello:

    I upgraded from Catalina to Big Sur on my macbook pro and salsa audio quit working. The box head demo still has all the proper movements, eye blinks, etc. on the box head demo except the audio is missing on the Big Sur implementation. I understand by talking to the Minnow Support Team that other users have had this same problem with audio disappearing for Salsa on the Mac with Big Sur. If any of you have found a solution, I would greatly appreciate hearing how you solved the problem.

    MacBook Pro, 2020
    Unity (most recent as of May, 2021)
    SALSA (most recent download)

    Thanks,

    Jrbourne
     
  37. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    As an update for other customers, the scenario that was being used (an emote configured with the same lids blendshape used in the Eyes blink sub-module) required persistence mode to be enabled in the emote configuration. The conflict occurred when both animations triggered an OFF animation at (nearly) the same time. Due to the timing, the conflict resolution algorithm matched with the auto-mergeback feature couldn't determine a 'winner' and assumed they were both finished. Enabling persistence on the emote (the lower hierarchy queue) forces the emote to remain in the animation queue and allows the blink to operate and merge back to the configured emote.

    Hopefully that makes sense to everyone. :)
    D.
     
    Last edited: May 14, 2021
    jorgeolothar likes this.
  38. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Sorry everyone, we were not getting notifications that all of these questions were being posted.

    Yes, this is a known issue and will be addressed in the next update, which should be coming out pretty soon. I have no idea why this issue wasn't brought up a long time ago, but it typically results when more than one change is made to a prefab instance. It seems the first change (typically) takes effect, but subsequent changes do not. And perhaps that is why it wasn't discovered in the past. Anyway, we have a fix and as mentioned, it will be available in the next release. Sorry for the inconvenience.

    Thanks,
    D.
     
  39. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello @Orbatrix are you squared away or still having problems? I know we've been chatting on a side-channel, but that was about other issues. Remember, SALSA is an approximation system and does not map Pixar-style lip-sync. So you won't see perfect phoneme mapping. Microphone, by nature also introduces a slight amount of latency, but in general, lip-synchronization should be timed pretty closely to your micinput. However, if your input buffer (which is a circular buffer for Unity's Microphone class) gets out of sync, timing can be extremely off for lipsync.

    The input buffer pointer can get itself very lost in some conditions, for instance when the Unity Player loses and then regains focus. You can alleviate most of this by simply setting your Player settings to operate in the background -- that way they don't pause when focus is lost. You can also implement some code to Stop/StartMicrophone() which will reset the pointers.

    Hope that helps (if you still needed it) ;)
    D.
     
  40. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    OK good. Glad you got it working. That was going to be the suggestion. Keep in mind, at this point, UGUI sprite controllers are only available for SALSA and EmoteR. We expect they will be implemented in Eyes at a future date.

    Thanks,
    D.
     
  41. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,314
    It would be great for eyes! Thanks :)
     
    Crazy-Minnow-Studio likes this.
  42. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    @jrbourne thanks for posting this for other Unity users. And to clarify, this is not a SALSA issue, but rather an issue between Unity and Mac OS (Big Sur). SALSA does not control or manipulate audio data. It simply reads and analyzes the data present.

    Thinking further on the issue, I know in the past, there was an issue with Unity installs on Mac (within the Unity Hub) that did not properly authorize the use of the microphone in the system. I don't know if the situation was resolved, but 'a' workaround solution was to circumvent SIP and insert authorization manually, then turn SIP back on. Again, I don't know if it is a similar situation, I just know there have been issues in the past with Unity and MacOS. This was the post, I am referring to -- again, totally different situation, but maybe there's something there to look into.
    https://forum.unity.com/threads/microphone-audio-is-silent.786497/#post-5735083

    Thanks,
    D.
     
  43. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    TimelineSalsaCore v2.5.0 Released
    TimelineSalsaCore v2.5.0 has been released on our downloads portal and includes support for the Eyes module.

    The new version includes usability and feature enhancements for all modules but namely there is now support to control Eyes.

    NOTE: The old SalsaAudio and Emoter tracks within the previous (legacy) version of TimelineSalsaCore are now deprecated and are not included in the new v2.5.0 package. There will be no further updates on those modules but you can continue to use them in lieu of, or along side the new modules until you convert them over. It is your choice to convert or not. The new package will not remove the old control tracks.

    I am working on new videos for each module. A "new features" video is released and the TimelineSalsaCore SALSA module tutorial re-make is also available.


    Enjoy!
    D.
     
  44. ChillX

    ChillX

    Joined:
    Jun 16, 2016
    Posts:
    145
    Seriously dude. I understand keeping things closed source.
    However when you do so then bugs like the following are simply unacceptable:

    Why?
    Because we have no option to fix it ourselves.

    Note: I've already switched from Salsa to SpeechBlend which is a billion times less hassle and way cleaner and simpler providing as good if not better end result. Was just trying to use Emotr and Eyes animation from your package before I can the whole lot and go write my own random blendshape animator and eye movements.

    Seriously??? index out of bounds??? In this day and age ??? Or is it some null reference that was not checked ???

    Unity Version: 2020.3.6f1

    Array index (-1) is out of bounds (size=107)
    UnityEngine.StackTraceUtility:ExtractStackTrace ()
    CrazyMinnow.SALSA.ShapeController:SetLerp (single,bool)
    CrazyMinnow.SALSA.QueueProcessor:processQueue (System.Collections.Generic.List`1<CrazyMinnow.SALSA.QueueData>,int)
    CrazyMinnow.SALSA.QueueProcessor:LateUpdate ()
     
  45. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello @ChillX,
    These errors have been greatly reduced or eliminated in v2.5.0. When they did or do occur, they are typically a result of custom runtime code deleting linked resources without the proper housecleaning or improperly building a configuration at runtime. Either way, we try to maintain a balance of error checking vs runtime performance and try to be as responsive as possible if someone has a problem with our software or the issue isn't covered in the documentation. Since it looks like you've already moved on to another product, I'm leaving this note for anyone else who has issues they would like to resolve. Please contact us if you have problems and we will quickly help you work through it.

    We realize our product is not for everyone and we don't try to portray it as such. @ChillX, we seriously wish you good luck with your project and with SpeechBlend.

    D.
     
  46. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    TimelineSalsaCore v2.5.0 - EmoteR Tutorial
    The tutorial video for the Timeline EmoteR module has been released.

    The EmoteR module tutorial has been released, adding to the available, updated TimelineSalsaCore videos:
    • New Features of v2.5.0
    • Timeline SALSA Module Tutorial
    • Timeline EmoteR Module Tutorial
     
    wetcircuit likes this.
  47. VideoTutorial

    VideoTutorial

    Joined:
    Jun 27, 2017
    Posts:
    29
    Hi - I hope you can help me...

    I'm using salsa with 2D sprites to lip-sync, and it works OK with a long audio file of speech.

    But I also have a different mode where the character audio is a series of many 0.5sec audio clips played through an audio source, similar to Animal Crossing. The Dialogue System (pixel crushers) typewriter is playing a sound for each letter, and I'd like the lip sync to move with the sounds...

    However, at the moment the mouth stay shuts and only opens occasionally.

    Any ideas please?
     
    Last edited: May 18, 2021
  48. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, I would imagine the issue might be AudioSource.PlayOneShot() calls. PlayOneShot does not work since it bypasses the AudioClip buffer and inserts further up the audio filter chain. Is that a possibility?

    D.
     
  49. VideoTutorial

    VideoTutorial

    Joined:
    Jun 27, 2017
    Posts:
    29
    Hi, thanks for getting back to me so quickly.

    I suspected PlayOneShot was a problem so I didn’t use that approach. Instead, I’m switching out the audioSource.clip.

    I’d be grateful if you can help me get to the bottom of this. I was going to try looking around the source code for a fix but it seems like that isn’t an option!
     
  50. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Are you stopping the AudioSource, changing the clip, then playing? Stop/change/Play is recommended to ensure the clip data pointer is reset.

    D.