Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, PlayOneShot does not use the normal AudioSource.AudioClip pipeline. If you want to use one shots, you'll need to intercept the data in the filter chain. https://crazyminnowstudio.com/docs/...alsa/delegate-processing-custom-filter-chain/

    D.
     
  2. jackyetz

    jackyetz

    Joined:
    Dec 28, 2022
    Posts:
    11
    Crazy-Minnow-Studio likes this.
  3. Romitas

    Romitas

    Joined:
    Nov 9, 2013
    Posts:
    1
    @Crazy-Minnow-Studio Hi!

    I'm having a bit trouble using Salsa API, can't figure it out from documentation

    Thing is, I have my AudioSources in a Pool, all created, requested and released on demand in runtime, so there's no single constant AudioSource in scene I could just assign to Salsa on setup phase.

    So this means I'd like to leave AudioSource field as null in the component initially, and then be able to assign and re-assign (probably many times) audioSrc in runtime from code, but if I do just that, it doesn't seem to work: lip sync animation is not played and this error keeps appearing in console:

    NullReferenceException: Object reference not set to an instance of an object
    CrazyMinnow.SALSA.QueueProcessor.Register (CrazyMinnow.SALSA.ExpressionComponent expressionComponent, System.Boolean isActivating, System.Single frac) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.RegisterComponentOn (CrazyMinnow.SALSA.ExpressionComponent component, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.TurnVisemeComponentsOn (System.Int32 triggeredIndex, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.TurnVisemeOn (System.Int32 triggeredIndex, System.Single dynamicsAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.LateUpdate () (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)


    Is there a right way to do this?
    Thanks!
     
  4. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, SALSA does some startup checks, so you cannot just leave the AudioSource reference as null. But you can enable the option to "wait for AudioSource". This will bypass the checks and behind the scenes it starts a coro to monitor the reference and continue when it's set. At this point, you can swap AudioSources at will and should not receive null-refs, AS LONG AS you leave the SALSA inspector collapsed. If it is open, it will check for the AudioSource while focused.

    The null-ref your error referenced came from the QueueProcessor and wasn't directly related to the AudioSource. However, when the AudioSource isn't configured, it prevents the data serialization process behind the scenes and the QP doesn't know anything about the controllers it is being asked to reference. So, putting SALSA in the "wait for AudioSource" mode allows the controller serialization to happen and all is well from that point on -- as long as the Inspector is collapsed.

    There are other ways to do this, like using External Analysis and letting SALSA tap your analysis as needed. Then you can control the AudioSource in your own way, but you'd need to perform and provide analysis on your own.

    Hope that helps,
    D
     
  5. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    3,948
    does salsa work with cc4 (character creator 4)? iclone 8? when will one clicks be available for them?
     
  6. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    It does work with CC4. TBH, I don't know about iClone 8. We don't actually use/own the Reallusion products and have not had anyone ask about iClone in years.

    D.
     
  7. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,187
    Hello,

    I have a quick question on Salsa performance. I've been profiling my game, and I'm seeing that every few frames, Salsa's LateUpdate is taking up a lot of time turning visemes on and off. I can't debug the code, since it's in the plugin, but here's what I'm seeing in the profiler:

    upload_2023-12-13_19-35-20.png

    This LateUpdate is taking usually between 1-2 ms per character. I'm not sure if that's just the expected overhead of Salsa, or if something unusually is happening. I'm not sure why it's not every frame (maybe Salsa only updates visemes a few times per second?) In any case, I was curious if this was indicative of an issue, or if there was anything I could do to improve the performance.

    Thanks.
     
  8. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, this is expected behavior, SALSA polls audio analysis at intervals set by the Audio Update Delay option in the Settings tab. It looks like you have a lot of components being registered (37), so this is either multiple characters or there might be an excessive amount of components in your visemes. Each time a component is registered it cycles through all existing components in the queue to see if there are conflicts to mitigate. If you are running a single QueueProcessor for all of your characters, I would recommend against this as it creates overhead that is unnecessary. The QueueProcessor should only manage a single character since there would not be conflicts between characters. If you have 37 components per viseme, I would recommend paring that down quite a bit, less is more. Also keep in mind, registering a component compares the component to all other components currently registered for a character, including Eyes and EmoteR since there could be conflicts between those configurations. Even still your numbers seem high.

    If I run a deep profile on my test environment using CC4 characters which generally have more components per viseme than most other model systems (3 - 8 per viseme), I see a 0.07ms process for registering a viseme's components (7). If I duplicate the character with 10 instances, I see a max of 0.86ms process for registering 80 viseme components (across the 10 character configurations). These are shape and bone components in the SALSA configurations. My computer is no slouch, but it is over 4 years old (Ryzen 3900x).

    Even tho your numbers seem a little high, 1-2ms is also not a serious frame chewer. 2ms is 500fps. Likely the QueueProcessor (doing all of the animation number crunching for the model) is eating a bit more. Your LateUpdate loop is sitting at <3ms. Mine is sitting at <10ms for 10 models all configured with Eyes, SALSA, and EmoteR -- doing a lot of stuff -- still 100fps+. I can increase to 20 CC4 models and sitting at <16ms (60fps) for the entire SALSA Suite. This sounds like a lot, but it's 20 detailed models all independently talking, emoting, and looking at stuff -- just stress testing.

    If you toggle the RegisterComponentOn method, what is the breakout? It looks like you aren't using EmoteR emphasizers which could take up more time since several emotes can be active at once (per model) depending on your configuration.

    Hope this info helps!

    D.
     
  9. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,187
    Thanks for the detailed breakdown. I'll look at trimming down the visemes, to see how much I can get away with before it starts looking strange. To answer a couple of your questions, I'm using one QueueProcessor per character. This scene had only a single character in it. The character uses Eyes, but not EmoteR, and the total Viseme components is pretty high, as you point out:

    upload_2023-12-14_9-42-41.png

    In looking at this, I realized I just had way too many viseme configs in there that honestly didn't need to be there. At some point, probably while I was testing out EmoteR, I found that some facial expressions didn't look very good unless the separate Eyebrow meshes were included in the viseme config. So I made sure those were there, along with all other separate Hair meshes. But now I'm not using EmoteR, just lip sync and eye blink, so those components really weren't doing anything.

    I also realized that the model contained an LOD1 mesh with blend shapes, and those were getting picked up by the OneClick, even though that's probably a big waste. So I got rid of those as well. I wonder if LOD awareness is something the OneClick should be made aware of? (I'm trying to decide whether there would be cases where you'd want the visemes on LOD1+ meshes. Seems just the Jaw Bone animation is "good enough" for that LOD level, at least for me.)

    So now I'm down to 10 components for a the viseme that had 37 on it. Thanks.
     

    Attached Files:

    Last edited: Dec 14, 2023
  10. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    OK, gotcha, that makes sense now. Yes CC4 can have a lot of additional meshes. As far as LOD's, if you are configuring a single SALSA instance for all LODs, yes it will try to apply them all. It is unaware of which LOD you are on and only knows what is configured. LODs are hard to be aware of since there are so many different implementations. It might be possible to just check if a mesh is enabled (if that is the way the LOD is being switched) and only firing off registrations for enabled LOD meshes. I'll have to look into that as an option. Let me know the performance you are seeing with the 10 vice 37. You could create a separate SALSA instance for your LODs and switch SALSA on/off with them. https://crazyminnowstudio.com/docs/...odules/#enablingdisabling-salsa-suite-modules

    I've been toying with the idea of adding a mesh inclusion array to the OneClicks that makes it more selective. It takes away the "one click" idea that way, but might be more useful considering some of these model systems.

    D.
     
  11. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,187
    The performance was much better after trimming down from 37 components to only 10 components. Maybe down to 0.2 ms to turn visemes on and off, instead of 1-2ms each time.

    As for LODs, I edited the OneClick code to look for an LOD group in the hierarchy, and to skip any renderers in LOD groups 1 or higher. But, that's probably pretty specific to my own needs. The code's simple, though. So at this point, everything's working well. Thanks again for the quick and helpful response.
     
    Crazy-Minnow-Studio likes this.
  12. mindfulmx

    mindfulmx

    Joined:
    Mar 21, 2017
    Posts:
    42
    Hello,
    I want to use Salsa in the simplest way possible in generic 2d faces, I already have the faces, I already have the different sprites mouth shapes for the phonemes, eyes open and closed. Could someone guide me on how to implement the face so that it performs the lipsync? Thank you in advance.
     
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Also replied to your email.
    We have two videos that discuss the setup and settings of SALSA.



    D.
     
    mindfulmx likes this.
  14. mindfulmx

    mindfulmx

    Joined:
    Mar 21, 2017
    Posts:
    42
    Thanks a lot for your fast and clear answer!
     
    Crazy-Minnow-Studio likes this.
  15. Graham-B

    Graham-B

    Joined:
    Feb 27, 2013
    Posts:
    329
    I'm getting the following error when I apply my custom one click to a spawned character at runtime:

    NullReferenceException: Object reference not set to an instance of an object
    CrazyMinnow.SALSA.QueueProcessor.Register (CrazyMinnow.SALSA.IExpressionController controller, CrazyMinnow.SALSA.ExpressionComponent+ControlType controlType, System.Single durDelay, System.Single durOn, System.Single durHold, System.Single durOff, System.Boolean persistence, CrazyMinnow.SALSA.ExpressionComponent+ExpressionType type, CrazyMinnow.SALSA.LerpEasings+EasingType easing, CrazyMinnow.SALSA.ExpressionComponent+ExpressionHandler handler, System.Boolean animatorControlled, System.Boolean useOffset, System.Boolean isActivating, System.Single frac) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.QueueProcessor.Register (CrazyMinnow.SALSA.IExpressionController controller, CrazyMinnow.SALSA.ExpressionComponent+ControlType controlType, System.Single durOn, System.Single durHold, System.Single durOff, System.Boolean persistence, CrazyMinnow.SALSA.ExpressionComponent+ExpressionType type, CrazyMinnow.SALSA.LerpEasings+EasingType easing, CrazyMinnow.SALSA.ExpressionComponent+ExpressionHandler handler, System.Boolean animatorControlled, System.Boolean useOffset, System.Boolean isActivating, System.Single frac) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.RegisterComponentOn (CrazyMinnow.SALSA.ExpressionComponent component, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.TurnVisemeComponentsOn (System.Int32 triggeredIndex, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.TurnVisemeOn (System.Int32 triggeredIndex, System.Single dynamicsAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
    CrazyMinnow.SALSA.Salsa.LateUpdate () (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)

    How can I fix this?
     
  16. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Looks like maybe the controller data wasn't baked. Possible you missed that step? https://crazyminnowstudio.com/docs/.../runtime-setup/#step-2-baking-controller-data

    D.
     
    Last edited: Dec 19, 2023
  17. Graham-B

    Graham-B

    Joined:
    Feb 27, 2013
    Posts:
    329
    Crazy-Minnow-Studio likes this.
  18. pradf4i

    pradf4i

    Joined:
    Nov 7, 2017
    Posts:
    39
    Are there plans to have a one click for models with ARKit blend shapes or any recommended approaches? thanks
     
  19. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    At this point, we don't have plans to implement an ARKit OneClick -- mainly because we cannot assume all models generally converted into the kit will have the same visual implementation. While this is also generally true for other model systems as well, it is particularly so with ARKit conversion. However, it is very possible if you have characters you want to implement with ARKit and your characters are standardized in naming and hierarchy, you could create your own custom OneClicks by configuring one character and using our 'Create Custom OneClick' add-on. You can then use the custom OneClick to apply to other models. https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/custom-one-click-tool/

    Hope that helps.
    D.
     
  20. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,187
    Quick question: I've been noticing that occasionally, one of my characters that has an Eyes controller on it, will end up with their eyes permanently "blinked". Their eyes stay shut, and never open again. I've been having a hard time diagnosing it, since it only seems to happen in a full build of my game, and it hasn't happened in the Editor so far. I was just curious if that's something you've ever seen in general?

    Normally the eye blink works fine. I've tried a few things to resolve this, like zero-ing out all blend shapes on the model before showing them, but so far the fix has eluded me.
     
  21. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, no we have not seen or heard of this happening. Is it consistently reproduceable or very intermittent? Nothing should be different between Editor and Build that I can think of. I would think that if you perhaps found a bug that conflicted the blink and caused the QueueProcessor to think it was finished with the blink when it actually wasn't, that it would correct on the next blink cycle. The only other thing I can think of is if you had a 'closed eye' configured in EmoteR, it would override the blink while it was active. Same goes for any other external influence (animation).

    D.
     
  22. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,187
    Thanks. It's very intermittent, unfortunately. I'm not using EmoteR, either. I'll let you know if I find a way to get it to happen reliably, or if it just turns out to be something weird I was doing to cause it.
     
    Crazy-Minnow-Studio likes this.
  23. varclifeuser

    varclifeuser

    Joined:
    Mar 3, 2021
    Posts:
    1
    Hello,

    We are using the OneClick setup with CC4 GameReady and the setup seems to not be correct.
    The mouth of the characters is more open than it should & the visemes quality is less than when we use the OneClick with CC3 Characters.

    Is this a known issue? Do you have any recommendations on the export of characters?
    How do you recommend that we should modify the one click script with a setup that fits us?
     
  24. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, it is hard to say without some visuals (video or screenshots). It isn't anything we are aware of -- of course, different models may be created differently with respect to the morphs they receive from the CC system. The OneClick is a starting point and can be modified directly if you find it consistently not to your needs. You could edit the OneClick directly or you could duplicate/rename the files and necessary parts and then change the duplicate, or you could create your own custom OneClick. I'd probably recommend duplicating/renaming the existing OneClick and making your changes there, just in case we ever update the official OneClick and you download it -- you won't lose your custom settings.

    Hope that helps,
    D.
     
  25. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    3,948
    Is a functioning CC4 oneclick on the roadmap? This really is the difference between the team I work with during my day job purchasing this and not.
     
  26. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    The current CC4 OneClick is fully functioning to our knowledge. Other than the post above which has had no follow-up by the poster, we have heard nothing but good things about the CC4 OneClick.I will say we have had some inquiries about some CC4 facial addons (i.e. facial hair) that utilize non-standard and non-predictable names that are not found. The simple fix is to edit the SMR search list for those instances.

    D.
     
  27. Trunksome

    Trunksome

    Joined:
    Jan 28, 2022
    Posts:
    13
    Hi there!

    I am using Salsa lipsync for a while now, and it's been continuously getting positive reactions from players. I'm really happy I chose your plugin, and I invested quite some time to setup things just right.

    Now I am in the process of exchanging Unity's built-in audio with the audio middleware FMOD, and as others before me in this thread I am now running into an issue with SALSA, because it relies on the Unity AudioSource that is no longer being used.

    I know that the option available to me is to set salsa.useExternalAnalysis to true and set the
    salsa.getExternalAnalysis delegate to provide the analysis float value myself. I then built a bridge component similar to what has been posted here previously, attaching a Digital Signal Processor (DSP) to my FMOD channel group, and then finding the peak values from the resulting spectrum data. So far, so good.

    The thing is that I was really happy with how SALSA looked with my characters before; now it is completely different. I am experimenting with an intensity multiplier on top of the peak spectrum values, and I guess I can get somewhere with this - but i'm wondering if I can somehow get closer to the original analysis SALSA is doing on the AudioSource without external analysis. Ideally I would like to stick with the way SALSA analysis the audio, or at least get as close to it as possible. As I have access to the raw PCM data, I'm wondering what AudioSource can provide SALSA that I cannot.

    Hope this makes sense to you. Any hint would be greatly appreciated!
     
  28. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Ah yes, implementing a 'gain' control of sorts can be fiddly. The analyzer we use in SALSA is basically the same as the one in our documentation (https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/custom-audio-analyzers/) except we do an absolute value conversion on it to catch downswing peaks. Ensure you are getting the raw data from FMOD, not post-effects/spatial, etc.

    Hope that helps,
    D.
     
  29. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Also, spectrum data is generally averaged and that might be the issue you're seeing. Can you get just the raw data?

    D.
     
  30. Ony

    Ony

    Joined:
    Apr 26, 2009
    Posts:
    1,977
    Been looking through the API, this thread, etc. but can't figure out how to get the name or index of the currently playing EmoteR emotion. Possible?

    Also, does EmoteR fire an event when a new emotion is triggered (I'm using random triggering)?
     
    Last edited: Feb 11, 2024
  31. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello, no due to the way the system works, there is no API call to find out what emotes are active. Basically, EmoteR registers emotes with the QueueProcessor as fire-and-forget. So, once fired...it's forgotten by EmoteR. The QueueProcessor isn't much help after the registration since it only knows about the components in the Queue(s), not what the components belong to. However, depending on your use-case, you can simply add an EventController component to the emotes you want to track and that will give you firing details on each component with respect to the four component animation phases. Turning on, ON, Turning off, OFF.
    https://crazyminnowstudio.com/docs/...cialty-component-controllers/#eventcontroller

    You can subscribe to any of the four phases and filter based on the component name. Hope that helps and satisfies your requirement.

    D.
     
    Ony likes this.
  32. Ony

    Ony

    Joined:
    Apr 26, 2009
    Posts:
    1,977
    Perfect, thank you. Looks like it should do the trick, so I'll try that.
     
  33. Object-Null

    Object-Null

    Joined:
    Feb 13, 2014
    Posts:
    70
    hello there.
    im working with character creator 4
    i also downloaded the latest software and one-click from salsa.
    but the issue i have is that the facial hair (eyebrows, beard) are not being seen with the one-click setup.
    is that part already a option in the one-click solution?
     
  34. Trunksome

    Trunksome

    Joined:
    Jan 28, 2022
    Posts:
    13
    Hi there, thank you for that tip with the spectrum data being averaged. Instead of an FFT DSP (which I read the spectrum values from), I went ahead and implemented a custom DSP to make sure I get the raw audio data (maybe I could have gotten that from an FFT DSP as well? I don't know). I followed this example, in case anyone reading this is interested.

    I then run this calculation on the raw audio data:

    Code (CSharp):
    1.  private float CalcSimpleMaxPeakValue(int channels, float[] audioData)
    2.     {
    3.         var maxVal = 0f;
    4.         for (int i = 0; i < audioData.Length; i += channels)
    5.         {
    6.             float absVal = Mathf.Abs(audioData[i]);
    7.             if (absVal > maxVal)
    8.                 maxVal = absVal;
    9.         }
    10.         return maxVal;
    11.     }
    .. which Salsa receives through the getExternalAnalysis delegate:

    Code (CSharp):
    1.     private float GetExternalAnalysis()
    2.     {
    3.         float val = Mathf.Min(1f, _peakValue);
    4.         _peakValue = float.MinValue; // Reset for next analysis
    5.         return val;
    6.     }
    @Crazy-Minnow-Studio does this mirror the "absolute value conversion to catch downswing peaks" you were talking about? It looks really similar to the previous implementation now, although I compared some frames and found some differences. If I'm doing the analysis right then this might be caused by differing volume though, I'm not sure.

    //edit: I ended up using pretty much that method above, just replacing Math.Abs with something a bit more efficient. So far I got the FMOD version to looks pretty much similar to the original!
     
    Last edited: Feb 23, 2024 at 7:46 AM
    SergS_Workshop likes this.
  35. SammmZ

    SammmZ

    Joined:
    Aug 13, 2014
    Posts:
    173

    My eyes have Z-forward, Y-up axis (which seems to be a proper orientation). I have set up Eyes module and eyes rotation/targeting seems to be working perfectly. Eyelid blinking also works fine. Unfortunately, I have issues with Eyelids Tracking.

    1. Direction Type Lower and Upper seems to be reversed. I had to set "Lower" to drive my "up blendshape" and "Upper" to drive my "down blendshape" to make it works correctly.

    2. Only Lower/Upper direction type seems to be working. I can't get any feedback from any other direction, so I can't get left/right eyelid movement. Any chance it's a bug or have I missed something in the setup?
     
  36. SergS_Workshop

    SergS_Workshop

    Joined:
    May 19, 2021
    Posts:
    3

    Hey @Trunksome, may you provide a bit more information, how did you combine SALSA with fmod?
     
    Last edited: Feb 23, 2024 at 5:38 PM