SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hi, what is the performance hit when analysing audio clips in real time? Is there an option to analyse in the editor, then export the blendshape animations to clip files? Thanks

 

Hello, SALSA Suite is very lightweight on processor usage and generates zero GC (per deep profile measurements). As with all things, the rate of processor usage depends on the configuration, platform deployment, etc. Only you can determine the level of 'performance hit' you are comfortable with. There is no option to bake animation analysis out-of-the-box; however, you could use third party options (or roll-your-own) to record the animations yourself if you wish. Likely the additional requirements of timing playback and control, etc. would negate any potential gains of letting SALSA Suite do its thing. We would highly recommend profiling your project using the deep option to dig into the Suite's usage, prior to going down a path that would likely not yield significant gains. Of course 'significant' is subjective and only you can determine the actual significance. Profiling SALSA Suite within your project will give you the best insight into SALSA Suite's performance impact within the scope of your project.

Hope that helps,
D.

 

Hi,
How can i change Eyes lookAt programmatically?
Thank's

 

OK solve it

 

I'd like to advise our community that I have, at last, started to read the manual.

 

That's great! Let us know if you cannot find what you are looking for.

D.

 

Any Chance of doing a Daz genesis 9 one click converter ? The 8 and 8.1 are so good but I got a lot of genesis 9 assets that would love the same treatment.

 

It's been out for a while and no one has said anything...we just assumed it was still working. I didn't think they changed anything on the morph side since 8.1. I'll take a peek at it (hopefully) tomorrow.

D.

 

What's morphs are best to use? I randomly choose some blend shapes, and her mouth forms words better.

Some viseme blend shapes seem to just come into existence? ie I don't export ee blend shape but it's in there. And it works. - She's a standard G8 character though so maybe that's why.

Also made a tutorial on how to export - but I think I got a number of things wrong. ie 8.1 seems to worse than 8? It doesn't make the all the ee,th visemes etc

 

OK, I've had a chance to look at it and they have definitely changed some things. Purely looking at the fbx (non-bridge) export, there are some mesh name changes and the blendshapes are similar, but a lot are missing. In your video and screenshots, you see the visemes, but not the entire configurations so a lot of the dynamics are lost. It would appear that the bridge exports some default shapes that are not in the fbx export or the standard filter we use to get the 8.1 shapes (doesn't include all of them). Looks like they've changed the prefix naming on some of the shapes as well. It would really be nice if they would exercise some consistency in the model versions. I'm going to be out of the office the rest of the week, so won't be able to get into this until next week. It's going to take some research and testing to figure out what they've done.

Thanks,
D.

 

No worries! Thanks for looking into it. Let me know how you go.

 

Hi McSwan,
I did not get a chance to look at this, this week -- had a personal/family emergency that left me emotionally drained this week. I should be able to get it wrapped up next week. Just wanted to let you know so you didn't think I forgot about you.

D.

 

I have a question about prefabs.

My avatars are used in several scenes, and I'm using Game Creator. So, SALSA is applied to the CC skin-made-prefab, but in another scene my SALSA settings are different. Opening the prefab doesn't show the Visemes I'd set on the prefab in another scene.
Hypothesis: SALSA settings are scene-level?

My SALSA avatar itself is within a Game Creator Character object, which now itself is also a prefab. So my SALSA avatar is a prefab in a prefab.

In scenes, I'm concerned about keeping consistent SALSA settings in the avatar from scene to scene.

TL;DR are viseme settings maintained in a prefab across scenes, or overwritten?
<second coffee brewing>

 

Hello. I'm not entirely sure I completely followed your thought process here -- whether you have an issue or are just pondering some different usages -- so please forgive me. SALSA Suite should follow the same usage application as anything else in a prefab or nested prefab. The only thing we are aware of (prefab related) at this point is applying an official OneClick will (currently) want to peel off one layer of prefab -- it doesn't need to and we will be changing that in the OneClicks soon. But this isn't the situation I believe you are pondering. If your changes to a prefab are not being reflected in your base prefab, my first guess would be they have not been applied. But this is all standard prefab stuff. Assuming these are all standard prefab usage quandaries, your questions might be better served in a more focused forum topic on the subject. But if you are having issues specific to SALSA Suite, we will need more information probably some video to demonstrate what you are experiencing. In that sense, it would be better to send us an email with more specifics (Invoice number, versions of every thing, video/screenshots, etc.).

D.

 

No worries! Thanks for looking into this for me and I hope everything is ok. I can work on other stuff in the meantime, and if I really need to, I can setup it up manually. Having a one click for daz 9 and 8.1 would speed things up a lot though.

 

Hi CrazyMinow!

Been really enjoying reading this topic and seeing your wonderful support for this asset. I'm absolutely ready to invest in it, the only problem lies, that we use WWISE for our project, which from what I understand doesn't really work with SALSA, any thoughts and ideas, on how to make lipsync work with WWISE and SALSA?

More specifically, is there a way to create an audioclip in runtime from pooling data from WWISE?
Or is there a way to play the relative audio and have it lipsync, with Unity's default audio system disabled. Elaborating on that, if Unity audio is disabled, besides the player not hearing the audio, does it also disable Salsa from reading the clip and applying the lipsyncing?

 

Hello and sorry for the delay, didn't get a notification on this post for some reason. Admittedly, I know very little about WWISE; however, if Unity's core audio system is disabled, it will be necessary to provide the data to SALSA using other means. If WWISE gives you access to the data, you can feed it to SALSA to analyze, or analyze it yourself and feed the analysis to SALSA. Please see the documentation on a custom filter chain for an example: https://crazyminnowstudio.com/docs/...alsa/delegate-processing-custom-filter-chain/

Of course, this example is written using Unity's implementation, but the theory would work for anything, assuming some sort of callback implementation. Without a callback, it would be necessary to operate on the audio data (if available) via one of the Unity loop callbacks (Update, LateUpdate, etc.).

Hope that helps!
D.

 

Hi there!

API documentation is a bit lacking. Especially when not using any audio analysis and using "getTriggerIndex".
I'm getting my speech and therefore also my visemes from Azure, and it's quite unclear how and when to trigger an index - especially when there's animation smoothing from SALSA.

To be more concrete, I can't find the API for getting "GlobalOverrides.Smoothness", which I guess I need in order to trigger the viseme ahead of the audio time. Maybe it's "globalDurON" ? But there's nothing mentioned in the docs about these values...

An example of manually triggering visemes while synching with audio playback would be great!

 

There are two general ways to control which viseme fires. The first, which you've already found in the documentation (https://crazyminnowstudio.com/docs/.../delegate-processing/#assign-a-viseme-trigger) is using the

getTriggerIndex

delegate and remapping that to your own viseme selection function. The second is to set SALSA to

useExternalAnalysis

and provide your own analysis values.

Using

getTriggerIndex

foregoes the ability to use Advanced Dynamics. This is something that was not in the documentation, but I have now added it. The reason is, selecting the viseme trigger without some sort of analysis does not allow for processing Advanced Dynamics. Therefore set:

salsaInstace.useAdvDyn = false;

If you wish to use Advanced Dynamics, it would be necessary to use the second method, providing your own analysis values. This can be accomplished by feeding the value directly, or by re-mapping the

Salsa.getExternalAnalysis

delegate.

If you place SALSA in external analysis mode, you are in control of providing analysis based on the trigger you want to fire. For example, consider the setup of a standard OneClick with 7 visemes setup:

​

Providing external analysis, to fire the 'th', you would set the external analysis value to something larger than .4285, and smaller than .5714. If you are using Advanced Dynamics, this range is important for providing more dynamics to the animations. i.e. .4285 would produce less of the 'th' extent, while .57 would produce nearly the full extent of 'th'. As long as you don't change this value, the viseme will stay engaged. If you want to turn it off, set analysis to 0.0f. NOTE: You do not have to set silence between each viseme. There is an API example of setting the analysis value in the documentation: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/api/#processed-values-inputoutput

Pseudo-code of what you are doing would be something like the following, assuming you are feeding the value to SALSA on a loop cycle:

(sorry for the way this formatted, but I think you get the point)
game-loop
get-viseme-from-source
if silence
set

salsaInstance.analysisValue

to 0.0f
if 'th'
// random for dynamics otherwise just set a value between the range.
set

salsaInstance.analysisValue

to random between (0.4286 .. 0.57)
if ...repeat for each viseme...
end-game-loop

Likewise, if you are re-mapping the delegate (preferred), you would have your process code outside of a loop and let SALSA call the delegate on the pulse cycle. The end result is the same, it simply allows SALSA to grab it when it needs it instead of feeding a value to it constantly while it only looks at it when it needs it.

Concerning timing. Smoothness is the animation ON timing. This does not control when the viseme fires. It controls how long the ON animation takes to complete. Longer time = more frames = smoother animation. The

audioUpdateDelay

field tells SALSA when to poll for audio analysis and subsequently, viseme change. This is explained in the API here: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/api/#operational-settings. The global overrides provide a more convenient way to tune the relationship between the ON, OFF, and processing pulse rate to set a look-and-feel for the project according to the designer's preferences. You can read more about this mechanism here: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/using/#global-overrides

There is no

smoothness

field in the SALSA API since this is an Inspector option and calculation. You can control the global override for ON and OFF timings using:

salsaInstance.globalDurON

and

salsaInstance.globalDurOFF

. But unless you are changing this value dynamically or setting up at runtime, there is no need to change what is set in the inspector. I have added this to the API documentation in the Operational Settings section: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/api/#operational-settings

As to how you control your viseme selection timings, this is up to you. As explained above, it will be a balance of adjusting the

audioUpdateDelay

verses the animation ON timings to provide the look-and-feel you desire. Your method for selecting the viseme index or providing analysis values that trigger a specific viseme will be polled by SALSA according to the

audioUpdateDelay

. The API documentation also describes this field: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/api/#operational-settings

The nature of SALSA, operating on set pulses, means you will likely not hit every viseme you receive from Azure if the pulse timing is too long. It will be necessary to experiment with the

audioUpdateDelay

cycle to find a good value for your use case. Keep in mind, you can set

audioUpdateDelay

to be a short timeframe value and as long as the visemes don't change to fast, the animations will have time to do their thing and your changes can be more responsive without being too jittery.

SALSA was designed to work with analysis values, yet is flexible enough to work with custom scenarios like this to an extent. In the distant past I did a proof-of-concept integrating OculusVR phoneme detection with SALSA and it worked pretty well. But Oculus is an analysis system and thus the timing of things worked along with the audio. Since you are getting your visemes, presumably in the form of a file or data chunk, likely with timing information associated with it, you will need to apply those viseme changes with the associated timings and the associated audio playback head position.

Hope that helps,
D.

 

Hi there!

First of all, great job on this library. It produces really high quality lipsync and the customizability is top notch. EmotR and Eyes work like a charm as well. So: thank you!

I came here with a question / problem: sometimes when my character model mesh and hierarchy changes, I struggle to replace it in the prefab and end up having to setup a new one, and then I find myself having to go through every reference of SkinnedMeshRenderers (for blendshapes) and bones (which is probably the most cumbersome, as I have to recapture min and max settings) in SALSA. Takes me over an hour and is prone to introduce errors. I tried writing an editor script yesterday to try and copy Salsa, and then update all references to the new gameobject programmatically. But I didnt manage to access those fields in the expression component. Is it possible? It would be a huge help if you could point me in the right direction.

 

Hi! Yes, it is certainly possible. Have you checked out the runtime configuration documentation? It should have the information on the data structures you are trying to copy over.
https://crazyminnowstudio.com/docs/...reading/runtime-configuration-data-explained/

Hope that helps you out!
D.

 

Ahh thank you, that was exactly what i was looking for!

 

Is there a demo with w-f-t-... visemes?

 

Hello, there is not one included in the package, but there are OneClicks available for supported model systems that you can try. For example you can get a DAZ model and quickly try it out with a OneClick. If you are looking for information on how to configure a custom model, we have information available on that as well: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/recommendations/

D.

 

Hello, in the Unity editor(2022.3.6f1), my character's lip sync functions correctly using the LipSync suite(2.5.5). However, after building and directly running the executable file, the lip sync doesn't work as expected. May I know the reason for this discrepancy?

 

Hello, we've responded to your email and have requested more information. Thanks,

D.

 

I want to know why I package webgl and it doesn't work, I already output it as a document. The version I use is 2022.3LTS and its mouth shape is not effective

 

Hello, we are going to need a lot more info. Please send us an email with reference to your post here and with all of the details of your project:

• How is your project configured?
• Where is the audio coming from?
• What kind of model, are you using a OneClick?
• Are you using the AmplitudeSALSA addon?
  
• Does it work in Editor?
• Are you getting errors in the web console?
  
• etc.

Also include your invoice numbers (SALSA and Amplitude) and screenshots of your configuration as well as video of what your project is or isn't doing.

Thanks,
D

 

Hi im getting the following error repeatedly when simply playing the sample audio along with my 1 clicked Daz3D model.


Array index (-1) is out of bounds (size -22)
UnityEngine.StackTraceUtility:ExtractStackTrace()
CrazyMinnow.SALSA.ShapeController: SetLerp (single bool)
CrazyMinnow.SALSA.QueueProcessorrocessQueue (System.Collections. Generic. LIst* 1 «Crazy Minnow. SALSA. QueueData» int)
CrazyMinnow. SALSA.QueueProcessor: LateUpdate ()

Any idea for a fix as I am unable to build with this error going on.
Thank you

 

Hello. This error is generally the result of an improper runtime configuration, but since you didn't mention anything about runtime setup, we will need more information about your project. Please send us an email with at least the following details and we will be happy to help you out:

• Send your invoice number.
• Version of SALSA, Unity, OS, etc?
• What generation DAZ model are you using?
• Did you get any warnings when you applied the OneClick?
• Is at least 1 viseme configured and the visemes section minimally configured (blue)?
• Include video of the issue.
• Include screenshots of the configuration and model hierarchy.
• Anything else that may be pertinent to your scenario.

Thanks!
D

 

Hello everyone,

I'm encountering an issue with the Unity Salsa plugin for facial animation, and I'm hoping someone can help shed some light on it.

In the Unity editor, I've configured the facial animations exactly how I want them. When I play the game within the editor, everything works perfectly as per my settings - no issues there.

However, when I create a build of the game and run it separately, it seems like the values I've set in the editor are completely ignored, as if the plugin defaults are being used instead. This inconsistency between the editor and the build is puzzling me.

Has anyone else faced a similar problem, and if so, how did you resolve it? I'd greatly appreciate any insights or suggestions on how to ensure that the facial animation settings in the editor are retained in the build.

Thank you in advance for your help!

 

Hello, we haven't seen this behavior but curious how you are measuring or coming to the conclusion build vs editor is different. A build (vs editor) run should be faster since the editor overhead is removed. But if you have your editor inspectors collapsed, that should be minimal. If anything, the effect of running a build should be smoother, a more positive experience.

D.

 

Make sure your building the correct scene ? Could be building an old version.

 

OneClick Refresh v2.7.2
All official OneClicks, along with the OneClickBase files, have been updated to version 2.7.2.

Except for the DAZ OneClick, the only changes are for supporting OneClickBase v2.7.2. The main changes to this collective update are as follows:

• OneClicks and OneClickBase are collectively v2.7.2 for clarification of requirements. Version 2.7.2 is only meant to be used with version 2.7.2 OneClicks (and vice versa).
• Prefab unpacking has been removed from all OneClicks.
• QueueProcessor and AudioSource components are added and configured last in the process and execution phase.
• If you call the OneClicks at runtime, be sure to call the OneClickBase static functions to configure the QueueProcessor and AudioSource in your calling script.
• It is probably best to remove all older OneClick Editor, runtime, and Base-core files prior to installation.

DAZ has some additional updates:

• Genesis 9 models are now supported.
• Genesis 8.1 models have been re-worked and utilize far fewer blendshapes.
• DAZ Studio morph export presets are available for Genesis 8, 8.1, and 9 models. This is a separate download in the downloads portal. These are csv files created within DAZ Studio. NOTE: if you load these presets, they will override any morph selections you have made -- so add them to your own presets, or create new presets including your personal selections if needed.
• Refactoring of class and file names for consistency and organization. Definitely remove old DAZ OneClick files prior to installing the new version.

ALWAYS BACKUP YOUR PROJECT BEFORE INSTALLING THIS OR ANY OTHER ASSET/ADD-ON.

I believe these packages are pretty solid, but since it is a pretty big update...if you experience issues other than console errors indicating you are trying to access OneClickBase with the wrong parameters--indicating you probably need to update your OneClickBase, please send us an email with your Invoice number and a detailed description of the issue you are experiencing.

Get the package(s) from our downloads portal.

Enjoy!
D.

 

Hi there (amazing asset by the way!)

I'm having some issues now I've ported my project to Android that the lip sync seems to be much less responsive and less convincing. Framerate in general is all good, just the animations that are off. Any ideas of settings I might have missed? Attached a little sample of what I mean (ignore the weird eyelids, that's a different issue)

 

Hello, we have responded to the email you sent concerning the same topic.

Thanks,
D

 

hello,

i'm working with the one-click from cc4
is see there are 2 options
cc4 and cc4 game-ready.
what is the difference between these two options?



thank you in advance

 

For a more definitive answer, you'll likely need to check with Reallusion, but it is possible to produce "game-read" models from their product. These models have different bone orientations; as such, we created a separate OneClick for them.

D.

 

Hi I am trying to use SALSA with a CC4 character in conjunction with Inworld.AI. I believe I have things setup correctly after watching some of your videos. I did a test with the CC4 character (after 1 click setup) and using the example audio and this worked as expected.

When I try to link SALSA to the Inworld audio source I do get some blendshape movements but most of the time the mouth barely moves or doesn't move correctly. Have you had any requests for support with this configuration? suggestions re: what to look at?

 

I had a brief discussion with a customer some time back that was using InWorld, but they didn't pass on any information about their scenario or about whether or not what I suggested did anything. It has been so long I don't remember much about it. I apparently learned enough to point them to the custom filter chain insert documentation. https://crazyminnowstudio.com/docs/...alsa/delegate-processing-custom-filter-chain/

This is generally the documentation we point all TTS service users to (except for Crosstales' RT-Voice). Most TTS services tend to stream data into the AudioSource instead of building a clip from it and supplying that to the AudioSource. Check out the linked page and try it out. I think it should be what you need. If you need further help, please send an email with your invoice number and project details.

Hope that helps!
D.

 

Thank you for the reply and suggestion. I do not know if seeing an object called "wav" in the the audio source slot when Inworld is playing back the response from the server is in fact a clip or not. I will review the information at the link you sent.

 

Hi,
there is a problem with the chracter that I would like to use.
It is a genesis8 original which i need to transfer via blender to unity, since otherwise i cant use certain
Morphs (they are Posecontrols in DAZ which are not exported).
So my workflow is:
Daz->blender->unity (import blenderfile)

first little problem is: In Blender the shapekeys have no Genesis8Female__ suffix, it has to be changed for all eCTRL shapekeys.This is just a bit extra effort, ok.
Now, when I use the DAZ Preset Gen 1,2,3,8 I get no errors with the character.
But when I run it there are a lot of runtime errors(see below) which i cannot analyze, because they happen in a SALSA DLL.

A character that is exported from DAZ directly, is working without errors.
Do you know where I must work on, what I have to change? will it be a problem in the game, later?
Thank you in advance.

Array index (-1) is out of bounds (size=79)
UnityEngine.SkinnedMeshRenderer:SetBlendShapeWeight (int,single)
CrazyMinnow.SALSA.ShapeController:SetLerp (single,bool)
CrazyMinnow.SALSA.QueueProcessorrocessQueue (System.Collections.Generic.List`1<CrazyMinnow.SALSA.QueueData>,int)
CrazyMinnow.SALSA.QueueProcessor:LateUpdate ()

 

Hello, are you doing a runtime setup? If so, it looks like the controller data was not set.

D

 

nope, i used the Oneclick - preset for Daz 1,2,3,8
btw, the lipsync is working, it just throws these runtime errors and continues.
the character has all eCtrl blendshapes that are available and all the bones from the original Daz character.

 

Interesting...And all of the modules (SALSA, EmoteR, and Eyes) show 'ready' (blue) headers?

Array index (-1) is out of bounds (size=79)
indicates a blendshape name wasn't found. Generally speaking, this wouldn't be possible to do in the editor since it's a dropdown list to select available shapes. If the OneClick didn't find the shape, it shouldn't have added it. Since you didn't get any notifications in the console window when the OneClick was applied, it would indicate it found the shapes the OneClick was looking for.

Do you possibly have an additional module on the character somewhere, that isn't properly configured?

D.

 

Now I found it, the missing blendshapes are in the Eyelashes Shape.
I think the blendshapes are not exported correctly with Daz->blender, i go and take a look.
Thank you very much for package

 

OK, glad you found it. Sounds like we need to take a look at the Eyes portion of the OneClick if it didn't catch it on application. Thanks!

D

 

sry, i think i didn't express myself correctly.
The Eyelash-Shape should have all the blendshapes that the Genesis8 Model has,
but with my workflow, the Eyelashes does not contain them.
That is the problem, your component seems to look for them.
May I suggest that you catch this Runtime error correctly and give the user some Information about the missing Object(Blendshape in this case) , since I looked back in this forum, and at least 2 other user have the exact same Error somewhere ?
Thank you,
Tom

 

Hi Tom, this is the age old quandary of how much runtime testing do we put in place, especially nullref testing, verses performance impact. Unfortunately, this isn't a transactional, one-time check and happens many, many times per frame depending on the setup. And since the QP is very decoupled from the core modules, it wouldn't provide much more information than what the system provides. I have been working on an analysis class that looks for common issues and provides more information when run. Ultimately, I'd like to put something like this as an option in the core modules and likely I'll place it as an option in the QP. I'll see about getting that implemented in the next major release.

D.

 

not work with AudioSource.PlayOneShot .
The soft works fine and the lip/mouth/eyes move with "audioPlay.clip = tmpclip; audioPlay.Play();" where tmpclip is non-streaming. Now I have a streaming audio which cannot be played using "audioPlay.Play();", yet well played by using "audioPlay.PlayOneShot(tmpclip);" The issue is the lip and humanoid head/eyes does not move when using "audioPlay.PlayOneShot(tmpclip);"