SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hello @CommunityUS,
I'm happy you found a solution to your problem; however, I have no idea why the scaled time ignore flag would solve an issue with your configuration or the data serialization process. We would love to help you out but will need to know more about your project and setup. Generally speaking, these types of errors happen when the model is setup and then referenced parts of the the model object are deleted; or a runtime configuration is not quite correct. In this instance, it looks like one or more components are setup, but not baked into serialized data within one of the modules. Things that could cause it? It's hard to say definitively, but I would suspect a bad blendshape index is configured in the controller and the QueueProcessor tries to animate the bad index. Possibly when the blendshape index was assigned, it was not found (by name) and assigned the return value of -1.

If you'd like us to help you troubleshoot it, please send your invoice number, version numbers, etc. and more details about your project, like how you are setting up the character (runtime, edit time) and the script you are using to do so. We'll be happy to help you get your configuration going.

D.

 

Hello,
Actually my scripts derived from OneClick DAZ were ok. I tried everything to isolate this problem with Eye.Initialize() error, but at the end it dissappeared once i deleted Eyes component and created new one manually + processed it with unchanged script derived from DAZ OneClick. Maybe it bugged somehow during numerous editor script iterations on my Salsa prefab GameObject when I was tunning it.
Also Emoter was not working (nothing has been shown in Queue Processor), even if everything has been set identical to character processed by OneClick which worked fine.

 

That's possible -- I've seen weirder things in Unity, not necessarily related to SALSA Suite. I cannot really provide anything useful for your EmoteR issues without seeing the script and some screenshots or videos showing what's going on (or not as the case may be). From the PM you sent, sounds like you're good to go for now. Let us know if you have further issues.

Good luck on your project!
D.

 

Sorry for misunderstanding, Emoter problem has been fixed by deleting/adding Eyes component too Evyrhing works fina now.

 

That's even better news!

D.

 

Hello,
Is it possible to change Global Dynamics valu in Salsa's Global Overrides via code? I'm unable to find it.
I would like to change how much characters open their mouth runtime, based on character settings and situation. Alternativelly I could multiple Visimes shape Max by some value I guess, but this looks like easiest way.

 

Yep, you sure can. globalFrac, advDynPrimaryBias, and hiCutoff are the main values to play with. hiCutoff is the upper bounds for the linear cutoff and it will allow you adjust how analysis is spread across the amplitude range. The bias adjustment dampens the max extent variability. The main one to tweak is globalFrac. It will linearly dampen the max extent. Use these values instead of re-writing your viseme configurations. Much easier and what they were designed for. You can read more about them in the API documents.

Hope that helps,
D.

 

It's not that I didn't look into OneClick and documentation, but I just didn't recognize that Global Dynamics is called globalFrac in API
Everything seems to be setup now. Thank you for great support and asset!

 

Hello. I imported salsa and "oneclickbase_2_5" and "oneclickcc_2_6_0" and I used in my cc3 model, but i get it a error and the emotes don't are load.
I use unity 2020.3.37 and tryed in many model, but i get the same error

 

Hello, it's hard to tell from the information presented. I would suspect, assuming the CC3 option is selected for OneClick, the model is missing the blendshapes the OneClick is looking for. If you check the code for the OneClick, the line it is stopping on is where the tweaks are applied to the emotes and since there aren't any, it's failing. So, either the OneClick isn't finding your SMR or isn't finding your blendshapes on the SMR. It will be necessary to compare the blendshapes being search with your SMR and ensure your SMR is being found by the SMR search from within the OneClick.

If this doesn't get you down the right track, send us an email with your invoice number and some more screenshots showing the scene hierarchy and the mesh SMR with blendshapes.

Thanks,
D.

 

Thanks. I created manually and salsa worked well

hello, is possible to disable (easy way) Salsa when the npc (with salsa) is far from the protagonist?

Thanks

 

Hello, please see our online documentation:
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/further-reading/enable-disable-modules/

D.

 

Does the OneClick support characters imported via the auto setup for unity from Reallusion Character Creator 4 ?
https://www.reallusion.com/auto-setup/unity/default.html

 

Hello,
We do not own or operate the Reallusion products and are far from Reallusion experts. But, the OneClick is based (minimally) on the 60 morph export. If your auto-setup does not have or is missing some of the specified morphs, you might need to modify the OneClick to work with your model. Hope this helps, but perhaps someone from this or the Reallusion forums can answer your question with more authority.

D.

 

I have seen that you have a character creator 4 preset in your import packages, which works wonderfully, pretty much out of the box, only a Jaw Bone has to be added manually to the Visemes. Apart from that it's just perfect!
I Made a video of the CC4 Test:
https://twitter.com/VWGAMEDEV/status/1620137931200667648

 

Actually, the new v2.6.1 that I uploaded within the past hour now uses the JawRoot bone and looks tremendously better! I also optimized the other shapes to provide better dynamics and more even movement based on another customer's input. The model in your link looks great -- but I didn't see any lipsync, just the head/eye tracking with the Suite.

Hope this update helps you and others using the CC4 OneClick.

D.

 

Reallusion CC OneClick v2.6.1
Release with improved support for CC4 character models.

Reallusion CC4 models have improved dynamics in this release (v2.6.1). Now uses the 'JawRoot' bone vice 'Teeth02'. The change in dynamics is dramatically improved. Also, tuned other viseme components for better, more even dynamics, keeping the mouth from being too 'toothy'. This OneClick will operate on 60/140 morph CC4 exports. 60 morph exports will not add a couple of tongue components to the configuration, but to be honest, it can only be seen if you are looking down the throat of the model, so do not feel it is necessary to export with 140 setting. The package contains OneClicks for CC3 and CC4. Ensure you are applying the correct OneClick on your model. NOTE: If upgrading from pre-v2.6 versions of the OneClick, please remove any existing Reallusion CC OneClick files prior to installing this one. There is one Editor file and 2 Plugins/OneClickRuntimes files to remove. Since I combined them, it felt necessary to modify some of the class file names to be generic -- as 'CC'.

Please also consider CC4 support to be beta. If you find issues with the OneClick for your CC4 models, please do not respond here, but send us an email, with your Invoice number and some details around what is not working. Also, please send your model so we can see what is missing and test any required changes. Reallusion changed quite a bit on the blendshape names and there may be other changes that we haven't seen with the test models we have.

Get the package from our downloads portal.

Enjoy!
D.

 

Thanks for the advice! Another thing that could be implemented is the eye blendshapes of CC4. CC4 provides incredibly realistic eye movement by not only rotating the eyeballs but also having blendshapes for the muscle areas around the eyes. this gives a charachter an extreme amount of realism. i haven't currently found a way to set this in Salsa. That would be great for a future update. Until then i can solve it with a custom code but it would be cool to have an option for this in salsa.

 

The eye shapes is a great idea, and could be implemented if the eye movement was shape-based. The implementation is bone-based for CC4 since all of the models we have as samples are bone-based movement. So, we have to go with what we perceive as the primary export option. We do appreciate the suggestion and will keep looking into it.

Thanks,
D.

 

Yes, the eyes themselves are bone-based, but in addition to the movement of the eyes, Character Creator provides 4 blend shapes per eye that work around the eye (on the body, not on the eyeball). I am attaching a video showing how the eyes move here and at the same time how the blendshapes work around the eyelid.

I wrote a little code this morning that implements the whole thing on the basis of the eye rotation. It works very well (as long as "Use Persistence" is activated in Salsa). Would be cool to have this directly in Salsa in a next update.

https://gist.github.com/valentinwinkelmann/4864b81a4210f8600ab70886d536c897
(Here is the code, probably not the most elegant solution, but it works for now.)

 

Understood, the problem is there is a blendshape option in Eyes, but the templates are separate from the bone templates (by design since they use a different processing algo). To allow for this scenario would require a lot of retooling in Eyes, a LOT. That is not to say it is impossible or will not happen, it just isn't an easy implementation. It is probably better implemented as an auxiliary process, much like SilenceAnalyzer is for SALSA and very much like your script below. In a scenario where the eyes are shape-driven, this effect could likely be implemented pretty easily in the shape-template.

This is very nice of you to provide for other customers. You mentioned it requires "Use Persistence" in SALSA, are you talking about the actual SALSA lipsync module? I cannot fathom how this would impact the eye movement. It seems like it could be a bug if that is the case. Could you send me an email (assetsupport@crazyminnow.com) describing this in more detail?

Thanks so much for the information you have provided!
D.

 

Hi there!

I am having an issue with Salsa's head/eye tracking when being used with FinalIK's AimIK.

As can be seen from my screen capture, Salsa yellow tracking line is quite off to the side and a green dotted line shows the discrepancies between the distorted tracking position and the correct tracking position (i.e., PC head).

Is there any efficient way to solve or circumvent this issue (besides disabling Salsa's head/eye tracking)?

Thanks!

 

Hello, I would think this has something to do with FinalIK influencing the twist up the spine to the neck and head and throwing off the Eyes calculations. Is there some way to cut FinalIK's influence on the head and neck bone to see if that resolves the issue?

D.

 

Hi there!
I removed the neck bones from aimIK but the issue persisted. I also bought IKPlus, hoping it might be a FinalIK issue, and see that this problem also arises when using the upper body chest IK module (i.e., no neck, no head). The only way for the head and eyes to work properly is to disable any upper body IK (ex: if using only the leg IK, your components work as expected).

 

Eyes is doing aiming calculation using the root bone as a reference for forward. It makes sense that adding additional offsets up the bone chain might lead to inaccuracies. Aiming calculations start with world space, but must ultimately switch to local space so that logical clamping restraints can be applied to prevent things such as a head from spinning all the way around, etc. We are also accounting for model designs that don't conform to Unity's axis orientation, so it's not a trivial problem. It's something we can look into, but don't have an immediate solution for. Please send an email to assetsupport@crazyminnow.com, reference your SI_007 forum name so we can connect the email to this forum conversation, include your invoice number, and send us a minimal test scene that demonstrates the problem clearly and concisely.

Thanks,
Michael

 

Alright, I found a less than perfect but workable option by strictly using your Head/Eye components when outside combat/aimIK situation. That being said, I do encourage you to look further into testing your Head/Eye components with FinalIK or IKPlus, as a proper solution would be very useful. Thanks again!

 

FYI: I have found that the asset "Realistic Eye Movements" has taken account of the issues that I am encountering with your components. As they provide significant details on how to make the head and eyes work with IK, I encourage you to make contact with them in order to help you find a similar solution for Salsa. Thanks!

 

Hello!
I have a 3d human face with an EmoteR collection based on facial blendshapes: Upset, happy. sad, etc, that are set to be manually triggered when needed.
When I I try them in Unity play mode, the emotes work. However, there is no smooth or fluent transition from default face blendshape to any of the manual emotes.
I realized that when using random or emphasys pools there is actually a fluent animated transitions (blincking, frown, etc.)
How can I make the manual emotes to be triggered on and off with a fluent transition?

I look forward for your reply. Thank you.

 

Hello and happy to help,
It sounds like the timings are possibly not configured appropriately. Check your timings and if that is not the issue, please send us some screenshots of your configuration and video demonstrating your issue. Since you are manually triggering emotes, please include the relevant script segments where you are triggering them. Of course, please also include your invoice number and versions of SALSA Suite, OS, Unity, etc.

Thanks,
D

 

Hi, thanks for replying. I wrote you an email with the information requested.
Should be continuing the support via email or in this chat?

 

Hello, email is probably easier. If it is something that might benefit the community, we can summarize it in the forum. Look forward to getting your email -- haven't received it yet.

D

 

SalsaMicInput v2.5.1
Release with improved support for prefab usage.

SalsaMicInput should now work with prefab instance changes. NOTE: It is always recommended to backup your project before updating this or any other package.

If you find issues with the add-on, please send us an email with your Invoice number and some details around what is not working.

Get the package from our downloads portal.

Enjoy!
D.

 

Hi! I bought Salsa few weeks ago and I'm very happy with it so far!
I'm using it with Character Creator 4 characters and the oneClick solution is very good! My only problem is: if, before exporting from CC4 I convert the character to game base (which is half size and keeps all the bones and facial expressions), oneClick doesn't work anymore.
The problem is the CC_Base_JawRoot facial bone of all the viseme that somehow doesn't set properly. The bones of the game version of the CC4 model are the same as the normal one, just the names are different.
Is there an easy way to fix it?
Thanks

 

Hello, I had a preset for the game-ready model exports that I was holding pending testing by another customer. I've posted it to the downloads portal. Download the CC OneClick v2.6.2 and test it out. BACK UP YOUR PROJECT BEFORE INSTALLING THIS OR ANY OTHER ASSET.

Hope that helps,
D.

 

Reallusion CC OneClick v2.6.2
Release with improved support for CC4 character models. Includes support for CC4 game-ready model exports.

Reallusion CC4 game-ready models have different bone rotations and some blendshape extents are also different. The package contains OneClicks for CC3, CC4, and CC4-game-ready presets. Ensure you are applying the correct OneClick on your model. NOTE: If upgrading from pre-v2.6 versions of the OneClick, please remove any existing Reallusion CC OneClick files prior to installing this one. There is one Editor file and 2 Plugins/OneClickRuntimes files to remove. Since I combined them, it felt necessary to modify some of the class file names to be generic -- as 'CC'.

ALWAYS BACKUP YOUR PROJECT BEFORE INSTALLING THIS OR ANY OTHER ASSET/ADD-ON.

Please also consider CC4 support to be beta. If you find issues with the OneClick for your CC4 models, please do not respond here, but send us an email, with your Invoice number and some details around what is not working. Also, please send your model so we can see what is missing and test any required changes. Reallusion changed quite a bit on the blendshape names and there may be other changes that we haven't seen with the test models we have.

Get the package from our downloads portal.

Enjoy!
D.

 

Hey, there new Salsa user here, bought it yesterday and it's great.

In a new project (Unity 2022.2.10), after importing the Salsa package (2.54) the demo runs fine.

However, I then:

Install the oneclickbase 2_5_0_2 package,

then install the oneclickcc 2_6_2 package,

I get these compile errors:

Code (CSharp):

1. Assets\Plugins\Crazy Minnow Studio\SALSA LipSync\Editor\OneClicks\OneClickCCEditor.cs(75,56): error CS0103: The name 'OneClickCC3Eyes' does not exist in the current context
2.  
3. Assets\Plugins\Crazy Minnow Studio\SALSA LipSync\Editor\OneClicks\OneClickCCEditor.cs(81,24): error CS0103: The name 'OneClickCC3Eyes' does not exist in the current context
4.  
5. Assets\Plugins\Crazy Minnow Studio\SALSA LipSync\Editor\OneClicks\OneClickCCEditor.cs(90,24): error CS0103: The name 'OneClickCC3Eyes' does not exist in the current context
6.  
7. Assets\Plugins\Crazy Minnow Studio\SALSA LipSync\Editor\OneClicks\OneClickCCEditor.cs(99,24): error CS0103: The name 'OneClickCC3Eyes' does not exist in the current context
8.  
9.  

I don't see a OneClickCC3Eyes.cs file. What am I doing wrong? If I comment out all usages of that in OneClickCCEditor.cs, it works and I'm able to use Salsa with my CC4 character ok, but I'd like to get the eyes going too.

I assume I'm missing some step in the setup, any help appreciated.

 

Hello, yes. Someone else just pointed this out to me too. I missed the Eyes component when I exported the package. I've fixed it and re-uploaded the v2.6.2 package. Please download again and it should work fine now.

Sorry for the confusion,
D.

 

Thank you very much, I will try it asap!

 

Works perfectly, you guys are awesome! Thanks!

 

Hi guys, I am not familiar with lip sync nor their terms. Based of the tutorials and documents, it seems that this plugin will only animate mouth shapes based of the loudness of the voice / words? Is this correct?

I am wondering if this can animate the lips word by word?

In the doc there is an example of 7 visemes: w, t, f, th, ow, ee, oo.

So can it actually animate these 7 words? If it does how do you actually set the viseme ow or ee to the sound ow or ee? I only see the trigger value settings.

Hopefully someone can explain this thanks.

 

Hello,

Just wondering what the standard method of using this is supposed to be in terms of not needing to use the persistant mode. With persistant it works fine, but the tooltip implies that its better to not use it.

But whenever you use it without persistant mode it never works right, it conflicts with the normal animator controller all the time. So what is the actual normal use case supposed to be for this?

Also what exactly is "auto merge with influencers" conceptually? Is that supposed to be the ability for example to have a happy or sad animation on the face on the base animator controller and at the same time have it lipsync talking while still displaying the base animation? Does this work with persistant mode, or just only non-persistant mode?

 

about readyplayerme ?

 

Hi,

I have a character configured with the latest release of SALSA, and all is working fine. Now I need to change the configuration of Eyes at runtime, using the API, to switch the useAffinity property from false to true, and vice versa, depending on certain other conditions. The problem is that as I try to setup a private reference to the Eyes component in my script, I'm getting an error message:
error CS0428: Cannot convert method group 'GetComponent' to non-delegate type 'Eyes'. Did you intend to invoke the method?

The error is referencing a line of code in my script here like this:
MyEyes = MyCharacter.GetComponent<Eyes>;

I can setup a reference to the Emoter on the character with no problem using a similar approach. I can't find any explicit Eyes API example code, but it is meant to work the same, correct?

 

SALSA is not a phoneme mapping solution. The visemes are simply chosen as a representation of the shape in the chain. You can read more about it in our online documentation and see how it works in our video tutorials.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/overview/

D.

 

Good questions. I believe this document should explain in detail most of what you are looking for.
https://crazyminnowstudio.com/docs/...g-with-mecanim-other-influencers/#persistence

Technically, it is not SALSA Suite conflicting with the animator, it is the other way around. The animator is undoing your intent -- what SALSA Suite is trying to do. For example, you are trying to walk down the street and your friend is pushing you in the opposite direction. Who is conflicting with whom? It is all about intent. What is the intent? You walking down the street or your friend pushing you backwards? The solution? If your intent is to walk down the street, your friend needs to get out of the way. It is the same thing in the case of SALSA Suite. The best way to handle this is for your friend to not push you in the opposite direction. Same with any animation. The best solution, since the Animator does not care about other animation systems, is to remove conflicting animations from the Animator. If this isn't possible for some design reason, SALSA has some mechanisms to try and work around it -- persistence is one method.

SALSA Suite does everything it can to work with external influences. Persistence mode was added to try to deal with Animators conflicting with SALSA Suite animations. You can feel free to use persistence mode if your situation requires it. The only reason we imply it should not be used is because if you don't need it, you are simply adding processor cycles to something that isn't necessary. (Refer to the document linked above.)

Merging with influencers is described in the linked document as well: https://crazyminnowstudio.com/docs/...ow-salsa-suite-works-with-external-influences. In short, SALSA Suite looks at the previous state of its controller animation progress. If something else has modified the animation progress, it assumes it is being influenced elsewhere and depending on which way the animation is going it will overwrite the current value or try to merge with the external influence. If the animation is moving in the ON direction, it will overwrite the values. If it is turning the animation OFF, it will try to merge to create a smooth transition. Influence can be internal to the Suite or external (Animator, etc.). Internally, SALSA Suite has a hierarchy or order of operation that should take precedence. For example, lipsync takes precedence over emotes. SALSA does a pretty good job of working with an EmoteR-configured emote (happy, sad), but it does require the emote configuration to be "persistent". This is all explained in the document, but consider this example: A SALSA viseme "EE" uses the smile blendshape. EmoteR uses the smile blendshape for its happy emote. EmoteR turns on "happy" and the animation exits the queue (see the linked document). SALSA is speaking, turning on viseme EE and then turning it off. The "haapy" emote just lost the smile blendshape because SALSA animated the "smile" blendshape to its ON position and then turned it off. So, enabling persistence on the "happy" emote keeps the smile shape controller in the QueueProcessor -- SALSA overrides it in the viseme component's ON direction and merges back to it in the OFF direction.

The next update to the SALSA Suite will have more tools and even better handling of these types of situations and a "mood" will be a first-class citizen that SALSA can work with.

Hope that all helps!
D.

 

I replied to your email, but for others here, you have a syntax error in your code.

Code (CSharp):

1. Eyes MyEyes = MyCharacter.GetComponent<Eyes>();

That should fix the error you are receiving. You may not need to 'type' MyEyes if you did it earlier.

D.

 

SALSA-driven AI co-host.

 

That error is usually the result of run-time setup without baking the serial data. Are you doing run-time setup? If so, check the docs here: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/further-reading/runtime-setup/, step #2.

D.

 

Hi, I just deleted my post - it turns out that avatar is missing its model geometry
Thank you for replying so quickly with your advice.

 

That would do it too! Good luck on your project!

D.