SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Awesome, if you need me to test anything let me know, or if you have any quick fixes I can do to the current integration before the new integration is released that would be great too. Thanks for looking into it, Salsa is currently the best and only option for realtime lip sync in VR!

 

A patch for our OneClick UMA DCS v2.3.1 add-on is now available in our downloads section:

OneClick UMA DCS - Patch v2.3.2
https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/downloads/

• Install UMA v2.10
• Install SALSA v2.4.1.
• Install OneClick Base v2.1.6
• Install OneClick UMA DCS v2.3.1
• Download OneClick UMA DCS - Patch v2.3.2, extract the zip and overwrite the existing UmaUepDriver.cs file found under the following path:

Assets/Crazy Minnow Studio/Addons/OneClickUMA/UmaUepDriver.cs

This is a temporary fix until we release SALSA v2.5.0, which contains tons of improvements that will require new releases of all of our one-click setups.

 

Thanks for the patch guys, works perfectly! Now I can continue development. Much appreciated, looking forward to Salsa 2.5

 

Hello Crazy Minnow Studio,

thanks for Salsa, it is working perfect in our game Biotopia 3D

Our new game will be a kind of space shooter. We would like to have our own heads in it talking, some kind of "voice radio". We would like to use Salsa for the animation of the heads.

- Any idea how we get our "real" heads into the game, or something that comes close?

And we need talking enemies, aliens or maybe monsters of whatevery. Heads would be enough.
- Any idea were to get Salsa compatible alien heads or something like that?

Thanks

 

Hi again!

While oneclick already makes everything much faster to re-import, we are often required to still perform some adjustments afterwards. Is there a way to export an existing SALSA setup as a 'new' oneclick? Or some other way to export a SALSA setup?

Thanks!

 

Hello! We do have a script that we utilize internally to initially create OneClicks and sometimes to tune existing OneClicks. We have been considering release of the script to the public as a means for customers to grab snapshots of their configurations. The script *only* captures SALSA and EmoteR configurations (it does not capture Eyes module configurations). And, it does not capture all SALSA and EmoteR settings, only the primary viseme and emote setups -- basically the same options that are configured with a OneClick. There are no current plans to update the script beyond its current implementation and it would certainly be a "use at your own discretion" deal. If there is interest, let us know.

Thanks,
D.

 

That would be great! We are totally fine with the 'use at your own risk' We often have to tweak the lipsync visemes and some other settings to match new character voices and it's a bit of a pain having to manually update the oneclicks each time that happens. Grabbing a snapshot of the current configuration is exactly what we are looking for!

 

@Crazy-Minnow-Studio Hello. I am trying to get things properly setup. I am using CC3 characters exported as GameBase. I used your one click setup. I can't really get the head to work right. If I just leave it as the one click did its always favoring looking down. If I add a look target - it doesn't work at all. This is screenshot where I put a transform right out in front of him. I did have this working before but am unsure how I messed it up.

 

Howdy, just thought i'd mention that the current integration for Adventure Creator seems to be busted, as there's a great deal of eyes errors.
"Assets\Crazy Minnow Studio\Addons\AdventureCreator\Actions\Eyes_SetAffinity.cs(81,21): error CS0103: The name 'eyes' does not exist in the current context"
Assets\Crazy Minnow Studio\Addons\AdventureCreator\Actions\Eyes_EnableAll.cs(133,8): error CS0103: The name 'eyes' does not exist in the current context
Assets\Crazy Minnow Studio\Addons\AdventureCreator\Actions\Eyes_SetRandom.cs(108,8): error CS0103: The name 'eyes' does not exist in the current context
ect

I know ya'll are in the middle of a big update for the eyes system so names and stuff are likely changing, is there a hotfix i could apply to this?

EDIT:
it looks like the error is that the "#if SALSA is present, enables these parts of the code" stuff isn't actually triggering so the eyes/handler variables never actually get created. I solved it by forcing SALSA to exist at the top of the namespaces instead of depending on an if, and commented out the "if exists" parts of private Eyes eyes and ect, and the error seem to be gone.

I have no clue if this will cause issues, but it's the best i got atm, so just a headsup about that.

editedit: Or maybe i should have read the manual closer and seen you gotta add a scripting define symbol. yup, that would do it. aaaaa. my bad.

 

Hi,

It sounds like maybe you changed the scale of the character using the transform scale? If you change the model scale, it should to be done in the Scale Factor field, on the Model tab of the character file in the project. The character transform scale should always be 1,1,1. Also, the one-click would need to be applied after this change, and some default values may not be valid if you change the scale.

Michael

 

Hi,

Glad you figured it out. Chris from Adventure Creator recommends using define symbols as a best practice when building add-ons for his system, so we use the define symbol "SALSAIsPresent".

Michael

 

Hi, for visemes should I implement a bone for the jaw and lower teeth, and then use blendshapes (morph targets) for lip shape? I'm not sure if I can use both bone position and shape together. Or would it be better to only use blendshapes for everything? Thanks.

 

You can combine both, but if you're already doing blend shapes I would not bother with the extra bone.

 

Yeah, what wetcircuit said, if you include your jaw bone you'd need to include it and and capture the position you want it in for each viseme.

 

I have not changed the scale. I have tried a bunch of different things and am getting the same results.

Update: I figured out my problem. The root was not set correctly. Manually set it and bam - everything is magic! Love this asset.

 

That's great, glad you figured it out.

 

Hey,

Any ETA for 2.5? Looking forward to the PUN integration

Thanks,
Nathan

 

It won't be long now as long as we don't find any more showstopper issues. 2.5 is functionally complete and we're currently testing and debugging.

 

@Crazy-Minnow-Studio I am receiving this error when I'm trying to import/reimport the SALSA-LipSync.dll file. Do you perhaps know what could be the cause of this issue?

 

@Crazy-Minnow-Studio I am having a serious workflow issue. I get my CC3 character into unity, I create a linked prefab so we easily import our animations later. When it comes to salsa - the one click basically destroys the prefab if I try to put salsa directly on it. I tried putting salsa on a parent object and get the weird head rotation - even if I manually setup the root again. Any idea on the best way to rig my characters?

 

can anyone let me know salsa only work with dissonance(Unet) Integration? How can I use salsa in Dissonance(Pun integration) .

 

Hello, just curious, will Salsa 2.5 include support for local avatar lipsync with Photon? I am currently doing remote lipsync and when I add the MicInput script it stops the remote lipsync from working.

 

Anybody else getting this in one click daz unity 2019.4 and mouth don't move

 

@Gatskop_Software

For the locked jaw:
Select your base character model in the Project window,
In Inspector, select Rig,
Click Configure.
This will open a new scene. In the inspector there is a green figure with a column of buttons on the left click the second top button labelled Head. Underneath this there will be a list of Head bones, the bottom one being Jaw.
Select the transform next to Jaw and delete it. It now should say None (Transform)
Click Apply, then Done.
You have now successfully disabled the jaw bone.

@troach06 I believe this is the case. Also looking forward to it. Currently using Dissonance but would prefer to streamline the process a bit.

 

Hello, is there a way to assign and read the SkinnedMeshRenderer and Blendshape Index values for Viseme Components through script? I'm looking at <ExpressionComponent> in the LipsyncExpression.expData and I can't find it where I can do that. Are those variables exposed?

And is it also possible to duplicate a viseme? It's quite time-consuming when I have to recreate and hook up all the viseme components data again whenever I create a new viseme.

Thanks!

 

Hello! Yes, we have many customers configuring their SALSA LipSync Suites at runtime (via script). Please read the Runtime Setup document under the Further Reading section for details.
https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/further-reading/runtime-setup

[EDIT] Duplicating visemes: you can copy/paste from one component to another within your SALSA instance.

Thanks,
D

 

I'm having a problem with the Timeline Salsa Audio Track. Basically it stops playing a few seconds before the end of the track. It seems to be incorrectly reading the length of the audio. It's possible to work around this by forcing the timeline to continue past the end of the Salsa Audio Track by adding some other track to keep the timeline alive. Alternatively you can force the timeline to "hold" and thus it will not reset at the end.

Would be nice if a future update read the correct time of the audio track so these workarounds aren't needed.

 

Send us an email with your Invoice number and project specifics (version numbers, etc.) and we can see if we can reproduce the problem. Please include video of the issue you are experiencing if possible. In our tests, the audio is spot on and no issues.

Thanks,
D.

 

Send us an email with your specifics (invoice and version numbers, etc.). Are you getting errors? What isn't working? Screenshots please. Also, will probably be necessary to send us your model for testing.

Thanks,
D.

 

Hello,
That looks like you're having a problem with your Asset database updating -- several plugins and packages. Looks like you were updating Unity or something?

D.

 

Try applying your OneClick configuration prior to creating your prefab. That should prevent the problem of applying to a prefab. If you already have a prefab, let the OneClick break it and then create a new prefab after you apply your OneClick.

Thanks,
D.

 

SALSA should work with any Dissonance setup. Our documentation simply demonstrates using UNET. The principle is the same. Configure SALSA on your spawnable prefab with Use External Analysis enabled. Then setup your network system of choice.

Thanks,
D.

 

Yes, we have an add-on that takes advantage of new SALSA 2.5.0 features, supporting local lipsync in PhotonVoice.

D.

 

Hey,
I'm trying to get salsa to work with Invectors 3rd person controller, but I'm having an issue with the head ik fighting each other. Is their a way to keep it from trying to animate the head? I'm also using the one click for character Creator 3.

 

You can simply leave the head not configured, set the head template back to none after the one-click setup.

 

I have found the reason for the difference between the recording timing and the timing detected by the Salsa audio source. I have an audio mixer which is dropping the pitch to 96%. This makes the track a little longer than the timing reported by the file itself as it is being played slightly more slowly.

 

Yup, that makes sense.

D.

 

I've been doing some experimentation with semi-procedural body animations to complement Salsa. The code is open source. I've put a video together explaining and demonstrating the technique

 

Awesome stuff!

BTW, the lipsync in SALSA v2.5.0 for UMA has received an overhaul and the resultant animation response is MUCH better (at least in my opinion ).

D.

 

@Crazy-Minnow-Studio can I please request that in your next update you create a Scripting Define Symbol for Salsa. Not having one means that I cannot set up #if defines statements in my code which means it's really hard to share libraries with other people who may not own Salsa.

 

There's an update I missed. Blast it! Oh well, good excuse for another video pretty soon

[EDIT] Just checked the store and I assume you mean a forthcoming update to 2.5. Looking forward to it and will certainly update the video, especially if there is such a big improvement.

 

Yes, sorry - in the upcoming 2.5.0 release. We are really close. UMA is being a little difficult (runtime recipe switching) with some of the new Eyes enhancements and Mike is trying to get it all sorted.

 

Just wrapped up putting this in place for ya. Will be available in the 2.5.0 release.

D.

 

I have version 1.5.5. Can I import 2.4.1 on top, or should I remove earlier version first?

 

Hi wood333,

You will need to remove the old version, SALSA LipSync Suite is a complete rewrite, and not compatible with the old product SALSA with RandomEyes.

 

Thank you for the prompt response! I look forward to using LipSync Suite.

 

SALSA LipSync Suite v2.5.0 is almost done!
We know it has been a long time coming, but we believe we are complete with all feature updates and fixes that will be in the new v2.5.0 release. As we mentioned in previous updates, we will need to update the OneClick add-ons to work with and support the new Eyes module -- things just work differently -- and that is what we are currently doing. There will be new versions of SalsaMicInput and SalsaDissonanceLink; however, they will not be necessary to run the new version. They will simply take advantage of a couple of the new SALSA "plugin" features and will subsequently, make them a little more efficient.

So, what is this big v2.5.0 update all about? Eyes!!!
Namely, the new version revolves around the changes in the Eyes module. Previously, to work around non-Unity-standard bone axis orientations, Eyes substituted known-correct bone scaffolding to perform its animation calculations. Unfortunately, this had an undesirable consequence to some external (non-Suite) animators/animations. And the workarounds were not always viable. So, we have removed all of that axis-fix stuff and implemented a new mechanism for dealing with bad axes. So much of the Eyes module code was re-written and re-designed, the module is pretty much brand new.

What about SALSA and EmoteR?
There are quite a few feature updates for these two. Track back through a couple of the update posts here and you'll get a feel for some of the updates. The latest update added is an Advanced Dynamics Silence processor. This new feature will be initially released as an additional component (likely to be incorporated into the SALSA module at a later date if well received). Basically, it runs alongside the SALSA analysis and can override the triggered visemes to create a more responsive perceived accuracy. The effect is subtle, but the result should be more responsive lipsync dynamics, especially while increasing the animation ON timings and the analysis delay pulse for smoother animations. I think you will definitely want to try it out and see for yourself. Remember, SALSA is not a phoneme baking solution. Our goal is a high level of performant, real-time, perceived accuracy.

I don't believe I mentioned it previously, but EmoteR is getting a new feature for LipSync Emphasis -- AlwaysEmphasize for individual emphasis emotes. Normal operation of emphasis emotes was a random selection of the emphasizer emote pool. This works great, but now there is an option to select specific emphasizers to always animate while others randomly animate. The benefit here is one or more emphasis emotes will consistently animate in time and magnitude with the audio, while others will add nuance and variation to the emphasis. Additionally, it is now possible to fire more than one random emphasizer from the pool on each pulse. Just another way to provide flexibility and designer choice.

A lot of additional work has gone into making the Suite better at working with other, external animation influences. Based on customer feedback, we began implementing some of these changes into the v2.3.0 release, but it has been overhauled again and the QueueProcessor has evolved to dynamically and automagically detect influence and smoothly work with or around it (in most cases). Of course, there are situations where design implementation decisions need to be made to resolve issues. By default, most conflict is simply dealt with, smoothly and efficiently. If you are using the SALSA Suite, it expects to override any other conflicting animations but should smoothly return to the underlying animation where detected and feasible. And, like most other Suite implementations, there are freedoms of choice for how the designer wants to deal with these situations.

Minimum Unity version changes:
It is doubtful this will affect many teams, but we have made the decision to change the supported Unity versions. This new version will drop support for v5.6.0 and will move the minimum Unity version to 2017 LTS for the core product. Of course, we do not wish to alienate any of our customers, but we do feel this is more than fair since Unity 2017 is technically end-of-life. Until further notice, we will be building against 2017.4.1f1 as the minimum Unity version requirement. NOTE: Add-ons may have additional minimum version requirements.

We are excited to get this new version into your hands and are hopeful it will further empower your designs and breathe even more life into your characters. Bare with us just a little longer while we test and document and finish the update.

Thanks,
D.

 

Hey, I was wondering if the v2 of this suite works with Mixamo character animations? Kindly let me know so that I can decide to buy it.

 

Is the one-click setup will support the new Character Creator 3.3 Base+ ? The rig is a bit different.

Thx

 

I'm not sure what you mean by this. Character animations typically animate the entire character while SALSA's focus is on animating the mouth, face, head, eyes, and eyelids. They are both animation drivers, and SALSA is designed to take precedence over character animation bones that conflict, but it can also handle smooth transitions between the two for most common scenarios.