SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

I was able to solve for this. It took three days lol.

I rewatched the demo and see that the Photon/Salsa tutorial was really showing a solution for first-person. By adding a new game object to my character, assigning a second audio source, salsa, and your mic input component, then disabling that object for all non-local clients works in my situation.

 

Oddly, the SALSAIsPresent define symbol tends to get deleted from the line; that just happened to me and I wondered why AC throws 10+ errors.

 

OK, great! Glad you found a solution that works for you. Is your voice still propagating to clients via Photon? Typically we have seen that microphones do not like to be shared between "applications".

D.

 

Update on the brand new SALSA LipSync Suite v2.3.1:

As a clarification to any customers that may be holding off on upgrading to version 2.3.1 of SALSA LipSync Suite and/or the add-on updates; there are no known issues with the current versions and it is highly recommended to upgrade to this version in current and new projects (see notes below). All issues we have seen since the release of v2.3.1 have entailed user error and a need to ensure the correct add-on versions are being used. Please ensure you are using the correct versions - the customer download portal includes requirements specific to each add-on.

NOTES:

1. In existing projects, it will be necessary to reconfigure the blink/lids section in Eyes.
2. If you are using the code-free add-ons, configuration changes may be required.
3. There are some breaking changes in the Eyes API pertaining to blinks/lids functionality.
4. It is necessary to use up-to-date OneClick, code-free, and micInput add-ons (the add-ons have been like-versioned with the Suite v2.3.0+ track).

The updates in this new-and-improved version of SALSA LipSync were the result of many requests by customers leveraging the SALSA Suite in advanced scenarios. These changes required some deep re-working of the underlying code and as such, did affect the OneClick and code-free add-ons to implement the new structures.

Quite frankly, even if you don't need the over/under-drive capability, or the flexible 2D rendering pipeline support, or built-in expression overriding and mergeback, or better persistence management...the general improvement to blendshape animations, especially for fractional registration changes is well worth the update.

D.

 

Hi,

I am trying to set up Eyes (random head + eyes) according documentation and I got this error (I set up everything right):

NullReferenceException: Object reference not set to an instance of an object

CrazyMinnow.SALSA.Eyes.Initialize () (at <ee0da2b3261f4ee6b6db8f46b0df6944>:0)

CrazyMinnow.SALSA.Eyes.Awake () (at <ee0da2b3261f4ee6b6db8f46b0df6944>:0)

UnityEngine.Object:Instantiate(Player, Vector3, Quaternion)

AC.KickStarter:ResetPlayer(Player, Int32, Boolean, Quaternion, Boolean, Boolean, Boolean, Boolean) (at Assets/AdventureCreator/Scripts/Game engine/KickStarter.cs:938)

AC.KickStarter:OnAwake() (at Assets/AdventureCreator/Scripts/Game engine/KickStarter.cs:1065)

AC.MultiSceneChecker:Awake() (at Assets/AdventureCreator/Scripts/Game engine/MultiSceneChecker.cs:50)

Do you please know what is it?

I am using Adventure Creator addon/package and SALSA LipSync is working great with AC, but eyes got the error above.

Unity/SALSA/AC - all latest versions.

Thanks,
Jakub

 

I had that problem when I tried to add salsa and a second audio source to the mesh - when I created a separate game object under the root and added a separate audio source, salsa and micinput +mic pointer I am pretty sure it worked with two players. I will double check. Photon offered some guidance here: https://forum.photonengine.com/discussion/15486/salsa-and-the-pun-voice-demo

 

Reading their article, it certainly sounds like they have the makings to somehow grab onto a local stream for processing. Thanks for the link!

 

Hi,

does SALSA LipSync consider (take into account) a white spaces in the sound files? Like if I have sound file with some spaces (with no sound), lips are still moving even if there is no sound hearing.

Like in these red circles lips are still moving:



Thanks,
Jakub

 

It does...however, it looks like you have some noise there that it is picking up on. You'll need to adjust the noise threshold (low end) in the Dynamics section -- Linear Scaled Cutoffs. https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/using/#dynamics

 

Hi Jakub-Pilar,

There is not enough information here to discern what kind of issue you are having. Please provide the actual version numbers for Unity, SALSA, AC, and the SALSA add-on for AC (if you're using it). What kind of character is this, custom or from a character generation system, which one? Are you using a one-click setup, is so, which one? Does the SALSA demo scene work without errors?

Also, please send your SALSA invoice number to assetsupport@crazyminnow.com and reference your forum name so we know it's you.

Thanks,
Michael

 

Thanks for the answer.

Actually, these are my sound files, there is mostly no noise between:


I tried many combinations in Linear Scale Cutoffs range, but it is still same, mouth is moving all the time with no waiting on speak.

These are my actual settings:


Do you see something wrong there or do you have some advice please?

Thanks,
Jakub

 

Unity: 2019.3.1f1
SALSA: 2.3.1
AC: v1.70.4
SALSA add-on for AC: v2.3.1
TextSync-AdventureCreatorListener: v2.0.0
Character: MB-LAB (Manuel Bastioni Blender character geration system)
No one-click setup
SALSA demo scene works perfectly

But as I am looking on the SALSA demo scene, I dont have same objects hierarchy, for example I have SALSA script (and all the scripts) on the parrent object, not on the object where Skinned Mesh Renderer is. And root object generated by SALSA is not main root object.. And it seems, that when I am playing with settings on the SALSA component, it has no effect..

One more question, are you about to make one-click addon for MB-LAB?

Thanks,
Jakub

 

Thanks for the additional information.

It shouldn't matter where the components are added, only what they are linked to. In the case of the Eyes component, it's important that the character root field in system properties is linked to the actual character root object, while the head and eye bones will typically be a child somewhere down the hierarchy from the root. Are any parts of the various SALSA related inspectors showing configuration errors (gold sections)?

Can you send fully expanded inspector screen shots of your character setup to assetsupport@crazyminnow.com?

We currently have no plans to provide a one-click setup for MB-Lab, though it's not too difficult to write your own.
https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/one-clicks/#creating-a-custom-oneclick

 

No gold sections there. Only blue.
I sent you my character setup to your email.
And maybe I will try to make MB-Lab one-click setup!

Thanks,
Jakub

 

If you are using TextSync with SALSA and audio with SALSA you're going to have problems. Looks like you might have a lot of things going on. Also, if EmoteR is active with emphasis or random emotes that use the same shapes as your visemes, it will move the mouth shapes. Maybe best to take some things off or start fresh and test along the way to ensure things are working.

Thanks,
D.

 

Hello!

We are trying to use SALSA with our characters and are facing some issues. Our characters exported from Maya via FBX. When we add Eyes module to character, it rotates eyes and head around wrong axis. This can be fixed by clicking Fix Axis button. But the problem is that it requires Prefab unpacking, which is not a good option for us, because we update our FBX file frequently (adding new animations, etc) and this changes cannot be incorporated in project if FBX prefab is unpacked (so it loses it's connection to original FBX file in assets). Are there any other options for us to use Eyes module and not unpacking FBX prefab? We tried to export model from Maya with different Up-axis (Z or Y) options, but results are the same.

Thanks,
Nikita

 

I think this might be your problem… your lowest setting is at 0 so it is active even when the audio is silent.

 

Thanks for reply.

I tried many combinations of these trigger settings, unfortunately with no effect.

But I still think so that when you set Linear Scaled Cuttofs min value to 0,2 (as picture above), that means that trigger for w is active when its more than 0,2 (because Linear Scaled Cuttofs overide 0 to 0,2) as it is explained in blue paragraph here: https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/using/#dynamics

Jakub

 

Hello Nikita,

If you fix the axis orientation of your head and eye bones to match unity's coordinate system, the SALSA orientation fix won't be needed, and no modifications to the prefab will be required.

Michael

 

Hey wetcircuit, great to see/hear from you!

Good thought, but actually "0" is valid and recommended for the first trigger value. We set all OneClick initial trigger points to zero. The analysis value must exceed the trigger threshold. This actually works out nicely as the trigger distribution is cleaner and (technically) the calculation is a teensie bit more performant. The lower linear cutoff value is the value that controls the noise floor. Anything less than or equal to the cutoff is set to zero and does not trigger the zero-based (first) trigger.

​

So in the example image above, any analysis calculation of 0.045 or lower is zero and does not trigger the 'w' viseme. However, an analysis value of 0.046 would exceed the lowcutoff and would calculate to 0.0014 (after scaling), which is greater than zero and would trigger the 'w' viseme.

And, as noted in the image, only one viseme is triggered -- the highest trigger point exceeded. In the example, a calculated analysis value of 0.18368 would trigger the fourth viseme (trigger value 0.18367). Whereas a calculated value of 0.18367 would not exceed the fourth viseme's trigger point and would trigger the third viseme instead.

​

Hope that helps clarify this a little bit!
D.

 

This is correct. In your example, anything 0.2 and lower is cut to 0. Anything above 0.2 triggers the appropriate viseme.

D.

 

Ha! That was it, thank you!

I removed TextSync script and lips moving now works fine - lips are moving only when there are speak in the sound file.

My Linear Scaled Cuttofs values are 0,1 <-> 0,75 now.

Jakub

 

New tutorial video available: SalsaMicInput Add-on

A refresh of the old micInput tutorial video, using the current versions of SALSA Suite, SalsaMicInput, and Unity.

 

Michael,

Yes, I know this ) The problem is I don't know how to fix it. The model is made in Maya. I know that Maya using different coordinate system, but I cant't find any tools in Maya or Unity to convert it to system SALSA is using. Maybe you know one? Also, exported model works just fine in Unity (plays animations and etc), except for SALSA's Eyes module. Are there any known pipeline to use characters made in Maya with SALSA?

Nikita

 

Hi Nikita,

We have many customers using models produced in Maya, and most all other 3D systems on the market, that have corrected orientation using export parameters, control bones, or using our Axis Fix. I'm not a Maya users, so I'm afraid I can't offer any assistant on how to accomplish this, perhaps a Maya forum?

Exported models with incorrect bone orientation work fine in Unity because you aren't doing any calculations against world coordinates, to track targets in Unity, world coordinate calculations are necessary. The Axis Fix feature addresses this by adding corrective hierarchy above the incorrectly aligned bones, but Unity 2019+ requires prefab unpacking to add the corrective hierarchy to the character.

Michael

 

Hi Michael,

I tried to export from Maya with different axis options (Z-up or Y-up), but it works the same in SALSA. For now we binded Eyes module on empty object with transform and using additional script to copy transforms from this object to character's head bone, applying addition multiplication (multiply by bone's original rotation if it's not 0,0,0) to correct head's orientation. I wonder why SALSA don't use this additional simple calculation insted of adding offset bones...

Nikita

 

TimelineSalsaCore has been updated to v2.1.0:

• Fixes display problems for EmoteR Timeline tracks in Unity 2019.3
• Minor bug fixes.

New tutorial video available: TimelineSalsaCore Add-on

A refresh of the old TimelineSalsaCore tutorial, using the current versions of SALSA Suite, TimelineSalsaCore, and Unity.

 

Plans for integration with Game Creator (catsoft studios)? or, any reason to believe Salsa won't work with GC?

 

Hello! We do not have any current plans for integration with GC. As to whether or not SALSA would work with it, what sort of integration would you be looking for or expecting?

 

Hi! Are you aware of any problems in PC builds? In the editor, Salsa runs smoothly, but when I build the game it does not run anymore. Audio is playing, but lips are not moving. I’ve used the isSalsaing boolean to try to get some info, and this one is not triggered in the build, but is in the editor.

 

[edit -- corrected reply recipient]

Sorry, no, not aware of any issues. I just performed a build in 2017.4 and 2019.3 and it all worked as planned.

D.

 

Hi there. My team and I have been using Lip Sync Pro for our current project but I get the impression they are no longer supporting the asset and we are considering seeking a refund and moving to Salsa. I have a few questions, if you wouldn't mind answering.

-Is Salsa supported? i.e if we have technical issues, will we be able to get answers from one of your team members within a reasonable time frame?

-We are using UMA. One of the issues we are running into with Lip Sync Pro is that it does not provide a delay option, so the audio and lip sync animation plays on load, before the scene is fully loaded or the UMA has even been fully generated. Does Salsa provide a delay function to get around this?

-Does Salsa allow for using multiple lip syncs within the same scene? For instance, if we wanted to call lip sync A or B on condition C?

-Is the source code provided in case we need to make changes that are relative to our project?

Thanks for your time.

 

Hello!
Sorry to hear about your experience with Lip Sync Pro.

SALSA is certainly supported and we have a significant set of online documentation as well as video tutorials. And are always adding more. We believe our support is top-notch, but we'd rather you take our customers' word for it, check out this forum and our ratings/reviews for a view on our quality of support. If we have documentation that covers issues/questions, we point customers to it. But we will certainly help out if or when documentation isn't enough.

UMA is a beast and there are so many ways to use it. Our 'driver' does wait for the system to be spun up. We do leverage the DCA and ExpressionPlayer. Depending on how you are using it, that should be sufficient. However, if you are using UMA differently, it may not support it out of the box, but all of the source for the UMA OneClick is provided in the add-on. SALSA Core is compiled, but all of the add-ons are in-the-clear.

Only one SALSA instance is necessary per character. Your conditions would apply to the AudioSource and that is easy to work with. There are a lot of ways to implement conditional dialogue. We also have add-ons for code-free systems, including Timeline for audio and emotes. You can watch our latest tutorial video on that implementation linked in a post above.

Source, no, only for add-ons. As mentioned, the core product is compiled.

Hope that helps!
D.

 

Thanks so much for the speedy reply! I'm Sold.

 

So, one more thing. SALSA is NOT a phoneme mapping system like Lip Sync Pro. Just want to ensure you are aware of what our system is and what its goals/strengths are. It is an approximation system that has very compelling, real-time lip-sync. You can get more of an idea of what it is and can do by watching some of the videos.

Thanks,
D.

 

I'm having trouble adding lip sync to my Mixamo characters. I'm using Unity 2017.4.25f1 and when I use the most recent 'one click setup' for Fuse, I see console messages to do with loads of expressions having been removed.

Upon playing the sound file attached to the gameobject, there is no mouth movement. The Viseme component has no entries and doesn't automatically allocate an appropriate skinned mesh renderer.

Any ideas?

 

Hi Zante,

It sounds like your character doesn't have any blendshapes. See the requirements section of the general overview in the documentation.

Michael

 

Couple of issues with UMA integration. For the most part it just works, but there are a couple of things not panning out for me.

First, the eyes don't open after the first blink. That is they start off open and there are some micro-movements in the eyes. But then they close (usually half close then moments later fully close) and do not open again.

Second, the mouth never really closes properly.

I'm using a standard UMA DCS character with everything set at the default. Salsa is being set up by the UMA DCS One Click solution. Any ideas?

 

Hi Michael, you were right. I'm sure I exported them with blend shapes but after redoing it, it seems to work. ;]

 

Hi fgardler,

Disable "Merge with influencer" in the QueueProcessor.
https://crazyminnowstudio.com/docs/...reading/queue-processor/#queueprocessor-class

Michael

 

That did it, thank you.

 

We have published OneClickUMA_2.3.1 to the downloads page. It sets the "Merge with influencer" flag in the QueueProcessor to false by default.

 

hi do you have any sample other that box hear? I'm using v2 but I want to use other sample

 

Hi domdev,

Only Boxhead and the 2D examples in the examples pack on our downloads page.

 

Hey guys!

Im starting to work on an old project here in my company, and the project uses the Salsa plugin, but i cannot make it run because of this problem:


Array index (-1) is out of bounds (size=50)
UnityEngine.SkinnedMeshRenderer:SetBlendShapeWeight(Int32, Single)
CrazyMinnow.SALSA.Salsa3D:LateUpdate()


Im trying to check what is causing this error, but i cannot find anything... Some help?

 

Hi LuizBeenoculus,

It sounds like a blendshape index is being used that doesn't exist. The only place we've ever really seen this issue is with MCS characters since they are the only characters that have blendshape indexes that can change between Unity sessions. Make sure all the blendshapes you intend to use are stable before setting up SALSA.

Michael

 

What is the default size of blendshapes??? Theses models has 50 blendshapes... All working great... The error log shows the Salsa script is trying to access the -1 index

 

Ok, i found the issue, these lines of code in CM_FuseSync.cs:

Code (CSharp):

1. /// <summary>
2.         /// Get the Salsa3D component
3.         /// </summary>
4.         public void GetSalsa3D()
5.         {
6.             if (!salsa3D) salsa3D = GetComponent<Salsa3D>();
7.             //salsa3D.saySmallIndex = -1;
8.             //salsa3D.sayMediumIndex = -1;
9.             //salsa3D.sayLargeIndex = -1;
10.         }

When i commented theses lines, it runs...

 

Hi LuizBeenoculus,

Support for SALSA v1x ended on 01/01/2020. SALSA 2x doesn't need any of the old sync scripts, all the mapping is built into the core product.

https://crazyminnowstudio.com/posts/salsa-with-randomeyes-end-of-life/

Michael

 

I'm trying to get Salsa to work with dissonance over UNET. Is there a way to have voice data lip synced on a GameObject which does not have a netid and only exists on one client?

So the voice data from one client is lip synced with whichever GameObject I'd like it to be on the receiving client?

The use case is fairly atypical but, nonetheless, is something I'd like to explore.