SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

There's nothing that can be done using the generation 4 figures since they have values that range from -4000 to +4000 (seems like they got lazy with this figure). Unity won't support negative or overshoot values greater than 100 although it's one line of code (limitRange(0,100) changed to limitRange(-1, -1) and the SetBlendWeight class is public, so theoretically it should allow any user input through code but the m_Range.Set{} and m_Range.Get{} are private, so hopefully the Blendshape API should be released soon (unless it was embedded within the new Animator API already) and we can make our own decisions on whether or not we want negative and high values applied to imported morphs.

 

Yeah, most 3D applications allow you to overdrive shapes, but sadly Unity's BlendShape API is limited.

 

I found a way to overshoot the morphs up to 200%, but it's not very intuitive. By exporting the EyesOpen-Close preset along with the EyeOpen-CloseL and EyeOpen-CloseR morphs you can get another 100% using the two morphs and another 100% using the preset since it appears that the Gen4 will apply 100% using the left, Right morphs and an additional 100% using the preset that can be exported as a morph. This will work in the DazSync script, but as far as the RandomEyes3D.dll to operate the eye of horus morph (sudo third eye) would need to be updated. maybe you can look into it.

Code (CSharp):

1. Under line 40 add
2. public string centerBlinkShapes = "EyesClosed,Blink";
3. under 45
4. public int centerBlinkIndex = -1;
5. under 48
6. public string centerBlinkShape = "";
7. under 329
8. int centerIndex = -1;
9. under line 330
10. centerBlinkShapes = centerBlinkShapes.Replace(" ", "");
11. under line 333
12. string[] centerNames = centerBlinkShapes.Split(',');
13. under 341
14. for (int i = 0; i < centerNames.Length; i++)
15.   {
16.   if (centerIndex == -1) centerIndex = ShapeSearch(skinnedMeshRenderer, centerNames[i]);
17.   centerBlinkIndex = centerIndex;
18.   }
19. Under line 121 of cm_DazSyncEditor
20. dazSync.centerBlinkShapes = EditorGUILayout.TextField(
21.   new GUIContent("Both Blink Shape Names", "Daz3D BlendShape names are not always " +
22.   "consistant, this is a CSV list of common variations contained in blink shape names."),
23.   dazSync.centerBlinkShapes);
24. and whatever line this was
25. dazSync.centerBlinkShape = "";
26.   if (dazSync.centerBlinkIndex != -1) dazSync.centerBlinkShape =
27.   dazSync.skinnedMeshRenderer.sharedMesh.GetBlendShapeName(dazSync.centerBlinkIndex);
28.   GUILayout.BeginHorizontal();
29.   {
30.   GUILayout.Label(new GUIContent("Both Blink", "-1 means no blink shape " +
31.   "index was found, check the SkinnedMeshRenderer to find the right eye blink shape name"));
32.   GUILayout.Label("(" + dazSync.centerBlinkIndex.ToString() + ") " + dazSync.centerBlinkShape);
33.   }
34.   GUILayout.EndHorizontal();

 

Actually, no change in code is necessary. I just set the head.CTRLEyesSquint to 50 and now when the script sets eyeL and eyeR to 100 it fully blinks. For anyone else with this query with generation 4 figures, when you export the morphs, you'll have PHMEyesSquintL and PHMEyesSquintR that exports with the head expressions, but also explicitly add head.CTRLEyesSquint and you can achieve the 150% value by statically setting it to 50% on you smr.

 

Hi guys! I have a figure setup issue maybe I could get your input on?

I have this idea that the voice should emit from the figure's mouth, but it's not quite as straightforward as I thought. Seems like a no brainer to parent a gameobject with an audiosource to the figure's head…. here's some issues I'm running into with SALSA:
– the volume falloff effects whether or not SALSA can hear the audio. Move away from the figure and the mouth stops moving.
– SALSA "just works" with the CM_Fuse Sync script at the top of the figure's hierarchy. Seems logical to leave that alone, and send the audio source signal (via mixer?) to another gameobject parented to the figure's head….
– err… this doesn't seem to be possible? there isn't a way to route an audio mixer to a gameobject audiosource…. Am I wrong?
– Maybe I should just play the audio twice simultaneously: one for SALSA and another for the audiolistener…? That seems wrong, inefficient….

Any ideas?

 

Hey Wetcircuit! If I'm understanding the root of your problem, you can actually link SALSA to any AudioSource -- it doesn't have to be parented to the same GameObject as SALSA. Put your AudioSource next to your listener and then link the source to SALSA. To link it in the Editor, place the Editor in debug mode and it will expose the AudioSrc slot. You can then link any source you want, keeping the source close to your listener so the audio doesn't fall off (doesn't change when you move the character near or far). See the attached image. I've placed an AudioSource on the camera and linked it to the SALSA component.



It is also very easy to do this in script if you prefer (substitute your GameObject containing the source and your SALSA instance -- in blue):

GameObject otherGO = someOtherGameObjectContainingTheAudioSource;
CrazyMinnow.SALSA.Salsa3D salsa3D = yourSalsa3DInstance;
salsa3D.audioSrc = otherGO.GetComponent<AudioSource>();​

Does this help?
>Darrin

 

Ahh, thank you… How do I switch the editor to debug mode? I'm going in circles searching the manual….

 

No worries, here's a screenshot.

 

Hello. New user and everything is moving along nicely; great job on SALSA. I am having just one issue but it is really getting in my way of delivering. I am experiencing random glitches on lip motion that I do not understand and I suspect a bug in CM_AutodeskSync.cs. I have attached an image that explains my situation and provides details of my setup. Any thoughts or help would be greatly appreciated.

 

Hi mdrajeske,

Thanks for your purchase. Can you check your RandomEyes3D instance (there are two on AutoDesk characters), it's the one that has the [Use Custom Shapes Only] check box checked near the top of the inspector. Please expand the [Shapes] section, and check if the [h_expressions.AE_AA_h] shape is set to [!Rand] (not random).

Thanks,
Michael

 

Actually, the h_expressions.AE_AA_h does not even show up in the Shapes list. It happens to be the first in the list when I explain the list under LoopUp Indext etc. Do we have an 'off by 1' unintialized situation?

 

(sorry; typos in previous - let me try again)
Actually, the h_expressions.AE_AA_h does not even show up in the Shapes list. It happens to be the first in the list when I expand the list under LookUp Index etc. Do we have an 'off by 1' unintialized situation?

 

Michael,
Update. Appear to have SALSA working properly. We may have had an export/import issue as we have two parts of our team in two locations. After deleting the SALSA and Random Eyes components and then reattaching and then re auto linking things appear to be working just fine.

Question,
Is it normal/expected that there would be two RandomEyes 3D script components on the Autodesk character after a 1 click Autodesk setup?

 

mdrajeske,

I'm happy to hear it's working for you now. We're a two-man team in two separate locations as well.

The short answer of why there are two instances of RandomEyes3D on AutoDesk, and several of the other character generation systems that we support, is that all of these character generation systems structure there characters in very different ways.

The longer answer is that the RandomEyes3D instance that is set to [Use Custom Shapes Only] links to the body SkinnedMeshRenderer where the master list of blend shapes live. This instance is used to drive shape and group based facial expressions. The second instance doesn't link to a SkinnedMeshRenderer but instead uses the exposed processed output of the eye position, tracking, and blink information, along with the CM_AutodeskSync script, to control blend shapes and translate our blend shape-based eye position output to bone transform rotations, since these characters use bones for eye movement but blend shapes for eye lids. We also use the CM_AutodeskSync script to synchronize blend shape value changes across multiple SkinnedMeshRenderers.

Thanks,
Michael

 

Hello. Thanks for the asset. I don't really need lip syncing and all that witchcraft in my project but I wonder if you solved the problem of Unity not recognising external microphone on iOS devices? Thanks a lot!

 

Hi Stepan, sorry, we haven't tested that scenario, so can't offer any opinion or help.

 

We're proud to announce that SALSA now has over 1,000 customer. Thanks everyone! We have some great new features in development for the next release.

 

Hello

Congratulation for the 1000th customer!

One call for help:


This is a furball, a flying guy and sidekick of my Player.

I took a primitive sphere and painted some eyes on it and used a fur shader.
This will be replaced by a blender mesh later.

Now I want it to have a moving mouse, do you have a Little mesh with correct bone names or whatever I can use to add it to give it a salsa compatible mouth? There will be much fur on it so you won't see it much, only when it is moved while talking...

Thanks and have a nice Weekend!

 

Hello Firlefanz! Sorry, we don't have this type of configuration. But please feel free to use the example assets that come with SALSA Lip Sync as a basis or example for creating your character. The boxhead models were quickly built in Blender and may be a good place to start.

 

We've been playing around with SALSA in a multiplayer setup with real-time speech and lip-sync using the awesome Photon Voice asset from Exit Games. Darrin put together a great video tutorial to help you get started with this setup.

Enjoy!

Photon Voice
https://www.assetstore.unity3d.com/en/#!/content/45848

SALSA
https://www.assetstore.unity3d.com/en/#!/content/16944

 

SALSA lip-sync is today's 24hr sale, get it now for 25% off.
https://www.assetstore.unity3d.com/en/#!/content/16944

 

Hello,
"Supports Unity & Unity Pro. Tested on Windows, Mac, Linux, legacy Web-Player, Android, IOS, Windows Store, WP8."

what about WebGL support?

 

@Crazy-Minnow-Studio is there any way of using this with a model that doesn't have blendshapes (MakeHuman)?

 

SALSA relies on the underlying FMOD audio system. This is the audio system used in all Unity distribution platforms except WebGL, where Unity uses a modified version of the Web Audio API. Although we are investigating solutions here, SALSA is not currently supported on this platform.

 

There is a third party Blender plugin that makes blendshape export possible with MakeHuman characters, and we've been evaluating possible workflows. However, they are not currently supported.

https://thomasmakehuman.wordpress.com/mhx2-documentation/

The main problem is that there doesn't seem to be enough diversity in the shapes available to deliver convincing lip-sync approximation.

 

I was considering picking it up, but after a bit more research, we seem to need to get characters created in Maya for Blendshapes to be exported correctly. Will pick this up when I eventually get round to finding someone who can do character art. Thanks for your response though.

 

Blendshapes can be created in most modern 3D editors, including Blender. We have a three part Blender video series that demonstrates this.

Working With Shape Keys in Blender:
http://www.youtube.com/playlist?list=PLcVmXGedVLuahpG6bRniRryIc1QBz11Nk

 

Okay might take a look at that. Cheers

 

was one click removed I don't see it anymore.

 

Hi Vortavasail,

Our 1-click character setups, and all of our free add-on's, are available on our website.

https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/downloads/

Michael

 

Hi Minnows!

I have a custom Fuse figure (actually it is the base mesh imported as a custom figure, so presumably this happens with all custom-sculpted Fuse figures) where the teeth are not attached to the body. They are in the figure hierarchy as their own object like hair or clothes…. As a result they don't work with SALSA. The "lips" move but the teeth stay put.



The teeth do have some useable blendshapes…:

but the CM Fuse Sync script didn't recognize the teeth when I tried to add them as a facial hair/beard….

As a separate issue, I have another Fuse figure (one of the included figures) where the mustache only partially syncs, or possibly some blendshapes are not found or not linking up (maybe a name issue? or blendshape ID mis-match?) so when he talks with SALSA the mustache moves kinda funny like classic Yosemite Sam – which isn't necessarily a bad thing, if I could control it….
http://giphy.com/gifs/yosemite-sam-Imlvje19ljYxW

So, I'm seeing situations where I want more control over the SALSA setup:
1. synchronize blendshapes across multiple objects (teeth)
2. adjust the blendshapes individually (prop whiskers, furry eyebrows)

I'd love something like a Generic Sync script that would be figure agnostic…. Thanks for considering.

 

Hi wetcircuit,

We have an add-on on our downloads page called SalsaSync that has been designed to be character agnostic, it allows you to link any number of BlendShapes from multiple different SkinnedMeshRenderers. It's still in beta, but was just updated to 1.2 after some customer feedback. This is likely the best place to start and find out if it meets your needs.

As for the Yosemite Sam mustache twitching, this is unfortunately the same Adobe bug that you helped us uncover a while back, where Adobe is generating some of their new scanned models with missing BlendShapes. I reported this, but they unfortunately can't seem to be bothered with it. The SalsaSync script should work around this issue since you can map to whatever SMR's and BlendShapes you want.

Give it a try and let us know how it goes.

https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/downloads/

We are heavy in development on some very big changes that will increase the flexibility when targeting these types of character generation systems and bring lots of other new goodies to the table.

Michael

 

Hey! SALSA Sync works perfectly! Thank you!
Added bonus, the tongue is now animating, so it's better than before!

 

That's Great!

 

Hello,

before I had a static Scene with all gameobjects created in the unity Editor.
I had two uma in my Scene with current components:

cm_uma Basic
cm_uma Sync
cm_umaexpressions

and one script "TalkingBehavior" which stores audioclips and Plays them using salsa.

This worked perfect.

Now I created a prefab because I wanted my uma objects in the game dynamic spawned into the Scene with salsa possibilities.

When I now spawn an uma prefab, it says:

AvatarBuilder 'SALSA_UMATroll': Transform 'Root' parent 'UMA_SALSA_1' must be included in the HumanDescription Skeleton
UnityEngine.AvatarBuilder:BuildHumanAvatar(GameObject, HumanDescription)
UMA.UMAGeneratorBase:CreateAvatar(UMAData, UmaTPose) (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBase.cs:196)
UMA.UMAGeneratorBase:SetAvatar(UMAData, Animator) (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBase.cs:152)
UMA.UMAGeneratorBase:UpdateAvatar(UMAData) (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBase.cs:117)
UMA.UMAGeneratorBuiltin:UpdateUMABody(UMAData) (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBuiltin.cs:312)
UMA.UMAGeneratorBuiltin:HandleDirtyUpdate(UMAData) (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBuiltin.cs:230)
UMA.UMAGeneratorBuiltin:OnDirtyUpdate() (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBuiltin.cs:246)
UMA.UMAGeneratorBuiltin:Work() (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBuiltin.cs:124)
UMA.UMAGeneratorBuiltin:Update() (at Assets/Standard Assets/UMA/Core/Scripts/UMAGeneratorBuiltin.cs:89)

and the uma Looks strange, like two of them overlapping. when I remove the cm_ scripts it is fine again. any idea what I did wrong?


Thanks a lot and have nice winter Holidays

 

I think I got it working by changing some of the uma creating routines.

But now my umaData.Skeleton is alway null, although I can see the uma and it has a complete Body and all Slots and overlays working perfect, even with Animation and talk expressions....

 

Hi Firlefanz73,

Please reference the instructions on our UMA2 page and use the included [DesignTime_Single] example scene for comparison with your scene.

http://www.crazyminnowstudio.com/posts/uma-lipsync-using-salsa-with-randomeyes/

Thanks,
Michael

 

Hi,
Is there any hope that this will be?

 

Hi Alex-3D,

We would love for this to be a reality, unfortunately we have verified with Unity that it's very low on their priority list. This means the only possible path for now is one that works around the Unity API and directly with the browsers underlying implementation of Web Audio API, and we don't have a solution that is simple enough and reliable enough to be supported yet.

That was a long winded version of we can't promise anything with WebGL at this time.

Michael

 

Michael,
Understood...
Well, there is some questions:
- Do you plan add in the ACG support basic emotions?
- What about Freedom Gage Characters?
Thanks

 

Complex emotions are already supported in ACG using our RandomEyes3D API SetGroup function.
Search [SetGroup] on the link below.
http://www.crazyminnowstudio.com/un...randomeyes-manual/classes-methods-properties/

Thanks for the tip about the Freedom's Gate characters, I had never heard of them. I imported SALSA and the free Adam character into a scene, along with our free SalsaSync add-on that makes it easy to link SALSA to many different character systems, and they work great.

Salsa3D:
Leave the SkinnedMeshRenderer unlinked.

RandomEyes3D:
Leave the SkinnedMeshRenderer unlinked.

I used the following settings in SalsaSync:
SaySmall: Phoneme_T, 100
SayMedium: Phoneme_IH, 100
SayLarge: Phoneme_OW, 100

Blink:
EyeCloseL, 100
EyeCloseR, 100

Eye Control: Bones
Local up rotation axis: X_negative
Local right rotation axis: Y_positive
Left eye bone: mc_Ad_lEye
Right eye bone: mc_Ad_rEye

 

what's news in 1.4

 

All version release notes can be found here.

http://crazyminnowstudio.com/unity-3d/lip-sync-salsa/release-notes/

 

For those of you using Daz characters and having problems with the limited range of the Blendshapes in Unity (0-100), you can try this:
http://www.daz3d.com/forums/discussion/94781/change-morphs-to-new-slider-ranges

Basically, you create new morphs in Daz that are now 0-100 but visually represent the old morphs at any value of your choosing. That's what I've been doing and it does work. Kind of a pain because you have to more or less go through the same process for each morph (you could combine some of them, I suppose) However, my understanding is that changing the values outside of 0-100 in Unity has been discussed and rejected by the developers, so it beats holding your breath and waiting for changes to SetBlendShapeWeight()

 

Thanks for the tip!

 

Hello I'm having trouble getting my randomeyes to work properly my setup involves a Fuse based model and I have followed the tutorial "Unity3D Lip Sync - SALSA and Mixamo Fuse" so my setup is the same. The issue is that when I add the eyes to the 'CM_FuseSync.cs' and press play my models eyes roll back and look zombie like. Another thing to note is that my model is laying down.

 

Since this character was not a standard Fuse character, I'm glad we were able to resolve the issue using our SalsaSync add-on to remap off-axis eye bones to Unity's axis system.

 

is there a support for makehuman model in salsa ?

 

Hi oded,

We do not have a supported workflow for MakeHuman characters. We have experimented a bit, and we were able to retain BlendShapes using Manuel Bastioni's add-on, but the exported models have issues around the eyes and mouth once imported into Unity. Do you have a clean workflow for getting these models into Unity with rig and BlendShapes intact? If so, please share it and we'll revisit the prospect of a supported SALSA workflow.

thanks,
Michael

 

Hello,

when I want to hear my PCs say something, I use

salsa3D.Play();

Works good, moves the lips, great!

But I have the Problem, I want to have many different voices with only few "real voices".

So I found the "voice changer":
https://www.assetstore.unity3d.com/en/#!/content/54963

With it you can Change pitch and some other value when playing the voice or Sound.
That's really cool!

But how do I Combine it? With my 4 real voices, I good do some voice recording with some common sentences and Change them to fit my many npcs...

Or does salsa have some way how I can Change pitch and others stuff to make voices Sound different?

That would be absolutely great!

Thanks and have a nice Weekend!