SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hi,

Try replacing line 321 in CM_SalsaSync.cs:

From this:
if (syncIndex != -1)

To this:
if (syncIndex != -1 && syncIndex < otherSMRs[othrSmr].sharedMesh.blendShapeCount)

If that does the trick, we'll update the source package as well.

Thanks,
Michael

 

I am still seeing the same error….

That was line 323 for me – is it possible I have an old version of SalsaSync?

 

The current version of SalsaSync is 2.2.0. You can check your version in the [ReadMe.txt] file at the path below. Please grab the latest if you're not already on it, and try that line modification if you're still getting the error.

Assets\Crazy Minnow Studio\SALSA with RandomEyes\Addons\SalsaSync\ReadMe.txt

Thanks,
Michael

 

SALSA Lip Sync is on sale this week -- get it now with our biggest discount ever, 50% off!
https://www.assetstore.unity3d.com/en/#!/content/16944

 

Hi, Michael. I was using an older version, but after updating and editing that line I still get the error (now it references line 320)

Code (CSharp):

1. IndexOutOfRangeException: Array index is out of range.
2. CrazyMinnow.SALSA.Sync.CM_SalsaSync.LateUpdate () (at Assets/Crazy Minnow Studio/SALSA with RandomEyes/Addons/SalsaSync/CM_SalsaSync.cs:320)

 

Sorry that didn't resolve it, please send you model and detailed configuration screen shots to assetsupport@crazyminnow.com so we can have a look at the model, and try to replicate the error.

Thanks,
Michael

 

I sent the model. I think Mixamo may have tinkered with the blendshape names during their revamp in August. My new figures are labeled differently than before.... I PM'd you with a sample figure. Thanks for taking a look.

ANYONE SHOPPING FOR SALSA DURING THE SALE this is the best support in the asset store!

 

Our SalsaSync add-on has been updated to version 2.3.0. This release improves the ability deal with character BlendShape naming convention variations. Thanks for discovering the issue wetcircuit!

http://crazyminnowstudio.com/posts/salsasync-flexible-salsa-map-for-custom-characters/

 

There got to be something wrong with the salsa inspector. It's really annoying.

 

Hi dearamy,

I see you're on a Mac, is this a retina display? Please respond via email to assetsupport@crazyminnow.com, and include your SALSA invoice number.

Thanks,
Michael

 

Hi,

Is there any documentation out there on how to make a character happy, sad and angry? Is this part of random eyes?

 

Expressions are handled by GROUPS in RandomEyes:


The PREVIEW TRUE/FALSE interface is a little clunky, but essentially you build all your expressions here and then call them by name. Each blendshape in a group can have it's own timing so the expression can be a little dynamic, but when you call a Group it is full-on, in otherwords you can not activate a Group 50% – it will always be 100%, so I made several versions of each expression with stronger and weaker settings to give more variety.

I use the Playmaker addon (Set Group, Set Group Duration), API code is here near bottom of page: http://crazyminnowstudio.com/unity-...randomeyes-manual/classes-methods-properties/

 

RandomEyes code example are available here:
http://crazyminnowstudio.com/unity-3d/lip-sync-salsa/manuals/randomeyes-manual/code-examples/

In addition to the SetGroup API method being available as a custom action in PlayMaker as mentioned by wetcircuit above, it's also available in the other 3rd party system add-on's we offer: Adventure Creator, Cinema Director, Dialogue System for Unity, and NodeCanvas.

http://crazyminnowstudio.com/unity-3d/lip-sync-salsa/features/

Michael

 

Tnx, I'll try it.

 

Hi @Crazy-Minnow-Studio, I'm having fun with Salsa - the only quirk been so far that I'm importing models from MakeHuman: after some messing around, I got some blend-shapes to Unity; and using SalsaSync, I can get lip and eye movement. The issue is with blinking.
My models use a transform for eye-lids that are rotated, where sync only allows blend-shapes. As we have the jaw bone, is it possible to have blinking from bone transform as well?
thanks

 

Hello ranaUK,

We have tested MakeHuman in the past, but we were not able to identify a clean export/import procedure that maintained both the BlendShapes and proper material conversions. If you can provide a detailed export/import procedure from MakeHuman to Unity, we're happy to reevaluate MakeHuman to see if we can provide a clean and easy support path with SALSA.

Thanks,
Michael

 

Hi @Crazy-Minnow-Studio, to be honest, I didn’t do anything additional; I’m using beta makeHuman with exchange plugin that creates mhx2 (not at my desk:might be wrong about extension name); blender with makehuman import plugin (had to override settings and allow shape: ie blend shapes). Previously I use to rename bones from blender, but now I’m skipping that as well. Export as ascii fbx. In this workflow, I have all materials + textures. I used unity rig as skeleton.
I can give more details (like versions and so on) once I get back.
Rana

 

Any status update on SALSA 2.0?

 

SALSA 3D is pretty much good to go, and we're currently planning the 2D rewrite. A customer provided us with some very awesome 2D character art with tons of phonemes and facial expressions, and it's helping us determine the benefit of our new 3D techniques when applied to 2D characters. 2D is challenging because there are so many different ways you can setup characters, but we're making headway.

Michael

 

Sounds good. My project development start date got pushed out to Q1 2018 so it looks like this will all be ready by then. Thanks for the update.

 

@Crazy-Minnow-Studio
Hi Michael,

here is my current workflow for MakeHuman:

MakeHuman v1.1.1
- export mhx2 + meter + feet on ground

Blender v2.7.8
- Import mhx2 + Override Exported + Base Face Shapes + Masking Apply
- Export fbx

all textures are exported at mhx2 stage. In this configuration, I get some blendshapes (except blink)

 

Hi ranaUK,

Thanks for posting the info, I'm already getting better results than I have with any previous MakeHuman version, so it appears things are improving. Are you adding a rig on import (which one), or building your own? I tried a Daz rig, but they don't include control for eyelids.

Thanks,
Michael

 

please add "executeineditmode" option on SALSA2D and RANDOMEYES2D components. I use SLATE in edit mode only so I'm forced to animate the "executeineditmode" property to ON.
thanks.

also quick question:
when eyes are targeting an object, is random eyes functionality disabled, is this dependent on the target reference being empty?

thanks.

 

Hi Micheal, I’m using a rig displayed as unity rig in MakeHuman. It’s been a while, and I can’t remember if it came with MH or not - if it’s not available in your copy, I think I can legally upload it here.

 

Hi ranaUK, I think I figured it out. It appears that you likely exported from make human with the "default" or "default no toes" rig, both of these include facial bone rigs. Please send your character for comparison to assetsupport@crazyminnow.com, and also include your SALSA invoice number.

Michael

 

Hello luniac, I don't think "executeineditmode" will work with SALSA because of it's real-time nature, but I will look into it to verify.

Random eye movement is overridden by eye tracking, but the target affinity option allows interspersed random eye movement during eye tracking for the specified percentage of the time. You can think of target affinity as attention span. When humans converse or engage, they typically gaze upon a target and randomly look away by distraction or thought periodically, target affinity tries to replicate this natural behavior.

Michael

 

Hi Michael, I think there are some misunderstanding: my request was to extend salsaSync to allow bone based blinking. I do have facial bones, but no blend shape for eye lids.

 

Hi ranaUK, I understand your request. For us to be able to support MakeHuman (or any character system), with either modifications to SalaSync, or a custom add-on, we really need to define a clean workflow we can document and support. Our small team obviously cannot support every nuanced variation of characters available, so we really have to nail down specific procedures. Our desire is to have a procedure that leverages MakeHuman, and provides as much linkage to SALSA and RandomEyes as possible. If you can send the character you've generated to assetsupport@crazyminnow.com, along with your SALSA invoice number, we can compare your output to ours to make sure our procedure's output matches yours.

Thanks,
Michael

 

Done.

 

ok thanks for clarifying.

SALSA seems to work fine with SLATE in editor once I flag executeineditmode on the SALSA2D component.

 

We're very excited to announce SALSA support for MakeHuman characters using our free SalsaSync add-on. This free add-on allows you to create a custom map to link SALSA and RandomEyes output to MakeHuman exported characters using the workflow outlined in the link below. If you're not familiar with MakeHuman, check the website for more information. SalsaSync has been updated to v2.4.0 and includes bug fixes, and the addition of a bone-based eyelid option to support this workflow. There are multiple programs used to achieve this workflow, and they are all documented in the link below.

Big thanks to @ranaUK for discovering, and sharing, a clean export/import procedure from MakeHuman > Blender > Unity3D.

https://crazyminnowstudio.com/posts/using-makehuman-characters-with-salsa-lipsync/

 

I've just started using SALSA and I'm wondering whether or not this asset allows for head turning and if so, what's the best way to do it? I'm using both Morph3D. I have my models blinking which is great, but I haven't tried having mouthes move while talking yet. That's next after I get the head turning thing worked out.

 

Hello EDarkness,

Head tracking is not a feature in SALSA 1.x, though you may be able to use Unity's LookAt method in the Transform API.

https://docs.unity3d.com/ScriptReference/Transform.LookAt.html

Michael

 

Thanks for responding so fast. I already use the LookAt method, but I wanted something specifically for the head. I'll look for another asset for moving the head specifically.

 

Hi,

I'm trying to implement Salsa with RTVoice and Morph3D for my ios game. RTVoice was not working properly on my IOS 11 device and fixed it but couldn't fix Salsa problem. Randomeyes seems to be working fine but when I press talk but for example it is not lipsyncing. I used one click setup for m3d, am I missing something? Do I need to tune it for IOS devices?

Thanks.

 

Hello developerEdibelo,

SALSA works fine on iOS, but as with all of our supported platforms, it needs access to your dialogue audio to function.

Our previous RT-Voice add-on provided two use-case examples:

• (Salsa_RTVoice.cs) uses the RT-Voice Speaker.Speak method to generate an audio file and passes the output clip to SALSA for lip-sync processing.
• (Salsa_RTVoice_Native.cs) uses the RT-Voice Speaker.SpeakNative method along with the OnSpeakNativeCurrentViseme event. It passes viseme data to our Salsa_RTVoice_Native.cs script where the visemes are converted to SALSA mouth shapes.

Today we discovered that RT-Voice on iOS generates audio outside of the Unity engine on the operating system, and viseme data is not available, it provides current word information inside Unity. Given this, we have updated our RT-Voice add-on to include a third use-case example script:

• (Salsa_RTVoice_Native_iOS.cs) leverages the RT-Voice OnSpeakCurrentWord event to receive the current word, and passes the current word to our free TextSync text-to-lipsync asset for simulated lip-sync on a per word basis. This is not lip-sync approximation since we don't have access to the dialogue audio or the viseme information, but it's better than random movement since it's simulating each word. This means pauses and silence in the dialogue will be honored. Salsa_RTVoice_Native_iOS.cs depends on the separate TextSync asset so it is included as a Unity Asset Package file inside the RT-Voice add-on so that it does not generate errors if TextSync is not present.

RT-Voice add-on
https://crazyminnowstudio.com/posts/salsa-and-rt-voice-for-runtime-lipsynced-text-to-speech/

TextSync add-on
https://crazyminnowstudio.com/posts/salsa-text-to-lipsync-using-textsync/

Michael

 

Hello,

I am using salsa with uma for my game and it works pretty well.
But because I think it costed too much performance in the game with many uma, I removed the salsa components again from the game scene.

I still have it in cutscenes, the character editor etc. there it works perfect.

So I have a new idea, if I talk to an uma in the game:

I pause the scene and have a popup (something between the camera and the paused game scene).
In the popup I have two gameobjects: the uma head of the player and the other one I talk to.
Both heads only including the salsa components, so they can talk.
Did you already do something like that for a demo maybe?

Thanks

 

Hello Firlefanz73,

Both our UMA2 and UMA DCS workflows rely on the UMAExpressionPlayer component. As long as you can generate UMA's that have this component, they should work with SALSA regardless of the rest of the scenario.

Michael

 

Hello @Crazy-Minnow-Studio,

I'm using Salsa on a MCS Character. During the runtime, when the lips sync is running, I have this error :



I investigate and it seems that my blendshapes used for the lip sync are choose with a bad index.
I made some researches and I think that it could come from a problem in the LipSync source code : https://forum.unity.com/threads/rel...l-animation-tools.309324/page-12#post-3219085

Could you confirm it ? If yes, how to fix it ?

 

Hi,
I have installed the latest Adventure Creator and Salsa 3D.
I have followed your "Using SALSA Lipsync in Adventure Creator" tutorial and I can't find the "Actions" folder as suggested in the AC third party path: "Assets/Crazy Minnow Studio/SALSA with RandomEyes/Third Party Support/Adventure Creator/Actions"

When I try to have Salsa lipsync on AC Dialogue : Play speech it doesn't fire. Is that because the actions are missing?
I have installed all packages that are listed in the tutorial.

Attached is my folder structure when everything has been installed.

Thanks,
Kay

 

Hello JDTaupiac,

Please send your run-time script, along with your SALSA invoice number to assetsupport@crazyminnow.com. Also include the version numbers you are using for Unity, MCS, SALSA, and the MCS 1-click setup. I suspect it has something to do with activating additional BlendShapes after SALSA is already setup and the index order being effected, but we'll need to see what your run-time script is doing to be sure.

Thanks,
Michael

 

Hello Kay,

It sounds like you did not download and install our free Adventure Creator add-on. Please use the [Download Files] link at the bottom of our Adventure Creator support page to download our free Adventure Creator add-on, then resume the tutorial to link everything up.

SALSA Adventure Creator support page
http://www.crazyminnowstudio.com/posts/latest-version-of-adventure-creator-supports-salsa/

SALSA Downloads
https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/downloads/

Thanks,
Michael

 

Hi @Crazy-Minnow-Studio, didn't get to test out updated sync with makeHuman until today. I found the following problem: if I have "Use Custom Shapes Only" in "RandomEyes3D" as suggested in web-site, I'm not getting any eye movements or blinking. If I un-check it, it works fine. Btw, all other settings are as specified in your guide.

Request: RandomEyes > Custom Shapes > Current Custom Shape drop-down.... can we get it to work in editor mode?
It would help greatly to setup which random shapes to play? Question: Does the random takes group into account as well?

 

Hi ranaUK,

The workflow is designed to use two instances of RandomEyes3D, one with [Use Custom Shapes Only] disabled (used for eye control), and the other with [Use Custom Shape Only] enabled (for facial expressions). Then be sure to link the correct instances to the correct fields in SalsaSync.

I understand the desire to have an in-editor preview of custom shapes, but in SALSA 2.0 custom shape and group have been combined into one system with greater control and flexibility, as well as in-editor preview. This was much easier to achieve in 2.0 since it is a complete rewrite. Random custom shapes does not apply to groups in 1.x, but it does in 2.0.

Michael

 

Thanks Michael, that was it.
Blew right past that download..

Now all is working great!

- Kay

 

This is a pretty awesome product. I'm using with Fuse and had to make a quick code change to get facial hair to work correctly. Apparently there was previously a "." in the blend shape names that's no longer there. This was causing a spamming of errors and prevented expected animation. You can fix this by changing some code in CM_FuseSync.cs LateUpdate.

syncIndex = -1;
string shapeName = body.sharedMesh.GetBlendShapeName(i);
syncIndex = ShapeSearch(facialHair[fh], body.sharedMesh.GetBlendShapeName(i).Split(new char[] { '.' })[1]);
if (syncIndex != -1)
facialHair[fh].SetBlendShapeWeight(syncIndex, srcShapeWeight);

To:

syncIndex = -1;
string shapeName = body.sharedMesh.GetBlendShapeName(i);
syncIndex = ShapeSearch(facialHair[fh], body.sharedMesh.GetBlendShapeName(i));

if (syncIndex != -1)
facialHair[fh].SetBlendShapeWeight(syncIndex, srcShapeWeight);

Not sure if this is the proper fix, I'll leave that up to the devs

 

Thanks a lot Joe, we'll update FuseSync with a solution that will work with old and new characters and push an update soon.

Michael

 

Here are the changes we made to the new Fuse 1-click release v3.5.0. Thanks again for pointing out the issue.

https://crazyminnowstudio.com/posts/using-mixamo-fuse-character-creator-with-salsa-with-randomeyes/

Code (CSharp):

1. syncIndex = -1;
2.  
3. if (body.sharedMesh.GetBlendShapeName(i).Contains("."))
4. syncIndex = ShapeSearch(facialHair[fh], body.sharedMesh.GetBlendShapeName(i).Split(new char[] { '.' })[1]);
5. else
6. syncIndex = ShapeSearch(facialHair[fh], body.sharedMesh.GetBlendShapeName(i));
7.  
8. if (syncIndex != -1)
9. facialHair[fh].SetBlendShapeWeight(syncIndex, srcShapeWeight);

 

Hi Michael,
I have a little syncing problem with blinking eyes and blinking eyelashes.
I'm using Fuse CC and last SALSA.
When the eyes are closed the eye blink blandshape value is 100 but the closed eyelashes blandshape is 80 (to best match the eyes).
Seems more a Fuse issue than SALSA but i would like to know if there is a way to fix.
Thanks
Max

 

Hi Max,

Our SalsaSync add-on is a more manual, but also more flexible, version our character system specific linkage scripts. It can solve this by setting the percentage value of each blink BlendShape.

https://crazyminnowstudio.com/posts/salsasync-flexible-salsa-map-for-custom-characters/

Also, please send us the model you're testing with, along with your SALSA invoice number, and we'll take a look and see if other options are available.

Thanks,
Michael