SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hi Hormic, it works with all of our supported character workflows. Just install the base Cinema Director add-on along with SALSA and the Fuse workflow.

 

LOL yeah, but it's an awesome game, we're playing it to.

 

First i have to say that i'm really grateful with full appreciation to your work!!
It is amazing, i have initialized Adventure Creator with Salsa / Fusechars and Cinema Director and it works together like a charm, which give me opportunities i hadn't dreamed about before, because normally it is a difficult and very time consuming thing to have good behaving characters not only speakwise but also with realistic faceexpressions which is incredible.

But i have a little issue with assigning Salsa Speech and FaceExpression in Cinema Director to my Player character.
It works perfect with the NPCs. I think it is because they are all appearing int the Scene Hierachy, but the Player Character is not in the Scene Hierachy, it is placed through Adventure Creator.
I guess it is therefore that it don't work with the the R3_expression command in Cinema Director.
I don't know maybe i should ask this question in the AC thread, but do you have an idea what i can do about it?
Thank you very much, Salsa is great!

 

Hello Hormic,

You're right, the issue your experiencing is likely because the character is instantiated at run-time. The good news is that SALSA and RandomEyes work with run-time instantiation, but it does require additional setup like most actions performed at run-time. Please send an email to assetsupport@crazyminnow.com with a detailed description of your scene, what you're trying to do, and where it's currently failing. We will work through the issue via email and I will summarize the results here once we're finished.

 

Hi, just a heads up.... The new version of Mixamo Fuse was released by Adobe yesterday and appears to be free while it's a "preview" app, all I needed was an Adobe CC id to download, (I don't have a subscription)..., and the Mixamo site still works if you were a member before.... (FYI there are some improvements, but it's not radically different. The main issue is that they've scrapped almost all the Fuse 1.3 figures and have all new figures.)

I tested a new figure with SALSA and RandomEyes... Even the CM Fuse 1-click script, and the Mic script.... Everything seems to work (or at least it did last night, but Unity giving me issues this morning ). I think some of the mouth shapes are a little different (the teeth are moving on two of the blendshapes: mouth up, mouth down) and I've been adjusting the settings.... All the different figures supported by SALSA have their quirks, but at the moment Fuse is still my prêt-à-porter figure of choice.... (until I get more confident at building my own)....

 

Cool! Thanks for the update wetcircuit! Have you tried the DAZ models? For the cinematic work you do, they might present very nicely. Of course, it all depends on the look and feel you're going after. Thanks again for the update!

 

As promised, here is the outcome to @Hormic's questions about run-time characters in Cinema Director, our next Cinema Director update will include two additional Actions to help with run-time characters:

A new expression Action that only requires text input to trigger advanced expressions. (This will get exciting with the upcoming SALSA 1.4 release)

A generic Action found in:
[Director Group] -> [Global Track] -> [Crazy Minnow Studio] -> CM_FindRuntimeActor

The [CM_FindRuntimeActor] Action really has nothing to do with SALSA, but does provide a way to find and link a run-time actor so that a Cinema Director cut scene time-line can effect a run-time character.

 

Ugh, DAZ....
The photos are attractive, but they never show their products in the actual render engines they are selling to, including their own... and I don't want to buy every individual garter belt just to get one new hair, and 150 werebat-werewolf-wereork-werealien morphs aren't in the least bit interesting.... Anyway it will all go "on sale" 50 times a year, and blah blah, I am done with their closed ecosystem and marketeers™. They hold their own products back more than they offer, imho. *cough*greed*cough*

Fuse will not be free forever, but at the moment still more accessible and I can make my own clothes for it.... The new models are better – direct path to Photoshop is not the worst thing in the world (although to read people complain on Mixamo's website you'd think Adobe has murdered all their puppies, hahaha)....

Ultimately I just need to get back in Blender and fumble my way through adding my own blendshapes to a rigged figure. It's been so long I think they released several new versions since I last tried.... In some ways a ready-to-wear figure is a big crutch, not a convenience.

 

Hehe...all good points. Wasn't sure if you were familiar with DAZ, but sounds like you are very well versed in their ecosystem. LOL The DAZ models do show nicer in Unity (in my opinion) due to their better textures and nicer models (at the cost of high poly counts). But it's all relative and FUSE models are nice too.

-D

 

the SCARS...!

hahaha!

 

Hello MCS/DAZ and SALSA users!

We just posted a patch release for our DAZ/MCS workflow (3.4) for SALSA that fixes a bug that caused the wrong LOD to be mapped during 1-click setup on MCS characters.

http://crazyminnowstudio.com/posts/using-daz3d-characters-with-salsa-with-randomeyes/

 

Hi, still using Fuse CC here... and my new issue is that the beard mustaches don't follow either the randomeyes expressions or Salsa (well, maybe 1 or 2 with Salsa, possibly mouthopen or jawdown...).

I thought this might be a known issue (?) but scanning back through forum posts I'm not seeing anything specific. Any basics, that I might be missing? (Did this work with Mixamo Fuse? I don't think I've used the beards much...)

 

Beards and mustaches work correctly with Mixamo Fuse. Are you using a legacy model in Fuse CC, or one of the new scanned humans? It's possible that Adobe changed the BlendShapes for the new characters. I grabbed the Fuse CC trial, but haven't done thorough testing yet. If it's only a BlendShape change, a simple shape map change will fix it. I'll follow-up once I know more.

Michael

 

Using the standard male figure in CC "Fit A", and the 1-Click Fuse Setup.

 

The Adobe Fuse CC models have slightly botched up BlendShapes on the moustache objects. I say botched because they include [BrowsDown_Left] but not right, plus the other missing shapes below makes me think this was not intentional. One of the advantages of the Fuse Character Generator system over another system like Autodesk Character Generator, is that they maintained the same set of BlendShapes on all morph-able objects, and with the same indexes, which made synchronization super fast. I posted a question on their forum to see if they're aware of the issue and understand the performance repercussions of non-matching BlendShape lists. If they don't fix it, we can re-work the CM_FuseSync.cs script to address the problem, but it will come at a higher processor cost (similar to CM_AutodeskSync.cs). I'm going to wait to hear from Adobe before proceeding.

Missing shapes
BrowsDown_Right
BrowsOuterLower_Left
BrowsOuterLower_Right
Squint_Left
Squit_Right

 

According to an Adobe forum post, they changed the legacy models as well but it sounds like this was just the first pass.

 

Ahh, thank you! I guess it's to be expected. I've got some open bugs with them too....

If it was easy to make a universal figure we would be seeing more of them, I suppose.

 

Can you email your [Fit A] Fuse CC character to assetsupport@crazyminnow.com? I'm working with Adobe to identify the issue, and I want to verify your model produced the same issue as my test. It appears that the breakdown is happening on the mixamo export since Adobe confirmed the BlendShapes still exist in the base model.

Thanks,
Michael

 

I double checked and made a new FitA with the accessories.... Definitely the wrong morphs in the beard.
Sent you the email.
Thanks!

 

Our popular micInput add-on for SALSA has received some updates. The newest version is 1.5.0 and is available now for testing. We're keeping versions 1.0.7 & 1.0.8-beta available for a short while since they operate quite differently from the new version.

What's Changed:

• Taking advantage of features introduced in SALSA 1.3.3, we've reduced the micInput package down to a single component that works with either Salsa2D or Salsa3D. It is no longer necessary to specify a 2D or 3D component.
• Dependency on SALSA has been removed. The only dependency micInput has at this point is on an AudioSource [oh, and a microphone of course ]
• Adding the micInput component to a GameObject no longer auto-adds SALSA and AudioSource components (if none were present). This increases flexibility for the component with the ability to add it wherever it makes sense for your project. Adding micInput to the same GameObject as the AudioSource used by SALSA continues to provide very automated behavior.
• micInput is much more run-time friendly. If the AudioSource component is not specified, micInput will spin in the background (non-blocking), waiting for an AudioSource component to become available on the local gameObject. This behavior is super beneficial when being used on run-time GameObjects such as UMA2 characters.
• Since Unity5.2+, behavior of AudioSource.GetOutputData has changed. To better facilitate this behavior, MuteMicrophone (isMuted) is now defaulted to false.
• v1.0.8-beta introduced the ability to select a microphone. This behavior still exists and resolves some issues seen with OS X microphone hardware. If you're running Windows, the ability to use multiple microphones attached to different lip-sync models appears to work rather well. NOTE: OS X does NOT seem to like this multi-mic mode at all. Sorry OS X users.

Check out the micInput page for more information and enjoy microphone-real-time-lip-sync-input-coolness!

 

We've also updated the video tutorial for real-time microphone lip-sync input to bring it up-to-date with the latest version. Enjoy!

 

Do you think this is a Unity issue? I mean should I create a Unity bug report? Or is it an OSX thing...?

 

I honestly think it's an OS X thing, but admittedly, I'm not really an OS X guru. I honestly didn't think it was going to work in Windows and was very surprised when it did. What version of Unity did you test it on? Might be worthwhile to try it on 5.3 and see if there's any difference. I believe I tested on 5.2.2 or 5.2.3 (will have to check).

 

A quick google and it seems this is an osX thing related to core audio...?
https://support.apple.com/en-us/HT202000

 

Ahhh, yes. It certainly looks like it. And unfortunately, an aggregate device would not serve the purpose you or I are looking for in this instance.

 

Below are the blend shape discrepancies I found on the MaleScan3 CC model, and your FitAMale CC model, when compared to a 1.3 model. You auto-rigged a five finger rig, and I auto-rigged a three finger rig. We both ended up with missing, but different, blend shapes. According to Adobe, these blend shapes are added during the mixamo auto-rigging process. What format did you export to, straight FBX or FBX for Unity? I did FBX for Unity.

MaleScan3

Missing BlendShapes

• Moustaches
• BrowsDown_Right
• BrowsOuterLower_Left
• BrowsOuterLower_Right
• Squint_Left
• Squint_Right

​

FitAMale

Missing BlendShapes

Moustaches

• Blink_Right
• BrowsDown_Right
• BrowsIn_Left
• BrowsOuterLower_Left
• BrowsOuterLower_Right
• Squint_Left
• Squint_Right

Beards

• BrowsDown_Left
• BrowsDown_Right
• BrowsIn_Left
• BrowsIn_Right
• BrowsOuterLower_Left
• BrowsOuterLower_Right
• BrowsUp_Left
• BrowsUp_Right
• Squint_Left
• Squint_Right

​

 

FBX for Unity.

 

Can you email your Fuse CC *.fuse file to assetsupport@crazyminnow.com? You might want to zip it first so our email system doesn't eat the file.

 

sent!

 

Starting to identify a pattern now. The issue appears to exist in the individual objects. I have confirmed this by using these objects (beards and moustaches) on different base characters with the same outcome. I've notified Adobe.

Alpha Handlebar moustache produces these missing blend shapes

• BrowsDown_Right
• BrowsOuterLower_Left
• BrowsOuterLower_Right
• Squint_Left
• Squint_Right

Alpha Stardard moustache produces these missing blend shapes

• Blink_Right
• BrowsDown_Right
• BrowsIn_Left
• BrowsOuterLower_Left
• BrowsOuterLower_Right
• Squint_Left
• Squint_Right

Alpha Thick Goatee beard produces these missing blend shapes

• BrowsDown_Left
• BrowsDown_Right
• BrowsIn_Left
• BrowsIn_Right
• BrowsOuterLower_Left
• BrowsOuterLower_Right
• BrowsUp_Left
• BrowsUp_Right
• Squint_Left
• Squint_Right

 

wow... Adobe owes you a paycheck... and I owe you beer and cake!

 

Mmmm...beer and cake!

 

Hey, Crazyminnow! I love SALSA and want to use it for my new game series, but it's not supported by WebGL? Have you found a workaround for this yet?

 

I managed to find this: http://forum.unity3d.com/threads/webgl-lipsyncing.328728/. Apparently a workaround for "GetSpectrumData". Any way to implement this into Salsa2d?

 

Hi nosajtevol,

We have not found a solution yet, so thanks a lot for the link, we'll definitely look into it.

FYI, I'm doing the Ludum Dare 34 game jam this weekend, so I won't be able to dig into this until after Monday.

Michael

 

SALSA with RandomEyes 1.4 has been submitted to the asset store.

The big new feature in 1.4 is Shape Groups. Groups allow you to combine multiple BlendShapes to create complex facial expressions right in the editor. We've created the following two new API methods to activate these expressions:

Code (CSharp):

1. // Activate a shape group by name for a specified duration
2. public void SetGroup(string groupName, float duration);
3. // Activate or deactivate a shape group by name
4. public void SetGroup(string groupName, bool status);

We're thrilled that the developer of Dialogue System for Unity will be releasing an update to extend access to the new shape group feature that should be released shortly after 1.4 goes live.

We're also very excited to add NodeCanvas to our growing list of integration options. With the release of SALSA with RandomEyes 1.4 and NodeCanvas 2.4, you can enjoy native SALSA lip-sync control with the [Say] and [Multiple Choice] dialogue Nodes in NodeCanvas's DialogueTree Graph. If you're not familiar with NodeCanvas, it's a superb behavior designer and state machine from ParadoxNotion. In addition to the native support for lip-sync, we're offering a free NodeCanvas add-on with over 30 custom nodes to expose deep control over complex facial expressions, settings, and more.

Read the release notes for 1.4.

We had to reorganized the folder structure a bit in 1.4 to make way for other exciting projects on the horizon. We removed the [Documentation] folder since the documentation is now online, and moved the [Examples] folder under [SALSA with RandomEyes]. Additionally, we compressed all example audio to ogg format and re-linked all the sample scenes to the new audio. We re-created the Ethan character with more BlendShapes to better demonstrate shape groups, and compressed some images to reduce the overall size of the package. Your best option for upgrading is to delete the [Documentation] and [Examples] folders prior to upgrading. Make sure you haven't saved any of your work in these folders.

 

SALSA with RandomEyes 1.4 now available!

New NodeCanvas add-on now available. Grab the download and check out the new video tutorial demonstrating SALSA lipsync with NodeCanvas Dialogue Trees.

http://crazyminnowstudio.com/posts/nodecanvas-actions-for-salsa-with-randomeyes/

 

New Cinema Director add-on 1.1 has been posted. This update includes two new actions for working with our new Groups feature in RandomEyes 1.4. Groups allow you to combine multiple Shapes to create complex multi-BlendShape facial expressions, and activate them using only the group name (e.g. "smile"). The new Cinema Director Actions are:

Re3D_SetGroup (Activate/deactivate a group by name (e.g. "smile") (true/false))
Re3D_SetGroupDuration (Activate a group by name (e.g. "smile") for a specified duration (1.5 seconds))

http://crazyminnowstudio.com/posts/salsa-and-randomeyes-actions-for-cinema-director/

 

Thank you, very nice additions, the group feature is absolutly outstanding!!
Now when this functionallity is also available with Adventure Creator its just amazingly perfect. ;-)

I have a question about Look Target:
How can i activate and deactivate the look target functionality?
As long as there is no generalized method to work within the Advenure Creator Action List, i guess it must be assigned and turned on and off via scripting.
Maybe it is also possible to assign it from the dropdownmenu as with NodeCanvas or Cinema Director when it will work with Adventure Creator?

 

Thanks, we're glad you like the new features. Adventure Creator actions are on my next to-do list, and should presumably provide similar capabilities to our PlayMaker, Cinema Director, and NodeCanvas actions.

Here is an example of setting a LookTarget in code for now.

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using CrazyMinnow.SALSA; // Include the SALSA namespace
4.  
5. public class RE_SetLookTarget : MonoBehaviour
6. {
7.     public RandomEyes3D randomEyes; // Public reference to the RandomEyes3D component to want to control
8.     public GameObject lookTarget; // Public refernce to the GameObject you want to track
9.     public KeyCode setKey = KeyCode.Space; // This key is a stand-in for whatever event trigger you want to set the look target. (e.g. collider, key stroke, etc)
10.    
11.     void Update ()
12.     {
13.         if (Input.GetKeyDown(setKey)) // When the set key is pressed, set the look target
14.         {
15.             randomEyes.SetLookTarget(lookTarget);
16.         }
17.     }
18. }

 

Thank you very much! That's awesome, looking forward to the Adventure Creator Integration!

 

About 2 months ago I had asked Justin from Opsive about integrating SALSA into Behavior Designer and he was planning to add it in the last released version. However, he wrote in early December and said that you were actually making the integration as a SALSA add-on and that good progress was being made. I wanted to stop in here and see what the status of that integration is.

 

Thanks for the inquiry Magique! That is correct! We're currently working on that integration. I don't have an exact release date to give you, but it should be pretty soon. The actions got sidelined for a brief moment to work on a couple of other projects. If you are interested, I can contact you when they're ready for a beta peek.

Thanks!
Darrin

 

Sounds good. I would love to give it a try when you have something to test. We are just starting to piece together the Player an NPC components of our game and SALSA is an integral part along with Behavior Designer. Being able to have behaviors trigger an NPC to speak something is going to be very cool.

 

Yes, that will be very cool! I'll try to spin some cycles up this weekend to get it further along. I'll start up an email thread with you for testing - hopefully very soon.

 

I am having an issue using Salsa and randomeyes with a MCS character...

Setting it up and using salsa with MSC Female as described in your demo video works 100%... but if I use the MCs morphs to customise the look of my character (which is what MCS is all about) as soon as I hit play - the face restores itself back to the default MCS character face.

Am I missing something here? Please help.

 

Hello CplMulder,

It appears that our custom shapes controller in RandomEyes3D is interfering with the BlendShape values being set by the M3D Character Manger. We're looking into this deeper to confirm. If you're using our 1-Click setup, then two instances of RandomEyes3D are added to the character, one controls the eye movement, and the other controls facial expressions. You can identify the instance that's controlling facial expressions because the option [Use Custom Shapes Only] is checked. As a temporary work-around to the issue, you can disable the facial expression RandomEyes3D component instance. This will restore eye control and lip-sync while we look into the facial expression issue. Sorry about the inconvenience but thanks for bringing this to our attention.

Michael

 

So there are a couple of ways to use RandomEyes3D for facial expressions on these characters without any interruption to the M3D Character Manger morphing. It depends on how you want to handle facial expression. First a little explanation of what RandomEyes3D offers:

• Eye movement and blinking (Manual or random.)
• Target tracking (Character looks at a target.)
• Shapes (Keeps reference to all BlendShapes you want to be able to expose to our API, animate, and optionally provides random facial twitches and movement.)
• Groups (Build complex, multi-shape, facial expressions that are exposed through our API. These are groups of Shapes from the previous bullet point.)

By default the 1-Click script links all available BlendShapes, sets all but a few brow shapes to [!Rand] which means not random (exclude from random selection), and enables [Random Custom Shapes]. Any shape that is not in override (active) is smoothly iterated back to zero, and any shape that is not excluded from random selection will be randomly activated using the Random Custom Shape settings. This is what is happening to your morph. For example, if I set my character to be 90% hob goblin in the M3D Character Manger, but the [Genesis2Male_FHMHobgoblin] shape is linked in RandomEyes3D and is not in override, then the shape will be smoothly iterated back to zero. If you don't want to manage this shape through RandomEyes3D, you can just remove it from the Shapes list. In fact, the MCS characters contain many shapes that I would probably remove from RandomEyes3D management since they are not likely to be used at run-time.

A performance warning about the editor. Unity does not handle large editor lists very well, so it's best to collapse these lists when your not actually editing anything in them. Collapsing the list prevents it from being rendered in the editor, drastically improving performance for very large lists. The editor performance has no effect in a build, since all editor code is automatically excluded.

Groups are made up of one or more Shapes, and can be activated using one of two different API methods:

Public Group functions – (Not required, but can be used to enhance your character with complex multi-BlendShape facial expressions)

Activate or deactivate a group of BlendShapes.

Code (CSharp):

1. RandomEyes3D.SetGroup(string groupName, bool status)

Activate a group of BlendShapes for a specified duration.

Code (CSharp):

1. RandomEyes3D.SetGroup(string groupName, float duration)

Classes, methods, and properties are documented here:
http://crazyminnowstudio.com/projec...randomeyes-manual/classes-methods-properties/

 

Thanks a million for your reply... I removed the character's LOD0 mesh from the randomeyes component (which was added there by 1-click) and now it is all working 100% - awesome!

 

Hi there. I've got a problem with DAZ integration and eye movements. Apparently RandomEyes does not recognize the look left-right-up-down morphs from DAZ genesis3 and genesis2. Perhaps I am missing something, I really cannot get it to work.
Browsing Genesis morphs I see a morph for looking up-down, and another for side by side movement. They accept values from -100 to 100, with zero as the center position. So 2 controllers while randomEyes wants 4.
Then when I import the DAZ figure into unity, moving these blendshapes sliders in unity inspector reacts only to positive values, so no looking left or down.
RandomEyes with 1 click DAZ integration cannot find the right morphs so it captures in "eye shape index" the first five morphs of the figure and.. does nothing.
I'm sure it must work, but I really don't know how!
Any clues for me?
Thanks!