SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Hi Phil,

Unfortunately, we are a small two man team with no free time. Maybe someone on this forum will respond with interest.

Good luck with your project!

 

Dude, the Adam character has no mouth…



I'll sell you a 5min clip of Adam's head bobbing around and you can just add whatever voiceovers you need.

 

I think he's talking about this one:


Just don't forget that these aren't free to use for commercial things. Not even if you put it on your website and market your stuff. You can use them strictly for personal. Which means nobody is able to make you stuff only you. Especially if you pay for it.

https://assetstore.unity.com/packages/templates/packs/adam-ep2-ep3-character-pack-115793

 

Ahh…! But that doesn't look like his cryogenics logo...

I would suggest Adobe Fuse because there's a 1-click setup with SALSA, it's very easy (and free)..., but I don't think there's a Linux version.

 

He was talking about his face. When I opened his website I immediately thought about this character when I saw his third photo there.

Edit:
@phil_unity464 I think you would be better off to advertise this job on the https://connect.unity.com/ you probably would find someone who can help you out. But please, read back about the usage of these Adam characters.

 

Good call!
How are the visemes on that figure? Anyone used it?

 

I was talking about this one:

 

Right - this would be non-profit but large-scale use - I will contact them to see if it is possible.

Thanks.

 

I will have a look - yes, not having a Linux version is a pain . .

Thanks,
Phil.

 

Yep, will have a look at that link too.

Thanks.

 

@wetcircuit: LOL, we both were wrong then.

@phil_unity464 Ok, we misunderstood you, the Adam name was used by Unity for one of their demo, so we automatically assumed that you want to use one of their characters. But apparently it's not the case, sorry about that!

 

To use Freedom's Gate characters with SALSA, we offer our SalsaSync custom character mapping add-on.

https://crazyminnowstudio.com/posts/salsa-lipsync-with-freedoms-gate-characters/

https://crazyminnowstudio.com/posts/salsasync-flexible-salsa-map-for-custom-characters/

 

OK, sounds good - now is there anyone here who could do the work for me?

Thanks,
Phil.

 

I have created an account and put up a first post summarising what I am looking for and I see there is an option to post a task but that asks for dollar hourly rates etc and I have no idea what would be fair / attractive enough etc - where could I get an idea about that sort of information?

Thanks,
Phil.

 

No worries - I should have been clearer - it didn't occur to me there might be more than one Adam character being talked about here!

 

I honestly think you should consider learning this yourself, especially if it will be your voice.

A figure talking at the camera with SALSA was one of my first projects learning Unity. You just need a figure (which you've found), Unity for Linux, and screen recording software. Follow the Crazy Minnow tutorials to set up the SALSA component. They even have a live-mic add-on if you want to do a livestream.

 

Maybe you are right - I will at least start looking at it and see how I go . . thanks!

 

OK, I have downloaded and successfully installed the Linux version (the only hiccup was I had to do a manual licencing).

Well I was going to use my voice - if I can reduce the nasallity a bit with Audacity or something . .

OK, but I need to eat before I get too absorbed . . but out of interest, how long did it take you to do what you wanted? Was it about the same degree of difficulty as what I am trying to do?

Yes, that is also of interest thanks!

 

I had worked with traditional 3D software before Unity, so Unity felt like a weird learning curve, but basic concepts were familiar… Like positioning a camera in 3D space, pointing a spotlight… This sort of thing has real life equivalents so it's not too hard, just time spent decyphering the Unity settings.

I don't write code, and I floundered getting anything much to "happen" for a while – that's one reason SALSA worked for me, code wasn't needed, just audio clips. SALSA can be set up to autoplay the audioclip when you start the scene.

Hating my voice has never gone away, LOL. That's just something you build up a tolerance for.

 

Right - I have a very small amount of experience with Blender - and I think that is a very steep learning curve . .

Right . . sounds good . . I used to write code but I don't really know how useful that will be for this exercise yet . .

I haven't had time to get back to it yet but I am getting more enthusiastic about the idea of being able to do this stuff myself. If I can make progress, I don't mind paying for more stuff . . I _am_ short of time though . .

Wow! I know I am a newbie but I really like that! . . and it had me smiling with the amusing stuff all the way through . .

Hmm . . I may end up having to pay someone yet . . but we will see . .

Hmm . . I am definitely much worse off than you are - I quite liked your voice and it suited your avatar I think.

Thanks very much for the detailed info - I really appreciate it!

Phil.

 

IBM Watson has a Unity SDK that you can use to have a Speech-to-Text service in your Unity project.

While I am in this scene, I right-click on another scene (where I have implemented Salsa with Live Microphone Input) and select: Open Scene Additive.

When I run this combo scene, I get an error that basically means the microphone cannot process input. I have checked, via debug messages, to confirm that the microphone is active and records, but it seems that the same microphone cannot handle the input for use by both Watson and Salsa.

Has anyone done this before? Is it the case that my laptop's only microphone cannot be used/shared between 2 different scripts that are both recording and receiving live audio input?

 

Hi, I found in your documentation that salsa basically just use open/close mouth ( SaySmall, SayMedium, SayLarge) and no visemes. Is it correct? If so, is there chance Salsa 2.0 will support different visemes?

 

SALSA 1 uses 3 visemes SaySmall, SayMedium, SayLarge. You customize what the bendshape (or combos) are for each figure.

I set mine as: SaySmall "ahh", SayMedium "woo", SayLarge "eee".

There's a great book on face animation called Stop Staring. It shows 2 systems to animate visemes: a set of 4 (including mouth closed) for "toon" characters, and the 10 Preston Blair phonemes for realistic figures. SALSA 1 is essentially the simpler "toon" system with fewer visemes.

Teaser videos of SALSA 2 show a lot more visemes (7 or 8...?). It will not be limited to just 3.

 

Thanks for response! Bit simplified Preston Blair set is basically what I created for my models (as I grouped E to C/D/etc and U to Q/W). 7 visemes for Salsa 2 seems cool though, I guess I ll just wait for release and try it.

 

Fildax, as wetcircuit mentioned, there are no logical limits on expressions that can be used with SALSAv2. However, there are practical limits based on trigger variation -- spacing out the trigger groups over the range (although we provide some tools to make that super easy). Please be aware that SALSA does NOT process or map phoneme sounds. It uses its audio analysis to fire expressions, creating a dynamic lip-sync mix. SALSAv2 builds on v1's mechanism and uses some new techniques and algorithms to create a much more dynamic real-time presentation while greatly increasing the perceived accuracy of synchronization.

 

Any news about Salsa V2?
I REALLY want it!

 

You will be supporting the IPhone X Face tracking ?

 

Hi,

All SALSA 2.0 updates are posted here.
https://crazyminnowstudio.com/posts/salsa-v2-devblog/

 

Hello,

We don't currently have plans to support iPhone X Face tracking, but we're always evaluating widely adoptable ways to improve.

Thanks!

 

Hello! I can't get the NodeCanvas Add-On to work. None of the scripts in it will compile with the latest versions of Salsa and NodeCanvas.

 

Hi @Crazy-Minnow-Studio
I'm a long term owner of the Salsa w/ RandomEyes package. I know that we can just throw an audio file at it and it will do its work pretty good.
Now I'm planning a desktop application (game) where I value the CPU cycles more than I value the memory, which means I would rather bake the phonemes than do it live when the system first meet with the audio. Obviously I would like to bake out specific audio files which will be played in the final product. It's all about reducing the CPU cycles used by the audio-processing and pay for it with a bit more memory.
Is it possible to do this with the current package? If not, is it possible in the 2.0 version?

 

Hello SPACE-ASU,

Please send your SALSA invoice number to assetsupport@crazyminnow.com. Sorry you're experiencing an issue, we'll look into this and get back to you soon.

Michael

 

Hi SPACE-ASU,

It looks like the NodeCanvas package had some name space changes, we're working on an update and will post it as soon as it's ready.

Michael

 

Our NodeCanvas add-on has been updated to v1.3.0 to remove deprecated references to [AgentType]. It was tested with SALSA 1.5.5 and NodeCanvas v2.9.0 on Unity 2018.2.6f1.

https://crazyminnowstudio.com/posts/nodecanvas-actions-for-salsa-with-randomeyes/

 

@Crazy-Minnow-Studio You are awesome! Thank you!

 

Hi Lurking-Ninja,

We do not support, and do not plan to support, audio baking in SALSA. However, if you're a coder, it would be easy enough to cook up your own solutions for this. I'm not sure if it would actually be any more efficient than just using SALSA. Here are two high level examples of roughly how it could be done.

I'm offering this information as is and without follow-up support. It's simply a topic that is outside the scope of SALSA support and beyond what our small team can offer.

• Method #1 - Use SALSA.average to drive blendshapes from recorded curve data.

• Setup a SALSA character, and for each dialogue audio you want to record, write the SALSA.average value and a time stamp to whatever data container you use.
  
• When playing back, feed the recorded average back into SALSA.average at the appropriate timestamp interval to make sure the curve data matches the audio playback play head. Make sure the playback SALSA instance is not linked to your AudioSource or it will still perform real-time processing.

​

• Method #2 - Use blendshape values from recorded curve data.

• Setup a SALSA character, and for each dialogue audio you want to record, write the blendshape index, value, and timestamp of each mouth shape to whatever data container you use.
• When playing back, feed the blendshape index, value and timestamp back into the SkinnedMeshRenderer directly. With this approach you have to manage much more data, but you wouldn't need SALSA at all for playback.

Michael​

 

Got it! Thank you, I will play around with it, will see if I can manage something more efficient than the real time audio processing. I really appreciate your support!

 

Hi there, this looks very interesting as I'm using mixamo fuse characters. I'm wondering if this will work on a character that is using one of the mixamo animations (ie idle) ? So for example, the character is idling / dancing etc but can also be made to talk at the same time? I ask cos the videos seem to show T-pose not animating characters. thanks!

 

Hi m0ds84,

Thanks for your interest. SALSA looks great with animations. The only step that is sometimes required, depending on what data in contained in your animation file, is that you may need to un-link the jaw bone from your avatar. Use the following steps:

• Select the character in the project folder.
• In the Inspector, on the Rig tab, select the Configure button.
• On the Mapping tab, select the Head, and un-link the jaw bone.
• Click the Done button.

 

Thanks for the reply! Sounds that it will work as I need it to then, so I'll be sure to pick this up very soon. Have a nice xmas!

 

Hi again! I have now bought it and out the box it's working very nicely with my mixamo characters, thanks, good job! I have one further question, as my game is 90% complete so the workflow of SALSA was not part of the process before.

I use Cinema Director, and this seems an excellent way to use it, however this limits it to 2D sound? My game is 3D worldspace and at the moment to ensure 3D sound (voices) projecting from their characters, the audio files are attached to gameobjects within their object in the scene (or another gameobject at their 3D location in the scene, so when the audio file plays, it is in 3D space, so you hear it to the left/right as you turn etc).

As such, I trigger gameobjects via Cinema Director (or Timed Object Activators) rather than placing audio in Cinema Director because that always plays it as 2D sound.

Is there a way to call their gameobject sound/voice file connected to an object? An alternative is that the SetAudioClip simply allowed me to mute it in Cinema Director's "Cutscene" (as in Director I am telling it to Enable the sound file in the scene object, so could just line up/sync that track to the actor track) but it doesn't seem possible to mute the actor track file (so SALSA effectively still reads the audio but doesn't play it out loud) - cos then I have my 3D sound play and also SALSA plays a 2nd audio clip over the top (in 2D). I understand it needs to read the audio file to process it, but would imagine there's a way for it to respect 3D spatial sound settings?

Basically, for closeups it's great, cos 2D sound there is normal, but for just general walking around, NPCs on the sidewalk that say something in spatial 3D, where voice files are specifically set to 3D locations, how do I get SALSA to play nice with that setup? I hope that makes sense, if not I'll try to explain further. But solving this one step will have it working 100% in my game and will get a 5 star rating, thank you!

Example:

Cutscene
Director Global Track 10 Seconds: Enable gameobject SpeechFile01 <--- attached to object in the scene, to ensure it is a 3D sound

Actor Track 10 Seconds: Salsa3D_SetAudioClip <--- will play a 2D variant over the top of SpeechFile01 3D object being played

As mentioned, one solution is that Actor Track simply plays a muted version so it reads the audio but doesn't play it back in engine. Maybe there's a setting like that? Another solution would perhaps be that the line of speech object, contained in a 3D space gameobject, can trigger a SALSA action somehow, ie when it's enabled? thanks for your help!

 

Hi m0ds84,

Thanks for being a SALSA customer. Are you using our Cinema Director Actions?
https://crazyminnowstudio.com/posts/salsa-and-randomeyes-actions-for-cinema-director/

Assuming you you have an AudioSource on your characters, and have added a SALSA setup to any where you want to add eye control or lip-sync, you can use our Salsa3D_SetAudioClip Action on an Actor Track Group that is assigned to your character. The Salsa3D_SetuAudioClip Action will set the AudioSource.clip to the AudioClip selected and optionally play that clip so that SALSA (which should already have a reference to the associated AudioSource) can perform lip-sync against the dialogue clip. In this case, SALSA will use your AudioSource as is set (2D or 3D). One thing to note is that SALSA 1.x only processes audio that is audible to the Audio Listener. If a player is far away and the Audio Listener can barely pickup the sound, then SALSA can barely hear it to process also. It's possible to work around this issue by giving SALSA it's own dedicated AudioSource, setting it's audio to 2D, and piping it's audio output to a mixer group with it's attenuation to -80, then basically play both clips at the same time, the Audio Listener will hear the 3D sound and SALSA will have a strong 2D signal for processing. This likely isn't an issue if your talkers are nearby.

If I missed what your trying to do, I apologize, please explain in more detail. You can send us an email at assetsupport@crazyminnow.com.

Michael

 

Hi again Michael, thanks for the quick response! Yes, you've answered pretty much what I was asking, thanks! After posting it occurred to me, all I need to do is set the AudioSource on the Fuse character that SALSA creates on it to 3D spatial sound (as you mention), and bingo, it's doing what I need it to do! I'll keep in mind what you say about moving out of audio range, that will be fine. So consider my question answered and that problem solved. Thanks for the further explanation, that will certainly help me should I need to fiddle around further into that aspect, but for now, everything is working perfectly and very well. I never thought I'd see those damned fuse characters move their mouths! So thanks for delivering a fine asset, I will leave a rating and review very soon. Now, I'll leave you alone so you can enjoy your day, haha! Thanks again, regards, Mods

 

Great! Let us know if you have additional questions.

 

To anyone using SALSA with Amplitude on WebGL

After a customer reported errors in 2018.3, we have confirmed that Unity introduced a regression BlendShape bug that will produce the following errors when trying to use SALSA and Amplitude on the WebGL platform:

"Platform does not support compute shaders"
"d3d11: failed to create buffer (target 0x1c mode 0 size 5360) since compute isn't supported at this feature level"

For now, please use 2018.2 or earlier if you are using SALSA with Amplitude on WebGL, and vote on the following issue.

https://issuetracker.unity3d.com/is...with-blendshapes-throws-the-error-and-warning

Thanks!

 

Can this plug-in be applied to any character with a blendshape-based face, or the lips should meet some requirements? I have obtained a face model that was created by a studio with high-quality scanning, etc... There are 12 blendshapes that I can manipulate; either manually by changing the array values from 0 to 100 or by applying a plug-in called Blendshape-Preset-Tool. So, I am wondering if I can also simply apply the SALSA plug-in to this character? Or perhaps if I wanted to have someone scanned in such a studio, what should I tell them to do with the character face, so when they prepare the face model, I can just apply SALSA to it? Thanks

 

Hi,

Thanks for your interest. SALSA 1.x is designed to work with the smallest blendshape set possible while still achieving a dynamically compelling approximation result. See the first page of the SALSA and RandomEyes manuals for recommendation details.

SALSA
https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/manuals/salsa-manual/

RandomEyes
https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/manuals/randomeyes-manual/

If your character doesn't have blendshapes to sufficiently match these recommendations directly, you can also use our free SalsaSync add-on which acts as a custom mapping solution between our recommendations and several different character designs.

SalsaSync
https://crazyminnowstudio.com/posts/salsasync-flexible-salsa-map-for-custom-characters/

 

Hi Crazy Minnow!
How is Salsa v2 going?
I read the devBlog but there is no info since 2 weeks ago and you PROMISED it would be done by the end of this month!
(Just kidding )

 

Hey @atomicjoe! I actually updated the post late last night. We're hoping just a couple of weeks more, maybe...NO PROMISES! LOL Seriously tho, we are really close. Mike is punching through the Eyes todo's with incredible speed and I'm trying to get all of the other things ready -- namely the add-ons and documentation. We have had a lot of SALSA v1 support requests over the past month that have slowed us down, but not halted us -- not making excuses, just facts -- 'cause there were NO PROMISES!

Since you brought it up and we are just about there, we want to remind everyone of our pricing plans for release. This is straight from the devBlog..."As we mentioned near the beginning of this development blog/log, we will be increasing the price of SALSA Lip-Sync v2 upon release. The expected release price will be $39. Shortly after release, the price will bump up. Also note, this will be a paid upgrade. As a thank you to those that have supported version 1, we will be offering a price-difference upgrade at launch, so you can continue to purchase version 1 now and upgrade upon release and not pay any extra over waiting for the new version. While the new-purchase price is $39, the upgrade price will be $4. Once the full price changes from $39, the upgrade price will increase and will no longer adhere to the price-difference model. So, get your upgrade quickly!"

Thanks,
Darrin

 

Hurry up! My game is bland and needs some salsa!