SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Website: CrazyMinnowStudio.com
Product site: SALSA LipSync Suite
Support email: assetsupport@crazyminnow.com
Youtube: videos

Version 2.0.0
Available Now: Asset Store
Release Notes

Works with:

• Adventure Creator
• Autodesk Character Generator (1-Click Setup)
• Cinema Director
• Cinematic Sequencer - SLATE
• DAZ3D (1-Click Setup)
• Dialogue System for Unity
• Fuse Character Creator (1-Click Setup)
• Freedom's Gate Characters
• iClone (1-Click Setup - beta)
• Master Audio - playlists for SALSA lip-sync input
• NodeCanvas
• PhotonVoice (multiplayer lipsync)
• PlayMaker
• Real-Time Microphone Lip-Sync
• RT-Voice (real-time text-to-speech)
• TextSync (text-to-lipsync)
• UMA 2 (1-Click Setup)

Includes two comprehensive products to simplify lipsync, eye movement, and facial expressions:

SALSA LipSync
Simple Automated Lip Sync Approximation provides high quality, language-agnostic, lip sync approximation for your 2D and 3D characters. Audio dialog files are processed in real-time to automate the lip sync process. Results are superior to basic jaw bone animations, without labor intensive phoneme mapping.

Custom component inspectors provide code-free configurable options, while our simple API provides code accessibility from any Unity supported scripting language.

Supports Unity & Unity Pro. Tested on Windows, Mac, Linux, legacy Web-Player, Android, IOS, and UWA.

 

Some questions before I need to sign up for your forums:

When you say you support morph targets from Blender, Maya, Maya LT, Modo, etc is there a means of assigning the proper morph targets in the FBX model to your SW control or does you SW search and find the morph targets because they all the same names and identifying information?

I'll have morph targets exported both from Blender and Maya LT. No problem?

Do you have pictures of how we should do our morph targets (shape keys / blend shapes) do have minimal fuss using your SW?

Thanks.

 

Can it synch lip motion by text ?

If there is no voice available for a dialog for exaple

 

Goat,

SALSA achieves quality results using four defined mouth positions: sayRest, saySmall, sayMedium, sayBig (these shapes are outlined in the forums and covered in detail in the manual). When you import your FBX with these BlendShapes into a Unity scene, Unity adds a Skinned Mesh Renderer component to you models GameObject. All of your BlendShapes are in this component. For 3D characters, you add our components to the same GameObject as the Skinned Mesh Renderer. SALSA lets you link to these four shapes by entering their index number into the corresponding index field in our custom inspector. These four indexes and an audio clip are all that's required for SALSA to work.

RandomEyes works the same way, but uses five positions: lookUp, lookDown, lookLeft, lookRight, and blink.

As for morph target support, as long as you can see the BlendShapes in the Skinned Mesh Renderer component after importing into a Unity scene, then SALSA and RandomEyes can link to those indexes. The order of the indexes doesn't matter, since you can set the index values in our custom inspectors.

I hope I answered all of your questions.

 

nasos_333,

SALSA performs lip sync approximation by analyzing audio only.
Sorry, no text analysis.

 

Hello Goat!
Here's a little more clarification on some of your questions. To minimize your workflow efforts with SALSA, the component will automatically assign indexes for shape keys as follows (matching the mouth positions identified in the screenshot of our previous reply):
- sayRest: 0
- saySmall: 1
- sayMedium: 2
- sayBig: 3

Likewise, RandomEyes automatically assigns indexes 4, 5, 6, 7, and 8 as:
- LookUp: 4
- LookDown: 5
- LookLeft: 6
- LookRight: 7
- Blink: 8

Keep in mind, these automatic assignments are easily changed to match your model information. We'll try to post up some more details and screenshots later today.

There also is no issue with using exports from Blender and MayaLT. They should work together just fine.

NOTE: When working with BlendShapes (at least with Blender), make sure to apply any modifiers *prior* to creating your BlendShapes. Additionally, modifying the mesh (adding/removing verticies) after creating your shapes, generally results in broken BlendShapes.

 

This looks really good! I was looking for another solution instead of the pricey FaceFX.

A couple of questions; can you create additional morph targets to achieve more accurate results?

Can this also be setup with using text-to-speech from this post? If you can get your system to work with that I'm all over it!

 

This does indeed look a lot lottt cheaper than the big name alternatives, like faceshift and such, and a lot less hassle - might not be accurate but 4 blendshapes instead of 1-2 dozen is so much easier and lighter and in this case, a hell of a lot cheaper. I'm planning to use daz/poser models as the art is getting excellent and very flexible, so i'm assuming i do my decimation in daz studio rather than whatever i'm importing it to. I suppose i could always test.

 

Thanks. I'll be buying.

The 4 lip syncs is good if you are not learning lip reading. This is an old project on Blendswap that uses all the phenomes of speak (InfameDavid) or there is, I think it's called targarela to base off of.

The other question I have need clarification for although I think you answered it above is when I make Blender models (I am still learning Maya) I often use subdivision to help guide me into using good topology however I leave subdivide unapplied when exporting as FBX because it makes the polygons too high. So to work with Salsa I will need to leave subdivision either turned off or delete that modified before exporting as an FBX.

In the future I'd like to see a whole gamut of emotions expressed such as the attached picture shows but honestly I'll have to create all the blendshapes and see how big that would make the file. It might make it too big complex for mobile.

 

Hi Goat,
Yes, SALSA's main goal is to provide great results with as little effort as possible. Making all of the phoneme shapes necessary for super-realistic mapping is very daunting and is not required of most game projects. We're trying to keep it simple but awesome!

As for creating your BlendShapes in Blender. Our experience is all modifiers (that modify geometry) need to be applied *prior* to creating *any* BlendShapes. The reason being, BlendShapes are very fussy about vertex counts. Even one vertex add/subtract usually prohibits the ability to add or change existing shapes. And most likely will prevent the BlendShapes from working properly, if at all, once exported to Unity.

Emotions/expressions are something that we are toying with for the future.

 

Thanks. You are right, I forgot the power of 'control-Z'. I've only created blend shapes with Blender so far and haven't tried them in Unity so your advice is helpful. I look forward to your emotion extensions.

 

Hey adamz!
Currently, SALSA only supports up to 4 shapes. You can use fewer shapes, but the max is four.

As for the text-to-speech option (which is very cool, btw), the tts module would need to produce an actual audio object that could then be linked into SALSA. After taking a brief look at this tts solution, it doesn't appear to do this; therefore it wouldn't work as an integrated solution with SALSA in its current form. We'd have to take a look and see if it's possible to link up with this sort of dynamic output. Obviously, using tts output files does work since there is an actual audio object to link to.

 

Hi lazygunn!
I'm only mildly familiar with the capabilities of Daz, but decimation is one of those processes that you could really use any tool to perform. The biggest considerations would be quality and output format. Blender certainly has decimation and re-topo features as do most full modeling packages. Of note to keep in mind is keeping vertex count/instance static while working with BlendShapes. While using Blender, any changes to vertex counts or instances (i.e. adding a vertex and removing another) will break or cause lots of problems with BlendShapes...pretty much creating a need to start over... So, creating BlendShapes should be the last step in the workflow pipeline.

 

lazygunn needs to know that DAZ FBX exporter isn't exporting the morphs of all the old Poser CR2 files I licensed. I haven't licensed all the DAZ owned content (as that's another $500) so I haven't tried Genesis morphs as an FBX export but I don't think they are supported either. You need MegaFiers/MegaDAZ ($150/$30) to import the morphs. MegaDAZ does not yet support Poser CR2s as he is working on that support but it does support the newer DAZ Genesis format.

Blender does a good job of decimating DAZ/Poser characters too although if you'll notice it's the areas that have the morphs that tend to come out the worse in decimation (DAZ Decimator or Blender Decimator).

Process for the Genesis is documented at west-racing.com and on his YouTube channel.

 

Awesome goat!
Thanks for the update! Automated decimation is never a perfect science. It's certainly getting better and better. For Blender, it is possible to take some control in the decimation process on where/what is decimated.
1. In edit mode, select the vertices that you want decimated, use CTRL+G to assign the selection to a group.
2. Add a Decimate modifier to the mesh and select the vertex group to apply the modifier to.
3. When the modifier is applied, it will operate only on the selected vertices.

 

Thanks for that. I'll have to look more closely at DAZ and see if it has similar capabilities.

 

I've updated the original post with some new overview information and images to help everyone get an idea of how the system works and what it's requirements are. Bear in mind that this is still just an overview, the manuals and example code go much deeper.

 

@Goat.

 

goat, genesis models export their blendshapes just fine with studio's fbx exporter, you just need to set up the morph export rules correctly, i havent triied the older poser formats yet. In the case of setting up the rules, for me it has been the case for me to set a match for anything based on the character you're using (your base fbm) and perhaps anything else youre using as specifically as you can and it will export the genesis model and the items its wearing perfectly fine, the bottom rule 'anything else' will have everything not specified baked and not available as a blendshape

I did actually buy mega daz only a few days ago not realising i didn't actually need it, i've contacted chris west and see if he's amenable to a refund as it would greatly aid in buying another of his products that id find extremely useful.

You have to be careful what you specify as the rules in the morph export however, keeping it as minimal as possible, or the model can end up extremely heavy and will crash something like max, and sometimes screw up the rigging too. On the other hand it's also been very amenable to putting on several layers of clothes to a figure and binding them all to the genesis skeleton in the exporter, which obviously makes the model heavier but means you can very simply swap out clothes as needed. If you wish to customise the entire model with blendshapes, you can, just add them as rules, but i'v found changing the absolute width or height of the figure will have it mismatch its skeleton

The export, i'll add, is perfect for things like this btw, as you can supply the shapes asked for easily - but you can also export full sets of phonemes or visemes if you like. Main thing ive noticed that dont export properly (or at all) are mcms which help non-standard fbms express properly cause their dimensions are different humans

To prove my point, however, here's the ribbit character in Unity, exported using the Daz Studio fbx exporter:

https://dl.dropboxusercontent.com/u/12598787/Showcase/Toady1.jpg


Sorry for the slight derail minnow, but i hope this information helps people in future

 

Not at all a derail lazygunn, BlendShape support in Unity only started in 4.3.3, so any information about BlendShape workflows is right on the rails!

 

I'm prevaricating on having voice acting in my game, at least 'real' voices, was maybe thinking more kinda simlish or the random gibbery nintendo style, but having a really strong manager for expressions would be brilliant, particularly with the workflow i mentioned above - the daz fbx exporter can easily supply various specified expressions as morph export rules too and being able to blend those in and maybe create triggerable macros and wotnot to automate them would be a godsend

Maybe out of your remit but to me, a cert buy, without a doubt - i think a formula of: create an expression definition, add the blendshapes for the expression(s) and their blending amount each, there being more than one blendshape possible for 'custom' expressions that the speech blendshapes could be modified by (to so some percent?), and that triggerable.. heaven

Supplying the blendshapes is made extremely easily by daz studio and daz3d's art used to be laughable and intractible for games, but now you can use it even as an in-game body customisation system with blendshapes it seems worth plugging, perfect marriage with something like this. I'd imagine you COULD update to include visemes etc but i neednt see the exact reason, if you add customisable expressions that are additive then you can modify the speech blendshapes by script or if youre feeling brave, some anim editor or curves then you can blend in lip movement and everything else to the speech without messing with the core system.

That is just an idea however, chances are with the next 'payday' as such coming along, it's a buy, i just do hope you keep updating this software as it offers a great deal in a really nice straightforward way

 

We have been discussing expressions outside the existing four speech shapes since the beginning, and we have a design plan that should allow an unlimited number of expressions that function independent of the speech shapes.

SALSA for 3D, and RandomEyes for 2D/3D already expose [Blend Speed] and [Range of Motion] options, so any expression system we would build would likely include similar type of control. We have worked very hard to make the system as easy to use as possible, with the smallest amount of requirements, while still exposing powerful programmatic control over it. We even built and included the basic dialog system that is used in the online demo scene so we could run the system through its paces and demonstrate one approach to a real-world implementation.

This doesn't get you integrated expressions today, but I hope it shows our commitment to the product.

 

Thanks for the reply, sounds perfectly fine! Facial animation may be a bit further on for me anyways, hence i havent got this yet, but even now it wouldn't be too hard to make a little thing that did what i described myself, it would just be a bit of a bodge job and only suit my project. So no worries, nothing expected this instant, but i'll probs grab this sometime soon anyways just to make development more fun if nothing else (mute characters with little animation to no animation while communicating isnt very enthusing to work with). Cheers

 

We decided to have a little fun with our second promo video, let us know what you think of the video.

 

Looks good. I have a simple game that needs these.

 

I think Chris West's MegaDAZ is needed because typically you need to decimate the character, export without the morphs, and then use MegaDAZ to import the needed morphs. As I need MegaDAZ to import Poser CR2s not the newer DAZ Genesis format I haven't tested this.

If I were to PM you could you take a look at a comple of older CR2 format files for me? I'll put them in a drop box. As of you, I've not seen an model in the asset store with blend shapes :-(

Thanks.

 

Well I have a simple game that needs the expressions more than the speech for now so maybe I can 'lie' and send expression blend shapes instead of speech blend shapes until you have finished.

...but that requires me getting CR2 morphs imported into an fbx and working....

 

I've got a bit of v4 and m4 stuff, no worries, i'll test them out now, but i'd genuinely ask you to consider Genesis instead, it's a notable move upwards in flexibility and quality. Still the usual gamut of blow up doll costumes for the lonely people but a lot of really strong art and lovely lovely blendshapes. Where i post the results is up to you but it's something that's worth being common knowledge right now (particularly any differences between cr2 and the Genesis format)

To the thread author, don't worry, not off-topic, as far as I can tell, Daz is easily the best place to get decent affordable stock with blendshapes, modelling them can be a pain and generally far more expensive in money-hours

 

Oh my are toon style characters. The licenses were expensive in total, over $500...they were licenses directly from the artist not DAZ. I'll PM you the link. I have an ideal for a game that I can program in one week (not counting the time to release it) I want to make.

 

Bloody hell, you money to spare! aha, no i understand when you've invested a lot and it proves to be a pain in the arse

I did own megadaz, im in the process of a refund as chris wests stuff is great (i have megafiers and megashapes), but mega daz is pretty irrelevant for at least genesis stuff now. PM me I guess, i'll look into it

 

The link I sent cost less to license then Genesis, but not much.

 

You should think about mileage! There's toon morphs for genesis that are pretty affordable.. although then again some aspects of these morphs i dont know how to export, like the mcms (morph controlling morphs) i have no idea how to link up in max etc so some very exagerated full body morphs relying on normal person shaped morphs through an mcm look a bit odd

 

I think you misunderstand how DAZ Game licensing works. Those DAZ prices only entitle you to use those characters as 2D sprites (very doable for some characters) and not as 3D meshes included in a released game. And certainly not for resale as a standalone mesh. To use DAZ in a game it costs $500. Of course you don't need to pay for the license until you see if you can actually get them to work and release the game to the public, something I should have done with what I licensed.

Example: There is a Chibi Girl Boy that I really like but that would be $500 to DAZ for their license (the Genesis base mesh) + whatever the man that make them wants for his male female. He might not even want to license his characters outside of thee 2D DAZ advertising print world.

What I need to figure out is out to access the morphs from the CR2 once exported from DAZ Studio as an FBX that I licensed as that has plenty of cartoon characters I can generate from the morphs...perhaps it's time I try to give the UMA base meshes a cartoon look by re-texturing them.

 

@goat
What you are describing would potentially be possible using RandomEyes. It does not require an audio file to be linked in and does have independent, programmable access to call each slot. You would have to call them by the intended action tho. So, if you loaded a smile into the lookUp slot, you'd have to call it using the lookUp designation. That would give you 4 slots to load, 5 if you count the blink slot, but it is a timed slot.

 

Thanks, I think I could load blank audio files too. I'll buy something this week if I can figure out how to get morphs exported with the CR2 in the FBXes otherwise it'll be another week or two while I get another model source...

 

SALSA may be more difficult to work with in that respect since it's main goal is to analyze a waveform and map to the appropriate shape key. RandomEyes is definitely the one to use for this. In fact, the new custom shape key system will be added to RandomEyes, since they share a lot of the same underpinnings. FWIW, making shape keys in Blender is pretty easy. I've started working on a video tutorial today, demonstrating how the Boxhead shape keys were put together for the asset. Will prob have it completed in another day or two. If you're just making expression shapes for toon characters, they should be pretty easy to do. I guess that's all relative, depending how many models you're using and how many expressions they should have.

 

Thanks then I'll use Random Eyes....and if the CR2 don't export nice I'll edit some meshes in Blender in the next week or two...

 

Short story: Poser stuff decimates and exports just fine via Daz Studio.

Longer: You'll need to clean the model up somewhere, Blender's probably fine, just Daz models always seem to be broken up into a ton of mat ids that need condensing and fixing the object names would make it nicer to work with too

Unless the clothing you're using has the same blendshapes as your base mesh, it wont follow your base mesh, so in a case like that have your body shape set in daz studio before export. The Genesis stuff i've used has always been pretty flexible with this but what i worked with in poser's formats wasn't so with the clothing i gave it. This might be a rare case though.

In short though, if you're motivated by the idea of an asset like this, it' should be very functional with the kind of products you'll find at www.daz3d.com and similar so if the licenses dont scare you and the idea of making your own blendshapes does, you have a huge resource there. And randomeyes implementing expression blendshapes (or any blendshape on the model) sounds great, especially if there's some nice level of control. You could count me in.

 

Thanks a lot for your help, as it so happens the clothing, hair, and accessories do conform and there is a decent size library too. I do want to learn to model and I have somewhat but I want to use what's I've licensed as well. They can be used in different niche games.

Thanks again.

 

Feature request time! Is it possible that instead of using the 4 positions you be able to supply a good set of visemes instead? I don't know how more complicated or less complicated that makes things for you but the visemes are quite easy to get with daz's Genesis model system and seem comprehensive

 

Hey lazygunn,
You could export visemes that you like, that roughly correspond to the recommended mouth shapes. Keep in mind, SALSA is *not* a phoneme mapping system. It's goal is to provide a great look without mapping complexity. Exporting the viseme morph targets that correspond to the recommendations sounds like a great way to make the simple workflow even simpler.

 

You're quite right I suppose! I've set myself a mission for the whatever amount of time i'm up for and this is a bit of a good test bed for one part of it. Fingers crossed paypal likes me should I come to it

 

Hey all! We posted a quick-tip on our devBlog, explaining how to use SALSA with fewer mouth shapes if desired.
You can check it out here!

 

Hey Everyone! We've got another QUICK-TIP posted on our devBlog today. This post is the perfect follow-up to yesterday's QT. Limit RandomEyes movement to specific quadrants. Great for keeping natural, random movement while limiting the general "look" direction.
You can check it out on our devBlog!

 

I'll make no bones about it, i'm strongly influenced by the game in development, Cradle, the interactions in this video sum up much of what i'd like from a system https://www.youtube.com/watch?v=2aFjdYxoCfs

With adding extra visemes to the selection i'd have more control over lip and jaw movement inherent to the sound being made, hardly expecting some massive support for this kind of thing but id wonder if you might consider broadening the allowed amount of shapes used during the process in future development, if possible with how you produce the movement. I'm very close to purchasing this but I do know there are free tools coming that suit this kind of thing, its just that will be later and i'd quite like to start learning now, so i guess i'm just wondering what you have on your roadmap, if you have one

Another feature for SALSA that would be very useful would be the option to 'modify' one mouth shape with another shape, maybe at a random point within a set range of the modifying shape each time the mouth makes the necessary normal shape. This would mean that you could add individual personality and characteristics to how the character talks, visually, say with a sneer or something like that, i'd probably also use it to group similar visemes together and when the mouth was in a particular shape, modify it to express slightly more variety. If this could be set programatically that could be great, as I could add tags as markup in any dialogue system I used for text on screen to prompt the expression. This might be a nice way to add expressions full stop, so long as you have audio as a cue. Or if one were to generate audio for quite a surreal effect (Which is something i'm considering), it could follow the written dialogue much more closely and expressions commonly added in the text as tags or markup or however you wish to define it. Theres a free unity tool for mapping expressions to parts of the audio which works on adding keys to audio, i guess having several options in how to implement things is always good

I'm going to investigate the system I intend to use for dialogue anyways and see how that takes care of it, or if the option is there to introduce these kind of cues

The next suggestion was very much aimed at random eyes, your examples are amusing and show the system works, i think with refinement you could get some very nice subtle effects it however, especially if the model has eyelid blendshapes. The Cradle video is a very nice example of how personality is being conveyed through body language and eye movement, the two characters you see having very different character. To me it seems your system at the moment could well get as nuanced as that and much like in the video and the tip you just gave, while not confining the look to a certain direction, weighting it towards particular points - say you had a grid and white vertices were strong weights for where someone might look and they got darker as they lose priority, This would be a great way to procedurally convey a variety of mood and character, and would, in the video seen, show a great replication of how strong that looks. The girl's weighting is a broad spread focussed on looking down and to the side (shy) with a very small but strong spread in the center - sudden looks towards you then back down again.

The old guy just keeps looking at you, you get a really particular idea of what kind of person he is just because he looks right at you (Eager, genuine, very self assured)

Then I guess you could modify weighting based on changing moods or development of characters, it would be fascinating to me to have characters whose body language and facial expression significantly convey how the events of the game have changed their demeanour, and that be done completely procedurally (FinalIK would be great for the overall body movement)

I'm maybe going way out of your original intended scope or you already have this functionality in there, but what i've suggested seems natural progression from what you already have, and as i'm feeling i dont have a lot of choice, currently in this area, your asset seems the most likely buy, if it were to continue to be developed that would be great, body language and facial expressions are very much undercooked in games and you're already sat on the foundations of something potentially pretty powerful there

I'm going to go and have a look at your website and a few other things to see what my requirements are exactly, i'm currently going through a very long process of sorting out the visuals for some characters who could well use this system, the first you can't see her eyes so i'd be interested in how salsa works with her, then the next will my pal mr frog if possible and i'd love to see how randomeyes does with him.

If you do have any kind of roadmap be sure to share it and if you dont think my suggestions quite fit into the aim of this particular product, or i'm being ignorant and it's all there already, then no worries, but it seems like you have a good couple of products here that are somewhat interlinked. I think if I was writing dialogue i would very much like to hook them both into it, but as said, something like the level of nuance of Cradle would be my direction ultimately

If that's a bunch more work, double your price, put Pro after the name and if you make it i'm sure folk can make some compelling implementations

 

I should add your system looks good and can no doubt get lipsync and facial animation up and running, with broad abilities to do as I describe. However I am interested in a system with a deal more granularity for a good deal of control and very particular nuanced effects, although the grid idea just came off the top of my head.

 

Much of what you're describing can be achieved to varying degrees in the current 1.0 release, and additional capabilities are being added now for our upcoming 1.1 release. I'll explain more about the new features below, but first I'll give some detail on current capabilities. Salsa3D and RandomEyes3D both use a fixed number of index slots that link to the applicable indexes of the referenced skinned mesh renderer. You are free to swap these indexes out programmatically if you wish, but our speech and look systems are designed to get the most accurate speech and broadest range of eye movement with the lowest number of Blendshapes, and without the need to manually map phoneme shapes. This is how we provide high quality with minimal effort.

We also designed the system to be programmatically accessible, with great documentation and tons of examples. For users who want to influence the results, we expose functions and properties to adjust shape change triggers, blend speed, range of motion, and a SALSA event system that fires off details about the Salsa3D instance, talk status, talker name, and clip name, at the beginning and end of each clip so you can programmatically respond to a dialog sequence. Here are the inspectors for Salsa3D and RandomEyes3D with some descriptions.





So you can have multiple sets of mouth shapes and you can dynamically swap them out as a response to a talk begin or talk end event, add in the ability to control blend speed and range of motion, and you get a lot of control. With RandomEyes, you can also control range of motion, the max interval between random eye look positions, the max interval between random blinks, blink duration, you can toggle between random and programmatically controlled look positions, and random and programmatically controlled blinking.

Here's a teaser shot of what's coming down the pipe for RandomEyes3D in the next update.



The fixed number of shapes used for speech and eye movement are an intentional design decision so that we can ensure automated quality with minimal fuss. The independently controlled unlimited custom shapes, with individual shape controls, coming to RandomEyes3D in the next update should fill in any gaps for users looking to implement highly emotionally expressive characters.

We are currently working on a video tutorial series, quick-tip blog posts, and the 1.1 update. For the latest information visit our website:
CrazyMinnowStudio.com

 

That's looking really good there, the ability to add any number of your own shapes into the system is great. I suppose the final thing i'd like would be to have a way of controlling randomeyes during an audio clip. I'm not that familiar with the animation editor in Unity but would it be straightforward to have a curve linked to the id of a randomeyes blendshape and use it to drive the shape over time? In terms of syncing it with salsa, an image and timeline of the audio would be handy, but with a few curves you could make it as simple or complex as you want - you have the basic 4 shape system, which is easy to understand, as is randomeyes, if i have the choice to do the above its quite straightforward, i would think, to add as much nuance as one would like, without the core idea being further complicated whatsoever. Well, lets see if my bank does still like me afterall, and maybe i'll look at putting in the curve thing myself if it comes to it

The improbable happened and my bank was agreeable afterall. I'm doing quite a lot at the moment to get my whole character making workflow (i hate that word) in order, but i should be able to have a go with this in the next few days, as it's now part of it. Do you have a general idea of when the update will land and if possible a suggestion of how challenging the curve thing might be to implement - im not alien to programming but i'm not particularly adept either

 

@lazygunn - That's great! We really appreciate your purchase! RandomEyes can be fully controlled programmatically, but currently isn't linked to audio or SALSA directly. However, there is a broadcast system that can be used to trigger actions between the components. For example, once a character begins talking, a msg is sent to RandomEyes and triggers a raiseEyebrows shape (v1.1 - although could still be implemented with v1.0 if you weren't using the eyes).

RandomEyes shapes can be triggered and given a speed and range of motion. Shapes can be triggered on and off. And (in v1.1), RandomEyes is getting broadcast events. In the current release (v1.0), the range-of-motion and speed can be programmatically set, so you could drive progression of a shape in that manner. However, as a caveat, the range-of-motion is clamped and setting the position of the shape via the range will immediately return the shape to the max range. For example, you could set the range to 0, then change it to 70 and the shape will progress to 70%. However, if you then set the range back down to 20, the shape will immediately return to 20% and will not smoothly animate down to that level.

As for how difficult implementation will be will be greatly determined by how complex you want your control to be. At their core, SALSA and RandomEyes are designed to give really nice output with very little complex programming required (outside of the programmer triggering the components when they want them to activate). We've got some pretty good examples included in the product that are completely documented.

v1.1 is going to greatly expand capabilities, it should be ready very soon -- finishing up code base and bug-checking now. Without putting an actual date on it, maybe a week or two unless we run into problems or decide to include more functionality. And we're already working on enhancement possibilities for v1.2. We don't have a roadmap per se since much of what is currently being added is based on user feedback and requests and our own imaginations.

 

Hmm, well the curve idea isnt really adding to the complexity, which is why i suggested it as a feature, it's something to uncover if you want more, and would pretty much make it for me. Is there any reason why the shape snaps back? I understood the diagrams i'm a grown up (you don't need italics either, just saying), and i like that the range is clamped because the feasible ranges can be large but i'd typically want them to end at 100, i'll try to explain exactly what I mean by controlling the randomeyes shapes this way, and i suppose there doesnt need to be any explicit link between the audio file and randomeyes, only that a randomeyes blend value be controlled by a curve over time

Well, having just looked at the docs and fiddled with the editor, it seems it'll probably be faster to just do it myself for now. But to explain in case you consider adding a far better implemented solution than i'll manage because i can guarantee the following explanation could surely be wrapped up in a more elegant manner that fitted in smoothly with the system, if you have a gameobject (maybe the skinned mesh itself) with a script on it with say, a public array of blendweights, the array index correspond to a blendshape on the skinnedmeshrenderer, opening the animation window will let you add a curve which can control any public values on the gameobject. Add a curve initially to create an animation and for it and further curves add for each blendshape you want to change by the blendweight in the array, add the animclip to an array in the script that matches the clips to the audio, attach the playing of the animation to the playing of the corresponding audio file, salsa does its thing, randomeyes does its thing and extra expressions during the lip sync be blended with these curves

Clunky as hell as I have it there but it makes sense to keep randomeyes as procedural as possible. Really so long as everything's exposed then i dunno why i'm going on about it! My one feature request now is to have the above in an elegant form, but i'll survive i think