[WIP] Cheshire Lip-Sync

Should the discussion of this tool continue in the asset forum? It would get a lot more attention there. Regardless of that, you're awesome, and I look forward to the update (even if it doesn't include automatic animation generation).

 

Oh, it's going to. That's one of the biggest draws for Cheshire, I would never leave that feature out. I'm throwing in support for programs like Papagayo as a bonus. The automatic timing detection and generation of animations is going to be front-and-center in the new update.

As to the discussion of the tool, I'm fine keeping my little progress updates here in the WIP forum. It is, after all, a WIP that I'm discussing. There's already a thread in the Assets forum about Cheshire, and I will start posting in that once I'm close to releasing. I'm not all that concerned about attention at the moment. Once I have the animation generation and playback working, I will start generating some example videos. Nothing gets people hyped like seeing the end results in action.

 

Still at it. Haven't posted much recently because there hasn't been much to show. I've been chipping away at things steadily, about an hour a day. I would have gotten further but my past two weekends haven't been very productive.

I've mainly been working on the Blend-Shape phoneme-set module. I decided to re-configure it a little to make it a bit more simple and user-friendly. I'm pretty pleased with the results so far. I also nailed down the interface for actually changing the values for individual blend-shape phonemes. And I rigged it up for adding and removing phonemes from each individual phoneme-shape. (for the multiple-blend shape approach) I just have to put in the code for refining the list of potential blend-shape names to the ones that haven't already been selected. I thought up a way to pull this off last night before I went to bed, so I don't think it will take very long.

With any luck, I'll be testing the creation and playback of animations sometime over the weekend. I don't have any major plans this weekend, so I should have a bit more time.

 

Were you able to get some good work done over the weekend? Though I'm not replying to every update, I am eagerly following your progress.

 

I didn't get as far as I would have liked. I ended up helping my parents out with some chores around their house on Sunday. However, it was not a lazy weekend, and I did actually get quite a bit of solid work done on Saturday. I finished up coding on the blend shape set module. I restructured it to just use a List of blend shape items, instead of a having a separate set of blend shape items for using just a basic 10-shape Preston-Blair set. And I was able to work out the interface in a satisfactory fashion, taking into account the various bugs that would have cropped up.

Now I've got a component that can be dropped on any blend-shape supported skinned mesh renderer and lets you select a set of different blend shapes for each standard Preston-Blair phoneme. Things are going well.

 

Still at it. I lazed around a bit last weekend. (played a lot of Crusader Kings II) But in the evenings this week I've increased my productivity. Last night I managed to put together an in-editor preview function. Once you've created an animation and added it to the animator component, you can press a button in the editor and preview the animation without having to switch over to game mode. It's just a beginning, but it should be a solid basis for further editing features.

 

Ugh, I was super-busy this week. I had a technical conference that I had to attend for work. (Dev Intersection) It was pretty interesting, and I got to see a lot of the new tools/apis for Windows 10 development. But it ate up a lot of time, and I've been doing lots of stuff with friends and family in the evenings.

Despite this, I have managed to get a little more time and work done on Cheshire. Last weekend in particular was very productive. After experimenting with the animation preview feature, I did a little more research and came to the conclusion that rolling my own animation solution was perhaps not the best course of action. I threw together a quickie prototype for testing and confirmed some of my suspicions.

While a custom data-type for handling Cheshire-specific animation playback is a good idea, and will continue, rolling my own animation data holder is not. I discovered that it is entirely possible for me to generate a Unity animation clip, extract the animation timing and reference data from it, and then playback all of that data at the pace of my choosing. I tested this with the in-editor animation preview I had cooked up and confirmed it. I was able to play a Unity-generated animation clip file synced to a selected audio clip. (well, I was able to playback the data from the file anyway)

I'm going to go back and alter the custom animation data type I created, and remove all of the references to animation timing. Instead, I'm just going to put in a public property for a Unity animation clip. Cheshire will be used to create Unity animation clips, just like before. But once those animations are created the new Cheshire Animator will allow users to create a new labeled Cheshire animation, and select one of those animation clips for playback.

I like this approach. It plays nice with the "Unity" way of doing things, while still providing the new functionality that I wanted to deliver.

 

Still at it. I've re-configured the interface to accommodate the new focus on Unity-specific animations. I'm currently working on some functions that take a Unity animation clip, and test it to see if it contains the curves that I've focused Cheshire on animating. (general blend-shape curves, the object reference curve specific to Sprites, and action events for methods) It's going a bit slow, but just because these current tasks are a tad tedious.

Given the new focus of the plug-in, I'm probably going to have to abandon any plans to create a custom animation editor. This raises a few issues for anyone wanting to customize their animations after-the-fact. They aren't big issues, more annoyances for the user-friendliness on my end. In order to edit animations in the Unity animation window, those animations have to be assigned to an object through an animation controller. This makes it much harder to add an interface for easily loading up created animations in the animation window.

Once I'm closer to a final version, I will be able to post some tutorials that will show how to handle this issue. It's not a big deal, more just an extra step. And it isn't required for using the plug-in, just for further tweaking and editing the animations after they're created.

I'm also giving serious thought to splitting the animation creator and animation player into two separate projects. It's kind of the direction that I've been going in anyway, and the final version is going to work that way anyhow. I'd love to hear any thoughts or suggestions on this.

Also, I wouldn't expect the re-branding that I'm planning to be ready for the next release. I have plans to upgrade the art assets for Cheshire, but development has been my primary priority, so I've only started on those art assets. I will put a little more effort into Blockhead, and clean up some of his texturing, but the new 3D model will have to wait for later.

However, I will be launching a new suite of documentation and tutorials with the new version of Cheshire. I will also be providing a full set of video tutorials that will walk the user through the entire process, step-by-step. This was a much-requested feature that was lacking in the initial Cheshire roll-out, and it will be addressed with this upgrade.

 

As far as editing the generated animations, it'd be occasionally useful if it was user-friendly, but my current opinion on it is a big "meh". You could create an elaborate editor akin to papagayo, and I'd probably use it maybe a handful of times. I'm sure others would use it more often for advanced tweaking, but I think it'd be better to do it by hand at that point anyway. That's just my opinion though - I don't have tons of experience in automatic lip-syncing.

Splitting into two projects sounds like a good idea. Though, all I would [humbly] want out of Cheshire is the animation creation. I guess I don't see the necessity for a player outside of convenience. It plays on the characters already - I just have a scene that is specifically set up to test lip syncing. Maybe I'm missing what the animation player would accomplish...?

 

Well, the player is designed to take an animation, and play it back perfectly tuned to the playback time of a specific audio file. This could have applications outside of just lip-sync animation.

But I'm probably not going to split it off. After thinking about it, I realized that handling all of the different use-cases for the myriad of different animations that Unity supports would be too much. I'll have my hands full already just handling blend shapes, Sprites, and action events. I will make sure that the functionality of the creation and playback are separate, so that the one does not require the other.

 

I'd like to hear your opinion on something. It isn't directly tied to Cheshire, but it's related. If I wanted complete control of how the character talks, like how the phonemes might vary depending on emotion, should I have an empty (or just barely assigned) set of phoneme blendshapes that I keep track of via script?

 

Well, emotions are a bit tricky. Especially where the mouth is concerned. A lot of emotional expression in the face actually comes from the eyes and eyebrows. Where the mouth is concerned you generally have smiles and frowns that can be applied while talking. There are other, more subtle expressions, but most of those wouldn't blend very well while the character is talking.

I would recommend making a smile and frown blend-shape for each side of the face (left and right). That would be a good starting point. You might also have a left sneer and a right sneer. (where the upper lip curls upward slightly) You should be able to combine those six shapes to make some decent emotional responses while the rest of the face is animating.

As far as controlling them with a script is concerned, it should be fairly easy. Just have a few emotional states pre-defined, and use some basic variables to determine what state your character is currently in. Pre-define some values for those blend shapes to match up with your emotional states, and transition between them dynamically. Ought to work like a charm.

 

Thanks! I'm trying to iron out a good system for controlling emotion, with an intensity-based hierarchy of blendshapes and bones. That, on top of lip syncing blendshapes, is my actual goal. I hope to eventually release a production set of characters that utilize Cheshire.

 

Cheshire is designed to be a little more general-purpose. This approach will be carried through to the upgrade. While I heartily encourage you to make models that play nice with Cheshire, if you are looking for a more general-purpose standard, I would recommend looking at Mixamo. They have a standard set of blendshapes that they recommend people use with their tools. The standard set they specify is quite flexible, and can help to produce a broad range of emotions and expressions when used together.

I'm a little sorry that I haven't finished the upgrade already. The new version features multiple blend-shape support per expression. It should be compatible with Mixamo's approach to blend-shape lip-syncing. I was planning on downloading a few of the Mixamo free models and making a few example/tutorial videos using them.

Mixamo provides a PDF that lists all of the standard blend shapes that they use for facial expressions. I'd take a look at that if you are wanting to make some standard blend-shapes. It's a good starting point.

On Cheshire news, I was able to get a lot of work done this weekend. I set aside a sizable block of time and really dug into the Method support. I hadn't done all that much, but I stripped down what I had and built it up with a much more efficient approach. I was even able to successfully test the creation of a Unity animation file using the Method approach. (an animation file with a whole bunch of AnimationEvents) I was even able to add support for entering optional parameters, which should significantly expand the power and capabilities of this feature. I was also able to focus down on stabilizing and simplifying the user interface for the phoneme timing processor and animation creation window. It's much more stable and straightforward now.

Next up is getting the in-engine playback running properly. I've run into a bit of a snag with the AnimationEvents. The AnimationEvent class has fields for all of the different parameters, but it doesn't have a field that tells you which parameter was actually input by the user. (AnimationEvents can only use one of their optional parameters) So I need a reliable method for sorting out which parameter needs to be used.

The Sprite playback is cake, I already tested it in the editor, and got it working. The BlendShape playback should also be relatively straightforward. After I sort out the Method playback, I will start work on the BlendShape animation file saving. That will be the last major task for the project. There's just a lot of extra logic that has to go into the BlendShape animation creation.

Things are going well though, and I am very encouraged. I'm hoping to have a functional upgrade available before July. After that I'll work on bug fixing, marketing materials, and other projects.

 

Thanks for the info and update on your progress! I'm looking forward to the next release. Fingers-crossed on you getting it done before July.

[EDIT::] Removed off-topic stuff.

 

Another good evening of effort last night. I was able to successfully implement the in-game playback for the Method Set. Seemed to be performing pretty well, but I'll need to do some more testing. I'll probably sort out the Sprite playback tonight.

I got around the parameter issue by simply making a second reflection query when the play function is called. It retrieves the necessary data for the functions and lets me know which ones need to be broadcast with which parameters. Again, more testing will be needed, and a few other use-cases that I've though of will need to be verified. But overall, things seem to be working as desired.

 

Ran into a little bit of a snag last night. When I went to build, I bumped my head on the run-time limitation of the Unity Editor classes. I commonly use the AnimationUtility to retrieve data from the Animation Clips. But this cannot be done when you build an application. (as all editor scripts are left out of builds)

Thankfully, the manner in which I had already structured Cheshire helps with this. I just have to pull the custom data model I had created out of mothballs and put it back to work. So long as queries for the animation data happen in-editor only, everything is fine. I'll have to shift a few things around, but this is a problem with a clear solution.

On a more upbeat note, I got the sprite animation running last night. Progress continues.

 

Still at it. Refactoring the user interface to handle animation updating on the editor side of things is just very boring work. I got the functions for the Sprite and Model readers working easily enough. Reading the model animation data is the easiest task. Animation Curves are part of the runtime, not the editor. So I just had to identify the Blend-Shape curves and hang onto them. The sprites were a little bit more difficult, but not that bad.

The Methods threw me for a little bit of a loop. I discovered that I need to refine my GetMethods search. Unity does not allow for overloaded methods to be used as Animation Events. But my current Method retrieval function allows for such methods to be selected. I've got to go back in and entirely remove any overloaded methods. I also don't think there are any Binding Flags for that, so it has to be done the hard way. (checking for repeats of the function name)

I'm hoping to be able to finish off the coding for that tonight, and be able to test a few of the animations. Then I can get back to working on the Model animation saving.

 

Once again, I'm still at it. As always, personal life interferes with my efforts. But I always return to my work eventually.

The model blend shape animation saving is pretty much all that remains. I sorted out all of the issues with the Method and Sprite animations. I even tested them both. And the refactoring for separating the editor and run-time scripts worked out very well. I successfully separated the data for the animations into their own classes, and optimized the update function to populate the animation data in the editor only. During run-time playback the engine just has to query the existing data. It's actually a much more performance-friendly means of handling the animation playback, so this is definitely a net gain.

I even got the model blend-shape animation playback running last night. The playback wasn't all that difficult once the data was configured properly. What was a bit more of a challenge was populating the data for the blend-shape animations by pulling it out of the Unity animation files. But even that wasn't as hard as I was expecting it to be.

Thanks to the editor re-factoring, I was able to build the example project successfully, and even tested the animation playback on my smart-phone. Worked like a charm. While there is a bit more work to do before a release, I am pleased with the progress I'm making.

 

Really looking forward to what you've cooked up since the last release. It sounds really promising.

 

Ugh, so much going on, I have to scrounge what hours I can to work on development. Thankfully, for the past three days I've been doing exactly that. I managed to put together all of the basics for saving out the 3D model animations. Performed the basic testing last night and got it working. I ran the test scene on my Android phone as well. I have a fully 3D talking character running on a mobile device!

There are one or two slight tweaks left to the coding. I specifically wanted to add an "intensity" slider to the animator for adjusting intensity of the blend shapes. It can be applied dynamically during playback, so I don't have to integrate it into the saving system. Should only take a little bit. After that it's just a matter of cleaning up, testing, and cooking up some early documentation/demonstrations. I should be able to have a YouTube video up sometime this weekend, and an Asset Store submission before the end of July.

Finally looking at the home stretch on this one.

 

Hi! How about your project? Did it progress since last year?

 

Sadly, no. 2016 had a few personal projects for me involving my family. Nothing too serious, but they ate up a lot of time. And I mean A LOT of time. I also now have several nephews and a niece, so that's eating up quite a bit of my time as well. I'm starting to transition back into development, but it is a slow process. One of the things I'm focused on at the moment is customizing my web presence, so that I can have a proper site for hosting tutorials. One of the big feedback things I got from the initial release of Cheshire is that proper, clear documentation is extremely important. So I'm going out of my way to insure that I have a more efficient means of providing such documentation in a timely manner.

I'm also focusing on adapting my development environment to support and test on multiple versions of Unity. Having to deal with compatibility issues for different Unity versions was also something I learned. Some of the early users of Cheshire had issues with the plug-in after upgrading Unity, or using an older version of Unity. I have installed several different Unity releases on my development machine, and I'm testing all of my currently developing Assets on all of them. Compatibility will be improved on future releases.

Thankfully, I'm pretty good about backing up, so all the progress I had made on Cheshire is still there. It will likely still take me a little time before I update it, but I have not abandoned my baby.

 

I take a peek at this thread every now and then just to make sure I've not missed anything. It's nice to hear you've not forgotten this project totally. I think I and others are concerned about it becoming vaporware sometimes, but life tends to happen so we can all relate at some point.

I've been looking at a few options for face animation and lipsync lately, but haven't settled on one yet. With this new Timeline + Cinemachine feature in Unity now, I've finally decided I'm ready to venture into the facial animation territory.

I just wanted to mention that, if there ever was a time to get some traction for this, this is it.

Also, is there any place I can access the old version in case I need to roll my own solution? The stuff on the Asset Store currently seems a little too obtuse for what I'm going for, but I've never had the chance to play with Cheshire so I don't know how it compares, either now or in the past. However I've seen loads of people recommending it despite it not being available in the Asset Store anymore... It really must be good. I really hope you'll continue to work on it..

 

I have a trip coming up this week, so I will be out of town until next Tuesday. I won't be able to do anything on this project until then. Past that point my schedule will free up a bit more, and I have no problem posting the legacy version of the plug-in on my web-page. The older version was not restricted in any way, you could access the source code and do whatever you wanted with it. So there's no real problem posting it for download. I just need the time to get around to doing it.

You're not wrong. But at the same time, I've never made money off of this plug-in. Cheshire has always been a labor of love for me, not a commercial project. Even the upgrade that I was working on was never intended as a for-pay component. Most of the Cheshire users like it because it was free, and it worked relatively easily. It was far from a perfect solution, but it played nice with Unity's built-in systems, and wasn't overly complicated. (as many such plug-ins tend to be)

So while gaining traction with the new animation tools would be a good idea, I'm more motivated by prestige than monetary considerations. This is a hobby for me. I would love to raise the profile and popularity of Cheshire. But such improved exposure would also lead to more demands for regular updates and bug fixes. There are always considerations to look at for these things.

 

That's totally respectable. I'm glad to hear that. I just noticed over at polycount you had posted a topic back in 2014 about potentially seeking to put it on the Asset Store and it implied you were looking to monetize it. However I totally understand priorities can change in a long-term project like this, so just know I'm totally glad to hear you don't plan to give this up even though you're doing it for free. After all, I'd love to learn from it since it probably took you some serious time to get to the point it's at now. Unity can be very unpredictable when it comes to writing editor extensions sometimes, and it takes time to learn those quirks.

I'm glad you're willing to help people learn this stuff by offering the source (no matter what state it's in!) because Unity is in dire need of decent lipsync support as is, and with your help, some of us will now be able to make our own to share.