Creative Motion Matching (MxM) Animation System for Unity (Released-beta)

Check it out on the Asset Store

Download the Standalone Demo

​

Motion Matching for Unity (MxM) is an alternative animation system to mecanim for characters. With motion matching you can achieve advanced, organic and fluid animation without the need for an animation state machine. You don't have to define transitions or set up conditions. Provided you have decent animations with enough coverage you can create a fully working character in about 15 minutes with starts, stops, plants and turns. With a little more effort and coding ability, its not hard to make very complex animation such as parkour, fighting and sports.

How Does It Work?
Motion matching is a relatively new animation technique that is not constrained by the concept of animation clips, states, pre-defined blends or transitions. Motion Matching allows animation to flow freely through your entire animation set, jumping to any pose at any time. Animation poses are chosen based on both the current pose and the desired gameplay input*. Whatever matches the pose and your desired input the best gets chosen and is rapidly blended in. By balancing the pose and the desired input, motion maching is able to achieve high fidelity animation while still achieving good input response**.

Industry Use:
Motion matching as a technique was originally developed about 2 years ago by Ubisoft for the game 'For Honor'. It has since been used in several EA and Ubisoft games and is even being used in 'The Last of Us Part 2' by Naughty dog. As you can see, the technique of motion matching is not novel and has significant AAA application. (I am not affiliated with any of the above mentioned companies or games)

Trailer

Introduction & Overview

Introduction & Overview

Motion Matching with Raw Mocap (alpha footage)

Note: This is raw mocap placed on a generic rig. The weird hands and arms are a retargetting issue and has nothing to do with MxM.
​

Performance:
Motion matching for Unity uses the cutting edge of Unity development to achieve solid performance while still being stable. It uses Unity's Job system, SIMD mathematics library and burst compiler, to achieve lightning fast, multi-threaded performance.

This video, which is already out of date, shows a performance benchmark on an i5 4 core system running motion matched characters above 60fps without LOD or infrequent updates.

Features:
Motion matching for Unity is not just a locomotion machine. It supports a number of features that allow the user to create almost any kind of animation. Features include:
- No animation state machine
- Fluid and responsive animation output**
- Support for cut clips as well as un-cut mocap
- Powerful event system for dynamic actions (e.g. vaults, sword attacks, pacour)
- Powerful tagging system to allow control over animations (e.g. stances etc.)
- Motion timing editor (change animation timing to match gameplay)
- Animation warping for event (precise environment contacts
- Layer system
- Use alongside mecanim (switch at any time)
- Compatible with custom animation playable graphs
- Compatible with IK systems

Useful Links
- Asset Store Page
- Unity Connect
- Project Roadmap
- Discord (for support)
- User Manual
- Quick Start Guide
- FAQ
- Tutorial Videos

*MxM needs gameplay code to tell it what to do. You have to write the gameplay code to do that. MxM is an animation system, not a gameplay system.

**responsiveness and fidelity of the resultant animation is dependent on the quality, responsiveness and coverage of your source animations. Watch this video for more detail.

 

Looks awesome! I'd love to use this some time. Do you have a target release date yet, or is it too early in development still?

 

Quite interested to see where this goes.

 

Thanks. It's moving along quite quickly now but it is very experimental in nature so its hard to say when it will be done.

 

Some significant improvements in quality over the last week.

 

Huge progress over the last week. Probably the most difficult part is out of the way now. i.e. getting the primary motion matching algorithm correct and error free. Most discrepancies are now down to animation quality.

As well as improving the stability of the current system there's a few things I need to do before the first release which will be a beta. These features include:

• Markup System (for stance etc)
  
• Event System
• Idle Animation smoothing (tricks to make idle animations work)
• Direct control matching policy (for things like interactions, e.g. pulling a lever

 

since it seems kinematica will take ages to release, im interested in this too. what are your asset plans? and also how much it cost on cpu right now?

 

Very interested in this, also wouldn't mind testing the alpha version. While this is probably far from your intended use case, would you think this technique could be useful in a scenario where you have lots of mocap takes where the pose at the end of one take does not match up to the pose at the start of the next one, and you need a convincing transition animation. Could a similar technique be used to pull bits from a large mocap database and create this transition?

 

Hi Razzraziel,

The plan is to release it when it's ready. That sounds like a huge cop-out but the whole thing is rather experimental so I don't really know when that will be. That being said, I'm hoping that won't be too long as I plan to release it as a beta on the asset store at a reduced price. For the beta I want to have enough features to allow different stances, combat and better idle animations as a minimum.

Regarding performance, with ~3000poses (5mins of animation total) the update on the MxMAnimator is about 3.6 - 3.8ms in editor. Note that the current approach is a completely brute force linear search. My focus has been on getting it working first. Optimization comes later. MxM actually has two different matching techniques, the Pose Culling technique performs better with 1.5 - 2.2ms un-optimised. However, the caveat here is that this technique seems to requires better / more animation data.

Following initial beta release there will be a big focus on jobifying and improving the data structures (kd-tree / voxel tree) to improve performance probably x100 or better. There will be some other minor improvements to performance before beta release which should all just drop nicely.

I'd say the matching algorithm is pretty well optimized itself. A lot of the data is pre-processed so there is less impact at run-time. It's just the search algorithm that just goes through every single pose that needs better optimization from this point.

It's possible. Motion Matching just picks whatever animation matches both the pose and trajectory the best. So as long as you have animations somewhere within your mocap data that could bridge the gap then it could work. But yeh as you said it's a not the intended use case and would likely need some customization to achieve that.

 

I'm not exactly sure what this is doing, but will the smooth running movement also link up with jumping and navigation over small obstacles and slopes? Just curious, because I'm looking at the running on a plane and thinking ... it works in a perfect world ...

 

Short answer.... yes

But its a bit more complex than that. Motion Magic will match whatever trajectory you give it provided that you have the animations that you need and you code your gameplay / trajectory model to detect such things. The trajectory model is not a part of MxM as it is tied too closely to your gameplay which will be unique for every game.

The trajectory model that comes with MxM will be for a simple use cases though it will be a bit more complex by the time of release. MxM provided hooks to easily make your own trajectory model and integrate it into the system easily. If you wanted to have your character say jump over small obstacles (maybe 0.3 - 0.5m high) you would have to code that behavior into your game-play and feed that prediction into MxM. I'll probably do an example of this later but it's not really a priority at the moment.

Also note that there's nothing stopping you from adding your own procedural animation (e.g. Foot IK) on top of the animation produced by MxM. Things like that are dealt with the same way as traditional systems.

 

@CptKen

Great, waiting for first optimized beta release then.

Also this can help. (Maybe you already know)
https://github.com/sebastianstarke/AI4Animation

I really want a performant & fluid next gen controller approach, but since i'm a solo dev, i cant afford the time that needs to develop something like this.

 

Cheers, deep learning animation selection is a whole different beast but the concept that you can jump to any pose is the same. Motion Matching's beauty is in it's simplicity (relatively). Last time I checked deep learning animation systems are really hard on performance.

 

Okay, so this could possibly be part of a solution for smooth parkour, with different forms of wall running, climbing, vaulting, leaping, etc., but we'd need to approach it kind of like how they do with the Motion Controller asset, by making a bit of code that tests for the criteria for change in motion (like slope, obstacle height, etc.), and then implements the movement mode change.

 

I am guessing this is a stand-alone system and cannot be integrated into Opsive Ultimate Character Controler?

 

I’m not completely familiar with how the Motion Controller asset works but you’re somewhat on the right track. Motion magic it simply jumps to whatever animation frame matches your current pose and desired future trajectory game-play. I suppose you could say that the current pose and desired future are the criteria for animation selection.

A character controller of some kind is needed to make that future prediction by simulating future movements within a single frame and recording the results. If the character controller is capable of vaulting over objects then motion matching should be able to choose an animation frame to animate that character appropriately… provided you have appropriate vaulting animations in your library.

It's important to note that Motion Magic isn't a character controller itself its an animation synthesis system.

I'm not familiar with how Opsive's Ultimate Character Controller works. However, it's important to note that Motion Magic is not a character controller, it is an animation synthesis system. The controller that is in MxM right now is only for testing purposes.


The system is designed to have custom gameplay controllers (not just motion) hook into it and provide it with a future prediction. These future predictions are usually done by simulating some kind of character controller code a number of timesteps in a single frame up to about 1 second in the future. If it’s possible to do this with the Ultimate Character Controller then it’s possible to integrate with MxM.


Once the asset is released, I fully plan to investigate integrations with other character controller type assets and hopefully get in contact with some of them for collaboration. However, for now I’m just focused on Motion Magic itself.

 

Oh, okay. So ... it is a bit like IK, then, in that it's a modifier. It lies in between the Actor (character) Controller and the Motion Controller (animation selector) in the ootii system, I guess.

 

Not quite, it's more like a replacement for Mecanim.

 

This is correct, Motion Magic completely replaces mechanim. Though I do plan to allow smooth transition between the two at run-time for those who desire that flexibility.

To understand motion matching you have to abandon all knowledge of state machine based animation and even the concept of animation clips. MxM takes raw mocap data as it's input, pre-processes it based on a few settings (no manual cutting of clips) and then uses that data to synthesise animation at runtime. There is no state machine, the system continuously jumps to any animation at any time to achieve the best results.

Here is the general workflow at the moment:

• Import your mocap animations (Literally drag and drop)
• Automatically pre-process them using MxM's PreProcessor Asset (this generates a new animData asset). <- (This is old school machine learning, not deep learning though)
• Add the MxMAnimator component to your character and slot in the animData asset you just created
• Tweak a few settings (calibration) <- I didn't even do this for the video I last showed.
• Play!

This is all I did to get the animation shown in the video I posted.

There is of course a lot more you 'can' do to tweak, finesse and control the animation to your liking. However, it gets you 80% of the way there really fast. Also this doesn't include the process of building the character controller which is up you as its tied too closely to your game play for me to make any reasonable assumptions.

There really is no state machine. Doing the above few steps takes maybe 5mins, and its enough to get locomotion with stops, turns, jukes, starts etc. That usually takes a long time to do with a state machine.

I might make a video showing the process.

I'll keep you posted on performance. In preparation for jobification, I refactored my data structures last night. This alone, without jobification, reduced the processing time on average by about 0.5ms (11% improvement) for the pose costing technique and 66% for the pose culling technique (pose culling went down from 1.5ms - 0.5ms process time)

This is without using fast searching algorithms or jobification so I'm pretty confident it will get down to a reasonable level.

 

I appreciate the extended description, it helps. But I think I still see it the same way. I'll be following the progress here, to try to continue to get a better idea of what this is. It sounds like something I would love to use, like the next step in animation evolution after the Playables API.

I'm thinking it still needs a character controller, plus a state manager of some sort. It may replace the animation state machine, but we still have to list internally what movement options we have available and manipulate our control modes or states of action ... like if we're in a stealth mode (crouching), or in combat, if we're jumping or falling, or if we're switching from one dance move to another ... right?

 

Correct, you will likely need a state machine for your game-play like any other game. MxM is purely an animation system and only replaces the need for an animation state machine which is usually far more complex than a gameplay state machine. You still need to tag animations, for example, tag sections that have crouching. However, this is a relatively simple process and you never have to define complex transitions or code the logic behind those complex transitions.

If it helps to understand, there are some AAA games that use Motion Matching as their animation technique. Check out 'For Honor', some of the newer 'Ghost Recon' games and the upcoming 'Last of Us 2'. Also if you search motion matching in youtube you will find some presentations that explain how it works.

 

Just an update on performance. I did some pre-preliminary optimisation / re-design of the data structures used for MxM in preparation for future jobification. This alone has increased the performance dramatically. Average 1.3ms - 2ms per update only sometimes jumping up to 3ms. Previously it was stuck between 3.5 - 3.8ms. The pose culling technique is down to 0.5ms per update but it currently doesn't have the same quality.

This is still way too slow for production but I'm not worried. There's so much more that I will do as incrementally optimise as I go including:

- Further improve data structures - Optimising CPU pre-fetching by not wasting bandwidth on cache-lines
- Kd-tree / voxel search algorithms - Currently the search is linear. This will reduce the search between 250 - 1000 poses even if there are about 70k poses in the database
- Jobification - the pose and trajectory calculations run many times per frame are perfect for jobification.
- LOD - Reduce the update rate of characters further away and update them out of sync to spread the load over multiple frames.

At the end of the day, I believe the combination of the above will make it quite reasonable and will be a key priority following initial beta. I just can't optimise too early as pre-mature optimisation can be difficult to re-work if I need to change anything.

 

You mentioned importing all of your mocap data. Do you use mocap animation cards like the Ubisoft demos, or would it work to import each mocap animation that is split up for use in mecanim?

Also, does this system work with masking? I could be wrong, but I believe for honour’s combat system used motion masking for the lower body movement, and used masks to play key framed animations for the upper body (sword swings, etc...)

Either way this looks great! I am excited to see more!

 

Typically this works better with un-cut mocap data. However, animation clips will be supported as well. The biggest problem with clips is that they don't have enough past and future on the each end of them making it difficult to calculate a past and future trajectory for poses. I plan to overcome this by extrapolating the motion state at the end and beginning of each clip so it should be doable.

For Honor doesn't use much layer masking at all. It is predominantly full body animation. Simon Clavet (Ubisoft Animation programmer), mentioned that its used very rarely in For Honor, and to quote him "motion matching is the church of full body animation". The reason for this is that for masking to work usually animations are kept in sync. However, with motion matching there is no way of syncing animations since it jumps all over the place.

They do however, use procedural animation on top of the matching in a number of instances:

• Foot IK
• Rotating of torso (x) on slopes to face target
• Rotating of torso (y) to ensure the player is looking at their target.
• Slight adjustments for interactions

Interestingly enough they don't even use a sword IK solution. They play a full swing animation depending on the point of contact and the time to hit.

MxM uses unity's playables system to actually animate the character so layer masking will be supported but highly un-recommended for most situations as it undermines the quality gains of motion matching. When you do a running turn your whole body is part of that motion. As soon as you mask over the upper body and add a sword swing or something else you lose that fidelity. Things like sword swings, attacks and interactions will be managed by an event system which I am working on right now. But yeh, as I said it will be supported so up to you how you do those kinds of things.

 

Just a quick update. Here's a preview of tagging and event UI. I've designed it to be as similar to the animation window as possibility for familiarity. It also behaves in more or less the same way.

This window allows you to easily markup any of your animation data with events and tags so the system can match them too in specific situations where game-play requires it.




I've also re-introduced support for cut clips. Typically the problem with cut clips compared to mocap is you can't determine the future or past trajectory of the clips because they are usually too short. I've added options so you can decide how the trajectory is determined so this problem can be overcome. You can decide from one of the following.
- Loop the clip
- Get the trajectory from another animation
- Extrapolate the physics state at the end of the clip
- Clamp the trajectory

Super keen to finish the back-end for events and show some results. I'm getting bored of just having locomotion XD

Here's a quick look at how you can define the clips you want to use in the system and how their trajectory is to be calcualted. Note this is not a state machine. It's just an alternative to having a list of clips and it allows you to specify how the trajectory should be calculated for cut clips. For mocap data you would simply drag and drop your animations into the graph without any linking or anything. Even for cut clips you don't have to link them, it's only if you want the future trajectory to be calculated from a different clip.

 

I was holding off posting this because needed permission from the video owner. However, one of the alpha testers had motion capture taken specifically made for motion matching. He threw it into Motion Magic and these are the results:

Note: the weird hands, arms etc. Are a retargetting issue for this model and not related to MxM in any way

 

Those arms are a little, uh... squeezed?

That aside, looks really fluid!

 

That's a re-targeting issue for this particular model. MxM doesn't change the animations provided to it, it will only play what it's given on the rig it's given.

I think it also looks a bit funny because there is no hand animation so they stay at that funny angle.

So the takeaway from the video is more the fluidity of motion, everything else is out of MxM's hands

 

looks nice! still needs a little more work i suppose.

cant wait to try

 

The video definitely lookg quite promising already. The turns sometimes look rather sharp/artificial e. g. around 00:05-00:07. I suppose that speaks for the state of MxM, though, that turns like that are noticeably worse than the rest of the video!

 

Cheers.

Aside from the tonne of work I still need to do to make it a complete system, I think the locomotion is pretty much working as intended. The only thing I'll probably add to it is anti-foot sliding which is usually a necessity for motion matching because of all the blending going on. Also keep in mind, despite the name, MxM isn't a magic bullet, if you don't calibrate it right, don't have a well re-targeted model and your controller is un-realistic you won't get a good result. The un-animated hands also don't help in this case but that's not MxM's fault.

Unfortunately I didn't have control over the setup in this video as I didn't make it myself. At the end of the day this took 5 mins from import to play. I think most people would struggle to get this kind of animation using mechnim in 5 days

Thanks!

MxM will match whatever trajectory you give it. If you give it unrealistic turns, it will turn un-realistically. For example in this particular video the position bias on the controller is set really high making the turns very sharp. This is probably the cause rather than MxM itself.

Also worth noting that any animation played by mechanim is an animation that exists in your animation set. Strange that a motion captured from a real person could be un-realistic but I suppose when you add re-targetting into the mix you may get different results, again not really up to MxM to resolve this.

 

What kind of calibration we're talking about? For motion capture or in MxM?

Btw as i understand, this is working on flat surface only now? How does it respond on Y axis change?

I need 2+1(1 optional) things for this kind of asset.

- First fluid animation blending for X, Y, Z inputs for bumpy surfaces. Which is you're working on right now. Also capable of handling climbing, jumping etc with additional inputs.

- Second, like Ubisoft GDC talk, able to react external forces (Colliding and hitting walls from sides and keeping balance) and blend that specific reaction animations. (Also PuppetMaster integration would be great, i dont know if i can do it since i dont know MxM yet).

- And optionally, ability to handle not only 2 feet, also 2 hands on the ground movement animations. Like crawling (Metal Gear Solid V opening). For example a damaged character, a broken bone. Or using hands with feet to move on sloped ground. Of course after given proper animations, not magically.

Then this is gold for me

 

For motion matching to work you need to calibrate it. For example, some things are more important to match than others. So its just a case of setting a bunch of weighting numbers. Ubisoft's motion matching requires this too but they brush over it quite briefly so not many people realise.

So I'll address your other points one by one:

1. MxM will be able to handle all of these. However, you need to understand that MxM doesn't handle gameplay. If you can code the gameplay, then MxM can match it. It's not a character controller or gameplay system. You typically don't want your games to be animation driven anyway. There's also options to integrate third party controllers which is something I plan on doing once MxM itself is finished.

2. This will be handled with the event system. One thing you need to know about the Ubisoft GDC talk is that the reaction to physics is all smoke and mirrors. The animation shown is recorded mocap and not synthesised with physics. They are simply taking physics impacts on the controller and matching those impacts to events in the animation set. You may be aware already but thought I'd just make that comment just in case.

Regarding things like PuppetMaster or FinalIK. MxM simply synthesises an animation like mechanim. Whatever PupperMaster or FinalIK does after that is not MxM's concern. I haven't tested it yet but I suspect it will need very little if any work to integrate.

3. MxM can match any bone in a humanoid or generic rig. During setup you choose which joints to match based on your desired use case. You'll also be able to define different matching rules (e.g. which joints to match) for different animation libraries. However, this kind of thing would be handled better with tagging anyway

At the end of the day, it's all about what tool is right for the job. I don't think Motion Matching should replace mechanim in all cases but it may be a better solution in some cases.That's just a judgment you'll have to make when you see the final features at release. However, there shouldn't be any kind of movement that isn't possible with MxM by the time it is finished.

 

PuppetMaster can significantly change the velocity/trajectories of bones of the controlled 'puppet'. It would be beautiful if MxM could interpret the physics state of the PuppetMaster puppet to sort of 'reconcile' the physics simulation to the animation state.

 

Haven't updated recently because I've been quite busy with MxM but a few updates:

1. First iteration event syatem is in. Here's an example. It uses animation warping to line up hands to obstacles and such. The warp is exaggerated in this clip so its easy to see. Its not perfect but its getting there.

Events can also be used for sword swing combos. In this clip the second swing has the same event id as the first but it works because MxM matches the best event to your current pose.

This can also be combined with warping if you want the attack to hit at a specific point in space

2. First iteration tagging is in.

3. Undertaken an optimisation pass using Unity's job system and Burst Compiler. The Update is some 40x faster than before on a 4 core system. (burst is incredible)

4. Support for cut clips added. You still need decent transitions though or it wont work. Thats just the nature of motion matching unfortunately.

5. MxM can now plug into any playable graph if you want to combine it with your own system or combine it with mechanim. Requires a little knowhow of the playable system but its doable.

6. Tested MxM where the character controller controls movement instead of root motion. I developed a technique to make this still look good. It's not really a part of MxM but I'll share an example at some point.

7. Tonnes of other minor things and slight improvements.

 

What?! Calling that 'optimisation' is a huge understatement. That's incredible indeed. Great job on the other additions as well!

 

This is really amazing. I thought MotionMatching is only good for the basic locomotion but event system changes my mind. Thanks for the great work and I'm looking forward to get my hands on it.

 

I'm curious when you said 40x times faster, are we talking about around ~0.05 ms for each update?

This is the last perf measurement you gave us.

0.05ms is very cheap and if it can scale, I can use MM not only for the main character but for all characters.

Honestly, I was worried that I still have to set up traditional animation if I were to have many players. When you have 100 players online game, I thought MM will be prohibitively expensive so that for the players over the network must use the traditional animation.
This means that I still have to setup Mechanims and there is very little I can gain other than the main character animation is more fluid.

If 0.05ms for each character update, it will give us a reasonable update (5ms) for 100 characters. This will make the animation set up so much easier.

Please let me know if missed something.

Thanks.

 

It really depends on a few things. Firstly how much animation data you have, and there is some overhead per character. What was sped up 40x was the main MxM loop which goes through all the animation data and picks the best pose. There is a small cost to setting up which is more or less constant and then there is the overhead from the playable animation system which I can't really change.

I'll do some additional optimosation at some point on the setup overhead but the real bottleneck now is Unity's animation system.

If you have about 65000 poses (40mins of animation) it will also still be too slow but I have a few future optimisations planned for high pose counts.

You also have to consider that you won't only have animated characters in your game. You need some CPU time to manage the rest of your.

MxM is however LODable. So characters further away can be updated less frequently. Meaning the load can get spread over multiple frames so it really depends on how a game might work.

 

Thanks. We can only tell for sure until we test them ourselves.
If the bottleneck is at Unity's Animation system, it will be the bottleneck whether we use MxM or not.
What matters is, then, how much MxM take more time compared to the traditional animation.
If you can tell us some benchmark number with many characters comparing them with Mechanim setup, I would really appreciate it.
BTW, when is the beta ETA that we can get our hands on for the first time?
I got lots of animation data and I'm really dying to test it out.
Perpahs I can do the benchmark test myself if it's soon enough.

Thanks

 

No ETA really. That doesn't mean it won't be soon though. I just don't like promising timeframes as many factors can come into play. There is one more necessary feature that I need to implement before I have a beta release candidate

Regarding benchmarks there are some more optimisations I want to do first.

 

Well, you've set the bar quite high

 

Here you go!

 

nice performance setup, so you can see even smallest impact easily. did you already jobified every costy part?

 

Wow, very nice. Good job! And I think normal Mechanim animation will probably take a similar amount of time. This means that I can replace Mechanim animation with MxM without worrying about the frame drop. And the best part is that it can scale with the number of cores whereas Mechanim cannot scale. Pleaes keep the good work.
Cheers!

 

Pretty cool. Is updating at less than 30 Hz an option for LODs or would that quickly lead to completely unacceptable spazzing etc.?

 

No, it won't. You can LOD all the way down to 10Hz even, the spazzing with the 200 is a bug not inherant with motion matching. Won't be a thing at release.

The trick is to make sure your characters are controlled by a controller and not root motion. Root motion looks good but its really bad for gameplay. There is a hybrid method to get both to work nicely together and I'll have an example of that on release.

 

Jobified the most costly part. Which now makes it the least costly part

 

Sounds good! It would be great to have a controller that takes root motion as a guide or template for movement, and not as the actual implemented solution.