Unite Now - Kinematica – Democratizing Motion Matching for Character Animation

Unite Now Session

In this session you’ll learn how to get started with Kinematica and implement it within your animation pipeline. Based on Motion Matching, Kinematica uses raw animation clips or motion capture data – not animation graphs – to produce character poses automatically, based on your description of the intended movement.

Update (May 29th)
The demo is now available! We've made the GitHub repository public as well.
The repo address is - https://github.com/Unity-Technologies/Kinematica_Demo

The release can be downloaded here.


When
Thu, May 7, 9AM PT


Where
You can watch the session here.
Make sure to sign up to get access to all Unite Now content.


______________________


Together with this Unite Now session we are also hosting a Q&A here in this thread to answer as many of your questions as possible.


A team of product experts from across Unity will be available starting after the session. Our Community team will continue to field questions to foster an ongoing discussion with the community. Please feel free to ask your questions in this thread.


Some basic rules

• Don’t bundle multiple unrelated questions in one reply. One question/topic per reply.
  
  
  
• Only questions related to the topics of the session are permitted.
  
  
  
• All questions will be fielded by our Community Managers (@LeonhardP and @AskCarol)
  • Replies will have to be approved by the moderators to show up in the thread.
    
    
    
  • Once approved, the questions will be forwarded to the relevant experts.

We really look forward to hearing from the community.

Thank you!

 

Edit: I misread your question. I'll leave the answer here because it is tangentially related, and answer your actual question at the end

At the moment, we have limited support for clips that are shorter than what Kinematica calls the "Time Horizon". The default for this setting is 1 second. We want to add implicit support for short, looping clips, but this hasn't made the cut for the first preview. For any clips that are shorter than the Time Horizon, you will need to stitch clips together to have at least 1 second of continuous motion.

For clips that are at least as long as the Time Horizon, you can provide "boundary clips", which will provide trajectory information outside of the range of your clip. This allows you to have valid frame and trajectory for the whole duration of your clip. Otherwise, your trajectory abruptly stops for last 1 second of your clip.

For the moment, we support only a single boundary clip for pre and post trajectory extrapolation, so you should use your most likely transition, or the same clip if it's meant to loop. While testing and building the demo content, this hasn't been a problem, but we may adjust if this becomes problematic to trigger correct transitions in specific cases.


Regarding your actual question: using Kinematica with a small set of clips:
Yes, you can use it with restricted set of clips, either like a classic state machine with motion matched transitions: "Find me a good pose inside 'Locomotion' to transition to", or by using trajectory matching. How smooth the result will depend on how well connected your animations are.

You can also use Kinematica as a "state machine if you want to. It's a bit overkill, but it's helpful if you want to force certain clips to play and you don't care about matching. There are samples in the package that showcase this.
If your set of clips for locomotion is limited to, for example a strict 8-direction motion set, and you want to use path-based motion matching, then you will get better results if you limit your desired trajectory input to strict, 8-directions

 

For your information, the package is already publicly available on the Package Manager , so you can go try it out as you wait for the talk to start.

Also worth mentioning, there is a Demo project mentioned in the video. Sadly, while the package is available, the demo is not yet available for public release, we need a bit more time to polish it.

We will do our best to have it available as soon as possible, as well as give targeted samples that detail the more advanced use cases like parkour.

 

Q1: It's up to you. Kinematica itself doesn't support partial animations at the moment, so doing this through motion matching would have a memory cost. But, Kinematica goes through the Animator and an PlayableGraph to write to GameObjects, so you can add other AnimationOutputs to overwrite parts of the hierarchy if you want to.

Q2: It depends. The workflow we have focused on for the first preview is the "Full Kinematica" workflow, where you only use Kinematica, and go through the Kinematica Component. This should interact nicely with things that postprocess the Animation, like the Animation Rigging package or FinalIK.

If you want to combine it with an AnimatorController, your own AnimationJobs, a Timeline, etc, it is definitely possible, but will require a bit of fiddling with PlayableGraphs. This is also an area where we have spent much less time, so feedback and bug reports are appreciated.

 

Puppeteering is an interesting use case. I don't think Kinematica will perform well (or at all) in your use case. It is clearly optimized towards finding the next best pose given your current pose (which is usually known ahead of time). In the case of puppeteering, I imagine your possible poses are endless.

Try it out, tell us if it works!

 

This is partially answered here, and in the video. You can definitely use Kinematica without a full dance card. All the samples in the project use a limited move set, for size reasons, and to showcase how to use Kinematica without a mocap studio

But Kinematica doesn't generate motion, so your results will be as smooth as your input motion. Kinematica will find the best match for you. It may happen that the best match is not a great match. You can still tell kinematica "go to my JumpForward animation, I don't care if it pops".

The more data you give Kinematica, for a given Tag (e.g.: Locomotion), the harder you will have to work if you want to get this specific animation, and the more specific you will have to be on what joints you use to match these poses.
This means more processing and more memory.

But Kinematica lets you force a specific sequence of poses without requiring you to always use the pose matching, or the trajectory matching. If you tag a section of a clip or a clip as "JumpForward3M", then you can tell Kinematica to give you JumpForward3M, and it will happily do so.


Regarding the demo content:
This is also mentioned here. I'm sorry, the demo content is not ready for prime time yet.
The video was produced a few months ago, and while we were hoping to have the content ready in time for today, a global pandemic happened, and our plans were a bit shuffled.

We're working to make it available as soon as possible

 

This is partially covered here. There are limitations on how short your clips can be before it becomes tricky. But besides that:

Runtime-wise, there is no impact. Kinematica only knows about each discrete segment you have decided to tag. There is no functional difference between tagging all your "clips" out of a long mocap clip, or having all your clips separate and the whole clip is tagged.

Workflow-wise, it's a balance. Longer clips are a bit easier to work with, but once your clips get long enough, it becomes harder to reason about them. It's easy to reason about "RunTurn45Deg: frame 45", but it becomes harder to figure out what animation is actually playing if the current frame is "RunTurnSpirals15To105Deg_mocap: frame 259".

 

Kinematica currently only plays one segment at a time. On any transition between poses or segments, it uses a version of inertial blending (see this talk for the general idea). So you will get some amount of blending, but you will not get multiple clips running at the same time. If you want layering, you can setup Kinematica to run in a PlayableGraph, and there you can layer Kinematica with other Animation sources, with IK, or with another Kinematica Synthesizer if you want to.*

This is partially also covered here

*As mentioned in the comment above, we have validated that it's possible but this is not where we have invested the brunt of our efforts, so the user experience is not as straightforward, and will require more programming.

 

The layman explanation for vector inertialization is: "you blend speeds, not positions". It doesn't really necessitate IK per se.

As for IK for the Animator, AnimationJobs lets you define your own animation post processing, and it's compatible with Kinematica. You can use these to drive transforms procedurally. If you want a bit less low-level, you can use the Animation Rigging package to setup an IK rig. This should work fairly well with Kinematica too. If there are serious friction points that make it really hard to use either of these solutions with Kinematica, we could have simple IK support directly in the Kinematica package, but I would like to avoid overlapping feature sets with existing packages if possible.

As I mentioned higher up, we haven't spent much time mixing Kinematica with Animation Rigging or Animator controllers yet, but we have a bit of time before 1.0, we can adjust if there are issues.

Multipede Kinematica?
Good question. We know it works for bipeds and quadrupeds. I don't see a specific reason why it wouldn't work with multipedes. If I was doing a millipede for example, I would use Kinematica to drive the body shape based on trajectory, then have a post-process IK controller that orchestrates the legs to do something sensible.

Optimizing the build time is next on our list of priorities. We know that we can significantly decrease the build time by using jobs and burst.

Better error messaging is also up there.

The biped sample matches on direction and foot position.
The bones to match on are defined in metrics (in the Kinematica Asset inspector), so it is all configurable.
Then there's some hysteresis (defined in TrajectoryHeuristicTask) to avoid constantly switching if the next frame in the "clip" is good enough (so that the head is not popping up and down for no reason).

Runtime is decently optimized. There are always opportunities, but it's already jobified and bursted. I think major gains could be achieved by integrating more DOTS tech, as a large part of the cost for Kinematica is to go through the Animator to write to Transforms. We want Kinematica to be 1.0 sooner rather than later, though, so we are focusing on GameObject-centric solutions for now.

Memory size is still mostly unoptimized. We have good ideas, but it's not the top priority at the moment. Usability, reliability and Samples are.
As mentioned higher up in this comment, build times are a big part of usability that we want to improve next.

 

The last minute improvements we made before shipping the public version have unstabilized the behaviour of the demo content, so we need to assess how many changes we need to make. But we have planned our next 3-week sprint to focus on this first. We expect this to be in a decent enough state to show by the end of these three weeks.

 

It's coming in the demo content. We expect to have it available in three weeks. Next minor version of Kinematica should be available in 6 weeks.

You can do that. If you are using tags, but you aren't using Markers, Trajectory or Pose matching, then it's a bit overkill, because it will essentially be the same as doing Animation.Crossfade or Animator.Crossfade. If you use pose matching, then you probably need your queries to be a bit more precise, so that Kinematica doesn't give you a pose in the middle of your clip just because it's a good match.

That's an apple to oranges comparison. The biped sample is doing a search through 35000 frames of animation to find the best next pose, and can handle plant-and-turns without doing anything more than dropping the clips in.

The SnappyLocomotion example is a bit closer to what Legacy does. I took some time to run the profiler on it, and the performance is reasonable. The pure Kinematica/Animation part runs 225 characters in the about same time it takes for OnUpdate to run on the 225 characters to set parameters on Kinematica. Of course, your mileage may vary based on the number of threads you have. Where it hurts the most is the debugger, which has no allocation pooling at the moment, and the code which runs on the main thread to tweak the root motion, which could probably use a bit of optimization.

At the end of the day, we are a bit constrained because we still need to go through the Animator and Playables to write efficiently to Transforms.

You can query the synthesizer as often or as little as you want. When you aren't asking anything, it will continue playing frames until the end of the segment. If you don't trigger a segment change, it will stick on the last frame of the segment until you trigger something else. If you don't query the synthesizer, it takes a lot less processing.

 

You need to use markers. Markers are a bit like AnimationEvents: they have a time, and an arbitrary payload of data.
In the builder, you go add the appropriate markers at given frames to mark important events (right hand contact, or left foot plant, etc..).

The way Michael has it setup in the demo, contact markers have a position and orientation in their payload. These represent the offset from the Root position of the contact, as well as the expected orientation of the surface.

What he does at Runtime is he processes the environment with raycasts and shapecasts, then he looks for segments with the correct tag based on what he found in the environment, then in these segments, he looks for markers in the future that have the right offset from the current position, and a decent normal. If he finds one, then he returns that pose.

While that is certainly useful code to have, it is really implementation-specific, which is why it's not included in the base package. We could have included it in the samples, but the amount of content necessary to get parkour going is fairly large, and it would make the package too big.

You can use IK like Final IK or Animation Rigging to place the hand more finely.

You can. Kinematica uses the Playables API to interact with GameObjects through the Animator. You can layer multiple AnimationOutputs together, in which you can use an AnimatorController, Animation Rigging, or your own Playables. It is a bit harder than using just one AnimatorController or just Kinematica though.

With Root motion.

For Locomotion, the Biped Sample uses a reasonably limited set of animations, and could probably run with way less, we have really just dropped in the files and tagged everything as Locomotion. I believe a Mocap Library named "Huge" should be fine to use with Kinematica.

As for a Unity one, we haven't decided yet.

 

That's a rather long post, and I can't address every single point, but you seem to have a few misconceptions:

Kinematica does have all the frames in the animation, at the given sample rate. It doesn't use inertial blending in-between poses, just between segments (essentially sub-clips). The default sampling rate of 30 frames per second may be a bit aggressive and may cause some minor loss of movement, but you can increase it if you think it lacks accuracy.

Also, in normal operations for things like a jump, it doesn't jump from pose to pose every frame, only at junction points. If your velocities match at the junction points, then there will be very little blending.

A lot of the floatiness of the samples comes from the fact that the camera is naively attached to the hips, which never looks good.

Most of the tools necessary to do this exist in Unity, or in Packages.

• The GameObjectRecorder lets you record transforms and save it to an AnimationClip. It has both a GUI and an API. It also supports keyframe reduction. You can also invoke it frame by frame at your favourite framerate.
• Playables let you blend clips with any weight you want, and you can adjust speeds to match.
• If Playables are too low-level, you can also use Animator Controller blend trees to do fancy blend trees, and you can drive them easily through parameters.
  
• You can also animate Animator parameters, so you don't even need to script your RunTurn_Spirals or your Walk_To_Run

The biggest thing we're missing is full-blown Character Animation Authoring, which we are interested in doing (Animation Rigging is one of the first steps in this direction), but we don't have any definite timeline on it. Animators are particular, and if you make an Animation tool that's halfway decent, no-one will really use it. It needs to be perfect.

At the end of the day, yes, having a fully supported tool that lets you generate your clips would be a nice addition to Kinematica, but it's in the territory of feature creep for 1.0. Kinematica already has 3 different windows (Task Graph, Debugger and Builder), as well as the Runtime. That's plenty enough to maintain for now.

 

The first and last "Time Horizon" frames cannot be used by Kinematica, because Kinematica needs "Time Horizon" seconds of past and future motion to calculate matching. This is indicated on the metric bar by the darker areas.

This is not normally an issue with mocap, which always has unused frames before and after, but it can be an issue if you have short clips.

If you need your whole clip to be used, you can use the Boundary Clip Selectors to provide additional frame information outside of the clip range.


You can toggle it using this control:


Assuming your clip is loopable, you can just put the same clip before and after.
But all of this will only work if your clip is at least "Time Horizon" long. If your clip is under 1 second, then you will need to stitch it together with another clip so that we have enough frames to work with. There are multiple ways of doing that, but an easy one is to copy/paste the keys from two clips to create a 3rd one.

For the moment, we are very much focused on improving the tool itself. We are taking your questions and feedback, and using this to make things easier or more evident. As Marketing and Content get involved, we will probably have better learning content available.

Do you mean on top of the Manual Page, the Scripting API Page and the samples?
The Scripting API is not fully fleshed out yet, but if you have suggestions of things you'd like to see explained, we can definitely improve the docs.

 

Kinematica cannot ingest Humanoid clips. It requires access to the curves of all the transforms, which Humanoid doesn't provide. For the same reasons, it also doesn't support animating a Humanoid character, because Humanoids cannot be animated at Runtime through Transform values. We should add some error messaging around this when you try to add a Humanoid clip.

That said, since we are aware that not everyone has access to their own mocap, and that users will need to get animations from multiple sources, in Unity 2020.1, we've added to the Animator the necessary APIs so that we can support using clips from different skeletons in Kinematica.

By using Humanoid at build time, we can retarget clips Generic from multiple sources to play to a single Generic character, as long as you can provide us valid Humanoid avatars for the source rig of the clip, and the rig you want to play on.

That feature still needs a bit more love though, since we've been focusing on features that we can showcase in 2019.3.

 

When you look at the same UI 8 hours a day for a year, some things that you think should be hidden by default might not be evident to a newcomer. It's a complex tool, and managing the amount of information to show is tricky. This is why we do previews and don't go directly to release.

 

The approach you take depends a lot on what you want to optimize for. Do you want to:

• spend less time tagging?
• have the best possible visual result?
• have the smallest possible memory footprint?
  
• get exactly the frames you want?
• get the best performance?

It will of course be a combination of these things, in an order that makes the most sense to you.

Assuming you have infinite patience for tagging, and you want to keep memory and performance under control, then here is how I would set this up:

1. Join your clips in a way that mimics how a human would showcase the specific motion. For example: Idle->IdleToWalk->Walk->-Walk->WalkToIdle->Idle.
   
2. Tag only the relevant frames: don't tag the Idle sections in this context; you probably don't want 40 copies of Idle with different in/out trajectories in your binary. But have them there so that Kinematica can use them to infer past and future trajectory for the motion you are showcasing.
   
3. For any animation that should loop, you should have at least two loops, so that there exists a set of frames where the past is a transition and the future is looping, as well as a set of frames where the past is looping and the future is a transition.
4. Avoid tagging the same AtoB animation multiple times if you can avoid it (for size purposes).
5. If you generate from blend trees, start with a limited set of angles. Kinematica can alternate from segment to segment with blending to achieve the trajectory you want. If you don't like the results, add more angles.

If you want to spend less time tagging, you can use a short "Stand" animation instead of Idle, and tag the whole generated clip. The idea here is to avoid tagging a long Idle clip multiple times in different contexts..

 

For things like blend trees for locomotion, Kinematica can jump back and forth between "clips" dynamically to support a blendtree-like behaviour and give you the trajectory you want. Of course, it requires enough data if you want, for example, smooth motion in all directions. You should try it with what you have, and see if you get the results you want.

As for ingesting blend trees directly: this is not aligned with the philosophy of Kinematica. The goal for Kinematica is to get the most out of your artist driven animations by specifying intent instead of wrestling with state machines. Automated blend tree import is pretty much the opposite of artist driven. It would also be a nightmare of UX to tag properly. We believe it's better overall to keep the Kinematica Builder solely focused on tagging AnimationClips.

That said, there is a definite need for better examples on how to stitch together Animation Clips that don't come from mocap, as well as a better set of tools to edit clips yourself, not only for Kinematica, but in general. But for now, as much as I am itching to go and hack together something that you can use today, we still need to focus on improving what we have (Builder, Task Graph and Debugger) before we start another set of tools, so that we don't end up with something like this:

 

It's been implemented in a way where when we want to transition to DOTS, it should be easy. But we haven't really tried it in this context yet.

There's no real reason why it shouldn't work, but you'll probably need to build a bit of scaffolding to transfer the poses to either ECS Transform system or to the DOTS Animation system.

I also think you may hit some issues with the debugging and the task graph, which rely in part on the Kinematica component at the moment.

Please tell us how it goes

 

There's nothing specific to be done about the clip, but you still need an avatar for your character.

 

You will get better results if you have trajectories that match the trajectories that you provide to Kinematica in your queries.

If you have no clip or segment that closely matches the trajectory you ask for, Kinematica may jump around a lot between clips, and unless the poses are really similar across the whole character, it will probably show a bit.
You'll always get the best result if you can play sequential frames from a single clip for continuous movement (like pressing straight forward).