Feedback A New Approach to DOTS Animation and Motion Matching?

After looking at this presentation, although a bit dry (and lacking on the details), their approach solves A TON of problems I saw with Kinematica, AnimationRigging, and DOTS Animation all at once. The approach used is general enough that, if allowed to be combined with DOTS Animation, any studio could have a small list of poses, without the need for much in the way of tagging, allowing that "tagging" and combining of tags to work in a similar way to DOTS tagging in queries when operating on standard programming to help "blend" poses that depend on both context and state tagging. Giving a bit of a "spring" to the limbs during transitions (based on a painted-on mask that allows some bones to have more influence between poses, like a shoulder having less "spring" when changing between extreme poses than an elbow or wrist for example).

At the end of the day, I could see this method, combined with DOTS and motion-matching / animation rigging being used for realistic motion in any style of game -- without the need to create HUGE databases of animations and mocap data (which was my main reason for not using Kinematica to begin with -- I prefer Animation Rigging or nothing at all).

As both an animator and a programmer, I know firsthand that the ideal path for democratizing animation for such extreme context-switching as a game requires is to enable fluid and realistic _procedural_ motion that is aware of context.


DOTS Animation is a perfect breeding ground for very performant interplay between realism, responsiveness, physics and gameplay.

These guys did a great job. As an animator, I sincerely believe it would be worthwhile for Unity to study (carefully) their solution in a way that would allow it to be combined with standard DOTS Animations (i.e. the way Animation Rigging currently works would be a great start for a visual-editing approach to a base interface -- treating character design for DOTS Animation as nicely UX-wise as one might treat scene-authoring tools like Terrain painting). Just food for thought.

 

I am not convinced that DOTS Animation is going to provide a solution that will allow for the context-aware tight interplay you and I seek. From what I have seen so far, Unity seems intent on providing an out-of-the-box code-free solution to animation, which unfortunately makes such tight integration more difficult if not impossible.

I am convinced that DOTS in general allows for this, which is why I built my own low-level animation solution with the goal of providing the basic utilities needed to implement a higher level animation feature set such as motion matching.

I have two challenging questions for you:
1) What low-level features would you need to prototype a motion matching implementation you want in DOTS?
2) What would be your technical requirements for a motion matching solution in DOTS, both from an authoring perspective and the perspective of a gameplay and integration programmer?

 

Considering how long it has taken them to actually talk about it (and their various business decisions and strategies as of late) -- I'm inclined to agree with you on that.

To be fair, "code-free" doesn't have to mean "performance versus flexibility".

A code-free solution would be ideal -- but Unity has a terrible track-record with UI and providing a flexible and performant underlying API (alongside that nice UI / UX) where necessary. The bigger problem is that Unity management tends to define "where necessary" themselves -- and they are often wrong.

This is, of course, because they don't eat their own dogfood. Gigaya proved this. Remaking their animation controller was still in the pipeline and to be robust enough for proper game development, they would have had to write tons of special tools and workflows to get it there. Not for the faint at heart. So at least some people at Unity got a taste. Then they tossed those guys. Apparently the higher-ups want Unity to be a mobile ad business, when it is, in reality, a game engine. And by being a game engine, one would think that helping devs (who'll distribute your ads) create optimized and playable _games_ would be the top priority. There are already too many advertisers, and sometimes I think, in the end, they just spin their wheels because some VIP secretly wants to tap out of any of the "hard" stuff that might make them look bad. :/

That's my current theory at least. -- But I digress.

These are definitely involved questions, but I'll just say off the top of my head what comes to mind:

The way I see it is that one should first start with the UX.

There needs to be a way in the UI / UX to shift around the offsets for IK targets. For example, hopping over a short wall of various heights, or placing the body and hands (i.e. as programmable frame "layers") to specify climbing a ladder programmatically. Then lerping or curving between these layers in a predefined set or frame order (via something like AnimationCurves) toward the two or more IK destinations defined on each "layer" in each "frame", and giving each a "weight" value and a "physics" value in each bone joint (to allow for springy ears or tails). This is somewhat how AnimationRigging is done -- but not quite. Recording keyframes across the timeline was how they did it, but defining a simple animation curve and duration would have been much better than forcing us to use Timeline and Playables IMO. Doing it that way took away the ease of simply programmatically changing or redefining the "frames" and "layers" since, at the end of the day, these were defined in completely separate and unrelated systems (with very heavy API and other boilerplate code necessary). At the end of the day, only a Technical Animator would stare down this system, meaning only AAA devs would ever actually use AnimationRigging for a full production, for example.

As far as "motion-matching" is concerned, I would do it in a similar way as they do here:

Only requiring a sort of "proxy rig" which consists only of the specific joints, bones, scaling/rotation of major poses, animations, pose weighting and transformation of the particular IK targeted bones you care about animating procedurally would allow a transferable "rig" that also remains programmable. To handle physics, having a "ball" or "sphere" physics object (like in the latest video above) that determines repetitive IK footstep positioning, animating cyclic timing behavior, and direction of the actual physical movement to which a walking animation should prioritize, and attempt to be applied (which is the only real thing Kinematica was actually good at) is done in that video very simply by handling the placement of the IK targets and blending them with preset poses and preset frame data of a "proxy rig" that only animated the IK targets, allowing nice, fluid movement that can be weighted in multiple ways (including programmatically) usually in some sort of visual inspector towards physics data, animation layer IK data, current or next/previous frame data, or only activated depending on what the overall motion "tag" actually is (which means one can tell the animation it is a "run" whether it is also tagged as a "crouch" or a "kick"), and this "blending" and weighting would be dependent upon which bodypart / bone / joint (or joint-chain) data the "proxy rig" is pointed at. After authoring once, that proxy rig can be applied to any other model.
And as far as DOTS goes, this can be utilized and made very performant when many animations require similar (or the same) proxy data on certain joints or proxy rigs. Rather than weighting based on different proxy rigs, it would be ideal to apply proxy "components" to joint chains and proxy rigs (i.e. floppy ears or tail having weighted physics, but NOT frame data, while preserving root transform -- but not tip transform -- based on frame or layer data). Also any sort of "ball" or "sphere" rotation (which would be cyclical or repetitive, such as footsteps) would be a DOTS "component" that favors physics data, but uses the weighted (sphere/ball rotation) physics data to weight the _frame_ data toward the next or previous frames. This "component" could be reused for anything -- including opening and closing of hands based on their motion toward a target (i.e. punching a baddie, or even just bobbing the head up and down if running).

At the end of the day, this approach solves "motion-matching" in a cheap, easy-to-author, way that doesn't require bloated APIs, and actually includes physics and programmable "frame" and "layer" states, as well as joints that react to both frame data and physics data, depending upon a user's requirement. And the best part is -- blendshapes, etc. (when Unity adds them into DOTS) should theoretically still work.

One important bit though -- Try not to get lost in the rabbit hole (pun intended) of trying to implement the exact solution in that GDC video. Use AnimationRigging and my suggestions above that improve upon it to help define the technical requirements to do this.

I answered most of this question in my above paragraphs. However, the gameplay portion of this is important to handle and should take into account stuff like what I mentioned about the AnimationCurve authoring, the "ball/sphere" rotation concept (to handle cyclical or repeating animations across a sequence or particular "set" of animation frames), and be sure not to write the transform until the physics system is done (for parts that use physics weighting, possibly using an invisible "Tag" that is applied to let the system know, programmatically, that it is using a different kind of processing for the individual frames during a particular "Tagged" animation -- such as a walk/run, which itself is tagged as something like "forward momentum" or something similar, using another tag that determines whether motion is in the "air" (which is typically the "special case" -- telling the physics object it is "grounded" has always seemed backwards to me).

Hopefully that gives you some idea of how something like this should be implemented.

If you do implement something like this -- all I ask is for it to be free, mostly so that a poor person like me can use it too.

Unity seems to have its thumb up its butt right now, so to anyone who wanted a great DOTS Animation solution, but was disappointed by Unity for teasing and then tossing anything related to animation aside (including non-DOTS stuff), you'd be a hero if you were to make something like this and release it.

Any design work or help you need on the UI / UX front, just ask me. I'll take care of that part if you like. Been needing something like this for quite a long time. Anything I can do to help, again, just let me know.

 

I think it's a good thing to put such a talk on the table again!
@awesomedata, I guess we can divide your post in two main point :

1. a Nodal GUI that expose animation stream handling api with Nodes : Conditions, Custom Properties bindind etc, as a state machine do
   
2. An way to add AnimationRigging to the DOTS stuff (which is far more complex than the first ask)

I do agree with the Awesome about the 1. CodeFree Interface (GUI) we need to have, just thinking about the community,

About - 1. :
I do agree, If the user got an easy way to create/script some custom Nodes to handle Animation Stream using the underlying Animation API, so I bet someone intelligent can adapt Kinematica's alike system to be used in a graph as in the GDC presentation of Control video posted before. But to be honnest, to make the DOTS Animation Authoring easy and understandable for all the Unity Community who want to dive in DOTS for 1.0, it have to get closer to the control of the actual Mecanim system in term of usability or they will be lost in the wood. So thinking about user-wide more than nich-case is more critical.
So classic Node first : Multi-Directionnal BlenTree, Nblend, LinearBlend, Root Handler, IKHandler, LookAtHandler, and ofc Editable Curve/time for all the transition state. If it's nicely made and extendable, it will be 100% a success story

About 2 -
Animation Rigging is another cup of tea, it is far more complex to do that, and so I would say it's not critical at all : not for 1.0 at least, AnimationRigging is more than just a simple IKcomponent handler, ok it is a usefull system but Procedural Animation is a nich-case of using animations too.

I had some good time using Animation Graph that was present in the very begining of DOTS (and DOTS VS) packages, as I pushed the test to script some custom blending Nodes and to make a little 3rd person controller, I can say that even is the system was far to be finished and polished, I could feel the strenght of the workflow, of what it could become, it was very easy to connect Animation and States with some entity Data & input system Data, it was just nice to begin with

 

I guess my point was that such an underlying API is necessary for really tight gameplay integration, and that consequently requires game-specific code. And in some cases, game code may need to completely control the execution of the animation rather than let it happen automatically by a Unity system.

But yeah. I think we are on the same page.

As much as I would love to, I can't for three reasons:
1) It's a pretty massive leap, and I don't have that kind of development time. I need smaller steps.
2) I don't understand half of what you wrote.
3) Even if I did understand everything, I don't believe it fully encompasses what you desire.

Unfortunately, the only person who can build what you desire is you. But what I can do, is build a solid underlying API that gets you a lot closer while also being useful for people who may have slightly different desires. I already have a base set of such an API written, working, and publicly released. So now I am trying to identify how I can expand that API to empower more people.

For example: Here are the types of underlying features I have already fulfilled:
1) Ability to render a skinned mesh based on bone transforms as entities or buffers (with automatic frustum culling)
2) Ability to sample an animation clip at any time point
3) Ability to blend animation clip samples
4) Ability to pack animation clips into a DOTS-friendly blob asset during conversion

And these are the requirements I was able to extract (somewhat), so please correct me if I am wrong or missing something:
1) Some form of IK solver (needs more definition)
2) General purpose animation curves that can be arbitrarily sampled and packed into DOTS-friendly blob assets during conversion
3) A mechanism for splitting off the IK targets into a separate skeleton during conversion
4) Some kinetics-based animation utilities for computing dynamic secondary motions