Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. We have updated the language to the Editor Terms based on feedback from our employees and community. Learn more.
    Dismiss Notice

Timelines with dynamically added outputs on creation.

Discussion in 'Timeline' started by Jakub-Slaby, Sep 20, 2019.

  1. Jakub-Slaby


    May 6, 2017
    Hi there, I'm trying to extend the timeline workflow for our artists to be able to specify tracks that affect multiple objects at once.

    Basically the use case is:
    We have effects and game events that have random number of involved elements, but we still want an artist to be able to specify how those elements (divided in to logical groups) are being animated and handled.

    I have to say I got pretty close to having this system running, I only have one issue: being able to specify animation tracks to run with this.

    I was able to specify custom tracks that when the TrackAsset.CreateTrackMixer is called, will pre cache all the objects that need to be involved. And when the TimelinePlayable invokes the `outputs` getter I return these objects wrapped in ScripPlayableBinding.

    The issue I hit over here was the lack of playerData set on that output when the timeline was executing. I tried to SetGenericBindings, but unfortunately I don't have access to the playable director anywhere from the TrackAsset creation (except from GatherProperties). And I didn't want to additionally add functionality on top of the PlayableDirector to pre execute these bindings as the code creating those groupings is enclosed in a query system on top of the tracks and custom clips.
    My solution, was to create a custom mixer that on ProcessFrame would actually call the output.GetReferenceObject() and output.SetPlayerData to that returned object - That worked great!

    I was able to run both custom movement scripts nicely, Spine animations and such...

    Until I went to integrate Unity's AnimationClips and AnimationOutputs.
    The complications came from both what the animations require to properly run and how the timeline playables are executed when generating the graph.
    Because the animation graph tree is quite complicated when I generate my custom graph I want to create my "wrapper" graph (for sorting out the output playerData) with a AnimationLayerMixer and AnimationMixer.
    Unfortunately the TrackAsset.CreateTrackMixer() only allows me to return 1 target mixer, hence I cannot connect these tree.
    I decided to try coming at it the other way, using the AnimationTrack as a base class, unfortunately that approach is also flawed as it overwrites the default creation of playable without giving any way of modifying it, which meant I never got access to the GameObject to fetch my dynamic objects.

    I was wondering if there is a flow that I might be missing, I want to keep the logic of creating these dynamic outputs as simple as possible with possibly no changes to the component / asset structure apart from custom tracks and clips with extra functionality.

    I already tried custom TimelineAsset extensions overwriting the CreatePlayable but it poses similar problems as when I do create the graph and want to process it later to potentially create the outputs, I'm missing all of the data from the tracks.

    I have created custom outputs from the TrackAsset.CreateTrackMixer() and bound them manually, though they're always added before the actual timeline track output and don't get the time control, which I had to manually do, and sync from the main track - this worked, and allowed for animation to be played out, but had at least 1 frame worth of delay, and the last frame of a clip had never executed.

    If there is nothing in place for allowing me to do this, would it be possible to consider some of these points:
    - Ability to create custom PlayableBinding types with our own Create methods for outputs - currently most of this is internal and I'm unable to customise that flow
    - Callback from TimelinePlayable.CreateTrackOutput() on the track, so we would be able to process these if needed
    - Ability to overwrite AnimationTrack.CompileTrackPlayable or a callback to process the created tree.

    We really want to use the timeline system for our in game animations, and it's proven to be quite nice and flexible - unfortunately our current implementation works around the above issues creating that delay in execution.

  2. seant_unity


    Unity Technologies

    Aug 25, 2015
    Awesome. That's great work. Targeting multiple items is a great feature to add! Feel free to correct me here, if I didn't quite follow the approach you got working, or if I'm misunderstanding the desired result..

    For custom tracks, a single binding that represent multiple outputs is possible because ProcessFrame gives you control over how many targets to write data to. You can either create playables for each target, or have a single playable (mixer) write to all targets, which ever works for your use case.

    Also, the GameObject passed in CreatePlayable() is the gameObject that contains the playableDirector being used to compile the object. (This is more reliable than GatherProperties, which isn't called outside of edit mode).

    If you are trying to run a single animation track on multiple targets, then that is definitely much more complicated. You will need to have an AnimationPlayableOutput per target. Unlike script playables, the animator needs to know which playables are targeting it, not the other way around. Timeline also assumes animation tracks have a certain format and does post processing on the sub-graph to propagate blending correctly.

    Basically, you need 1 to generate track per target. Instead of making the track compile to multiple outputs, compile the track multiple times.

    Some thoughts/hints about how to do that.
    TimelinePlayable.Create takes an arbitrary list of tracks. I haven't tried, but I see no reason why passing multiple copies of the same animation tracks wouldn't work. This could be useful in a post-processing step after the PlayableDirector makes it's graph (via a custom monobehaviour). Or in some kind of Proxy track, maybe?

    Use that to make a second timeline playable and connect it with the first, using the existing PlayableGraph. You might need to create and bind the outputs yourself, including a script output for the timeline playable. It would also need a mechanism to sync the time between them. Playables have a method that tells them to pass all explicit calls that set the time on to their inputs. Enabling that on the existing root playable might work here.

    Anyway, that's just a thought. Maybe you've tried that path, but that should be easier than overriding AnimationTrack, which, while possible, wasn't really intended.
  3. JakubTrailmix


    Jun 1, 2018
    Hey @seant_unity thanks for your reply.
    This will be long... but I'll try to get a lay down of how we ended up working around the issue

    We're able to wrap the functionality in and get the result. I think my question would be if the Timeline team could consider in future feature set extending a bit more the control over what get's created, or extend the creation a bit more.

    Just to give you an example what we've ended up doing...
    I'll call all of our dynamic objects that the tracks are created for Targets, to make sure it's not mistaken with the Bindings
    We have two main track classes:
    1) Basic one that modifies the TrackAsset.outputs getter and creates a mixer with multiple outputs.
    2) A wrapper track which creates custom outputs for each Target in the CreateTrackMixer method and custom mixers for synchronising the weights with the output and mixer that's attached to the timeline.

    1. The implementation overwrites the CreateTrackMixer at which point it creates the list of dynamic Targets.
    Creates a mixer that has to extend from an abstract class (will mention later why), returns it, and proceeds.
    Next we overwrite the TrackAsset.outputs getter, which instead of returning the default getter will return custom list of
    PlayableBindings with those Target objects.

    When that track is executed, it hits the custom mixer ProcessFrame and checks if the playerData is null - first time it always is as I'm unable to write the correct user data for the output.
    It then writes that userData from PlayableOutputExtensions.GetReferenceObject(info.output); and it proceeds to execute the mixer.

    2. The wrapper approach comes from the need of creating a new playable tree per item, when it cannot be just a lot of outputs attached to a single mixer, like when using Unity Animations or Spine Animations.

    In the TrackAsset.CreateTrackMixer we not create custom outputs on the graph, which we cache and we create a custom set of mixers. We attach those mixers to the output, and we create a Wrapper mixer which get's the custom mixers cached in - and that's the mixer we return.

    When the track creates playables for each of the clips, we attach them to the custom cached mixers as well as the default connection to the Wrapper mixer.

    Here comes the problem, we now have let's say custom animation tracks for 6 Targets, but they're not connected to the timeline, they're connected to custom outputs, so the connection between our custom mixers and the clips will never get it weight set.

    That's where the Wrapper mixer comes in, having the list of custom mixers it will iterate trough all inputs, and then update the weights on all cached mixers.
    Unfortunately, because the custom created outputs are made before the actual timeline output, the execution of the weight update is always after the custom mixer update, causing the objects to update 1 frame to late, or never to execute the last frame if it's the last frame of the timeline.


    I have tried the post processing of a graph after it's created, but it proves to be more tricky, as all of the logic for finding targets and assigning them needs to be unified in that processing class, that makes further extensions more complicated as the targets have different rules and types of object being affected - keeping that to the track is a much nicer way to work with.

    Tried using SetCustomBinding for all of the objects involved, but similarly it makes a big "master class" with a lot of junk and we additionally need to cache these lists for each instance as we're caching all of our effects, the bindings don't just get reset, and than random things start animating.

    I've tried creating a new AnimationOutput for each animator and assigning to the same track and as you mentioned, that was definitely not the right way. Got some interesting glitches.

    I don't really want to modify the asset dynamically to create new tracks, as I mentioned, each time we spawn that effect, it's coming from a object pool, it than needs to be reset, and keeping track that the asset has no modifications after each play is also a hurdle.


    So in conclusion, this approach works for us, and we're able to get around it, we've been able to mostly solve the last frame not executing by adding additional OnGraphStop() and some value caches in some of the Mixers.

    My question though would be, would the Timeline team consider adding some more flexibility around track creation and or being able to extend some of the internal classes a bit better.

    Creating custom PlayableBinding types with custom CreateOuput methods <- that would be actually great, would solve the ability to assign custom set user data.

    TrackAsset.CreateTrackMixer - to have additional flow in which we can specify more than just one playable, basically to return one that get's assigned to the Timeline playable, and one that the clips get assigned to.
    That way we could easily create a AnimationLayerMixer and AnimationPlayableMixer combo on a track, now I'm only able to return one playable :/

    Ability to add outputs that get assigned to the Timeline playable - as mentioned before, when I create a custom output, I'm unable to set in which order it's created and the playables I assign there don't have a reference to the Timeline playable
    As a workaround, I'm ok having the Synchronisation Wrapper Mixer that I mentioned before, but if my custom outputs get executed before that, so an ability to create those outputs after the default ones are done, would be amazing. Even a custom callback.

    Thanks! Hope this sheds some more detail on what we're doing with the timelines, and maybe gives you ideas for consideration around dynamic features of the Timelines.
  4. JakubTrailmix


    Jun 1, 2018
    Just noticed I posted the messages from different Unity accounts ;P
  5. seant_unity


    Unity Technologies

    Aug 25, 2015
    Thanks for sharing that. I'll try to address some of the issues below.

    Definitely a yes to this one - we've noticed it's a tricky limitation as well.

    Probably not - the bindings are typically implemented by subsystems that implement native playables (animation, audio).

    You can do this but, I think the limitation you will run into is the ability to add playables to the scheduler. That is currently internal.

    I was curious about the post process approach - so I threw a small script together as a test. I was able to get multiple objects animating from a single track, although this is by no means a final or polished solution. I'll post it so you can see how I did it - and if it helps you.

    I discovered that TimelinePlayable.Create does not handle multiple of the same track, so instead I have one timeline playable per track. Not ideal, but it does show the idea of how to do post processing of a timeline to create arbitrary outputs. It's animation track only, but this would be applicable to any type of track with an binding.

    Now, not to spill the beans too much, but the single track -> multiple target is something we have been investigating as part of having a DOTS-compatible timeline. We have a working demo (still very early in development), and it takes a different approach. I mention it because many of the issues you've raised, we've addressed there and it's something you may want to look out for in the future.

    Code (CSharp):
    1. using System;
    2. using System.Linq;
    3. using UnityEngine;
    4. using UnityEngine.Animations;
    5. using UnityEngine.Playables;
    6. using UnityEngine.Timeline;
    8. [Serializable]
    9. public struct AdditionalBinding
    10. {
    11.     public AnimationTrack track;
    12.     public Animator binding;
    13. }
    15. [ExecuteInEditMode]
    16. [RequireComponent(typeof(PlayableDirector))]
    17. public class AdditionalBindings : MonoBehaviour
    18. {
    19.     public AdditionalBinding[] additionalBindings;
    21.     private PlayableGraph m_lastGraph;
    23.     public void Update()
    24.     {
    25.         var director = GetComponent<PlayableDirector>();
    26.         if (director.playableGraph.IsValid())
    27.         {
    28.             if (!director.playableGraph.Equals(m_lastGraph))
    29.             {
    30.                 m_lastGraph = director.playableGraph;
    31.                 var root = director.playableGraph.GetRootPlayable(0);
    32.                 root.SetPropagateSetTime(true);
    34.                 for (int i = 0; i < additionalBindings.Length; i++)
    35.                 {
    36.                     if (additionalBindings[i].track == null)
    37.                         continue;
    39.                     var other = TimelinePlayable.Create(m_lastGraph, new[] { additionalBindings[i].track }, gameObject, true, false);
    40.                     root.AddInput(other, 0, 1);
    42.                     var output = AnimationPlayableOutput.Create(m_lastGraph, additionalBindings[i] + " i ", additionalBindings[i].binding);
    43.                     output.SetSourcePlayable(other, 0);
    44.                 }
    45.             }
    47.             m_lastGraph = director.playableGraph;
    48.         }
    49.     }
    51.     public void OnValidate()
    52.     {
    53.         // unhide the tracks so they can be picked in the inspector
    54.         var director = GetComponent<PlayableDirector>();
    55.         if (director != null)
    56.         {
    57.             var timelineAsset = director.playableAsset as TimelineAsset;
    58.             if (timelineAsset)
    59.             {
    60.                 foreach (var t in timelineAsset.GetOutputTracks().OfType<AnimationTrack>())
    61.                 {
    62.                     if ((t.hideFlags & HideFlags.HideInHierarchy) != 0)
    63.                     {
    64.                         t.hideFlags &= ~HideFlags.HideInHierarchy;
    65.                         UnityEditor.EditorUtility.SetDirty(t);
    66.                     }
    67.                 }
    68.             }
    69.         }
    70.     }
    71. }
  6. IARI


    May 8, 2014