How to have both Timeline and a custom playable graph play nice?

We are trying to create an idle system for a character which builds a PlayableGraph that plugs a number of timelines into an AnimationMixer which then gets piped out to an AnimationOutputPlayable attached to an Animator. We want to be able to override the idle system output with timelines in the scene that drive narrative sequences. The narrative sequences are authored as plain old AnimationTracks so that our designers/artists get full use of the native UI. The problem is that if we target the same Animator from our custom PlayableGraph as well as the AnimationTrack in the timeline, the timeline no longer plays. The playable director attached to the timeline reports that its graph is invalid.

From what I understand the Animator is supposed to layer these inputs and the custom PlayableGraph is supposed to override the timeline so it sounds like this is a proper use case. Is there anything else we should consider here?

The idle system's graph and narrative timeline graph are attached below in respective order. The red nodes on the left are AnimationPlayableOutputs.

 

The idle system graph is created like this:

Code (CSharp):

1.  
2.         playableGraph = PlayableGraph.Create();
3.  
4.         // Graph outputs to the animator driving the character
5.        AnimationPlayableOutput playableOutput =
6.             AnimationPlayableOutput.Create(playableGraph,
7.                                            "Character Animator",
8.                                            animator);
9.  
10.         // Create a mixer that will blend any registered playables. Should start
11.         // with 0 inputs.
12.         animMixer = AnimationMixerPlayable.Create(playableGraph, numMixerInputs);
13.  
14.         playableOutput.SetSourcePlayable(animMixer);
15.         playableOutput.SetSourceInputPort(0);
16.  

The idle system has several subsystems that each have their own mixer but those subsystem mixer outputs are connected to the inputs of the mixer listed above like this:

Code (CSharp):

1.  
2.     public void ConnectPlayable<T>(T playable, int outputPort)
3.         where T : struct, IPlayable
4.     {
5.         // Hook up the playable to the next available input in the mixer
6.         animMixer.SetInputCount(numMixerInputs + 1);
7.         playableGraph.Connect(playable, outputPort, animMixer, numMixerInputs);
8.         numMixerInputs++;
9.     }
10.  

Where a subsystem would call ConnectPlayable like this:

Code (CSharp):

1.     ConnectPlayable(subsystemMixerPlayable, 0);

The narrative sequence timelines are separate graphs that output to the same Animator and are being driven by a playable director. As part of our debugging efforts we called IsValid() on the playable director's graph which returns true at init time (Awake) but once the director tries to start playing it returns false. Attempting to perform operations on any playables that were part of this graph also elicit the following error in the console:

InvalidOperationException: This PlayableOutput is invalid. It may have been deleted.

I saw that you mentioned rewiring the output in this thread:
https://forum.unity.com/threads/fade-out-a-playabledirector-so-an-animator-can-take-over.513800/

We tried applying that to the narrative timeline graph like so and calling it on Start:

Code (CSharp):

1.  
2.         // Redirect the timeline graph output to another output that we control
3.         PlayableGraph nisGraph = nisDirector.playableGraph;
4.  
5.         if (nisGraph.IsValid()) {
6.             AnimationPlayableOutput oldOutput = (AnimationPlayableOutput) nisGraph.GetOutputByType<AnimationPlayableOutput>(0);
7.             if (oldOutput.IsOutputValid() && oldOutput.GetTarget() != null) {
8.                 output = AnimationPlayableOutput.Create(nisGraph, "fake", oldOutput.GetTarget());
9.                 output.SetSourcePlayable(oldOutput.GetSourcePlayable());
10.                 output.SetSourceInputPort(oldOutput.GetSourceInputPort());
11.                 output.SetWeight(1f);
12.                 oldOutput.SetTarget(null);
13.             }
14.         }
15.  

This doesn't change anything though. The narrative timeline still doesn't play and the nisDirector's graph becomes invalid and stops after it starts to play.

I should also mention we are on 2017.4.2f2

 

The timelines in the idle system are created using TimelineAsset.CreatePlayable(). The idle system also creates a mixer for blending between multiple idle timelines and then the output of that mixer is plugged into the CharacterAnimator's playable graph. So all of that is one graph.

The narrative sequence is driven by a PlayableDirector and will generally be part of a timeline that could control a number of characters across multiple moments. I think that means the narrative sequence needs to stay a separate graph unless it's possible to rewire the outputs of just some of the tracks as inputs into the mixer on the CharacterAnimator's graph.

Our designers would like to be able to author idle states with timelines so that they can synchronize audio tracks and other things easily with the idle animations. If there's no other way to do it we could fall back on the state machine route for these but it's definitely not our first choice.

So given all of these factors what is the right way to hook this up?

 

Ok so that means I would have to call SetTarget on the CharacterAnimator's graph before any PlayableDirectors that are driving narrative sequence timelines setup their graphs?

This also still doesn't explain why the narrative sequence graph becomes invalid. If both the CharacterAnimator's graph and the narrative sequence PlayableDirector have graphs with AnimationPlayableOutputs targeting the same animator, is it possible that calling SetTarget on one AnimationPlayableOutput could invalidate the graph of the other AnimationPlayableOutput?

 

Unlikely. We don't call Stop() on the playableDirector anywhere and since the timeline never even starts playing it doesn't reach the end. I just tried not starting the PlayableDirector at all and it seems that sometime after swapping the output, the graph's playable handle turns null. Any idea what could cause that?

 

So after some more digging I found that the root of the problem is that one of the idle subsystems which does some procedural animation is calling SetCurve on a generated AnimationClip. After this call, the narrative sequence's timeline becomes invalid. Is this expected behaviour? If it is, what would be the recommended way to blend procedurally generated animation (through either linear interpolation or a skinned mesh) with other animation clips?

 

2018.2 beta is unfortunately not an option based on where we are in our dev cycle. We've been experimenting with blending in a LateUpdate but the trouble is we want to blend the procedural anim with the resulting pose after the idle system and any narrative sequence timelines have all run. We can't find a good place to capture this state after the idle system and timelines. Any recommendations?

 

I guess the real challenge is that we're trying to blend poses by interpolating the resulting quaternion from the state-machine and Timeline update with a procedurally generated pose calculated in LateUpdate. The discrepancy is in the fact that a transform that has been touched by the Animator will have been "reset" at the start of the LateUpdate call by the state-machine/Timeline update whereas a transform that was not touched during that update will retain the procedural pose from the previous LateUpdate call. We can't find a reasonable way to differentiate between the two cases. We tried using Transform.hasChanged but that is not robust as it is set to true for any number of reasons. Is there a better way to approach this that we're not thinking of?

 

@seant_unity by this do you mean use this https://docs.unity3d.com/Manual/TimelineEasingClips.html to adjust start and end of clip to be sloped start / end for blending, and then play a timeline on the director when needed?

If the timeline finishes will that mean the graph is now null / empty? or will it transition back to the graph that was playing. trying to get my head around what having a playable director component does, as I know it has its own graph. Does playing a timeline asset on a animator via a director replace the graph or simply connect new things to it?

Essentially from the quoted part above, I think I have massively misunderstood playables and timelines and believed them to be far more complicated than they actually are.

I just want an idle animation system, but one where I can trigger one shot timeline clips that have audio and other tracks, have those clips play, then transition back to the animation system. What you wrote makes it seem like that is very much in grasp and without too much headaches so eager to clarify this!

EDIT: Anyone interested, the answer is yes. It is this easy. Color me happy

 

@seant_unity @mikew_unity
I have a question about the part where you say to call TimelineAsset.CreatePlayable() and to bind the PlayableOutputs.

I'm able to create the playable from the timeline, connect it to my PlayableGraph, and get it to play my animations and particle effects that are stored in the timeline, but I'm not able to get the sounds to play. Do you have any samples of how to bind the PlayableOutputs when using TimelineAsset.CreatePlayable?


I was previously just playing my PlayableDirectors directly instead of trying to put them into the PlayableGraph, which worked pretty well, but I wasn't able to blend out of the Timeline at an arbitrary point, hence why I'm trying to switch to using a PlayableGraph to run the timelines.

I followed @mikew_unity 's example to do Timeline Output binding from this forum post which worked well for PlayableDirectors, but I can't figure out how to bind outputs for when I use TimelineAsset.CreatePlayable

 

For reference, here's what it looks like when I play the Timeline via TimelineAsset.CreatePlayable connected to my PlayableGraph:


Perhaps the reason I don't hear any audio when I'm playing via the PlayableGraph is because I'm not connecting the audio playables to an audio output? It seems wrong that I'm connecting the audio playables to the AnimationOutput root.



And here's what it looks like when I play the same Timeline directly through a PlayableDirector:

 

@seant_unity

I can't thank you enough! I did what you said, and it totally worked:


The red circle at the bottom of the graph is my shiny new AudioOutput, and sounds from my TimelineAssets are successfully playing! I had to remove the PlayableDirectors from all my gameobjects, since the Editor was crashing when both my PlayableGraph and the PlayableDirector were trying to play the same audio track (some crash in FMOD). I see you mentioned that playing both is a bad idea above, but I wouldn't have expected the whole editor to crash from doing this (instead, I'd hope the editor would report an error to the Console instructing on the best practice, since everyone might not have the luck to find this awesome forum post). Here's the callstack in case it's helpful:

Code (csharp):

1. ========== OUTPUTTING STACK TRACE ==================
2.  
3. 0x0000000141C3DF33 (Unity) OnApplyFMOD
4. 0x00000001406D65F2 (Unity) DirectorManager::ExecuteStage
5. 0x00000001406D69A2 (Unity) `DirectorManager::InitializeClass'::`2'::PostLateUpdateDirectorLateUpdateRegistrator::Forward
6. 0x0000000140967E07 (Unity) ExecutePlayerLoop
7. 0x0000000140967EB6 (Unity) ExecutePlayerLoop
8. 0x000000014096A75C (Unity) PlayerLoop
9. 0x0000000141447E4A (Unity) PlayerLoopController::UpdateScene
10. 0x0000000141446063 (Unity) Application::TickTimer
11. 0x00000001415E45A5 (Unity) MainMessageLoop
12. 0x00000001415E698C (Unity) WinMain
13. 0x0000000142423D0A (Unity) __scrt_common_main_seh
14. 0x00007FFABFC33034 (KERNEL32) BaseThreadInitThunk
15. 0x00007FFAC1ED1431 (ntdll) RtlUserThreadStart
16.  
17. ========== END OF STACKTRACE ===========

and here's a snippet for anyone else trying to do this:
Essentially, I was looping through the timelinePlayable inputs (mostly for debugging to know what I had to work with) and then wiring up to the AudioSource when I found one.
The inner for (j) loop can mostly be ignored - the important items to hook up to the graph were found at the root of the TimelinePlayable, thankfully.

Code (csharp):

1.  
2.         Playable timelinePlayable = timelineAsset.CreatePlayable(m_Graph, gameObject);
3.         int count = timelinePlayable.GetInputCount();
4.         for (int i = 0; i < count; i++)
5.         {
6.             Playable curPlayable = timelinePlayable.GetInput(i);
7.             if (curPlayable.IsValid())
8.             {
9.                 Type curPlayableType = curPlayable.GetPlayableType();
10.                 if (curPlayable.IsPlayableOfType<AudioMixerPlayable>())
11.                 {
12.                     // TODO: check for the existence of an AudioPlayableOutput instead of just creating one directly
13.                     AudioMixerPlayable audioMixer = (AudioMixerPlayable) curPlayable;
14.                     AudioSource audioSource = this.GetComponent<AudioSource>();
15.                     var output = AudioPlayableOutput.Create(m_Graph, string.Empty, audioSource);
16.                     output.SetSourcePlayable(audioMixer);
17.                 }
18.                 int innerCount = curPlayable.GetInputCount();
19.                 for (int j = 0; j < innerCount; j++)
20.                 {
21.                     Playable curInnerPlayable = curPlayable.GetInput(j);
22.                     if (curInnerPlayable.IsValid())
23.                     {
24.                         Type curInnerPlayableType = curInnerPlayable.GetPlayableType();
25.                     }
26.                 }
27.             }
28.         }
29.  

 

I still have 1 question - what is an "UnityEngine.Animations.AnimationMotionXToDeltaPlayable"? It seems protected, and I'm wondering if I can remove the AnimationMotionXToDeltaPlayable, AnimationOffset, and the AnimationLayer to avoid all of the strange root motion offsets from the Timeline?

 

@seant_unity that's great! thank you! I'll experiment with trying to remove those, since our application of Timelines is purely gameplay so we aren't ever using offsets.

I'm still having some trouble wrapping my head around how to blend TimelinePlayables together, mostly when it comes to VFX now.

Just for some context, my goal is to create a PlayableGraph that is completely responsible for coordination of character animations, character foley audio, VFX, and gameplay events. I want our team's artists to be able to author timelines that contain all of these elements for the different "Skills/Abilities" in our game (a character action RPG), and whenever a skill is activated, use the PlayableGraph to blend from a base AnimationControllerPlayable that contains our locomotion blend tree into a TimelinePlayable that contains all of the visual and gameplay aspects of an ability. I need to be able to blend out of an ability TimelinePlayable into another TimelinePlayable (for combos) or back into the base AnimationControllerPlayable if we get cancellation input (e.g., pressure on the joystick to break out of an ability early). These breakouts/blends can happen at any point in the TimelinePlayable, not just at the end or at a fixed point. I've written custom tracks (the red tracks in the image below) that define these breakout windows.

Here's an example of one of our Ability timelines:

I was able to get the first animation track ("@Attack_Combo1") to be inserted into my large AnimationOutput graph and blend the animation by using TimelineAsset.CreatePlayable and hooking that into the AnimationOutput.

I was also able to get the audio track ("run_combat_attack_sw...") into my graph via the method you proposed (creating a new AudioOutput and wiring it up to the AudioMixerPlayable from the Timeline.

Which brings us to VFX - I'm not sure how to play my VFX tracks without using the PlayableDirector. All of the VFX objects are parented under a child GameObject of my character, so I'm not sure how get my custom PlayableGraph to control the particle systems/VFX objects.


I see 3 possible paths to move forward: (I ordered them in terms of how difficult I perceive them)

1. Continue to use PlayableDirectors, but when I instantiate the PlayableDirector gameobject into my Character's hierarchy, I need to iterate through the outputs, fetch the first track (the Character animation), destroy it from the PlayableDirector (so that the director doesn't touch the Character's Animator and just does gameplay events, audio, and VFX) and insert the TimelinePlayable from the PlayableDirector.playableAsset.CreatePlayable into my custom PlayableGraph that does the blending with the base AnimatorControllerPlayable/other TimelinePlayables
2. Somehow take all of the outputs (incl. the ScriptOutputs and AudioOutputs) from the PlayableDirector, and move them into my PlayableGraph, making sure to take the AnimationOutput that controls the Character's Animator and slot that into my existing AnimationOutput so that I can blend it
3. Remove the usage of PlayableDirectors from my project entirely and somehow figure out how to mimic the way the PlayableDirector enables/disables gameobjects and controls particle systems

What would be your suggestion for how to get a TimelineAsset's VFX tracks to play when using a custom PlayableGraph instead of PlayableDirectors? I'm not sure which APIs I should be using to let the ScriptPlayableOutput to "know" which VFX gameobjects it should be controlling. I had a look at `scriptOutput.SetUserData` and `scriptOutput.SetReferenceObject` but those didn't seem to do anything.

Thanks again!
- John

 

@seant_unity you're the best man, thank you

If you come to Unite LA this year, please let me buy you a coffee or a beer

I actually managed to get my option #1 to work pretty well last night. My approach was basically this:

1. On instantiation of the PlayableDirector's GameObject that holds my Character's ability's animations, events, audio, and VFX,
2. Search the outputs of the playableDirector.playableGraph for the first AnimationPlayableOutput (as you can see in my image above of my Timeline structure, the first track is always the character animation)
3. Disconnect that AnimationPlayableOutput's Playable from the graph (so that the PlayableDirector is no longer in charge of Character animation)
4. Then use playableDirector.playableAsset.CreatePlayable to connect the Timeline into my custom PlayableGraph that holds my other AnimatorControllerPlayables and TimelinePlayables that are used for animation blending

The code looks like this:


And the resulting before/after graphs:




Hopefully there's nothing glaringly wrong with this approach... It's actually quite nice since I can blend away from a Timeline's character animation in my PlayableGraph, and have separate control over whether I destroy the PlayableDirector that's playing out the audio/VFX. In a lot cases, I still want the VFX or audio to hang around after the animation is done playing, e.g. for foley audio on a bunch of quick combos in succession. The part I don't like about it is that I'm not sure if the Timeline in my custom PlayableGraph is actually synced up still to the PlayableDirector's graph.

FYI, at first I tried calling playableDirector.playableGraph.DestroyOutput on the AnimationPlayableOutput and I got tons of errors from some internal part of PlayableDirector. I don't think it can handle it when developers DestroyOutputs, but calling Disconnect didn't seem to cause any issues... As much as I love all this new Playables stuff, it seems like it's very easy for a developer to poke around at Playables and either a) get LogErrors from within Unity internal libraries that you can't address or b) bring down the whole editor. When there's not a lot of example projects around, it's pretty easy to run into this kind of issue.

If I was to go a little deeper with option #3, what you mentioned is key for me to be able to do that, thank you! It seems a little daunting to recreate the entire functionality of the PlayableDirector.

Is there any easy way to do my option #2 where I grab all of the PlayableOutputs from the PlayableDirector's graph and dump them into my custom PlayableGraph? I couldn't find an easy way to copy a PlayableOutput from one PlayableGraph to another without recreating them...