Question How to create a custom Track/Clip that plays audio besides other things

I have a usecase where I'd want a custom track / clip system that contains both audio and other data in a single datastructure (e.g for dialogue). The structure works fine, but I have trouble understanding how to create a custom track that in turn creates both some PlayableOutput and AudioOutput.

After looking at what an AudioTrack is doing it became clear that a ton of the code in there is purely internal, and while it's not sealed the properties inside aren't inheritable either.

Is there any documentation or hints, or an example, on how to get a custom track working that does both custom things and play audio? Bonus points for in-editor playback and scrubbing....

 

You can have 50 tracks that are simply synched and handle the individual pieces. Or you can simply run a script or have them communicate with signals. So you can have your text boxes/dialogue on one animation track or activation or whatever. And an audio track.

 

Hm, no, that won't work in my usecase. There's more to it than simply having an audio track in sync, there are different audio clips that might be played back, etc.
Also, this is a workflow tool for artists, so I'd like it to really be a "drop this thing here in the timeline to get the result" tool.

 

There are a few blogs on this concept. Where you can jump all around the Timeline kind of like Animator state changes. However, IMHO, that creates a kind of spaghetti code. Flash animation used to be the same way. You had this enormous animation deck and could have code embedded anywhere that could do anything and bounce you around to infinity. And when I saw that, I was horrified. I have a long, corporate programming background. And Timeline should be a...Timeline. An L->R time-based organization. If you're going to have like signals popping off all over and jumping around the Timeline, I think you undermine the concept of a Timeline. If you read the first few sentences/paragraphs describing Timeline, its purpose is clear. You can stick code anywhere, but then you've made monster code that other people can't follow or maintain.

IMHO, and this is just me, if it's not:

Use the Timeline window to create cut-scenes, cinematics, and game-play sequences by visually arranging tracks and clips linked to GameObjects in your Scene
​

Then Timeline isn't the best solution. If it's not a L->R sequence that you scroll through, but complex game logic that can punt out whatever and wherever and lead to anything, I don't personally think it's a Timeline.

BUT I might not understand what your case is.

 

Yeah I don't think you do in this case

It's a purely L>>R sequence. It's just about playing dialogue and making the editing of it suitable for an artist / director. But there can be different languages switchable at runtime, subtitles, all the content comes in from external systems, plus you want to preview all that in Timeline, because that's what it's great at. So the idea is, instead of having an AudioTrack + AudioClip, you have a DialogueTrack + DialogueClip that internally manages language, which audio clip to play, which subtitles to show, ... because nobody wants to care about that when designing.

I got a basic POC working by now (by duplicating the actual AudioTrack etc. and modifying that as needed), might share the code later once I'm happy with it.
Some learnings/pains already went here: https://forum.unity.com/threads/ext...ngs-are-internal-private.906620/#post-5947178

 

I haven't had to deal with localization. But yes, Timeline would be great at that.

I don't think the TRACK itself would be and should be changed. There's no reason to not have some big track group of Audio that has all your languages in it. Mute all the stuff you don't need. That way you can visually synch everything and it's not swapped around via code. You COULD swap out clips inside a single track or you could simply not enable the tracks based on languages or whatever your critera is.

* TIMELINE
AudioDialogue Group Track
-English 4.3 seconds
-Spanish 5.2 seconds
-Swedish 8.6 seconds
-PigLatin 128 seconds
^Only one of those will play with the corresponding text/animation

You can run into problems because depending on your actors and language, they will have totally different lengths and synching, and you will get misaligned.

I had something like this happen to me where I switched voice actors. And they're reading the exact same text in the exact same language. But even a 1% difference in length caused trouble. A lot of trouble. I was like, "can I just throw out an um or uh or pause here and there to get back to baseline?"

 

The reason your outlining above with the different lengths is one of the main reasons why I am going with a separate Track/Clip - it can handle that. It just uses the max of all supported languages as duration.

The Track Group + many muted/unmuted pieces of audio is very cumbersome to handle - even with automation, I would not want my artists to deal with that...

Here's a sneak peek at a first draft of this... This is a custom track with clips that contain scriptable objects, that have multiple audio clips each. The right ones (depending on selected language) are being played back from the timeline.
The different lengths are shown directly on the clip as little grey lines, to make multi-language editing MUCH easier. Also, the text is pulled from the scriptable object as well.



Coming together! There's still some issues around making this nice, as outlined in the other thread (e.g. haven't figured out how to do another binding type while still being able to play back audio), but that will probably solve in time.

 

hey @fherbst have you found a solution to this eventually?
I'm currently making a point&click game and I heavily use timeline for all my interactions. I've localized my subtitles and am now thinking about how to localize the audio. Would love to have some pointers as to what my best options are.
I was contemplating the idea of having a localized asset table for my timelines and have a separate timeline for each language, but if someone invented the "localized speech" track wheel, I'd love to know about it

 

Did you ever manage to finish it? I am trying to make the same thing, to localize audioclip inside timelines, but I'm having a few issues with it