Video Use cases for Audio experimental sample provider on the VideoPlayer in 2018.1

Ideally looking for access to the audio clip components for specific video player clips, I was excited to see the new experimental audio sample provider in the new unity 2018 beta. The initial thought was that I could grab the audio samples from the buffer and construct an audio clip which could then be synced up to the video player component to provide a separate linked audio clip that would allow more flexibility in our current media system.

However in my initial tests, I ran into a couple roadblocks around the way. Specifically I learned that you can't construct audio clips outside the main thread. When trying to bring the sample data into the main thread to construct the clips, I realized the experimental nature of the system, with after several unity editor plays, the video player refuses to playback the video and the buffer fails to return any frames. This is resolved with a restart of unity, but has left me perplexed with how to properly use this new system.

What are the use cases to use the samples to properly provide audio playback? Has anyone else played around with this system?

Ideally Im pushing towards taking the audio samples and bitrate, constructing a clip which gets linked to an audio source and then hooked up to the video player frames to provide synced audio playback but with the ability to say unlink the audio clip with the video player system and visualize the waveform.

Is this taking the audio sample provider class and bringing the samples to an audio clip possible/ planned in the roadmap or should I just use easy movie texture or avvidpro?

Thanks!

 

Ok! Just an update for anyone looking into this. I was able to get the audio clip constructed in the main thread with https://github.com/PimDeWitte/UnityMainThreadDispatcher

A stupid oversight on my part, but once I found this resource the audio clip creation and syncing works pretty much as expected in my brief testing. Still a little perplexed on how to bodge the video player to generate the audio clip without having to playback the entire video first, or get it to playback on runtime, but this new experimental audio sample provider is looking to fill my needs perfectly with a little bit of hacking.