Video Audio processing callback for video

Hello,

I was wondering if there is any way to extract the audio from a playing video from a callback or similar mechanism.

To do this I've tried routing the video audio output to an audio source and implementing the OnAudioFilterRead() callback of the audio source but unfortunately at that point the audio has already been mixed down to the output channel configuration set.

Ideally what I was looking for was a feature that would expand the "Direct", "Audio Source" and "None" audio playback behaviour from the VideoPlayer to contain a "Custom" option that instead of doing the routing internally would just handover the unmodified audio frame to the user in a callback or similar mechanism.

Is there a way to do this already and if not would it be a feasible addition?

Thank you,
Pedro

 

Hi,

I'm also trying to figure out ways of extracting audio data (as in AudioClip.GetData) from a video file, because I need to a library for processing.
As I see it VideoClip or VideoPlayer doesn't give any access to the audio data of a video-file loaded, except passing it directly to the AudioSource?
So far I only found out, that the deprecated MovieTexture lets you access the AudioClip, but trying to get the raw data gives me the error:

Well I cannot set the load type for a video clips audio data also, right?

So is there even any way to work with the audio data of a video, preferably not with deprecated api?
Any other tips on how to get audio data in raw wave format except splitting my video and audio into two files (optionally with other libraries)?

 

Hi Pedro,
I'm not sure what you mean with output channel configuration set. However it seems that the native audio plugin gives you a kind of callback to code your own effects and such. Also maybe you can have a look at the spatializer sdk which claims following:

This helped me utilizing a custom audio playback library instead of Unity's builtin (FMOD).

 

Hey Jahuu, sorry I didn't get a notification of this.

Yes what you are describing is effectively the issue I was having. There doesn't seem to be a way to access the raw wave data contained in a video file. The VideoPlayer plugin can route to an audio source to which you can attach a callback but when you get the audio data from that callback the audio will already have been in the format that Unity outputs (for example, if you have multi-channel audio inside the video file the callback will only give you a stereo buffer if that's what Unity is outputting) which is not what I require as I want all the channels to do some processing. Using the spatializer gives you the same thing specially regarding your comment where it specifically says that Unity will only provide the raw source as a stereo signal and indicating that even if the source is mono or multi-channel it will up/down mix.

Splitting the audio and video in two files is the most obvious workaround but then this brings up issues with synchronisation etc. that would be already handled by the video player and would now have to be handled outside.

 

Do you have an example (implementation) for routing the audio from Unity to Pure Data or Max?

 

I am currently having a similar problem with 2017.4.3 I have a VideoPlayer that is sending its audio to an AudioSource, and I have a script on that AudioSource that implements OnAudioFilterRead. But the data in OnAudioFilterRead is all zeros, and the VU meter doesn't show any audio. In my case, I'd like to pipe this data to multiple other AudioSources. Any tips on what I am doing wrong?

 

I'm trying to read samples from .webm that contains only audio but I get error:

Error: No video tracks were found in <my_correct_path_to_file>