Question How can I extract the audio from a video tag in the js side?

@The_Island @brendanduncan_u3d Could you help me out here on webgl? I am trying to figure out where on the build.framework.js the audio from a video can be redirected to an audio source. Or how can I create an audio clip in the js side from a video html tag. So the user can use an audio node with an audio clip from the audio of the video. I'm looking at functions such as "jsAudioCreateUncompressedSoundClipFromCompressedAudio", etc.
The reason I'm asking is because I have this plugin and there's a client that wants to mess with the audio from the video, so I need to make this scheme of extracting the audio from the video and put it into an audio clip/audio source work.

For example, I have this video tag <video src="somevideourl.mp4"></video> . How can I extract an audio clip or audio source out of it, or redirect the audio from the video to an existing audio source/audio clip on the scene? This video tag is from my plugin, not the normal Video Player. Thank you.

 

Hey @brendanduncan_u3d . I managed to take an HTML5 video and extract its AudioBuffer, convert it into a wav file format, then into an object url, which is then downloaded into a Unity Audio Clip. I basically followed this tutorial. It works, but it's slow, the wav file is huge, and not optimal.

However, I am pretty sure Unity doesn't do this. It must be a much simpler and effective process of redirecting the audio from a video into an audio clip, instead of converting the audio into a wav file first. Do you know how I can redirect the audio from an HTML5 video into an existing/new audio clip in Unity? I know that all the audios in Unity 2021 are stored at WEBAudio.audioInstances in the build.framework.js, and the HTML5 videos are stored at videoInstances . What I need to understand is how the audio from the videos in these video instances can be redirected into an audio from an audioInstance. Because then I will do it with one of my HTML5 videos. I know that WEBAudio.audioInstances only exist if there's at least one audio source node with an audio clip in the scene, but that's ok. I can have a dummy audio clip. I want to learn how can I hijack one of these audio instances with an audio from a random HTML5 video.

Can you give a code snippet or something?

 

Just want to voice my support for this! My studio uses @Marks4's Video Player WebGL plugin, and we'd really appreciate any insight you might have in this matter, @brendanduncan_u3d & @The_Island. Thanks!

 

Thank you both for your replies! I've been doing some testing related to this problem myself. I tested a WebGL build set up with a 3D audio source linked to Unity's video player and found that when you build for WebGL, even if the Unity video player is set up to use a 3D/spatialized audio source, it only outputs via direct audio. This appears to align with @The_Island's findings that in a WebGL build, the VideoPlayer is not aware of the AudioSource. I also found this thread from 2020 that discusses this a bit.

That being said, is there actually an opportunity for improvement here if a video element can actually be connected into the audio graph in order to use the WebGL-specific implementation of an audio source? Spatial audio is quite important to us in WebGL for the multiplayer platform we are building out, and we'd love to be able to spatialize the audio for the video player (and for @Marks4's plugin).

 

I understand that the audio is decoupled. How can I redirect the audio from an HTML5 video to an audio source in Unity? That's what Unity does for spatialization to work using an Audio Source. The audio from the video is piped to an audio source(or audio clip?). I just need to understand exactly how/where this is done in Unity 2021 on the build.framework.js so that I can hijack an audio instance and use it to play an audio from my video. Thanks!

 

@The_Island @brendanduncan_u3d Ah, interesting that it was a regression! So, if it is actually possible for the video player to route its stream into the WebGL implementation of an Audio Source, could you (or one of your colleagues) please provide @Marks4 with a code snippet or something, so that they are able to add this functionality to their video player plugin? We'd really appreciate it!

 

Hey @brendanduncan_u3d, I appreciate you looking into this and understand about things getting busy from vacations and sick days backing things up. Looking forward to hearing a solution for this when ready. Thanks!

 

This forum thread can be closed. I discovered how to successfully hijack an audio source node to play audio(and manipulate it) from any HTML5 videos, including those from my plugin.

 

And how?