Audio DownloadHandlerAudioClip.streamAudio is ignored?

Hi everyone!
How do I load a .mp3 file at runtime so that it is streamed from disk (not preloaded)? Here is the code I use for loading predownloaded mp3 files from Application.persistendDataPath:

Spoiler: Mp3LoadingCoroutine

Code (CSharp):

1. private IEnumerator CoR_LoadAndPlayAudio(string audioFilePath)
2.     {      
3.         UnityWebRequest uwr = UnityWebRequestMultimedia.GetAudioClip(audioFilePath, AudioType.MPEG);
4.    
5.         uwr.uri = new Uri(uwr.uri.AbsoluteUri.Replace("http://localhost", "file://"));
6.         uwr.url = uwr.url.Replace("http://localhost", "file://");
7.         if (Debug.isDebugBuild) Debug.LogFormat("Loading audio file for {0} from {1} ", App.State.TextPieceToRead.Title, uwr.url);
8.    
9.         DownloadHandlerAudioClip dhac = uwr.downloadHandler as DownloadHandlerAudioClip;
10.         dhac.streamAudio = true;
11.         yield return uwr.SendWebRequest();
12.  
13.         if (uwr.isHttpError || uwr.isNetworkError)
14.         {
15.             Debug.LogErrorFormat("Could not load audio file for {0}. Error: {1}", audioFilePath,  uwr.error);
16.             yield break;
17.         }
18.  
19.         _currentClip = dhac.audioClip;
20.    
21.         if (_currentClip != null)
22.         {
23.             if (Debug.isDebugBuild) Debug.Log ("Clip load type: " + _currentClip.loadType);
24.             if (Debug.isDebugBuild) Debug.Log ("Clip load state: " + _currentClip.loadState);
25.             _audioSource.clip = _currentClip;
26.             _audioSource.time = 0f;
27.             _audioSource.Play();
28.         }
29. }

This works, however despite setting the DownloadHandlerAudioClip.streamAudio to true the output log says (both in Editor and on Android Device):

Clip load type: DecompressOnLoad

Clip load state: Loaded

I've tried various combinations of setting the properties of the download handler but every time the result is the same.
I'm using Unity 2019.1.2f1.
Thanks in advance

 

I can confirm the issue occurring on Unity 2019.1.8f1 too. Am I doing something wrong or is this a bug?

 

Still a problem in 2019.4.11f1 it really irks me that I have to waste memory keeping the clips fully loaded

 

Here is my code, for reference: https://github.com/JimmyCushnie/Jim...b/master/Scripts/AudioLoadingUtilities.cs#L72

 

Was experimenting this on 2019.4.18f1 LTS, but it was my code lol.
Be sure to use the result of you request even if the request is not yet marked as done !!!
request.isDone = false when you are streaming !

 

Hi,
I am having issues with this feature.
If I wait for isDone then it starts playing, if I play let's say when it is 50% loaded I get htis error:
InvalidOperationException: Cannot get content from an unfinished UnityWebRequest object
My code as follows:

Code (CSharp):

1.     IEnumerator GetAudioClip()
2.     {
3.         using (UnityWebRequest www = UnityWebRequestMultimedia.GetAudioClip(_fullUrl, AudioType.OGGVORBIS))
4.         {
5.             (www.downloadHandler as DownloadHandlerAudioClip).streamAudio = true;
6.             var operation = www.SendWebRequest();
7.             while (www.downloadProgress < 0.5)
8.             {
9.                 Debug.Log("progress: " + www.downloadProgress);
10.                 _loader.transform.localScale = new Vector3(_loader.transform.localScale.x, www.downloadProgress*LOADER_MAX_SCALE, _loader.transform.localScale.z);
11.                 yield return null;
12.             }
13.  
14.             if (www.result == UnityWebRequest.Result.ConnectionError)
15.             {
16.                 _status = 4;
17.                 Debug.Log(www.error);
18.             }
19.             else
20.             {
21.                 _status = 2;
22.                 Debug.Log("Audio clip loaded.");
23.                 _clip = DownloadHandlerAudioClip.GetContent(www);
24.                 _audioSource.clip = _clip;
25.                 _audioSource.Play();
26.             }
27.             yield return operation;
28.         }
29.     }

Maybe I am missing something basic but I am quite new to Unity...

 

Gotcha, my fault.

_clip = DownloadHandlerAudioClip.GetContent(www);

Should have been

_clip = (www.downloadHandler as DownloadHandlerAudioClip).audioClip;

 

I'm also running into this trying to stream audio from a remote resource API. seems as though it's being ignored. Using 2021.3.x

 

Sure! I'm streaming from a web server using 2021.3.x

https://github.com/RageAgainstThePi...kages/com.utilities.rest/Runtime/Rest.cs#L689

When I add a break point to observe the DownloadHandlerAudioClip.audioClip, the properties show that it's not streaming, and I get FMOD errors saying not enough of the clip has been downloaded.

Let me try to get a min repro using one of my other packages for you.

 

Bump. I'm using @StephenHodgson-XRTK s github api's, They're great! I would like to stream ElevenLabs voices.

 

Ditto, audio bug remains. The AudioClip streamed cuts off because the download handler has not yet finished downloading it, the streaming stops (instead of buffering or having any sort of paused notification).

 

Much appreciated on the complement.

I have in fact submitted multiple bug reports, but they've been closed by QA several times.

To be clear, these aren't personal projects. Each of these are backed by my clients, and all work/fixes get rolled back up and into existing products used by some pretty big companies (like Pfizer, Microsoft, Lenovo, etc).

This is a flaw that's been known about and pointed out by multiple people using the engine.
We respectfully ask that it gets fixed.

Hell, give me the access and the ability and I'll fix if for you!

IMHO ever since the IPO unity seems to be only focused on putting in engineering effort into new features that increase the profitability, instead of important bug fixes and and quality of life improvements for developers in general. Kinda sad to see, since I've been a paying customer of this Engine since the very beginning (circa 2006/2007).

Found other posts that could be somewhat related:
https://forum.unity.com/threads/dow...-system-is-out-of-memory.572677/#post-8903148
https://forum.unity.com/threads/how-to-play-an-audio-stream.526186/#post-8903370

 

Ah no worries, I didn't feel like you were diminishing my work by saying that. I'm sorry I made you feel that way.
I really appreciated the complement before!

Yes coffee would be a must sometime.

I'll submit yet another bug report just to get a fresh one going, since the others have likely made it to the heap.

As for a min repro project, one could easily clone my repo for ElevenLabs and tweak the demo enough to get the errors we discussed in this thread, although it does require an API key etc etc.

If there was a more "Open" and alternative remote resource to test with that would be easier but I'm unaware atm.

 

Here is the logs from the last attempt. I also updated my strategy... One thing to note is that the length of the audio clip is unknown...

Code (CSharp):

1.  
2. Playback of audio clip not yet possible; headers are done, 341245/? (96.04%) bytes downloaded but size is still not known
3. Error: Cannot create FMOD::Sound instance for clip "" (FMOD error: The HTTP request timed out. )
4.  

 

One thing to note, I tried to download the clip first to disk, then stream it from disk as a way to possibly get around the fact that it might be an issue with the server, but still running into the same problems.

 

Any updates?

 

Still not working. Unity 2022.3.3f1 LTS

 

Apologies for necroing this thread, but I also ran into issues trying to stream audio of unknown length on a Windows app. In the end I resorted to using the NAudio approach outlined here to play back audio: https://github.com/AstralSkies/RadioUnityStream/blob/main/Assets/Scripts/RadioManager.cs.

This works out of the box to play audio, but it bypasses using an AudioSource so you can't spatialize or postprocess the audio.

If you need to do further postprocessing of the data or to attach it to an AudioSource as an AudioClip, you can store the streamed data in a temporary NAudio sample provider class that implements NAudio's ISampleProvider.

E.g:

Code (CSharp):

1.  
2. using NAudio.Wave;
3. using System;
4. public class BufferedSampleProvider : ISampleProvider
5. {
6.     private readonly ISampleProvider sourceProvider;
7.     private readonly float[] sampleBuffer;
8.     private int bufferReadPosition;
9.     private int bufferSize; // Set an appropriate buffer size
10.  
11.     public BufferedSampleProvider(ISampleProvider sourceProvider, int bufferSize)
12.     {
13.         this.sourceProvider = sourceProvider;
14.         this.bufferSize = bufferSize;
15.         sampleBuffer = new float[bufferSize];
16.     }
17.  
18.     public WaveFormat WaveFormat { get { return sourceProvider.WaveFormat; } }
19.  
20.     public int Read(float[] buffer, int offset, int count)
21.     {
22.         int availableSamples = bufferSize - bufferReadPosition;
23.         int samplesToCopy = Math.Min(availableSamples, count);
24.  
25.         // Copy from our buffer
26.         Array.Copy(sampleBuffer, bufferReadPosition, buffer, offset, samplesToCopy);
27.         bufferReadPosition += samplesToCopy;
28.  
29.         if (samplesToCopy < count)
30.         {
31.             // Need to refill the buffer with new data from the source provider
32.             int samplesRead = sourceProvider.Read(sampleBuffer, 0, bufferSize);
33.             bufferReadPosition = 0;
34.  
35.             // Recursively call Read to fill the remainder of the output buffer
36.             int remaining = count - samplesToCopy;
37.             return samplesToCopy + Read(buffer, offset + samplesToCopy, remaining);
38.         }
39.  
40.         return samplesToCopy;
41.     }
42. }

This then gets implemented in a coroutine within your class to stream the Audio into the sample provider:

Code (CSharp):

1. private IEnumerator PlayRadio()
2. {
3.     yield return null;
4.     try
5.     {
6.         mediaFoundationReader = new MediaFoundationReader(radioUrl);
7.         // Create the buffered provider
8.         bufferedProvider = new BufferedSampleProvider(mediaFoundationReader.ToSampleProvider(), bufferSize: 4096);
9.         waveOut = new WaveOutEvent();
10.         waveOut.Init(mediaFoundationReader);
11.         waveOut.Play();
12.     }
13.     catch (System.Exception ex)
14.     {
15.         Debug.LogError($"Error playing radio: {ex.Message}");
16.     }
17. }

and you setup another coroutine to trigger capturing a slice of the buffer into an AudioClip:

Code (CSharp):

1. IEnumerator AutomaticCaptureRoutine()
2. {
3.     while (true) // Adjust the loop condition if needed
4.     {
5.         yield return new WaitForSeconds(captureIntervalInSeconds);
6.         Debug.Log("Capturing audio clip");
7.         CaptureAudioClip(captureDurationInSeconds);
8.     }
9. }

This coroutine calls CaptureAudioClip() to take a slice of samples from the buffer and create an audio clip:

Code (CSharp):

1. public void CaptureAudioClip(int durationInSeconds)
2. {
3.     Debug.Log("Starting auto capture of internet radio stream...");
4.     if (bufferedProvider != null)
5.     {
6.         int desiredNumSamples = durationInSeconds * bufferedProvider.WaveFormat.SampleRate * bufferedProvider.WaveFormat.Channels;
7.         Debug.Log($"Capturing {desiredNumSamples} samples...");
8.         AudioClip audioClip = audioClipCreator.CreateFromBufferedProvider(bufferedProvider, "RadioCapture", desiredNumSamples);
9.         audioSource.clip = audioClip;
10.         audioSource.Play();
11.     }
12.     else
13.     {
14.         Debug.LogWarning("Cannot capture audio. Radio stream not initialized yet.");
15.     }
16. }

This uses the following helper class to do the dirty work:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class AudioClipCreator
4. {
5.     public AudioClip CreateFromBufferedProvider(BufferedSampleProvider provider, string clipName, int desiredLength)
6.     {
7.         var audioClip = AudioClip.Create(clipName, desiredLength, provider.WaveFormat.Channels, provider.WaveFormat.SampleRate, false);
8.  
9.         float[] allSamples = CollectSamplesFromProvider(provider, desiredLength * provider.WaveFormat.Channels);
10.         audioClip.SetData(allSamples, 0);
11.  
12.         return audioClip;
13.     }
14.  
15.     private float[] CollectSamplesFromProvider(BufferedSampleProvider provider, int totalSamplesNeeded)
16.     {
17.         float[] allSamples = new float[totalSamplesNeeded];
18.         int currentSampleIndex = 0;
19.  
20.         while (currentSampleIndex < totalSamplesNeeded)
21.         {
22.             int samplesRead = provider.Read(allSamples, currentSampleIndex, totalSamplesNeeded - currentSampleIndex);
23.             currentSampleIndex += samplesRead;
24.         }
25.  
26.         return allSamples;
27.     }
28. }

You may need to adjust the sample provider bufferSize, plus the desiredLength of samples when creating the final AudioClip otherwise you'll get choppy or stuttering audio for the final output.

Following on from this you can apply this to other Unity libraries that do lip sync (like Meta's lipsync library for character mouth movements) or visualise the resulting wave form.