Question Byte[] to AudioClip

Hello all,

I've been messing around with sockets on unity and had the idea to make an online voice chat system, It sends the audio clip to the server as bytes, how do I convert the bytes back to an audio clip?

Thanks,
Daniel

 

Haven't tested but the idea is to convert the bytes to floats and store them in an array. Then you can create a new AudioClip and set the data from the array of floats. Ideally you wouldn't create an AudioClip to play the audio, but rather store the received data in a buffer, convert it to floats and then read the buffer containing floats in the OnAudioFilterRead function (check documentation for examples on this).

Code (csharp):

1.  
2. float[] samples = new float[receivedBytes.Length / 4]; //size of a float is 4 bytes
3.  
4. Buffer.BlockCopy(receivedBytes, 0, samples, 0, receivedBytes.Length);
5.  
6. int channels = 1; //Assuming audio is mono because microphone input usually is
7. int sampleRate = 44100; //Assuming your samplerate is 44100 or change to 48000 or whatever is appropriate
8.  
9. AudioClip clip = AudioClip.Create("ClipName", samples.Length, channels, sampleRate, false);
10. clip.SetData(samples, 0);
11.  

Couple notes you might want to think about:

- A float is 4 bytes, you could shave off 2 bytes and save half of the bandwidth if you use Int16 instead because that is only 2 bytes. Audio using floats sits in the range -1.0 to 1.0. Using Int16 means your values should be in the range -32768 to 32767. Before sending the data you convert it to Int16, and then back to bytes and send it. On the receiving end you convert that back to Int16 and then back to float so the AudioClip can use it.

- Microphone input usually is mono so 1 channel would be a safe bet to go with.

- Samplerate can be different on every machine. It's also not uncommon to use a low samplerate for voicetransmission because you typically don't need the full audible frequencyspectrum. A lower samplerate means you'd need less data to recover the sound.

 

Thanks for your reply, Willemsenzo,

I will try this out when I get home and get back to you on how it works, this looks like it might work.

Thanks, HurdOfDaniels

 

No problem. I made some changes to my previous post that you might want to consider as well.

 

I tried to do the same
my criteria is to read the mp3 which I previously stored as byte array using WriteAllBytes function
when I read and convert it to float array, I created an audio clip with the data, and when I played it ,I'm getting TV static noise sound

Code (CSharp):

1.  
2.  
3. string saveFile;
4.  
5. void GetSong()
6.     {
7.         saveFile = Application.persistentDataPath + "/EncryptedSongs/SongData.txt";
8.  
9.         string songLoc = Application.persistentDataPath + "/SongData.mp3";
10.  
11.         ass = GetComponent<AudioSource>();
12.  
13.         if (File.Exists(songLoc))
14.         {
15.             if (!isRead)
16.             {
17.                 byte[] songData = File.ReadAllBytes(songLoc);
18.                 //string songString = Convert.ToBase64String(songData);
19.                 File.WriteAllBytes(saveFile, songData);
20.                 Debug.Log(saveFile);
21.             }
22.             else
23.             {
24.  
25.                 byte[] songData = File.ReadAllBytes(saveFile);
26.  
27.                 using (Stream s = new MemoryStream(songData))
28.                 {
29.                     AudioClip audioClip = AudioClip.Create("SongName", songData.Length, 1, 48000, false);
30.                     float[] f = ConvertByteToFloat(songData);
31.                     audioClip.SetData(f, 0);
32.                     ass.clip = audioClip;
33.                     ass.Play();
34.                 }
35.             }
36.  
37.             Debug.Log("Complete");
38.         }
39.     }
40.  
41.  
42.     private float[] ConvertByteToFloat(byte[] array)
43.     {
44.         float[] floatArr = new float[array.Length / 4];
45.         for (int i = 0; i < floatArr.Length; i++)
46.         {
47.             if (BitConverter.IsLittleEndian)
48.             {
49.                 Array.Reverse(array, i * 4, 4);
50.             }
51.             floatArr[i] = BitConverter.ToSingle(array, i * 4) / 0x80000000;
52.         }
53.         return floatArr;
54.     }
55.  

 

if its actual mp3 file, that would need to be parsed, its not raw audio data:
http://www.multiweb.cz/twoinches/mp3inside.htm

also raw data should be within -1 to 1 range:
https://docs.unity3d.com/ScriptReference/AudioClip.SetData.html

can also check
https://docs.unity3d.com/ScriptReference/Networking.UnityWebRequestMultimedia.GetAudioClip.html

 

One more thing I need to encrypt the song and save it in my local device and decrypt while the song is played
and I need this to be done quick is there any way this can be done.
lets say each mp3 file size is approx. 6 megabytes

 

could try do processing in another thread(s).

 

is there a way to process it as a continuous byte stream? I've been using the methods above alternating between two buffers and audio sources, but this is sub optimal and has gaps in between playbacks

 

What changes did you do in your code to fix the noise sound?

 

I used the plugin NAudioPlayer https://github.com/aykanatm/NaudioPlayer
using this I converted from byte array to audioclip

 

Can you please show us the code that you used to do this?

 

Hi guys i need to send Loopback audio from wasapi and cscore

But i cant get bytes red and converted to a sound clip can anyone help ill post the 2 files below that control everything
I just cant seem to figure out whats happining here

It plays the audio so the sprectrumdata and visualizer but i cant seen to get the bytes to an audio clip in game witch i need to have to send over webrtc to the client so they can here the live desktop audio

I want Loopbackaudio.cs to read the bytes from realtimeaudio.cs and output them to audio clip or directly to the webrtc audio streamer.

Loopbackaudio.cs

Code (CSharp):

1. using Assets.Scripts.Audio;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using System.Linq;
5. using System;
6. using CSCore;
7. using CSCore.SoundIn;
8. using CSCore.DSP;
9.  
10.  
11. public class LoopbackAudio : MonoBehaviour
12. {
13.     #region Constants
14.  
15.     private const int EnergyAverageCount = 100;
16.  
17.     #endregion
18.  
19.     #region Private Member Variables
20.  
21.     private RealtimeAudio _realtimeAudio;
22.     private List<float> _postScaleAverages = new List<float>();
23.  
24.     #endregion
25.  
26.     #region Public Properties
27.  
28.     public int SpectrumSize;
29.     public ScalingStrategy ScalingStrategy;
30.     public float[] SpectrumData;
31.     public float[] PostScaledSpectrumData;
32.     public float[] PostScaledMinMaxSpectrumData;
33.     public float PostScaledMax;
34.     public float PostScaledEnergy;
35.     public bool IsIdle;
36.  
37.     // Set through editor, but good values are 0.8, 0.5, 1.2, 1.5 respectively
38.     public float ThresholdToMin;
39.     public float MinAmount;
40.     public float ThresholdToMax;
41.     public float MaxAmount;
42.     public float recivedaudiobytelength;
43.     #endregion
44.  
45.     #region Startup / Shutdown
46.  
47.     public void Awake()
48.     {
49.         SpectrumData = new float[SpectrumSize];
50.         PostScaledSpectrumData = new float[SpectrumSize];
51.         PostScaledMinMaxSpectrumData = new float[SpectrumSize];
52.  
53.         // Used for post scaling
54.         float postScaleStep = 1.0f / SpectrumSize;
55.  
56.         // Setup loopback audio and start listening
57.         _realtimeAudio = new RealtimeAudio(SpectrumSize, ScalingStrategy,(spectrumData) =>
58.         {
59.             // Raw
60.             SpectrumData = spectrumData;
61.  
62.             // Post scaled for visualization
63.             float postScaledPoint = postScaleStep;
64.             float postScaledMax = 0.0f;
65.  
66.             float postScaleAverage = 0.0f;
67.             float totalPostScaledValue = 0.0f;
68.  
69.             bool isIdle = true;
70.  
71.             // Pass 1: Scaled. Scales progressively as moving up the spectrum
72.             for (int i = 0; i < SpectrumSize; i++)
73.             {
74.                 // Don't scale low band, it's too useful
75.                 if (i == 0)
76.                 {
77.                     PostScaledSpectrumData[i] = SpectrumData[i];
78.                 }
79.                 else
80.                 {
81.                     float postScaleValue = postScaledPoint * SpectrumData[i] * (RealtimeAudio.MaxAudioValue - (1.0f - postScaledPoint));
82.                     PostScaledSpectrumData[i] = Mathf.Clamp(postScaleValue, 0, RealtimeAudio.MaxAudioValue); // TODO: Can this be done better than a clamp?
83.                 }
84.  
85.                 if (PostScaledSpectrumData[i] > postScaledMax)
86.                 {
87.                     postScaledMax = PostScaledSpectrumData[i];
88.                 }
89.  
90.                 postScaledPoint += postScaleStep;
91.                 totalPostScaledValue += PostScaledSpectrumData[i];
92.  
93.                 if (spectrumData[i] > 0)
94.                 {
95.                     isIdle = false;
96.                 }
97.             }
98.  
99.             PostScaledMax = postScaledMax;
100.  
101.             // Calculate "energy" using the post scale average
102.             postScaleAverage = totalPostScaledValue / SpectrumSize;
103.             _postScaleAverages.Add(postScaleAverage);
104.  
105.             // We only want to track EnergyAverageCount averages.
106.             // With a value of 1000, this will happen every couple seconds
107.             if (_postScaleAverages.Count == EnergyAverageCount)
108.             {
109.                 _postScaleAverages.RemoveAt(0);
110.             }
111.  
112.             // Average the averages to get the energy.
113.             PostScaledEnergy = _postScaleAverages.Average();
114.  
115.             // Pass 2: MinMax spectrum. Here we use the average.
116.             // If a given band falls below the average, reduce it 50%
117.             // otherwise boost it 50%
118.             for (int i = 0; i < SpectrumSize; i++)
119.             {
120.                 float minMaxed = PostScaledSpectrumData[i];
121.  
122.                 if(minMaxed <= postScaleAverage * ThresholdToMin)
123.                 {
124.                     minMaxed *= MinAmount;
125.                 }
126.                 else if(minMaxed >= postScaleAverage * ThresholdToMax)
127.                 {
128.                     minMaxed *= MaxAmount;
129.                 }
130.  
131.                 PostScaledMinMaxSpectrumData[i] = minMaxed;
132.             }
133.  
134.             IsIdle = isIdle;
135.         });
136.  
137.         _realtimeAudio.StartListen();
138.  
139.         AudioSource audioSource = GetComponent<AudioSource>();
140.         float[] samples = new float[_realtimeAudio._soundInSource.WaveFormat.BytesPerSecond * _realtimeAudio._soundInSource.WaveFormat.Channels];
141.         audioSource.clip.GetData(samples, 0);
142.  
143.         for (int i = 0; i < samples.Length; ++i)
144.         {
145.             samples[i] = samples[i] * 0.5f;
146.         }
147.  
148.         audioSource.clip.SetData(samples, 0);
149.         audioSource.Play();
150.  
151.     }
152.  
153.     public void Update()
154.  
155.     {
156.  
157.     }
158.  
159.     public void OnApplicationQuit()
160.     {
161.         _realtimeAudio.StopListen();
162.     }
163.  
164.     #endregion
165.  
166.     #region Public Methods
167.  
168.     public float[] GetAllSpectrumData(AudioVisualizationStrategy strategy)
169.     {
170.         float[] spectrumData;
171.  
172.         switch (strategy)
173.         {
174.             case AudioVisualizationStrategy.Raw:
175.                 spectrumData = SpectrumData;
176.                 break;
177.             case AudioVisualizationStrategy.PostScaled:
178.                 spectrumData = PostScaledSpectrumData;
179.                 break;
180.             case AudioVisualizationStrategy.PostScaledMinMax:
181.                 spectrumData = PostScaledMinMaxSpectrumData;
182.                 break;
183.             default:
184.                 throw new InvalidOperationException(string.Format("Invalid for GetAllSpectrumData: {0}", strategy));
185.         }
186.  
187.         return spectrumData;
188.     }
189.  
190.     public float GetSpectrumData(AudioVisualizationStrategy strategy, int index = 0)
191.     {
192.         float spectrumData = 0.0f;
193.  
194.         switch (strategy)
195.         {
196.             case AudioVisualizationStrategy.Raw:
197.                 spectrumData = SpectrumData[index];
198.                 break;
199.             case AudioVisualizationStrategy.PostScaled:
200.                 spectrumData = PostScaledSpectrumData[index];
201.                 break;
202.             case AudioVisualizationStrategy.PostScaledMinMax:
203.                 spectrumData = PostScaledMinMaxSpectrumData[index];
204.                 break;
205.             case AudioVisualizationStrategy.PostScaledMax:
206.                 spectrumData = PostScaledMax;
207.                 break;
208.             case AudioVisualizationStrategy.PostScaledEnergy:
209.                 spectrumData = PostScaledEnergy;
210.                 break;
211.         }
212.  
213.         return spectrumData;
214.     }
215.  
216.     #endregion
217. }
218.  

RealtimeAudio.cs

Code (CSharp):

1. using System;
2. using CSCore;
3. using CSCore.SoundIn;
4. using CSCore.DSP;
5. using CSCore.Streams;
6. using System.IO;
7. using Assets.Scripts.Audio;
8. using System.Collections.Generic;
9. using UnityEngine;
10. using System.Linq;
11. using System;
12.  
13. namespace Assets.Scripts.Audio
14. {
15.     public class RealtimeAudio
16.     {
17.         #region Constants
18.  
19.         private const FftSize CFftSize = FftSize.Fft4096;
20.         public const int MaxAudioValue = 10;
21.  
22.         #endregion
23.  
24.         #region Private Member Variables
25.  
26.         private WasapiLoopbackCapture _loopbackCapture;
27.         public SoundInSource _soundInSource;
28.         private IWaveSource _realtimeSource;
29.         private BasicSpectrumProvider _basicSpectrumProvider;
30.         private LineSpectrum _lineSpectrum;
31.         private SingleBlockNotificationStream _singleBlockNotificationStream;
32.         private int _spectrumSize;
33.         private ScalingStrategy _scalingStrategy;
34.         private Action<float[]> _receiveAudio;
35.  
36.         #endregion
37.  
38.         #region Constructor
39.  
40.         public RealtimeAudio(int spectrumSize, ScalingStrategy scalingStrategy, Action<float[]> receiveAudio)
41.         {
42.             _spectrumSize = spectrumSize;
43.             _scalingStrategy = scalingStrategy;
44.             _receiveAudio = receiveAudio;
45.         }
46.  
47.         #endregion
48.  
49.         #region Public Properties
50.  
51.         public BasicSpectrumProvider BasicSpectrumProvider { get { return _basicSpectrumProvider; } }
52.  
53.         #endregion
54.  
55.         #region Public Methods
56.  
57.         public void StartListen()
58.         {
59.             _loopbackCapture = new WasapiLoopbackCapture();
60.             _loopbackCapture.Initialize();
61.  
62.             _soundInSource = new SoundInSource(_loopbackCapture);
63.  
64.             _basicSpectrumProvider = new BasicSpectrumProvider(_soundInSource.WaveFormat.Channels, _soundInSource.WaveFormat.SampleRate, CFftSize);
65.  
66.             _lineSpectrum = new LineSpectrum(CFftSize)
67.             {
68.                 SpectrumProvider = _basicSpectrumProvider,
69.                 BarCount = _spectrumSize,
70.                 UseAverage = true,
71.                 IsXLogScale = true,
72.                 ScalingStrategy = _scalingStrategy
73.             };
74.  
75.             _loopbackCapture.Start();
76.  
77.             _singleBlockNotificationStream = new SingleBlockNotificationStream(_soundInSource.ToSampleSource());
78.             _realtimeSource = _singleBlockNotificationStream.ToWaveSource();
79.  
80.             byte[] buffer = new byte[_realtimeSource.WaveFormat.BytesPerSecond / 2];
81.        
82.             _soundInSource.DataAvailable += (s, ea) =>
83.             {
84.                 while (_realtimeSource.Read(buffer, 0, buffer.Length) > 0)
85.                 {
86.                     float[] spectrumData = _lineSpectrum.GetSpectrumData(MaxAudioValue);
87.  
88.                     if (spectrumData != null &&  _receiveAudio != null)
89.                     {
90.                         _receiveAudio(spectrumData);
91.  
92.              
93.  
94.                     }
95.                 }
96.             };
97.  
98.             _singleBlockNotificationStream.SingleBlockRead += singleBlockNotificationStream_SingleBlockRead;
99.         }
100.  
101.  
102.  
103.         public void StopListen()
104.         {
105.             _singleBlockNotificationStream.SingleBlockRead -= singleBlockNotificationStream_SingleBlockRead;
106.  
107.             _soundInSource.Dispose();
108.             _realtimeSource.Dispose();
109.             _receiveAudio = null;
110.             _loopbackCapture.Stop();
111.             _loopbackCapture.Dispose();
112.         }
113.  
114.         #endregion
115.  
116.         #region Private Methods
117.  
118.         private void singleBlockNotificationStream_SingleBlockRead(object sender, SingleBlockReadEventArgs e)
119.         {
120.             _basicSpectrumProvider.Add(e.Left, e.Right);
121.         }
122.  
123.         #endregion
124.     }
125. }
126.  

AudioStreamer.cs

Code (CSharp):

1. using System.Linq;
2. using Unity.WebRTC;
3. using UnityEngine;
4.  
5. namespace Unity.RenderStreaming
6. {
7.     [RequireComponent(typeof(AudioListener))]
8.     public class AudioStreamer : AudioStreamBase
9.     {
10.         private MediaStream m_audioStream;
11.  
12.         void OnDisable()
13.         {
14.             WebRTC.Audio.Stop();
15.         }
16.  
17.         protected override MediaStreamTrack CreateTrack()
18.         {
19.             m_audioStream = Unity.WebRTC.Audio.CaptureStream();
20.          
21.             return m_audioStream.GetTracks().First();
22.         }
23.  
24.         private void OnAudioFilterRead(float[] data, int channels)
25.         {
26.             WebRTC.Audio.Update(data, channels);
27.          
28.         }
29.     }
30.  
31. }
32.  
33.  

 

SOLVED:

Code (CSharp):

1.     void Start()
2.     {
3.         var audio = GetComponent<AudioSource>();
4.         audio.clip = Microphone.Start("Stereo Mix (Realtek High Definition Audio)", true, 10, 44100);
5.         audio.loop = true;
6.         while (!(Microphone.GetPosition(null) > 0))
7.         {
8.  
9.         }
10.  
11.         audio.Play();
12.     }

 

Did you get any updates?

 

Just retrieve your data filepath:-

StartCoroutine(GetAudioClip());
IEnumerator GetAudioClip()
{
using (UnityWebRequest www = UnityWebRequestMultimedia.GetAudioClip("file://" + filepath, AudioType.WAV))
{
yield return www.SendWebRequest();
AudioClip myClip = DownloadHandlerAudioClip.GetContent(www);
audioPlayer.clip = myClip;
audioPlayer.Play();
}
}