PCMReaderCallback, Native Audio Plugins & Spacialization

I'm working on a plugin for Unity that (amongst other things) streams audio from an external source. This is a slightly unusual use-case but there seem to be a couple of ways to go about it, both with drawbacks.

CreateAudioClip with PCMReaderCallback
This seems to work well enough and interacts the way you'd expect with 3D sound and any Spacialization plugin you have active. You can place an AudioSource anywhere and the streamed data will naturally just play from it. This is the most "natural" way of doing it.

Unfortunately there's quite high latency between callbacks and--as far as I can tell--the timing information in AudioSource.timeSamples is not at all accurate. I haven't evaluated performance but I can't imagine the native-to-managed-to-native call overhead is great.

Native Audio Plugin
This seems to be the more modern and performant way to stream data into the Mixer system. Its native C++, which is perfect in my case.

The drawback is that it seems to be completely separate from the AudioSource system. Since the mixer happens after spacialization I'd have to write my own solution for panning or 3D sound.

DIY
(I suppose I could use some completely other native audio plugin, which I've noticed several plugins like video players will use. This seems like the worst of both worlds, although I suppose it's the most flexible.)

Is there any other way to stream or otherwise generate audio so that it participates correctly in 3D sound? What's the best practice for generated or streamed audio?

 

I'm using onAudioFilterRead to send audio generated procedurally(sort of a sequencer). It works well but in Unity 5.3 there is a bug that allocate memory at runtime continuosly for the temp buffer it pass you.


How where you able to use PCMReaderCallback to stream audio? I tried but did encounter too many issues.
First it was pre buffering without me telling it so.
Second it was only called for the length of the created audioclip, but I need to generate an infinite stream.

 

Yeah, it pre-buffers in a way that I can't control, but it will keep reading data if the AudioSource it's playing on is set as "looping". You can get an infinite stream that way--it will keep reading data forever.

That sucks about the memory leak. This stuff all seems pretty new and not well-tested.

 

Actually except native plugin they are quite old except native plugins.
I'll have a look at the looping way. but if it's not called constantly it's no good for me. I need to play audio in real time, onAudioFilterRead is called at regular intervals.
If you want to vote for a fix (maybe it will speed up the resolution), my bug report is here:
https://issuetracker.unity3d.com/is...very-frame-instead-of-reusing-the-same-buffer

 

Don't use

AudioSource.timeSamples

. It's not plugged into the actual audio back-end, and will basically return the approximate playback position calculated by multiplying

AudioSource.time

by

AudioClip.samples

.

Calculate your sample position manually in your PCMReaderCallback. Store a

position

field somewhere, and increment it by the length of the sample buffer each time your callback is called. Additionally, implement the

PCMSetPositionCallback

, which is called every time the playback position is changed. This will allow you to keep track of the true sample position, and eliminate the delay on your custom audio stream.

Code (CSharp):

1. private void PCMReaderCallback (float[] data) {
2.     // ...    
3.     // Do some stuff
4.     // ...
5.  
6.     position += data.Length;
7. }
8.  
9. private void PCMSetPositionCallback(int position) {
10.     this.position = position;
11. }