Official DOTS Audio Discussion

So, i've tried the latest unity and beta builds. But none of them have the package 'DSPGraph' available. Will it be available soon? I'm guessing it's been tested lately and prepared for a release?

 

you have to set your package manifest to the staging repository, its not on the main one yet

 

Hi, what is string-name of package to import?

I've tried this in 2019.2.0b6:
"com.unity.audio.dspgraph": "0.1.0-preview.1"

 

No that's not needed actually.
Its on the release one, just the list is not public so you cannot install with the PackageManager
Use this link: https://bintray.com/unity/unity/com.unity.audio.dspgraph
The version you see there (currently 0.1.0-preview.3) needs to be manually added to Packages/manifest.json

 

I'm getting this on PS4:

Code (CSharp):

1. InvalidOperationException: Incompatible buffer passed to ReadMix, buffer of size 256 does not match previous read length 1024
2.  
3. (Filename: currently not available on il2cpp Line: -1)

Does that mean I have to set the dspBufferSize to 256 on PS4?

 

And now I'm getting this on Xbox One

Code (CSharp):

1. InvalidOperationException: Incompatible buffer passed to ReadMix, buffer of size 512 does not match previous read length 512

So basically 512 != 512
Hmm, riddles are fun

 

So what I found so far:
- Update and Dispose jobs cause Unity crash hard. Fortunately I can get by without them for now.
- Arrays allocated in IAudioKernel's Initialize() should NOT be disposed in Dispose() - it causes Unity crashes
- Setting the output to Stereo on Xbox One does not throw an error. It causes hearing damage instead...
- I found that fading out a streaming music and starting to load a scene causes an abrupt stop in the sound on PS4 and Xbox One. I haven't investigated more though.
- Legacy audio no longer works in 2019.2 along with DSPGraph. (Not a big issue we only have a few places to convert.)

Good news is that our project now runs without crashing (streaming audio, 3d sounds, etc.)

Now I have to write a DSPGraph based Audio Playable for Timeline, and find a way to connect the VideoPlayer to DSPGraph and everything should work in 2019.2

 

What's currently the best place to start looking into the DSP graph? Still the megacity demo, or is there an up to date sample anywhere that I just can't find? Or @Tak, can't you make those "simpler examples" available somewhere already?

 

Now I checked without Burst and I get the following results:

If I allocate an array in my audio kernel Initialize() with the

AudioKernel

allocator, but do not dispose it then I got complaints about leaking DSP buffers:

Code (CSharp):

1. 1 leaked DSP node allocations
2. Unity.Audio.DSPGraphInternal:Internal_Update(Handle&)
3. Unity.Audio.DSPGraph:Update() (at Library/PackageCache/com.unity.audio.dspgraph@0.1.0-preview.3/Runtime/DSPGraph.cs:177)

If I allocate an array in my audio kernel Initialize() with the

AudioKernel

allocator, and try to dispose it in Dispose() I get a hard crash in Unity:

Code (CSharp):

1. 0x00007FF6C280D49E (Unity) FreeArrayForDSPGraph
2. 0x00007FF6C526AB24 (Unity) UnsafeUtility_CUSTOM_Free
3. 0x000001793F3DFCA7 (Mono JIT Code) (wrapper managed-to-native) Unity.Collections.LowLevel.Unsafe.UnsafeUtility:Free (void*,Unity.Collections.Allocator)
4. 0x000001793F3DEE13 (Mono JIT Code) [C:\buildslave\unity\build\Runtime\Export\NativeArray\NativeArray.cs:165] Unity.Collections.NativeArray`1<single>:Dispose ()
5.  

Finally if I allocate my array with the

Temp

allocator, I got no crashes, but I got audio corruption instead (the buffer is probably freed and reused from time to time)

 

I removed the

[NativeDisableContainerSafetyRestriction]

attribute to see if I get anything.

I got the following error:

Code (CSharp):

1. The job VoicePlayerResampleJob has changed the assigned container VoicePlayerResampleJob.resampleBuffer. This is not supported.
2. Unity.Audio.DSPCommandBlockInternal:Internal_CreateDSPNode(Handle&, Handle&, Handle&, Void*, Void*, Void*, Int32, Void*, Int32)
3. Unity.Audio.DSPCommandBlock:CreateDSPNode() (at Library/PackageCache/com.unity.audio.dspgraph@0.1.0-preview.3/Runtime/DSPCommandBlock.cs:46)
4.  

I see the Tests has only two use cases:

Code (CSharp):

1.     struct LeakyKernel : IAudioKernel<NoParameters, NoProviders>
2.     {
3.         [NativeDisableContainerSafetyRestriction]
4.         private NativeArray<float> m_Buffer;
5.  
6.         public void Initialize()
7.         {
8.             m_Buffer = new NativeArray<float>(200, Allocator.AudioKernel);
9.         }
10.  
11.         public void Execute(ref ExecuteContext<NoParameters, NoProviders> context)
12.         {
13.         }
14.  
15.         public void Dispose()
16.         {
17.         }
18.     }
19.  
20.     struct AllocatingKernel : IAudioKernel<NoParameters, NoProviders>
21.     {
22.         public void Initialize()
23.         {
24.         }
25.  
26.         public void Execute(ref ExecuteContext<NoParameters, NoProviders> context)
27.         {
28.             using (var buffer = new NativeArray<float>(context.Outputs.GetSampleBuffer(0).Samples, Allocator.AudioKernel))
29.                 buffer.CopyTo(buffer);
30.         }
31.  
32.         public void Dispose()
33.         {
34.         }
35.     }
36.  
37.  

But allocating locally in Execute doesn't really sound useful...
Well I try to implement the resampling player differently...

 

It tried to implement a resampling player without permanent allocation by reading as much sample date as needed.
Now I'm getting:

Code (CSharp):

1. Reading/writing memory blocks that are not multiple of tuple size is not allowed on a tuple_ringbuffer!

What's the "tuple size" ???

By the way this trial and error programming is really slow because Unity crashes on each code recompile...

 

I reported (Case 1170153) containing a reasonably cut-down version of our project.
For me the crash occurs 100% within seconds after entering play mode.

There's a sound attached to every footstep of the squad so it's a reasonable load test.

I have disabled Burst. With Burst the Editor crashes without a callstack and the bug report tool doesn't appear.

 

Can we use this to create MidiSynthesizer?

 

I've been watching all these lovely developments from the sidelines for a while, but started to try out the code today.

Using information from this thread and the api docs, I'm almost at the point where I have some audio going, but I can't quite get it to run without errors. Here's what I have so far, with the aim to play a simple sine wave:

Unity 2019.2.0f1, all packages updated to latest.

Code (CSharp):

1. public class DspApplication : MonoBehaviour {
2.     private DSPGraph _graph;
3.     private DefaultDspGraphDriver _driver;
4.     private AudioOutputHandle _outputHandle;
5.     private DSPNode _node;
6.  
7.     private NativeArray<float> _output;
8.  
9.     private const int SAMPLERATE = 48000;
10.     private const int BUFFSIZE = 512;
11.     private const int NUMCHANNELS = 2;
12.  
13.     private void Awake() {
14.         _graph = DSPGraph.Create(SoundFormat.Stereo, NUMCHANNELS, BUFFSIZE, SAMPLERATE);
15.         _driver = new DefaultDspGraphDriver();
16.         _driver._graph = _graph;
17.         _driver.Initialize(NUMCHANNELS, SoundFormat.Stereo, SAMPLERATE, BUFFSIZE);
18.  
19.         var cmd = _graph.CreateCommandBlock();
20.         _node = cmd.CreateDSPNode<MyKernelParams, MyKernelProviders, MyKernel>();
21.         cmd.AddOutletPort(_node, NUMCHANNELS, SoundFormat.Stereo);
22.         cmd.Connect(_node, 0, _graph.RootDSP, 0);
23.         cmd.Complete();
24.  
25.         _output = new NativeArray<float>(NUMCHANNELS * BUFFSIZE * 4, Allocator.Persistent, NativeArrayOptions.ClearMemory);
26.     }
27.  
28.     private void Start() {
29.         _driver.BeginMix(BUFFSIZE);
30.     }
31.  
32.     private void OnDestroy() {
33.         var cmd = _graph.CreateCommandBlock();
34.         cmd.ReleaseDSPNode(_node);
35.         cmd.Complete();
36.  
37.         _driver.Dispose();
38.         _output.Dispose();
39.     }
40.  
41.     private void OnAudioFilterRead(float[] data, int channels) {
42.         _driver.EndMix(_output, BUFFSIZE);
43.         _driver.BeginMix(BUFFSIZE);
44.         for (int i = 0; i < data.Length; i++) {
45.             data[i] = _output[i];
46.         }
47.     }
48.  
49.     public enum MyKernelParams {
50.         Param
51.     }
52.  
53.     public enum MyKernelProviders {
54.         Provider
55.     }
56.  
57.     public struct MyKernel : IAudioKernel<MyKernelParams, MyKernelProviders> {
58.         private float _phase;
59.         private float _phaseStep;
60.  
61.         public void Dispose() {
62.  
63.         }
64.  
65.         public void Execute(ref ExecuteContext<MyKernelParams, MyKernelProviders> context) {
66.             for (int outIdx = 0; outIdx < context.Outputs.Count; outIdx++) {
67.                 var sampleBuffer = context.Outputs.GetSampleBuffer(outIdx);
68.                 var buffer = sampleBuffer.Buffer;
69.                 Debug.Log("OutIdx: " + outIdx + ", buffer exists: " + (buffer != null) + ", size: " + ((buffer != null) ? buffer.Length : -1));
70.                 for (int i = 0; i < buffer.Length; i++) {
71.                     buffer[i] = math.sin(_phase);
72.                     _phase += _phaseStep;
73.                 }
74.             }
75.         }
76.  
77.         public void Initialize() {
78.             _phaseStep = ((math.PI * 2f) * 440f) / (float)SAMPLERATE;
79.         }
80.     }
81. }

(Note: I know the kernel is not handling interleaved stereo correctly and all that)

When I try to write to the output buffer I get an exception within the buffer's native array:

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. Unity.Collections.NativeArray`1[T].CheckElementWriteAccess (System.Int32 index) (at C:/buildslave/unity/build/Runtime/Export/NativeArray/NativeArray.cs:130)
3. Unity.Collections.NativeArray`1[T].set_Item (System.Int32 index, T value) (at C:/buildslave/unity/build/Runtime/Export/NativeArray/NativeArray.cs:146)
4. DspApplication+MyKernel.Execute (Unity.Audio.ExecuteContext`2[DspApplication+MyKernelParams,DspApplication+MyKernelProviders]& context) (at Assets/Dsp/DspApplication.cs:90)

Now, that's a buffer allocated by the audio system internally, so I don't think I have control over how it's done. The debug print shows that the array itself exists, and has the expected length.

I'm not writing out of bounds; if I try to write only to buffer[0] I still get the exception.

I'm probably missing something, but what?

Anyway, excited to make a few prototypes with this, and possibly pipe it into an ASIO stream through a PortAudio wrapper. Very nice that such a thing is now possible.

Edit 1: Ah! I get audio in a build. Turns out that despite turning Burst safety checks off in the editor, this code path is still executing:

Code (CSharp):

1. // NativeArraySOA.cs
2. 134 #if ENABLE_UNITY_COLLECTIONS_CHECKS
3. 134            AtomicSafetyHandle.CheckWriteAndThrow(m_Safety);

But the safety handle is never allocated. I can work around this now, but is this expected behaviour?

Edit 2: By taking more care to match buffer size to what the Unity audio system is actually expecting I've brought it back to a few null refs at program start. After that I'm getting a smooth, stable sine wave.

Here's a gist with my code, which I'll update with any further changes.

Now to try and pipe this into a low latency API!

 

Nice, thanks.
Are you using the same DefaultDspGraphDriver struct as here:
DOTS Audio Discussion? Could not make the code work yet.

 

Have just updated the gist to include the DefaultDspGraphDriver:

Code (CSharp):

1. [BurstCompile]
2. public struct DefaultDspGraphDriver : IAudioOutput {
3.     public DSPGraph _graph;
4.     private int _channelCount;
5.  
6.     public void Initialize(int channelCount, SoundFormat format, int sampleRate, long dspBufferSize) {
7.         _channelCount = channelCount;
8.     }
9.  
10.     public void BeginMix(int frameCount) {
11.         _graph.BeginMix(frameCount);
12.     }
13.  
14.     public void EndMix(NativeArray<float> output, int frames) {
15.         _graph.ReadMix(output, frames, _channelCount);
16.     }
17.  
18.     public void Dispose() {
19.         _graph.Dispose();
20.     }
21. }

(Note that it isn't actually doing much, and I'm wondering why you'd have this thing sitting around, heh.)

Let me know if that works!

 

Just needed to change

Code (CSharp):

1. private const float TWOPI = Mathf.PI * 2.0f;

to avoid double casting complier error, and now there is no error.

But I forgot how DSPGraphs work in general, so I am relearning (also general DOTS) from https://drive.google.com/file/d/1Y7go2Aq6SVxvKVMlipDIqLGsnFnJyN4H/view. Will report when everything works.

PS: Thanks for Gist update and also your unity project template at https://github.com/mzandvliet/UnityTemplateProject

 

I second this wish. Megacity has many helper classes, impossible to learn the basics from there. @Tinus gracefully shares his code, but I don't know what to make of the script. Attaching it to a game object, making the game object a prefab, importing entities package all result in error. @Hyp-X mentions a project and shares some wisdom, but also those do not help me to get started. So, @Tak, looking forward for those simple examples. Cheers, I remain excited on this.

 

Hey, I try to add Jobs, Math, Burst via package manager but I don't see where to add DSPGraph. Which package contains it?

Thank, Marek.

 

Hi Marek, use a text editor to open your projects Packages/manifest.json and add the following line somewhere in dependencies list

Code (CSharp):

1.     "com.unity.audio.dspgraph": "0.1.0-preview.3",

Mind the comma and closure of curly brackets. When you reopen the project in the Unity Editor the package will be installed, and also be visible in the Package Manager. This may add also burst, jobs, and mathematics preview packages as dependencies.

 

Ok, I checked Unity 2019.2.0f1
Looks like even stuff that used to work do not work anymore.

In the following job:

Code (CSharp):

1. //[BurstCompile]
2. struct PassTroughJob : IAudioKernel<PassTroughJob.Params, PassTroughJob.Providers>
3. {
4.     public enum Params
5.     {
6.     }
7.  
8.     public enum Providers
9.     {
10.     }
11.  
12.     public void Initialize()
13.     {
14.     }
15.  
16.     public void Execute(ref ExecuteContext<Params, Providers> context)
17.     {
18.         SampleBuffer inputBuffer = context.Inputs.GetSampleBuffer(0);
19.         SampleBuffer outputBuffer = context.Outputs.GetSampleBuffer(0);
20.  
21.         int numSamples = outputBuffer.Samples * outputBuffer.Channels;
22.         var src = inputBuffer.Buffer;
23.         var dest = outputBuffer.Buffer;
24.  
25.         int srcOffset = 0;
26.         int destOffset = 0;
27.  
28.         for (uint i = 0; i < numSamples; i++)
29.             dest[destOffset++] = src[srcOffset++];
30.     }
31.  
32.     public void Dispose()
33.     {
34.     }
35. }
36.  

I get an exception inside NativeArray:

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2.   at Unity.Collections.NativeArray`1[T].CheckElementReadAccess (System.Int32 index) [0x00031] in C:\buildslave\unity\build\Runtime\Export\NativeArray\NativeArray.cs:117
3.   at Unity.Collections.NativeArray`1[T].get_Item (System.Int32 index) [0x00003] in C:\buildslave\unity\build\Runtime\Export\NativeArray\NativeArray.cs:139
4.   at PassTroughJob.Execute (Unity.Audio.ExecuteContext`2[PassTroughJob+Params,PassTroughJob+Providers]& context) [0x00047] in E:\Work\Project.UnityBeta\Assets\Engine\Kite.Audio\Runtime\DSP\Upmix.cs:117
5.   at Unity.Audio.AudioKernelExtensions+AudioKernelJobStructProduce`3[TAudioKernel,TParameters,TProviders].ExecuteKernel (TAudioKernel& kernel, System.IntPtr audioDataPtr) [0x00173] in E:\Work\Project.UnityBeta\Library\PackageCache\com.unity.audio.dspgraph@0.1.0-preview.3\Runtime\Extensions\AudioKernelExtensions.cs:250
6.   at Unity.Audio.AudioKernelExtensions+AudioKernelJobStructProduce`3[TAudioKernel,TParameters,TProviders].Execute (TAudioKernel& kernel, System.IntPtr audioDataPtr, System.IntPtr functionIndex, Unity.Jobs.LowLevel.Unsafe.JobRanges& ranges, System.Int32 ignored2) [0x0001d] in E:\Work\Project.UnityBeta\Library\PackageCache\com.unity.audio.dspgraph@0.1.0-preview.3\Runtime\Extensions\AudioKernelExtensions.cs:190
7.   at (wrapper delegate-invoke) Unity.Audio.AudioKernelExtensions+AudioKernelJobStructProduce`3+ExecuteKernelFunction[PassTroughJob,PassTroughJob+Params,PassTroughJob+Providers].invoke_void_TAudioKernel&_intptr_intptr_JobRanges&_int(PassTroughJob&,intptr,intptr,Unity.Jobs.LowLevel.Unsafe.JobRanges&,int)
8.  

 

I tried this in 2019.2.0f1 and it crashes the editor right away, without anything in the logs or the crash handler appearing.

If I disable Burst I get a similar exception as in my own project:

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. Unity.Collections.NativeArray`1[T].CheckElementWriteAccess (System.Int32 index) (at C:/buildslave/unity/build/Runtime/Export/NativeArray/NativeArray.cs:130)
3. Unity.Collections.NativeArray`1[T].set_Item (System.Int32 index, T value) (at C:/buildslave/unity/build/Runtime/Export/NativeArray/NativeArray.cs:146)
4. DspApplication+Osc.Execute (Unity.Audio.ExecuteContext`2[DspApplication+OscParams,DspApplication+OscProviders]& context) (at Assets/DspApplication.cs:122)
5. Unity.Audio.AudioKernelExtensions+AudioKernelJobStructProduce`3[TAudioKernel,TParameters,TProviders].ExecuteKernel (TAudioKernel& kernel, System.IntPtr audioDataPtr) (at Library/PackageCache/com.unity.audio.dspgraph@0.1.0-preview.3/Runtime/Extensions/AudioKernelExtensions.cs:250)
6. Unity.Audio.AudioKernelExtensions+AudioKernelJobStructProduce`3[TAudioKernel,TParameters,TProviders].Execute (TAudioKernel& kernel, System.IntPtr audioDataPtr, System.IntPtr functionIndex, Unity.Jobs.LowLevel.Unsafe.JobRanges& ranges, System.Int32 ignored2) (at Library/PackageCache/com.unity.audio.dspgraph@0.1.0-preview.3/Runtime/Extensions/AudioKernelExtensions.cs:190)
7.  

Seems like DSPGraph doesn't work in 2019.2.0f1 at all.

Edit: Just checked in 2019.2.0b10 and works correctly with and without Burst.

 

Oops, I'm reading your posts in reverse order.

Anyway as I said in the previous post, if you want it to work in the Editor, just use Unity 2019.2.0b10
https://unity3d.com/unity/beta/2019.2.0b10

 

I made an updated gist which works without an AudioSource, and uses the AudioOutputHandle.
Looks like you were halfway there already.

Works in 2019.2.0b10 in the editor.

 

Just tested 2019.3 versions and 2019.3.0a8 is the last one that works.
a10 and a11 both crashes.
If someone wants to try it with 2019.3.0a8 than you need to downgrade Burst to 1.1.1 because 1.1.2 doesn't compile with a8.

 

Will DSPGraph have built in sample providers for microphones?

 

I wonder if it would be possible Googles Resonance Audio with DOTS. At least partially ? https://resonance-audio.github.io/resonance-audio/

 

That may be better to work with, but without any experience with either system's implementation it is hard to give a good estimation of feedback.

That being said, this is how I thought it was going to work anyway

 

Sounds good, it would be easier to write AudioKernels that work with different channel configurations.

While we are at redesigning the API, could you add a version of

CreateDSPNode

that takes a

TAudioKernel

kernel parameter instead of creating a blank one with

new TAudioKernel()

? That would make it easy to initialize constant data in the AudioKernel.
Doing an update job for this is awkward, even when it doesn't crash (which it did last time I checked)

 

According to Unity QA, the crash issue is fixed in Unity 2020.1.0a1
Since that build is not public, I'm waiting for the fix to be back-ported.
I don't see any new packages on bintray so I assume the fix is in core.

 

Wow, Good news.
It looks like the package has been updated to 0.1.0-preview.5 and it fixes the crash!
Thanks!

 

I assume this is a staging package and not on the package manager?

 

Nope.
It's on release.
https://bintray.com/unity/unity/com.unity.audio.dspgraph

But if you haven't added the package yet, than you won't see it, you have to add

"com.unity.audio.dspgraph": "0.1.0-preview.5",

to manifest.json manually.

 

So now that we don't crash right away, we are back to the crash I submitted on July 17
It is: (Case 1170153) FreeArrayForDSPGraph crash
Any news on that?

Edit:
I think I can work around my issues using Persistent buffers instead. So I'm back to using Create/Dispose jobs, because you can't create/dispose Persistent buffers from Initialize/Dispose

 

Definitely do this. The JUCE audio library works this way, and it is much more pleasant than the Windows 10 AudioGraph I was working with previously, which used interleaved audio streams.

Edit: The primary benefit is much greater ease of independently routing channels. For example, separate channel buffers let you (for example) write a mono plugin that expects just one input channel, and instantiate two of them to cover two stereo channels with no code changes. This is not possible with interleaved buffers. Plus interleaved buffer math is tricky and easy to get wrong, with excruciating results.

 

Thanks for providing a sample of how to use this new, much appreciated. Are there any more examples showing how this new audio system can be used in a more complex setup?

https://gist.github.com/Hyp-X/a5df43224f09a3ccb8faee57c7220e53#file-dspapplication-cs-L35
At this point, you are sending the audio to the default device. Is it currently possible to control which device is being used? Can you access the device via ASIO?

Yes, do it!

 

I reported a crash bug (Case: 1184507)
This pretty much makes our life miserable while trying to use DSPGraph in Edit mode.