Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Enter the 2020.2 Beta Sweepstakes for a chance to win an Oculus Quest 2.
    Dismiss Notice

DOTS Audio Discussion

Discussion in 'Data Oriented Technology Stack' started by harini, Mar 28, 2019.

  1. Hikiko66

    Hikiko66

    Joined:
    May 5, 2013
    Posts:
    1,022
    Hi

    Is there anything built in for scheduling audio at this point? How would I accomplish that?
    I don't see anything in Megacity or the script reference.

    Thanks
     
  2. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    What are you looking for in scheduling?
    Starting audio at a specific point in the future?
    Or synchronizing something to audio playback?

    For staring audio at a specific time, since you buffer the DSP commands, it is supposedly picked up the next mixing cycle after the buffer submission. There's an unknown delay here depending of the mixing buffer size.
    For large time intervals I would delay the command submission, for more precise stuff I would implement it in the DSP kernel.

    If you mean synchronizing this is something I also need to look into for cutscenes...
     
  3. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    The fix for this one will land in 2019.2.8f1 (and 2019.3.0b5), as well as version 0.1.0-preview.6 of the dspgraph package (which is actually available now, but requires those unity versions anyway because of other fixes needed in unity).
     
  4. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    For general precise scheduling, you can schedule a parameter key for a precise time.
     
  5. Hikiko66

    Hikiko66

    Joined:
    May 5, 2013
    Posts:
    1,022
    I want to schedule audio to play at a specific time relative to other audio that is already playing. Really, I want the sample position of the clip being played, because pitch shifting might have happened. How could I query the current sample position of a clip that is being played?

    Something like AudioSource.timeSamples

    I am basically confused as to how to get parameter values and other data back out of dots into the main thread so I can query it. Are there any resources I can look at?

    Thanks
     
    Last edited: Oct 2, 2019
    Lars-Steenhoff likes this.
  6. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    I tried
    0.1.0-preview.6
    with
    2019.3.0b5
    , but it doesn't compile in build mode, I get the following errors:

    Code (CSharp):
    1. Library\PackageCache\com.unity.audio.dspgraph@0.1.0-preview.6\Runtime\SampleProviderContainer.cs(222,17): error CS0117: 'SampleProvider' does not contain a definition for 'Safety'
    2. Library\PackageCache\com.unity.audio.dspgraph@0.1.0-preview.6\Runtime\SampleProviderContainer.cs(222,26): error CS0103: The name 'Safety' does not exist in the current context
    3.  
     
  7. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    Argh, you're right, somebody just found that internally.
    You can work around it locally by wrapping the offending line in: #if ENABLE_UNITY_COLLECTIONS_CHECKS
     
  8. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    I think I'll wait for the fix...
    I tried to use preview.5 but that threw an error during runtime.
    I'm back to developing in 2019.2.7f2 in the meantime (our Main branch), there's plenty of things to do there as well.
     
  9. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    979
    Slightly tangential question - hope this is not too off topic here -

    will existing audio API surface be preserved in DOTS Audio ?
    I am assuming it won't go away completely on a whim, just want to find out what level of compatibility can we expect going forward

    I'm mainly concerned about existing OnAudioFilterRead and PCM callbacks for AudioSource, and possibly Audio Mixer and its plugins interface (this will probably change)

    (though I would expect user facing Editor stuff wrt to audio manipulation not to change drastically it would be nice to have existing API directly tied to it to stay as well - if it's possible to tell at this time/at all)

    Thank you !
     
  10. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    Since 2019.3.0b5 is out already, I had better publish the DSPGraph changelogs:

    [0.1.0-preview.7] - 2019-10-03

    Fixes and improvements
    • Reenable bursting of output jobs
    • Improve synchronization of unattached subgraphs
    • Fix safety handle error when building player

    [0.1.0-preview.6] - 2019-08-26
    Migrate DSPGraph implementation to C#
    • Migrate DSPGraph implementation to C#
    • Add optional support for executing subgraphs with no outputs
     
    mitaywalle and Griz like this.
  11. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    Cool, 0.1.0-preview.7 seems to work with b5, great job!

    Next challenge for make VideoPlayer audio work with DSPGraph:
    So far what I have:
    - 2019.2.7f2/preview.5 on Windows: Works with both in Build and in the Editor
    - 2019.2.7f2/preview.5 on PS4/XB1: No audio (haven't tested myself as I'm home with a flu, but I'll investigate/report if needed
    - 2019.3.0b5/preview.7 on Windows Editor: Audio starts but cuts off and I get a lot of messages like:
    AudioSampleProvider buffer overflow. 128 sample frames discarded.

    - 2019.3.0b5/preview.7 not tested in build yet

    My implementation is:

    Code (CSharp):
    1. public static void PlayClip(this VideoPlayer videoPlayer, VideoClip videoClip, bool looping)
    2.         {
    3.             videoPlayer.clip = videoClip;
    4.             videoPlayer.isLooping = looping;
    5.             videoPlayer.audioOutputMode = VideoAudioOutputMode.APIOnly;
    6.             videoPlayer.prepareCompleted += VideoPlayerOnPrepareCompleted;
    7.             videoPlayer.Prepare();
    8.         }
    9.  
    10.         private static void VideoPlayerOnPrepareCompleted(VideoPlayer videoPlayer)
    11.         {
    12.             // Create the AudioKernel here
    13.             videoPlayer.Play();
    14.         }
    My kernel determines the audio is over when I don't get a full buffer.
    This is what happens in b5, but I don't exactly know what to do here.
    Maybe this is what happens on consoles...
    I'll try to work around for now.
     
  12. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    I tried 2019.2.8f1 with 0.1.0-preview.7
    While it looks nice on Windows (so far), I get compile errors on PS4/XB1 (same error on both platforms):

    Code (CSharp):
    1. IL2CPP error for method 'System.Void Unity.Audio.DSPNode::SetupPortBuffers(Unity.Audio.DSPGraph,System.Int32,System.Single**)' in DSPNodeImplementation.cs:369
    2. Additional information: Sequence contains no matching element
    3.  
    Edit: The issue also appears on Windows if I compile with IL2CPP

    Edit2: Looks like 0.1.0-preview.5 doesn't work with this Unity version...

    Edit3: Looked at 2019.3.0b5 and it has the same IL2CPP error

    Edit4: Looked at 2019.3.0b6 and it has the same IL2CPP error
     
    Last edited: Oct 7, 2019
  13. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    Also 2019.2.8f1 with 0.1.0-preview.7 crashes for us at every script recompile, so we reverted to 2019.2.7f2 with 0.1.0-preview.5 for now.
     
  14. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    Reported (Case 1189661) Can't build DSPGraph project with IL2CPP
     
  15. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    Reported (Case 1189666) DSPGraph project crashes on script recompile
    It appears to have a 100% repro rate
     
  16. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    Thanks for the reports, we're looking into these. :)
     
  17. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    907
    Would this also replace onAudioFilterRead?
    If yes can we have an example of how to implement it?
     
  18. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    Like everything with DOTS, the capabilities will be similar, but the API will be different.
    You'll still be able to use the existing API for things in the GameObject/MonoBehaviour world, but the DOTS world will require a different workflow.
     
  19. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    You'll be able to do similar things with DSPGraph as with OnAudioFilterRead.
    We're working on shipping a set of simple examples - in the mean time, the best example is probably our Megacity project.
     
    00christian00 likes this.
  20. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    979
    Oh right the question wasn't necessarily about DSPGraph (/artist tooling/HAL) but you answered it for existing API anyway
    Thanks !
     
  21. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    Note that this issue (1184507 DspGraph: Crash when entering Play mode) is still crashing in
    2019.2.8f1

    I have a workaround of shutting down audio from
    EditorApplication.playModeStateChanged
    , but if I disable the workaround I still crash.

    The issue I reported yesterday (1189666 DSPGraph project crashes on script recompile) crashes with the same call stack, but I wasn't able to find a workaround.
    At least my attempt to use
    AssemblyReloadEvents.beforeAssemblyReload
    didn't help.
     
  22. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    At least in the project you sent, the trigger for the behavior was the
    AudioSystem
    being
    ExecuteInEditMode
    (not that we won't make that work, but that's why we missed it). You should be able to work around by removing that, if that's a palatable workaround.
     
  23. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    I need editor mode audio for our Sound Engineer to place and test ambient sounds on the level.
    But you are right it doesn't need to be always on.

    I already support the sound on/off icon on the scene view toolbar, to turn audio on/off.
    In my test project I changed it to not even initializing DSPGraph unless scene audio is turned on.

    Most workflows don't need audio and our Sound Engineer will not change code so this fixes 99% of our workflow.

    The remaining issue which prevents us to move to 2019.2.8f1 or 2019.3.0b6 is that DSPGraph doesn't compile to PS4 and XB1 (IL2CPP issue).
    (Actually now that 2019.3.0b6 has cache server support, we might move our production there.)
     
  24. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    I'm experiencing similar behavior. It looks like I'm seeing a buffer underflow (inside the native sample provider) near the start of the video playback.
    Nothing with sample providers has changed between
    preview.5
    and
    preview.7
    , so I'm not sure if it's a coincidence or if there's some incidental change in timing etc. that causes different behavior. We'll look into it.
     
  25. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    It looks like this is actually a bug in il2cpp, but we'll have a workaround on our side for
    preview.8
    .
     
  26. Griz

    Griz

    Joined:
    Aug 27, 2013
    Posts:
    129
    Gave it a go and made a small synth based on the Tinus example. Thanks Tinus!
    Snap 2019-10-11 at 00.27.38.png
    It is playable with both, keyboard (FL studio style) and the UI keys.

    Lowering the buffer size, which is currently 1024, would make it more responsive, but i haven't found a way to get smaller sample sizes than 1024 from OnAudioFIlterRead.
     

    Attached Files:

    Cloudless-Rain, deus0, twobob and 3 others like this.
  27. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    Nice! You can avoid OnAudioFilterRead (and the extra buffer copy) altogether if you attach your driver to the default output in Awake. (Although the default output is still limited by the minimum dsp buffer size in the audio settings.)
     
    deus0, elbows and Griz like this.
  28. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    3,847
    can ECS audio do binaural beats? = control of each channel output
     
  29. Nothke

    Nothke

    Joined:
    Dec 2, 2012
    Posts:
    108
    I've done this by using
    _driver.AttachToDefaultOutput();
    , it works, but editor now crashes when exiting play mode. Is there something that needs to be detached/disposed on quit?

    Thanks @Griz for an example btw.
     
  30. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    It should never crash, but we have had some bugs in this area. :-|
    Are you disposing the graph in the output job's
    Dispose
    method?
     
  31. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    979
    graph is being disposed in the @Griz sample above @Tak
    after attaching driver to default output and removing audio filter b8 crashes in EndMix for me too after exiting the play mode
    you can guard Begin/EndMix with graph.Valid, it then throws c# NRE in atomicqueue but can exit play mode at least *

    * sometimes
     
    Last edited: Oct 27, 2019
  32. Nothke

    Nothke

    Joined:
    Dec 2, 2012
    Posts:
    108
    I literally took @Griz's example, removed OnAudioFilterRead and added attach in Awake (line 128):

    Code (CSharp):
    1. using UnityEngine;
    2. using Unity.Collections;
    3. using Unity.Audio;
    4. using Unity.Mathematics;
    5. using Unity.Burst;
    6. using System.Collections;
    7. using System.Collections.Generic;
    8.  
    9. /*
    10.     For allocations inside AudioKernels, use Allocator.AudioKernel
    11. */
    12.  
    13. public class AudioTest : MonoBehaviour {
    14.     public const int SAMPLERATE = 48000;
    15.     public const int NUMFRAMES = 1024;
    16.     public const int NUMCHANNELS = 2;
    17.  
    18.     private DSPGraph _graph;
    19.     private DefaultDspGraphDriver _driver;
    20.     private AudioOutputHandle _outputHandle;
    21.     private DSPNode _node;
    22.  
    23.     private NativeArray<float> _output;
    24.  
    25.     private bool _isRunning;
    26.  
    27.     public float volume = 1.0f;
    28.     public float pitchShift = 0.0f;
    29.     public float pitchShiftAmount = 400.0f;
    30.  
    31.     public bool playWithMouse = false;
    32.  
    33.     #region Keys
    34.  
    35.     List<KeyCode> keysPressed = new List<KeyCode>();
    36.     Dictionary<KeyCode, int> keyIdDict = new Dictionary<KeyCode, int>();
    37.     float [] frequencies = new float []{
    38.         130.81f, //C3
    39.         138.59f,
    40.         146.83f,
    41.         155.56f,
    42.         164.81f,
    43.         174.61f,
    44.         185.00f,
    45.         196.00f,
    46.         207.65f,
    47.         220.00f,
    48.         233.08f,
    49.         246.94f,
    50.         261.63f, //C4
    51.         277.18f,
    52.         293.66f,
    53.         311.13f,
    54.         329.63f,
    55.         349.23f,
    56.         369.99f,
    57.         392.00f,
    58.         415.30f,
    59.         440.00f,
    60.         466.16f,
    61.         493.88f,
    62.         523.25f //C5
    63.     };
    64.  
    65.     KeyCode [] keys;
    66.  
    67.     void InitKeys() {
    68.         keys = (KeyCode[])System.Enum.GetValues(typeof(KeyCode));
    69.         PopulateDictionary();
    70.     }
    71.  
    72.     void PopulateDictionary() {
    73.         int i = 0;
    74.         keyIdDict.Add(KeyCode.Z, i++);
    75.         keyIdDict.Add(KeyCode.S, i++);
    76.         keyIdDict.Add(KeyCode.X, i++);
    77.         keyIdDict.Add(KeyCode.D, i++);
    78.         keyIdDict.Add(KeyCode.C, i++);
    79.         keyIdDict.Add(KeyCode.V, i++);
    80.         keyIdDict.Add(KeyCode.G, i++);
    81.         keyIdDict.Add(KeyCode.B, i++);
    82.         keyIdDict.Add(KeyCode.H, i++);
    83.         keyIdDict.Add(KeyCode.N, i++);
    84.         keyIdDict.Add(KeyCode.J, i++);
    85.         keyIdDict.Add(KeyCode.M, i++);
    86.         keyIdDict.Add(KeyCode.Comma, i);
    87.  
    88.         keyIdDict.Add(KeyCode.Q,      i++);
    89.         keyIdDict.Add(KeyCode.Alpha2, i++);
    90.         keyIdDict.Add(KeyCode.W,      i++);
    91.         keyIdDict.Add(KeyCode.Alpha3, i++);
    92.         keyIdDict.Add(KeyCode.E,      i++);
    93.         keyIdDict.Add(KeyCode.R,      i++);
    94.         keyIdDict.Add(KeyCode.Alpha5, i++);
    95.         keyIdDict.Add(KeyCode.T,      i++);
    96.         keyIdDict.Add(KeyCode.Alpha6, i++);
    97.         keyIdDict.Add(KeyCode.Y,      i++);
    98.         keyIdDict.Add(KeyCode.Alpha7, i++);
    99.         keyIdDict.Add(KeyCode.U,      i++);
    100.         keyIdDict.Add(KeyCode.I,      i++);
    101.     }
    102.     #endregion
    103.  
    104.     List<int> freqIDsToPlay = new List<int>();
    105.  
    106.     private void Awake() {
    107.         InitKeys();
    108.  
    109.         var audioConfig = AudioSettings.GetConfiguration();
    110.         Debug.Log("Unity Audio Config:");
    111.         Debug.Log(audioConfig.sampleRate);
    112.         Debug.Log(audioConfig.dspBufferSize);
    113.         Debug.Log(audioConfig.speakerMode);
    114.  
    115.         _graph = DSPGraph.Create(SoundFormat.Stereo, NUMCHANNELS, NUMFRAMES, SAMPLERATE);
    116.         _driver = new DefaultDspGraphDriver();
    117.         _driver._graph = _graph;
    118.         _driver.Initialize(NUMCHANNELS, SoundFormat.Stereo, SAMPLERATE, NUMFRAMES);
    119.  
    120.         var cmd = _graph.CreateCommandBlock();
    121.         _node = cmd.CreateDSPNode<OscParams, OscProviders, Osc>();
    122.         cmd.AddOutletPort(_node, NUMCHANNELS, SoundFormat.Stereo);
    123.         cmd.Connect(_node, 0, _graph.RootDSP, 0);
    124.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Frequency, 440f);
    125.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Volume, 0.0f);
    126.         cmd.Complete();
    127.  
    128.         _driver.AttachToDefaultOutput();
    129.  
    130.         _output = new NativeArray<float>(NUMCHANNELS * NUMFRAMES, Allocator.Persistent, NativeArrayOptions.ClearMemory);
    131.     }
    132.  
    133.     private IEnumerator Start() {
    134.         /*
    135.             Note: when we try to start drawing audio from the graph right at application start
    136.             we get some errors related to NativeArray buffers with invalid state. If we wait
    137.             a tiny bit we're fine.
    138.         */
    139.         yield return new WaitForSeconds(0.5f);
    140.  
    141.         _driver.BeginMix(NUMFRAMES);
    142.         _isRunning = true;
    143.     }
    144.  
    145.     private void OnDestroy() {
    146.         var cmd = _graph.CreateCommandBlock();
    147.         cmd.ReleaseDSPNode(_node);
    148.         cmd.Complete();
    149.  
    150.         _driver.Dispose();
    151.         _output.Dispose();
    152.     }
    153.  
    154.     private void Update() {
    155.         KeyboardUpdate();
    156.         PlayUpdate();
    157.         if(playWithMouse) MouseUpdate();
    158.     }
    159.  
    160.     void MouseUpdate() {
    161.          if(Input.GetKey(KeyCode.Mouse0)){
    162.             float pitch = 440f + Input.mousePosition.y;
    163.             float volume = Input.GetKey(KeyCode.Mouse0) ? 0.5f : 0.0f;
    164.             SendCommand(pitch, volume);
    165.         }
    166.         if(Input.GetKeyUp(KeyCode.Mouse0)) {
    167.             SendCommand(440.0f, 0.0f);
    168.         }
    169.     }
    170.  
    171.     void PlayUpdate() {
    172.         if(freqIDsToPlay.Count > 0)
    173.             SendCommand(frequencies[freqIDsToPlay[freqIDsToPlay.Count-1]] + pitchShiftAmount * pitchShift, volume);
    174.         else
    175.             SendCommand(440.0f, 0.0f);
    176.     }
    177.  
    178.     void KeyboardUpdate() {
    179.         for(int i = 0; i < keys.Length; i++) {
    180.             if(Input.GetKeyDown(keys[i])) {
    181.                 if(keyIdDict.TryGetValue(keys[i], out var value)){
    182.                     PlayKey(value);
    183.                 }
    184.             }
    185.         }
    186.  
    187.         for(int i = 0; i < keys.Length; i++) {
    188.             if(Input.GetKeyUp(keys[i])) {
    189.                 if(keyIdDict.TryGetValue(keys[i], out var value)){
    190.                     StopKey(value);
    191.                 }
    192.             }
    193.         }
    194.     }
    195.  
    196.     void ClearKeys() {
    197.         freqIDsToPlay.Clear();
    198.     }
    199.  
    200.     public void PlayKey(int id) {
    201.         freqIDsToPlay.Add(id);
    202.     }
    203.  
    204.     public void StopKey(int id) {
    205.         freqIDsToPlay.Remove(id);
    206.     }
    207.  
    208.     void SendCommand(float pitch, float volume) {
    209.         var cmd = _graph.CreateCommandBlock();
    210.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Frequency, pitch);
    211.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Volume, volume);
    212.         cmd.Complete();
    213.     }
    214.  
    215.     private const float TWOPI = math.PI * 2f;
    216.  
    217.     public enum OscParams {
    218.         Frequency,
    219.         Volume
    220.     }
    221.  
    222.     public enum OscProviders {
    223.         Provider
    224.     }
    225.  
    226.     [BurstCompile]
    227.     public struct Osc : IAudioKernel<OscParams, OscProviders> {
    228.         private float _phase;
    229.         private float _phaseStep;
    230.         private float _freq;
    231.         private float _volume;
    232.  
    233.         public void Initialize() {}
    234.  
    235.         public void Dispose() {}
    236.  
    237.         public void Execute(ref ExecuteContext<OscParams, OscProviders> context) {
    238.             for (int outIdx = 0; outIdx < context.Outputs.Count; outIdx++) {
    239.                 var sampleBuffer = context.Outputs.GetSampleBuffer(outIdx);
    240.                 var buffer = sampleBuffer.Buffer;
    241.                 for (int i = 0; i < NUMFRAMES; i++) {
    242.                     _volume = math.lerp(_volume, context.Parameters.GetFloat(OscParams.Volume, i), 0.01f);
    243.                     _freq = math.lerp(_freq, context.Parameters.GetFloat(OscParams.Frequency, i), 0.01f);
    244.  
    245.                     _phaseStep = (TWOPI * _freq) / (float)SAMPLERATE;
    246.  
    247.                     buffer[i*NUMCHANNELS] = math.sin(_phase) * _volume;
    248.                     buffer[i*NUMCHANNELS+1] = buffer[i * NUMCHANNELS];
    249.                     _phase += _phaseStep;
    250.                     if (_phase > TWOPI) {
    251.                         _phase -= TWOPI;
    252.                     }
    253.                 }
    254.             }
    255.         }
    256.     }
    257. }
    258.  
    259. [BurstCompile]
    260. public struct DefaultDspGraphDriver : IAudioOutput {
    261.     public DSPGraph _graph;
    262.     private int _channelCount;
    263.  
    264.     public void Initialize(int channelCount, SoundFormat format, int sampleRate, long dspBufferSize) {
    265.         _channelCount = channelCount;
    266.     }
    267.  
    268.     public void BeginMix(int frameCount) {
    269.         _graph.BeginMix(frameCount);
    270.     }
    271.  
    272.     public void EndMix(NativeArray<float> output, int frames) {
    273.         _graph.ReadMix(output, frames, _channelCount);
    274.     }
    275.  
    276.     public void Dispose() {
    277.         _graph.Dispose();
    278.     }
    279. }
     
  33. Zikoure

    Zikoure

    Joined:
    Sep 18, 2019
    Posts:
    2
    How would one go about playing audio clips ?
    In the megacity demo you use CreateAudioSampleProvider from the AudioClipExtensions class to get an AudioSampleProvider from an AudioClip. This doesn't seem to exist in Unity 2019.3.0b8 using the 0.1.0-preview.7.
    Are there any examples on how to get a SampleProvider directly from an AudioClip ?
     
  34. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    325
    The API changed in 2019.2 you can pass the AudioClip directly to SetSampleProvider
    So you don't need to get a SampleProvider
     
  35. Musonica

    Musonica

    Joined:
    Jul 25, 2017
    Posts:
    7
    Hey, I'm developing an AI Music system in Unity for a PhD. I'm working on re-architecting the software framework using DOTS and looking for the optimal way to approach to a flexible audio event scheduling / threading architecture.

    Naturally the highest resolution music note event timing is essential, with other level threads (such as DSPGraph, an Instrumental Control rate thread (i.e. MIDI type controller messages) as well as AI processing threads running in the background).

    Can anyone suggest the best approach to ensure note timing stability is rock solid?

    Would calling scheduled Jobs in the Main thread work well enough or is there a way to schedule future Jobs internally in the DOTS framework?
     
  36. Zikoure

    Zikoure

    Joined:
    Sep 18, 2019
    Posts:
    2
    Playing audio from a Video Player works fine with 2 audio channels, but when the VideoClip has 4 audio channels the sound doesn't play correctly. The SampleProvider is fine, it reads everything correctly but the audio sounds stretched and the audio/video aren't synchronised.
    Are VideoClips with 4 audio channels supported yet ?
     
  37. toneTechnican

    toneTechnican

    Joined:
    Oct 22, 2019
    Posts:
    5
    Hey team!

    I'm getting crazy excited about this project! And slow to the uptake as have only seen it now. Really keen to jump in with it.

    I was wondering if there will be support for non-mixed raw audio output to the sound card? I see there is a RAW SoundFormat, but not entirely sure what this means. Maybe it already provides a functionality I'd like to see.

    Ideally, a DSP graph that can route game object audio directly to soundcard channels (i.e audio obj 1 -> channel 1, audio obj 2 -> channel 2).

    I think something that really excites me is being able to playback Ambisonic audio files out to the soundcard and do the decoding to with an external decoder to be able to get immersive loudspeaker sound. This is particularly useful for 360 videos with VR caves and headsets.
     
    5argon likes this.
  38. janusl

    janusl

    Unity Technologies

    Joined:
    Aug 8, 2018
    Posts:
    19
    We don't have a control rate system. Most likely you want a queue with events, that are popped from a note every N samples (which will be your control rate then). You then buffer the queue up with a gracious window, depending on your real time needs.

    This in particular is one of the primary problems DSPGraph solves. It takes care of scheduling the jobs with the lowest possible latency.
     
  39. janusl

    janusl

    Unity Technologies

    Joined:
    Aug 8, 2018
    Posts:
    19
    Awesome!

    Yes, outputting directly to hardware channels will be possible with the HAL integration (if you have a sound card setup that makes it possible).

    But...
    it won't automatically be routed through game objects like you mention. Certainly that could be an easy stack to build on top, but from the hal / dspgraph side it would just be an array of buffers you mix into.
     
    5argon likes this.
  40. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    283
    I'm assuming the HAL aspect isn't available just yet.

    Looking into handling routing asio channels from a device and IO to and from it from within Unity.
     
  41. janusl

    janusl

    Unity Technologies

    Joined:
    Aug 8, 2018
    Posts:
    19
    Not just yet, but we'll make a preview as soon as it is usable to some degree.
     
  42. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    ## [0.1.0-preview.9] - 2019-11-15
    ### Fixes
    - Update com.unity.media.utilities dependency to publically-available preview.2

    ## [0.1.0-preview.8] - 2019-11-06
    ### Improvements
    - Extract collections into their own package
    - Add DefaultDspGraphDriver implementation
    - Improve granularity of profiler markers
    ### Fixes
    - Fixes and workarounds for il2cpp compilation
    - Fix embedded image in manual

    Additional known issue: due to an issue with il2cpp, you can currently build for burst or il2cpp, but not both. If you're building to an il2cpp platform, you'll need to disable burst aot for that platform in your burst settings.
     
    deus0, mitaywalle and Zikoure like this.
  43. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    ## [0.1.0-preview.11] - 2019-12-02
    ### Improvements
    - Fix playback in PlayClip sample

    ## [0.1.0-preview.10] - 2019-11-29
    ### Improvements
    - Add a set of some simple samples
    - Update com.unity.media.utilities to preview.3
     
  44. Nothke

    Nothke

    Joined:
    Dec 2, 2012
    Posts:
    108
    EDIT: My mistake, looks like both
    _graph.Dispose()
    AND
    _driver.Dispose();
    cause crashes. Once you take them out, it seems to be working. Try with code below.

    I found the cause of the crash that I had when using AttachToDefaultOutput(). It was because I called
    graph.Dispose()
    in OnDestroy(). It seems to be automatically disposed, so no need to call it, riiight? Nevertheless, making this mistake shouldn't just straight hard crash Unity.

    Here's an example of a simplest graph that just plays a sine wave, without OnAudioFilterRead, (note the "uncomment either of these to crash Unity"):
    Code (CSharp):
    1.  
    2. //#define USE_ONAUDIOFILTERREAD
    3.  
    4. using UnityEngine;
    5. using Unity.Collections;
    6. using Unity.Audio;
    7. using Unity.Mathematics;
    8. using Unity.Burst;
    9.  
    10. public class SimplestGraph : MonoBehaviour
    11. {
    12.     public const int SAMPLERATE = 48000;
    13.     public const int NUMFRAMES = 1024;
    14.     public const int NUMCHANNELS = 2;
    15.  
    16.     private DSPGraph _graph;
    17.     private DefaultDspGraphDriver _driver;
    18.     private AudioOutputHandle _outputHandle;
    19.     private DSPNode _node;
    20.  
    21.     private NativeArray<float> _output;
    22.  
    23.     private void Awake()
    24.     {
    25.         var audioConfig = AudioSettings.GetConfiguration();
    26.         Debug.Log("Unity Audio Config:");
    27.         Debug.Log(audioConfig.sampleRate);
    28.         Debug.Log(audioConfig.dspBufferSize);
    29.         Debug.Log(audioConfig.speakerMode);
    30.  
    31.         _graph = DSPGraph.Create(SoundFormat.Stereo, NUMCHANNELS, NUMFRAMES, SAMPLERATE);
    32.         _driver = new DefaultDspGraphDriver();
    33.         _driver._graph = _graph;
    34.         _driver.Initialize(NUMCHANNELS, SoundFormat.Stereo, SAMPLERATE, NUMFRAMES);
    35.  
    36.         var cmd = _graph.CreateCommandBlock();
    37.         _node = cmd.CreateDSPNode<OscParams, OscProviders, Osc>();
    38.         cmd.AddOutletPort(_node, NUMCHANNELS, SoundFormat.Stereo);
    39.         cmd.Connect(_node, 0, _graph.RootDSP, 0);
    40.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Frequency, 440f);
    41.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Volume, 0.0f);
    42.         cmd.Complete();
    43.  
    44. #if !USE_ONAUDIOFILTERREAD
    45.         _driver.AttachToDefaultOutput();
    46. #endif
    47.  
    48.         _output = new NativeArray<float>(NUMCHANNELS * NUMFRAMES, Allocator.Persistent, NativeArrayOptions.ClearMemory);
    49.     }
    50.  
    51. #if USE_ONAUDIOFILTERREAD
    52.         private void OnAudioFilterRead(float[] data, int channels)
    53.         {
    54.             if (!_isRunning)
    55.             {
    56.                 return;
    57.             }
    58.  
    59.             _driver.EndMix(_output, NUMFRAMES);
    60.             // Todo: use low level memcopy
    61.             for (int i = 0; i < data.Length; i++)
    62.             {
    63.                 data[i] = _output[i];
    64.             }
    65.             _driver.BeginMix(NUMFRAMES);
    66.         }
    67. #endif
    68.  
    69.     private void Start()
    70.     {
    71.         _driver.BeginMix(NUMFRAMES);
    72.  
    73.         SendCommand(440.0f, 0.5f);
    74.     }
    75.  
    76.     private void OnDestroy()
    77.     {
    78.         // Uncomment either of these to crash Unity:
    79.         //_graph.Dispose();
    80.         //_driver.Dispose();
    81.  
    82.         _output.Dispose();
    83.     }
    84.  
    85.     void SendCommand(float pitch, float volume)
    86.     {
    87.         var cmd = _graph.CreateCommandBlock();
    88.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Frequency, pitch);
    89.         cmd.SetFloat<OscParams, OscProviders, Osc>(_node, OscParams.Volume, volume);
    90.         cmd.Complete();
    91.     }
    92.  
    93.     private const float TWOPI = math.PI * 2f;
    94.  
    95.     public enum OscParams
    96.     {
    97.         Frequency,
    98.         Volume
    99.     }
    100.  
    101.     public enum OscProviders
    102.     {
    103.         Provider
    104.     }
    105.  
    106.     [BurstCompile]
    107.     public struct Osc : IAudioKernel<OscParams, OscProviders>
    108.     {
    109.         private float _phase;
    110.         private float _phaseStep;
    111.         private float _freq;
    112.         private float _volume;
    113.  
    114.         public void Initialize() { }
    115.  
    116.         public void Dispose() { }
    117.  
    118.         public void Execute(ref ExecuteContext<OscParams, OscProviders> context)
    119.         {
    120.             for (int outIdx = 0; outIdx < context.Outputs.Count; outIdx++)
    121.             {
    122.                 var sampleBuffer = context.Outputs.GetSampleBuffer(outIdx);
    123.                 var buffer = sampleBuffer.Buffer;
    124.                 for (int i = 0; i < NUMFRAMES; i++)
    125.                 {
    126.                     _volume = math.lerp(_volume, context.Parameters.GetFloat(OscParams.Volume, i), 0.01f);
    127.                     _freq = math.lerp(_freq, context.Parameters.GetFloat(OscParams.Frequency, i), 0.01f);
    128.  
    129.                     _phaseStep = (TWOPI * _freq) / (float)SAMPLERATE;
    130.  
    131.                     buffer[i * NUMCHANNELS] = math.sin(_phase) * _volume;
    132.                     buffer[i * NUMCHANNELS + 1] = buffer[i * NUMCHANNELS];
    133.                     _phase += _phaseStep;
    134.                     if (_phase > TWOPI)
    135.                     {
    136.                         _phase -= TWOPI;
    137.                     }
    138.                 }
    139.             }
    140.         }
    141.     }
    142. }
     
    Last edited: Dec 15, 2019
    deus0 likes this.
  45. Kandy_Man

    Kandy_Man

    Joined:
    Mar 8, 2014
    Posts:
    46
    I'm very new to this so apologise if this has been answered before. Will we be able to set loop points on audio clips? For example, having an intro to a piece of music that then has a loop later in the piece (think Final Fantasy battle musics and such). Also it would be really cool to be able to import VST plugins into Unity so we can play instruments or apply effects in real time, though I don't see that happening any time soon.
     
  46. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    976
    Thanks, we've improved our validation for the next release.
     
    deus0, mitaywalle and Nothke like this.
  47. ZexScal4

    ZexScal4

    Joined:
    May 25, 2018
    Posts:
    8
    I had a question on how I could add the DSPGraph package to my unity editor project. Please show me the way. Thanks in advance and happy holidays!
     
  48. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,510
    Hello, I would like to ask a few questions!

    I have thought of a DOTS Audio training project for myself : I'ts a tracker module player. I have assumed the following which I would like to confirm are they true, so I do not make the same thing as official solution (that is yet to come?) :

    - The one Unity had since before Unity 5? https://docs.unity3d.com/Manual/TrackerModules.html is not under active development or getting any updates for long time.
    - There is no active team or any roadmap in porting/improving tracker module support to newer architecture/version (due to low demand, etc.)

    I understand that it is a very niche feature and making one for your game is even harder since it require a composer that know how to do it properly. I just love the potential of having a really long, progressive/techno style song in your game in very little space. (Ideal for Tiny build) If there is no demand I better work on it myself.
     
    GliderGuy and sngdan like this.
  49. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    1,012
    5argon likes this.
  50. Zeroneth01

    Zeroneth01

    Joined:
    May 15, 2016
    Posts:
    2
    In your unity project folder, there will be a 'Packages' folder, with a file called 'manifest.json'. In that file, add the following line to the list of 'dependencies':

    "com.unity.audio.dspgraph": "0.1.0-preview.11"


    The package should automatically be added to your project, along with any required packages. As they update the package, you can increase the version number also (the current version number is 11).

    The package is currently working for me using Unity version 2019.3.0f1, but it probably works on many other versions too.
     
    imaginadio, WAYN_Games and ZexScal4 like this.
unityunity