Search Unity

Unity DOTS Audio Discussion

Discussion in 'Data Oriented Technology Stack' started by harini, Mar 28, 2019.

  1. Rich_XR

    Rich_XR

    Joined:
    Jan 28, 2020
    Posts:
    9
    Thanks for the latest release and continued support in this difficult time of Version 0.1.0-preview.13 of DSPgraph and also wanted to mention the work of @florianhanke incredible example code which incorporates a spatialisation approach. @zollenz you mentioned “new system will most likely follow the same approach” on 3rd party spatialisation (resonance for example) does this mean current 0.1.0-preview.13 can be spatialised by a 3rd party post DSPGraph? Any advice on the approach to this, even if hypothetical at this stage is greatly appreciated.
     
    deus0 and florianhanke like this.
  2. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,351
    I just found out about this (DOTS) today. I'm the author of the Master Audio plugin. I am wondering a few things:

    1) What is the release date? Surely there must be one by now, you've been working on this for over a year and a half.
    2) Is this an alternative to current Unity Audio or does it or build on top of it? Will current Unity audio still be a choice to use?
    3) Where can I find an overview of what all it does or what the differences are between current Unity audio and DOTS?

    I'm anticipating our thousands of users are going to want integration with it, so I'm getting ahead of that.

    Thanks!
    -Brian
     
    GliderGuy, BigRookGames and Shinyclef like this.
  3. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    1,001
    Hi Brian,
    1) Sorry, we don't have anything to announce right now. DOTS Audio has been in progress for a while, but we're a small team with a lot of responsibilities, and the DOTS landscape is still shifting quite rapidly around us.
    2) It will be a DOTS-native way of interacting with Unity's audio system. You'll still be able to use the current system as long as you're able to use GameObjects etc.
    3) We'll provide this documentation when we ship the first versions of the DOTS Audio package. You can see what we have now for DSPGraph (mainly in the form of API reference) in the package documentation.

    I want to stress that the experimental DSPGraph packages that we're publishing today are still very early and experimental - there's no user interface, and the API is still expected to change significantly before a "final" version.
     
  4. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,351
    Ok thanks for the info!
     
    Tak likes this.
  5. joeri_07

    joeri_07

    Joined:
    Dec 18, 2015
    Posts:
    44
    Hi everyone,

    I want to make a granular synthesiser in Unity. It is important to have as much flexibility in the execution of the "grains" in terms of the interval and their lengths and also at the same time have it as stable as possible (no hickups).
    I am currently experimenting with executing unity's audioClip's play-method, but I think after discovering this thread I could benefit from concurrency by using this newer approach.

    I have a few questions:
    - could I use the DOTS/ECS programming style to share one audio buffer to be used by multiple AudioSources? I think a lot of memory could be saved this way, since now in my current synchronous approach I believe every AudioSource has its own copy of the AudioClip data... (I found out about Shared Component Data, maybe the audio-buffer could be shared in this way?)
    - If the previous is possible, should I use the DSPGraph framework or use a "roll my own" DOTS implementation?
    - where can I find the most relevant information / test project / code specifically for my use-case?
    - anybody has a similar project or any hints they could share?

    Thanks a lot - really excited about this new direction of audio unity is going in!

    Cheers
     
    Last edited: Mar 3, 2021
  6. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    DSPGraph is sample-accurate on the mixing thread, so that probably is a good place to start.

    Are you currently using ECS? DSPGraph doesn't have anything special for ECS. It is just as valid in a MonoBehaviour project. Also, DSPGraph allocates intermediate buffers between nodes in the graph which might be just as much memory as your audio sources. What makes you believe that the AudioClip data is not shared?

    If you are looking to leverage ECS and manually control buffer allocations, I may have a project that is close-ish to what you are looking for.
     
  7. joeri_07

    joeri_07

    Joined:
    Dec 18, 2015
    Posts:
    44
    ah interesting, so this differs from scheduling things on the main thread as then I would be stuck to the interval of unity's FPS and wouldn't be able to schedule anything at a higher rate, correct?

    No I am not using ECS now, so it would be awesome to benefit from DSPGraph without the need to go into ECS territory.
    I am not sure why I believed that, I guess I was assuming it wasn't.

    If multiple AudioSources are referencing the same AudioClip data in a normal monobehaviour project anyway, could you explain me what exactly would be beneficial of implementing buffer allocations manually. Isn't GraphDSP leveraging the power of multithreading for audio? What exactly is it that it's doing in a multithreaded way, and especially related to my use-case (playing grains of the same soundfile at a high rate)? I am trying to understand.

    And yes I would love to get a look at your project! Please DM if you like :_)

    As for general references, is it correct to assume I should get all info about DSPGraph here:
    https://docs.unity3d.com/Packages/com.unity.audio.dspgraph@0.1/api/Unity.Audio.html

    And by disecting the mega-city project.
    https://unity.com/megacity

    Or are there other official and non-official sources I should know about?
    Thank you for your help!
     
  8. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    The update-interval on the mixing thread is typically platform-dependent. But yes it is not tied to your graphical framerate.
    None. It is something I did in DSPGraph world where the temporary buffers weren't shared.
    I just released it officially: https://github.com/Dreaming381/Latios-Framework/blob/master/Documentation~/Myri Audio/README.md
    There's a few of them earlier in this thread.
     
  9. joeri_07

    joeri_07

    Joined:
    Dec 18, 2015
    Posts:
    44
    Thank you for all the help and sharing your project!

    I can actually backup my "non-sharing of the audioclip-data"-claim with some analytics :p
    When I create a grain in my world which is essentially an AudioSource with some visual components, and I use the profiler it shows that it uses 4.5mb, when I increase the amount of simultaneously sounding grains with a factor of 100, the memory usage also increases with that same factor. Wouldn't data-sharing prevent this? Or maybe I'm misunderstanding.

    Also right now I am limited to use 255 soundsources in my project, does GraphDSP overcome this? and how?

    That's my final question :)

    Thank you!!
     
  10. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,084
    Megacity used its own DSPGraph-Megacity package - it's not very useful initial reference since
    - it's tied to the specific Unity beta version at the time of its release
    - it's not compatible with later DSPGraph versions
    - was/is not being maintained going forward
    UT would have to upgrade the whole Megacity project to latest DOTS/DSPGraph versions (and possibly redo some parts to a large extent)
     
    Last edited: Mar 4, 2021
  11. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    It's likely the source samples are shared but the destination buffers when sampling the clip with pitch shift, volume, and all those other effects are not. DSPGraph has a similar problem. I was able to avoid this only because I do sampling in jobs dispatched from the main thread, which has its own downsides.

    Yes it does. It is a different audio engine not bound by the same limitations.
     
  12. joeri_07

    joeri_07

    Joined:
    Dec 18, 2015
    Posts:
    44
    Interesting, is this the approach used in the Myri engine?

    That's good news :)
     
  13. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    Yes. Myri schedules sampling from the main thread and there's a configurable amount of oversampling it can perform. It is more work, but I am able to schedule that work when the worker threads are idle due to ECS sync points, so the extra work isn't felt in frame times.
     
  14. joeri_07

    joeri_07

    Joined:
    Dec 18, 2015
    Posts:
    44
    Hi DSPGraph gurus,

    Is it correct to assume that the Parameters of IAudioKernels (GraphNodes) are operating at the block-size level? Linearly interpolating between blocks?

    If so, would it then not be possible to run commands at an even higher rate? Say that I want to trigger the PlayClip command (of the DSPGraph PlayClip example) at a really fast interval of 0.0001, how would/could I accomplish that?

    Since if I can only interact with a kernel from the outside on the block-level wouldn't I be limited to blocksize/samplerate seconds?
     
    Last edited: Mar 11, 2021
  15. djeedai

    djeedai

    Joined:
    Sep 28, 2018
    Posts:
    6
    Hi-

    Sorry I didn't read the full 6 pages of discussion :) and couldn't find any other relevant info. Is there any visual editor for DSPGraph? I understand this is a lot of work, but that would be really awesome for quick prototyping! Maybe something branched from the ShaderGraph of URP, but adapted for DSPGraph?

    Thanks!
     
  16. Rich_XR

    Rich_XR

    Joined:
    Jan 28, 2020
    Posts:
    9
    Hi

    ECS clearly gives the opportunity to have 10,000's of entities, however what about DSPGraph nodes and unity voicing limitations (255) for example? Is there a way to have 10,0000's of audio emitters playing at the same time? Are they able to get sub mixed somehow into more singular voices to get around this limitation Unity 255 voice limit at all? Megacity demo has lots of emitters but not playing at once, if this audio was generated programmatically in C# code procedurally rather than using an audioclip or .wav material would that be more beneficial? I also have spatialisation concerns, could 4 different sounds from 4 different locations in 3D space be played through 1 voice simultaneously in 3d spatialized audio for example ported out through stereo 3D output? Apologies if my terminology is incorrect?

    @djeedai have you seen the megacity demo pdf documentation from Unity ECS page, it has some instructions of how to visualise the DSpGraph - https://drive.google.com/file/d/1Y7go2Aq6SVxvKVMlipDIqLGsnFnJyN4H/view


    Many thanks
     
  17. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    You are limited by your hardware when using DSPGraph, not some hard-coded threshold.

    Yes, assuming by "different sounds" you mean different emitters playing the same sample set at the same time offsets. I've done this. It is tricky to get right, but the payoff is worth it!
     
  18. Rich_XR

    Rich_XR

    Joined:
    Jan 28, 2020
    Posts:
    9
    Ok, thank you! that's really promising on the spatial voicing scenario as the project is only one sound being used as many times as possible in a 3D scene, I'm really interested in how you managed to sum that signal into one voice! Is it along the lines of submixing DSPgraph nodes into eachother?
     
  19. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    I actually do sampling outside of DSPGraph and send the individual spatial channels to DSPGraph for filtering and mixing. This is the summing logic I am using: https://github.com/Dreaming381/Latios-Framework/blob/v0.3.1/MyriAudio/Jobs/Batching.cs
     
    datne and Rich_XR like this.
  20. Rich_XR

    Rich_XR

    Joined:
    Jan 28, 2020
    Posts:
    9
  21. Hikiko66

    Hikiko66

    Joined:
    May 5, 2013
    Posts:
    1,156
    Metasounds is out, and it's pretty good.
    It's been 2 years since this thread was started.
    Has this been put on the back burner? Are we ever going to see this?
     
  22. thelebaron

    thelebaron

    Joined:
    Jun 2, 2013
    Posts:
    672
    I just gave it a test and want to thank you for this. Brings me one step closer to a monobehaviour free(runtime at least) workflow. I also wanted to report a bug? Or maybe its incorrect use.
    In just an empty project using the Injection bootstrap, I noticed Looping seems to restart the audio before its completed playing, seems like after 0.1/2s it starts again.
     
    GliderGuy likes this.
  23. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    Thanks for reporting this. I just confirmed the issue. As a temporary workaround, make sure your looping audio source's sample rate matches the output.
     
    thelebaron likes this.
  24. webbgb

    webbgb

    Joined:
    Jan 26, 2019
    Posts:
    1
    I have been playing with DSPGraph for the last week so this might be a stupid question but before I start going deeper I wanted to make sure I was thinking about Dataflow correctly. I have looked at all the ways the data can be interchanged between the DSP Nodes and the main thread. But am I correct in thinking that the only way to pass data between DSP Nodes directly is through the Outlet/Inlet ports or have I missed any alternatives (eg some way to pass data at say a block level)?

    Thanks
     
  25. LaurentGibert

    LaurentGibert

    Unity Technologies

    Joined:
    Jan 23, 2020
    Posts:
    144
    Hi @Hikiko66,

    Thank you for your message! Yes Metasounds looks great. We are still dedicated at developing and improving audio capabilities in Unity, and we don't lack ideas, though DOTS development has taken more time than we hoped, and DOTS specific Audio development is dependent on it.

    We are currently reevaluating our options to find a way to accelerate the availability of new Audio development to all Unity users. Stay tuned!

    Laurent
     
  26. vildauget

    vildauget

    Joined:
    Mar 10, 2014
    Posts:
    101
    How I imagine the evolvement of DOTS now.

    upload_2021-6-15_9-14-7.png
     
  27. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,027
    So basically you are saying that it is not happening now and we can expect progress to be entirely rebooted due to having lots of ideas that took too much time. Re-evaluating means you start from scratch for the nth time.

    Doesn't sound brill mate. And I think TBH people have stayed tuned for a while now, it's Unity that's been lacking on openness and progress.

    A bit of candour is best. DOTS is taking a year off according to the official sticky, so we should not be staying tuned for another year of no news. It would be better if we come back and see the state of DOTS in a year or two and give Unity the valuable time they need to properly work on all this.

    We can take real news, we are logical developers. People only get upset if they're given the run around.
     
  28. JamesWjRose

    JamesWjRose

    Joined:
    Apr 13, 2017
    Posts:
    530
    INDEED!! I have been a software dev for 20+ years, and communication is the Alpha and Omega of ALL of it.

    It has taken WAY too long for audio to get to ANY point in this process. Unity NEEDS to get it $#!t together about DOTS, and I REALLY hope they do. I do VR dev now, and DOTS is SO important.
     
    Thygrrr and april_4_short like this.
  29. Tak

    Tak

    Unity Technologies

    Joined:
    Mar 8, 2010
    Posts:
    1,001
    Yes, currently the only way to pass data from node to node is via ports/connections.
     
    webbgb likes this.
  30. LaurentGibert

    LaurentGibert

    Unity Technologies

    Joined:
    Jan 23, 2020
    Posts:
    144
    @hippocoder @JamesWjRose thanks for the follow up, I hear your frustration. Let me expand on this statement I made:
    The words were chosen carefully on my end, as I was trying to acknowledge the point being made:
    This is accurate, we made announcements that haven't been fully realized with actual systems, features, and workflows you can all use in production, and it is unclear where we are going. The current "reevaluation" that is happening is in part a retrospective action that tries to clarify what went wrong, and make adjustments to our way of working so we can serve you better, with actual real deliverables that help you create your games and other experiences.

    We will come back to you with an updated plan, that will describe how we fix this situation.

    Thanks for being engaged in this conversation.

    Laurent
     
  31. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,743
    If I can make a suggestion, I'd say it would be very useful to get a simplistic DOTS AudioSource/AudioListener equivalent released as soon as possible, even if it's just a temporary solution while we wait for the real one.

    Dealing with anything "Hybrid" greatly complexifies any DOTS project at the moment, and for audio we are currently forced to go the hybrid route with the GameObject AudioSource (or figure out how to make an AudioSource using DSPGraph, but that's a lot of work and the doc isn't sufficient). Having a temporary DOTS AudioSource would allow us to at least start implementing audio in our dots project in a way that won't necessitate huge code architecture changes once real DOTS audio solutions become available

    Just my 2 cents
     
    Last edited: Jun 17, 2021
  32. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    2,522
    Out of curiosity, have you tried the community solutions posted on this thread? If so, what makes them inadequate that you believe Unity's temporary solution would improve upon?
     
  33. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,743
    I'll admit I didn't read much of this thread, I'll check out your Myri!
     
  34. lucasparis

    lucasparis

    Joined:
    Mar 3, 2020
    Posts:
    2
    Unity's audio has been a let down, I'm working on audiofocused games and was putting a lot of hope in DSPgraph for the future. Unreal's Metasounds is making a strong argument for me to make the switch...
    Please focus on procedural audio, audio synthesis, and custom driver access (for low latency professional solutions like asio4all)
     
    ll3v3ll likes this.
  35. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    1,000
    Hi @LaurentGibert. Actually for me I just want the dots packages will continue to release incrementally and frequently just like previously. I'm not sure why seems like starting this year the release frequency become way too slow.
     
  36. l33t_P4j33t

    l33t_P4j33t

    Joined:
    Jul 29, 2019
    Posts:
    214
    i keep getting this error randomly, then unity crashes after a few seconds.
    anyone know what it mean? think its a bug. happens on subscene import
     

    Attached Files:

    • fmod.png
      fmod.png
      File size:
      57.6 KB
      Views:
      2,105
  37. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,293
    You should ask FMOD devs, not Unity.
     
  38. l33t_P4j33t

    l33t_P4j33t

    Joined:
    Jul 29, 2019
    Posts:
    214
    i don't even have fmod
    wtf is fmod.
    seems unity uses it underneath or something
     
    Last edited: Jun 28, 2021
  39. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,293
  40. april_4_short

    april_4_short

    Joined:
    Jul 19, 2021
    Posts:
    489
    Could we just get a Jobs/Burst backed AudioSource/AudioClip with sampling and effects processing done across as many threads as the system has so we can scale effects and play masses of samples/clips/sources?

    Audio doesn't really need to be ECS/DOTS bound... it only needs to be Jobified because it's such simple, linearly occurring, logic free streaming of samples and effects.
     
  41. Eden1230

    Eden1230

    Joined:
    Feb 23, 2020
    Posts:
    5
    my system currently converts wav to
    AudioClip
    in runtime. This is to allow for modding.

    The
    AudioClip
    wont be altered after the use of
    AudioClip.Create()

    is there a way to use that
    AudioClip
    in DSPGraph? i.e. lock the
    AudioClip
    to read-only
     
  42. Eden1230

    Eden1230

    Joined:
    Feb 23, 2020
    Posts:
    5
    Will this be out by the end of 2021 (audio package)?
     
  43. LaurentGibert

    LaurentGibert

    Unity Technologies

    Joined:
    Jan 23, 2020
    Posts:
    144
    Hi @Eden1230, there is currently no plan to release this in the calendar year 2021.
     
  44. JamesWjRose

    JamesWjRose

    Joined:
    Apr 13, 2017
    Posts:
    530
    Not ecstatic about that, but I am highly appreciative for letting us know.
     
  45. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,027
    Yes - thanks, very dependable to have concrete answers even if one must wait for them :)
     
  46. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    3,268
    @LaurentGibert Thank you for being candid and replying. Its really hard to get information out of unity regarding what is actually happening, and that alone is what really builds the frustration you see in users. The more we get information like this earlier, the less of a "blow" it feels and therefore the less frustration you see vented in the forums and other channels like discord etc.

    As long as we know what the truthful state of things are, we can plan around it and that is what really matters to everyone here. Greatly appreciated to have someone clarify this, shame it took digging through the forums to work this out.

    Right now everyone in Tiny forums has been sat for ages with no news, and are using this thread as "news. Whoever you can speak to at unity to rectify that would be great - its not good to leave them hanging.
     
    bb8_1 likes this.
unityunity