native audio plugins in unity 5.0

obviously, native plugins need to be compiled for their respective platforms, however, are there still security issues/enforcements for native audio plugins with web player builds etc? what platforms can i develop audio plugins for besides mac/pc desktop builds? or are those the only ones?

what other gotchas might i encounter as well? the docs are a little light on those aspects in regards to unity 5

 

no one?

 

I have similar questions, Just been through the docs and demo sdk but a lot of things are unclear atm.

Potential sounds good though! *see what I did there*

 

I orginally posted this to answers, but was told to post it in the forums:

I know the SDK is new and probably still a work in progress, but I need some answers to a few questions. The docs are 'basic' to say the least, but enough to get up and running and get into trouble

I also have not found anything pertaining to this on the answers or audio forums.

I have a granulation component that i have working in unity 4.6 via the OnAudioFilterRead callback, and i used that callback to push data back to the audiosource. I handle all of the positioning, spatialization (along with audio occlusion planned in the near future) and additional filters etc. For this, i need the entire audio buffer due to the calculations as well as drawing etc. My main issue was that the callback wasn't realtime (~7ms latency) as i'm assuming this new version is, which i think was causing some additional weird phasing artifacts.

My thinking is that i could basically break things up a bit and have a 'granulation' filter that would handle the base audio, with multiple side chained 'granulators'.

• For this to work however, i'd need to be able to set/get position on the audio for each granulator, but i'm not sure how i would go about that if they were filters. Would it just be an exposed property on a mixer? Or could i get directly to a filter?
  
  
• I also have the ability to fire back events when a user defined 'region' is entered/exited. Am i able to fire events from a filter? Because i'm assuming the filter GUI is only used in the editor, right?
  
  
• I'm assuming "GetFloatBufferCallback" is capable of returning the entire audio buffer as well? And how would i go about that? And who calls it (the associated gui, correct)? do i need to 'feed' the filter the buffer or is there an internal call/callback etc?
  
  
• On the drawing side, to be able to show a waveform, would i use 'AudioCurveRendering' stuff or is there something more appropriate?
  
  
• Additionally, in the filter itself, is there a way to determine where my plugin is running from? Some sort of native callback? IE: the editor or a compiled version? (this helps in determining licensing, functionality etc)
  
  
• What about webplayer builds? It was a non-starter before due to security issues. Has that changed?
  
  
• In the processing callback, is it a constant size now? i noticed some weird issues with the buffer sizes sometimes changing during processing.
• Also, there is not a lot about the set position callback nor anything that uses it in the sample filters. How does this work exactly?

Lastly, i'm not even sure if this all would work the way I've laid it out, but it sure would be cool if it could.

If some kind soul from UT could take pity on us and explain in *detail* about how to go about actually using all the cool new stuff in the new audio for 5, it would go a long way for those of us trying to make assets etc

 

I took a few minutes and built the SDK demo for web player, trying different build settings and dll combos and it didn't work (it built and ran but no plugins are audible). The option is there but I'd be surprised if native dlls were allowed now, especially considering the impending demise of the web player?

So far I've managed to create my own plugins in a fresh project only by dragging and dropping the SDK demos, first into XCode and then Unity, and tearing them apart. Trying to build my own bundle in xcode based on, but not copied from, the SDK demos has so far been fruitless. Just reuising the Unity provided code has worked fine but I would love some directions on properly compiling. I know the documentation touched on this but I'm struggling to understand proper work flow. I have to recompile my plugins and then restart Unity to test them?

 

Exactly. What I feel is needed is a base template with *only* the headers needed for a clean filter build. I'm not sure what utility methods are supposed to be there or not, and thus I'm potentially adding a lot of crap I might never use. I can make guesses etc, but tbh if someone already has this knowledge it'd help and speed things along instead of fighting unresolved references etc..

I personally would like to know who calls and when the callbacks are fired. Is it constant like portaudio or is it something else.

 

Btw jeff, do we make x64 builds for the new editor? Or is the output from the x64 editor also x64? I'm a bit confused on that point. I thought that it was just the editor that was 64 bit now, no? so if that's the case then what's the point of x64 bit builds if the compiled output is going to be 32 bit?

 

Lol sorry dude but I'm the wrong person to ask.
I'm currently failing to get the SDK demo to run the plugins on my Android device. Same result as the web player. Building the project freshly downloaded from the Doc, Unity throws an error saying the x86 and x86_64 plugins are "colliding with each other".

Found plugins with same names and architectures, Assets/Plugins/x86/AudioPluginDemo.dll () and Assets/Plugins/x86_64/AudioPluginDemo.dll (). Assign different architectures or delete the duplicate.
UnityEditor.AndroidPluginImporterExtension:CheckFileCollisions(String)
UnityEditorInternal.PluginsHelper:CheckFileCollisions(BuildTarget) (at /Users/builduser/buildslave/unity/build/Editor/Mono/Plugins/PluginsHelper.cs:25)
UnityEditor.BuildPlayerWindow:BuildPlayerAndRun()

Removing one or the other gets the project to build but still fails to produce plugin sounds. Messing with different check boxes (Android, Editor, all platforms) made no difference.

 

Yeah, that was kind of my experience as well ;(

Btw, aren't Unity's filters in native format as well? And if so, do they work in a webplayer build? (Haven't tested that yet), but if they *do* work then theoretically ours *could* too right? I mean if UT would allow it.

 

Both the 4.x pro filters and the "new" 5 filters work in web player (and on my device). I added flange and echo to the SDK demo project audio clip and can hear the results in web player while the custom demo plugins are inaudible.

 

Hmmm..

So unless we're able to use the SDK for as many platforms as we can (dunno about Xbox etc either) then that means pc/mac builds only. So while we might gain some lower level access (remains to be seen) we have to give up additional platforms, which limits us hugely if we're going to make filters for the asset store, which was my whole intent. So now my only choices are to *try* to go this route, or make .net dlls with my code and use the OnAudioFilterRead and all that that entails..

Ideally, what we *need* is a way to get lower level access to the dsp/callbacks which don't limit us at the same time. I've said it before but the same kind of access as I have in portaudio for example. Right down to being able to select my asio device without having to change my default os sound settings. A reliable callback with a constant size with as close to realtime as possible. the default latency on my asio device is ~7ms.

While I think the mixers and groups etc are a huge upgrade, I would've liked the OnAudioFilterRead function to be addressed or something new Like it. Shrug.

 

Yeah I was doing cart wheels when I first heard about this functionality but so far I'm pretty underwhelmed, especially given the lack of documentation. Custom native plugins must work on mobile, it'd be insane if that's not the case, but I'm at a loss as to how to make it happen.

I was so ready to say goodbye to OnAudioFIlterRead...

 

http://docs.unity3d.com/Manual/UpgradeGuide5-Plugins.html
Just stumbled across this, might provide clarity?

 

Just chiming in with my 2 cents:

- As I understand it, audio plugins can be developed for all platforms which support native plugins. That should include mobile, but not the web player for the very same security reasons that unsafe code isn't allowed to run in the web player. Yes Unity does it, no we can't because that would open up pandora's box( potential exploits by buffer overflow and the likes ).

- It is a pain to have to compile for every platform we wish to support, but that's the price to pay for lower level access.

- Processing callbacks happen on the audio thread. Even if no GUI is implemented, nothing forbids us from creating bindings to our filters / synths / granulators that can be accessed by scripts. As long as we're thread safety aware, and avoid locks on the audio thread( compare and swap atomic operations shoulld be preferred imo ), we should be fine.

- No comments regarding the AudioMixer, Group and FX APIs, there's so little to comment upon...

Cheers,

Gregzo

 

thanks jeff, greg

"32-bit native plugins will not work in the 64-bit editor. Attempting to load them will result in errors being logged to the console and exceptions being thrown when trying to call a function from the native plugin."

from this am i to infer that the 64 bit editor now creates 64 bit executables? or must we use a 64 bit plugin for the editor and a 32 bit for compilations? or (ugh) revert to a 32 bit editor? still a bit confusing..

"but not the web player for the very same security reasons that unsafe code isn't allowed to run in the web player. Yes Unity does it, no we can't because that would open up pandora's box( potential exploits by buffer overflow and the likes )."

agreed. however they could institute some sort of certification program if this is the way they want to go with plugins in general. i would have no problem doing that to potentially gain additional platforms shrug. btw greg, is the webplayer going bye bye permanently or just to the new gl version?

it'd be nice if Wayne could peruse this thread and throw a few answers our way, or mebbe where all this information is contained? i completely missed the upgrade part that jeff posted. it helps, but it also bring up more questions lol

 

amen

 

i was referring to the callbacks in the native sdk. sorry ;(

the callbacks there are called by whom? the audio engine first then scripts or....? i was hoping to move all of my 'script' dll code from .net back to c++ and use filters instead. still trying to figure out the best way to do it, or *if* i can do it at all, without limiting myself to just 2 platforms.

i mean, i agree, it all *sounds* great etc, especially the blurbs here:

excerpt:
"But what if you want more DSP control that just the inbuilt effects of Unity? Previously this was handled exclusively with the OnAudioFilterRead script callback, which allowed you to process audio samples directly in your scripts.

This is great for lightweight effects or prototyping your fancy filter ideas. Sometimes though, you want the ability to write native compiled effects for the best performance. Allowing you to write more heavy weight ideas, perhaps like your custom convolution reverb or multi band EQ.



Unity now also supports custom DSP plugin effects, with users having the ability to write their own native DSP for their game, or perhaps distributing their amazing effect ideas on the Asset Store for others to use. This opens up a whole world of possibilities, from writing your own synth engine to interfacing other audio applications like Pure Data. These custom DSP plugins can also request sidechain support and will be supplied sidechain data from anywhere else in the mix! Hawtness!"

LOL, i agree, it *is* hawtness! now *how* do we do it, and what do we have to give up *to* do it?

 

Hi Marionette,

Just tried building the X-Code demo project, targeting 64 bit architecture only. The bundle works fine in my editor.
Targetting iOS is just a matter of building a static lib instead.

I just realised one major issue: in the create callback, we have no reference to the AudioGroup or Mixer the filter is being added to, so it'll be hard to identify which instance of the filter we're tweaking from our own scripts. One really tacky workaround springs to mind, I hope it's not the only one: use a float ID that's set in the editor and can be queried via GetFloat API to identify filters. Hmmm...

 

yuck ;(

i mean, it sounds like it'll work, but we're back to the whole having to look something up that we should already get a reference to..

that's excellent about the x-code tho, jeff will like that too
btw, did you only run it in the editor or a compilation too with 64 bit?

 

btw, and this just occurs to me, but the end point of the audio filter chain is spatialized, right? onaudiofilterread wasn't, and thus i had to implement my own hybrid vbap/ambisonics thingy..

 

@Marionette

Tried an OS X x86_64 build which worked fine, one and one only bundle for both the editor and the build. It should be trivial to build a universal 32-64 bit OSX bundle.

What will take much more work imo: per platform optimisations, i.e. SIMD instructions. Instruction sets are very different depending on the platform, and not all have the excellent Accelerate framework to provide a clean wrapper.

About spatialisation: the native SDK's callbacks are post spatialization: 3d settings have no effect, nor does distance, doppler, or pitch shift.

This is where it gets interesting: OnAudioFilterRead's position in the processing chain has changed, and is now post distance attenuation, and can pre pitch shifting( in certain circumstances... ). All in all, quite the mess! If only we'd be given a simple chart detailing these callbacks' positions in the chain, it would be much, much simpler to work with.

Also, in my experience OAFR is somewhat unreliable in the editor: sometimes the method simply won't be registered and just won't fire at all...

 

Progress report:

It seems that the GetFloat / SetFloat API doesn't directly interact with the native plugin - all values are cached user side probably in managed land. This can be easily demonstrated by simply returning UNITY_AUDIODSP_OK in the SetFloat / GetFloat parameter callbacks native side: the GUI for parameters is still reactive, although of course the parameters aren't applied. Calling GetFloat confirms this.

On a slightly brighter note, I was able to hack around and grab a reference to a filter's data and write my own getters and setters which do interact directly with the native plugin. The strategy is super hacky, but incurs no performance penalty and just works. I'll post if anyone is interested.

The main issue imo is that the docs for the native audio SDK mention future developments of the API which will cover these problems. But I'm impatient, and want to play now, not later

 

Yes please

In exquisite detail

 

Patience, friend, patience... When the sun will set over the glorious alps, thou wilst have what thy heart covets.

 

Lol

I feel like the plant in little shop of horrors. *feed* me! Lol

Seriously though, I'm about to hit the keys myself and will post some of my discoveries as well

 

Ok, here we go, and may you all forgive the hacks!

General strategy:

- declare an ID parameter in the native filter
- piggy back the SetFloatParameterCallback to grab a ref to the EffectData struct and fire a callback to managed code including both the ref and the ID
- in the editor, expose the ID param of the target effect
- call SetFloatParameterCallback with the exposed ID param label and ID, and wait for the callback to fire with the results. As SetFloatParameterCallback is not directly exposed( calls are delayed until the audio thread kicks in, I guess ), the native to managed callback strategy is necessary.

Anyways, it's all implemented below.

First, we need a native manager to store pointers to the EffectData structs and provide bridges to managed land.

NakedFXManager.h, C header

Code (CSharp):

1. /*  Called from SetFloatParameterCallback if param index is P_ID
2.     returns false if id is taken, or if the registration callback
3.     hasn't been set.
4. */
5. bool RegisterNakedEffectData( void * fxData, const unsigned int ID );

Implementation, NakedFXManager.cpp:

Code (CSharp):

1. #include "NakedFXManager.h"
2. #include "AudioPluginUtil.h"
3.  
4. static void ** __fxInstances;
5. const int MAX_FX_INSTANCES = 128;
6. static void (*__registrationCallback)( int, void * );
7.  
8.  
9. bool RegisterNakedEffectData( void * fxData, const unsigned int id )
10. {
11.     // protect against setting ID manually
12.     if( __registrationCallback == NULL )
13.         return false;
14.  
15.     if( __fxInstances == NULL )
16.     {
17.         __fxInstances = ( void ** )malloc( sizeof( void * ) * MAX_FX_INSTANCES );
18.         memset( __fxInstances, 0, sizeof( void * ) * MAX_FX_INSTANCES );
19.     }
20.  
21.     if( __fxInstances[ id ] != NULL )
22.         return false;
23.  
24.     __fxInstances[ id ] = fxData;
25.  
26.     if( __registrationCallback != NULL )
27.         __registrationCallback( id, fxData );
28.  
29.     return true;
30. }
31.  
32. extern "C"
33. {
34.     // No checks for paramIndex out of bounds, handle with care!
35.     float __GetEffectFloatParamValue( const void * effectData, unsigned int paramIndex )
36.     {
37.         float * fxParams = ( float * )effectData;
38.         return fxParams[ paramIndex ];
39.     }
40.  
41.     // idem
42.     void __SetEffectFloatParamValue( const void * effectData, unsigned int paramIndex, float value )
43.     {
44.         float * fxParams = ( float * )effectData;
45.         fxParams[ paramIndex ] = value;
46.     }
47.  
48.     // Called by the managed NativeAudioHelper singleton
49.     void __RegisterManagedCallback( void ( * callback )( int fxID, void * fxData ) )
50.     {
51.         __registrationCallback = callback;
52.     }
53.  
54.     // idem, clean up when entering play mode is necessary.
55.     void __CleanRegisteredEffects()
56.     {
57.         if( __fxInstances != NULL )
58.         {
59.             memset( __fxInstances, 0, sizeof( void * ) * MAX_FX_INSTANCES );
60.         }
61.     }
62. }
63.  
64.  

Onwards to our filter, defined in the NakedFilter namespace:

Code (CSharp):

1. #include "AudioPluginUtil.h"
2. #include "NakedFXManager.h"
3.  
4. namespace NakedEffect
5. {
6.     enum Param
7.     {
8.         P_FREQ,
9.         P_MIX,
10.         P_ID, // Compulsory to grab reference to EffectData
11.         P_NUM
12.     };
13.  
14.     struct EffectData
15.     {
16.         struct Data
17.         {
18.             float p[P_NUM];
19.             float s;
20.             float c;
21.         };
22.         union
23.         {
24.             Data data;
25.             unsigned char pad[(sizeof(Data) + 15) & ~15]; // This entire structure must be a multiple of 16 bytes (and and instance 16 byte aligned) for PS3 SPU DMA requirements
26.         };
27.     };
28.  
29.     int InternalRegisterEffectDefinition(UnityAudioEffectDefinition& definition)
30.     {
31.         int numparams = P_NUM;
32.         definition.paramdefs = new UnityAudioParameterDefinition [numparams];
33.         RegisterParameter(definition, "Frequency", "Hz", 0.0f, kMaxSampleRate, 1000.0f, 1.0f, 3.0f, P_FREQ);
34.         RegisterParameter(definition, "Mix 2", "%", 0.0f, 1.0f, 0.5f, 100.0f, 1.0f, P_MIX);
35.      
36.         // Sadly, for now we need to expose the ID parameter here: it's the only way we can expose it
37.         // to the Unity editor and then expose it to script...
38.         RegisterParameter(definition, "Public ID", "", -1.0f, 256.0f, -1.0f, 1.0f, 1.0f, P_ID );
39.      
40.         return numparams;
41.     }
42.  
43.     UNITY_AUDIODSP_RESULT UNITY_AUDIODSP_CALLBACK CreateCallback(UnityAudioEffectState* state)
44.     {
45.         EffectData* effectdata = new EffectData;
46.         memset(effectdata, 0, sizeof(EffectData));
47.         effectdata->data.c = 1.0f;
48.      
49.         state->effectdata = effectdata;
50.      
51.         InitParametersFromDefinitions(InternalRegisterEffectDefinition, effectdata->data.p);
52.      
53.         return UNITY_AUDIODSP_OK;
54.     }
55.  
56.     UNITY_AUDIODSP_RESULT UNITY_AUDIODSP_CALLBACK ReleaseCallback(UnityAudioEffectState* state)
57.     {
58.         EffectData::Data* data = &state->GetEffectData<EffectData>()->data;
59.         delete data;
60.         return UNITY_AUDIODSP_OK;
61.     }
62.  
63.     UNITY_AUDIODSP_RESULT UNITY_AUDIODSP_CALLBACK SetFloatParameterCallback(UnityAudioEffectState* state, int index, float value)
64.     {
65.         EffectData::Data* data = &state->GetEffectData<EffectData>()->data;
66.         if(index >= P_NUM)
67.             return UNITY_AUDIODSP_ERR_UNSUPPORTED;
68.      
69.         // setting ID, time to cache the pointer to effect data
70.         if( index == P_ID && value >= 0.0f )
71.         {
72.             // if false( ie already registered or no registration callback to managed land ),
73.             // an error will be outputted to the Unity console
74.             if( RegisterNakedEffectData( data, ( unsigned int )value ) == false )
75.                 return UNITY_AUDIODSP_ERR_UNSUPPORTED;
76.         }
77.      
78.         data->p[index] = value;
79.         return UNITY_AUDIODSP_OK;
80.     }
81.  
82.     UNITY_AUDIODSP_RESULT UNITY_AUDIODSP_CALLBACK GetFloatParameterCallback(UnityAudioEffectState* state, int index, float* value, char *valuestr)
83.     {
84.      
85.         EffectData::Data* data = &state->GetEffectData<EffectData>()->data;
86.         if(index >= P_NUM)
87.             return UNITY_AUDIODSP_ERR_UNSUPPORTED;
88.         if(value != NULL)
89.         {
90.             *value = data->p[index];
91.         }
92.          
93.         if(valuestr != NULL)
94.             valuestr[0] = 0;
95.         return UNITY_AUDIODSP_OK;
96.     }
97.  
98.     int UNITY_AUDIODSP_CALLBACK GetFloatBufferCallback (UnityAudioEffectState* state, const char* name, float* buffer, int numsamples)
99.     {
100.         return UNITY_AUDIODSP_OK;
101.     }
102.  
103.     UNITY_AUDIODSP_RESULT UNITY_AUDIODSP_CALLBACK ProcessCallback(UnityAudioEffectState* state, float* inbuffer, float* outbuffer, unsigned int length, int inchannels, int outchannels)
104.     {
105.         EffectData::Data* data = &state->GetEffectData<EffectData>()->data;
106.  
107.  
108.         float w = 2.0f * sinf(kPI * data->p[P_FREQ] / state->samplerate);
109.         for(unsigned int n = 0; n < length; n++)
110.         {
111.             for(int i = 0; i < outchannels; i++)
112.             {
113.                 outbuffer[n * outchannels + i] = inbuffer[n * outchannels + i] * (1.0f - data->p[P_MIX] + data->p[P_MIX] * data->s);
114.             }
115.             data->s += data->c * w; // cheap way to calculate a steady sine-wave
116.             data->c -= data->s * w;
117.         }
118.  
119.         return UNITY_AUDIODSP_OK;
120.     }
121. }

You'll notice that it's the RingModulator filter, minus the PS3 specific junk, and with an extra P_ID param and a call to RegisterNakedEffectData in the SetFloat callback.

Don't forget to adjust PluginList.h accordingly:

Code (CSharp):

1. DECLARE_EFFECT("Naked Effect", NakedEffect )

Now, the managed code: NativeAudioHelper.cs, a Singleton class

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using System.Collections.Generic;
4. using System;
5. using UnityEngine.Audio;
6. using System.Runtime.InteropServices;
7.  
8. public class NativeAudioHelper
9. {
10.     public delegate void OnEffectDataGrabbed( IntPtr fxDataRef, int fxID );
11.  
12.     public int GetEffectDataRef( string idParamExposedName, AudioMixer mixer, OnEffectDataGrabbed callback )
13.     {
14.         int id = _nextValidID;
15.      
16.         _callbacks.Add( id, callback );
17.      
18.         _nextValidID++;
19.      
20.         mixer.SetFloat( idParamExposedName, id );
21.      
22.         return id;
23.     }
24.  
25.     public static NativeAudioHelper SharedInstance
26.     {
27.         get
28.         {
29.             if( __sharedInstance == null )
30.                 __sharedInstance = new NativeAudioHelper();
31.          
32.             return __sharedInstance;
33.         }
34.     }
35.  
36.     #region Native Bridge Public Interface
37.  
38.     #if UNITY_IPHONE
39.     const string PLUGIN_NAME = "__Internal";
40.     #else
41.     const string PLUGIN_NAME = "AudioPluginNaked";
42.     #endif
43.  
44.     [DllImport (PLUGIN_NAME)]
45.     public static extern float __GetEffectFloatParamValue ( IntPtr fxRef, int paramIndex );
46.  
47.     [DllImport (PLUGIN_NAME)]
48.     public static extern void __SetEffectFloatParamValue ( IntPtr fxRef, int paramIndex, float value );
49.  
50.     #endregion
51.  
52.     #region Private
53.     static NativeAudioHelper __sharedInstance;
54.  
55.     // internal delegate: we'll route all callbacks delivering grabbed fxData refs to
56.     // a static method( OnRefGrabbed ) to maintain compatibility with AOT platforms
57.     delegate void InternalGetRefCallback( int fxID, IntPtr fxDataRef );
58.  
59.     int _nextValidID;
60.  
61.     // dictionnary of delegates to notify when the ref has been grabbed
62.     private Dictionary< int, OnEffectDataGrabbed > _callbacks = new Dictionary<int, OnEffectDataGrabbed>();
63.  
64.     private NativeAudioHelper()
65.     {
66.         // Important in the editor, where native static arrays won't be cleaned up when entering / exiting play mode
67.         __CleanRegisteredEffects();
68.      
69.         // Must do to grab fx refs when setting ID!
70.         __RegisterManagedCallback( OnRefGrabbed );
71.     }
72.  
73.     // the static callback method we use to route and dispatch grabbed fxData refs
74.     #if UNITY_IPHONE
75.     [ AOT.MonoPInvokeCallback( typeof( GetRefCallback ) ) ]
76.     #endif
77.     static void OnRefGrabbed( int fxID, IntPtr fxRef )
78.     {
79.         OnEffectDataGrabbed callback = __sharedInstance._callbacks[ fxID ];
80.         __sharedInstance._callbacks.Remove( fxID );
81.         callback( fxRef, fxID );
82.     }
83.  
84.     [DllImport (PLUGIN_NAME)]
85.     private static extern void __RegisterManagedCallback( InternalGetRefCallback callback );
86.  
87.     [DllImport (PLUGIN_NAME)]
88.     private static extern void __CleanRegisteredEffects ();
89.  
90.     #endregion
91.  
92. }
93.  

Usage is reasonably simple:
1)In the mixer editor, expose your NakedFilter's ID parameter and name it with a unique, per instance name.
2) From any Monobehaviour's Start method( not Awake, god knows why... ), register your filter like this:

Code (CSharp):

1.  
2.  
3. public AudioMixer audioMixer; //assign in inspector
4. public string idParamExposedName; // the name of the exposed ID param
5.  
6. void Start() //for some reason, doesn't work in awake...
7. {
8.         NativeAudioHelper.SharedInstance.GetEffectDataRef( idParamExposedName, audioMixer, OnFXDataRefGrabbed );
9. }
10.  
11. void OnFXDataRefGrabbed( IntPtr fxRef, int id )
12. {
13.         Debug.Log( "Gotcha!" );
14.         // do stuff with the ref: associate with AudioMixerGroup, cache for further use, etc...
15. }

Once you've grabbed a nice IntPtr to your filter's EffectData struct, you can get / set parameters following your own rules, not Unity's:

Code (CSharp):

1. NativeAudioHelper.__SetEffectFloatParamValue( _effectRef, paramIndex, value );

Now, this seems like not much, but for my use case( scheduling audio data to be mixed in my own mixer ), I can use this ref to pass much more than floats.

Amen, and may the gods forgive my impatience!

Gregzo

 

you rock

I'll check it out when I get back from the store

 

running into all kinds of frustrating issues..

there seems to be a disconnect between edit mode/run mode and the filter GUI stuff. so while in the editor if i change a param in edit mode, even with their filters, the visual doesn't update or change until it's run.. so for example if i wanted to update the inspector i could invalidate it etc to force a refresh.. doesn't look like i can do anything like that in the filter GUI stuff, which becomes problematic.. you can see this behavior if you look at the demo convolution reverb filter and adjust the time or reverse params.. it doesn't update the GUI.. now do the same thing in run mode and adjust those params.. see how the wave display updates now? i can't find a way to update it in edit mode, which is where i need it ;(

additionally, why isn't there a clean way to iterate thru a mixer's filters?? or get back to the audio source's clip data from a filter? there should be a parent or top level reference from the filter itself like you talked about.. seems like the only way i'm going to be able to get this to work properly (for my specific needs) is to do exactly what you've outlined above. but that only covers a few issues, not to mention, what happens when they change or update the api? probably means constant updates every patch ;(

mebbe i'm making this more difficult than it has to be. all i want to do in this first step is to send the complete clip data buffer to my filter in edit mode, to be able to draw an associated wave in a GUI to be used as an editor. similar to the correlation meter GUI or convolution reverb GUI etc

does the native set position callback only work within the constraints of the buffer passed in? and is this callback fired when i set position on the clip via what? source.time?

 

GUI:
Yeah, it seems that the GUI is completely reliant on cached data. When you rebuild a filter's bundle, even if Unity is not open when you replace the bundle, the GUI for filters won't update...

API( or lack there of):
Now you understand the irresistible need for the hack.

Frustration:
Yes, here too, lots! Never crashed Unity as much, but that's to be expected with native code that's not unit tested... ( talking about my code, not the audio SDKs )

Native set pos callback:
I'd say not implemented yet? Nothing in the docs...

My little complaint: AudioEffectState->currdsptick doesn't seem to be synchronised with AudioSettings.dspTime, so more hacks in order to get the delta between the 2.

I had posted an (angryish) rant about the lack of feedback request regarding the new audio features during beta testing, but no one replied / backed it up. I guess we get what we ( don't ) ask for...

 

Progress report:

Did I just invent beep driven development?

 

More seriously, starting to get the hang of this.
Piggybacking Unity's spatialisation happily, got a granulator working( spatializedy, youhoo! ).

Workflow tips for quick iteration of native audio plugins builds:
0) Have 2 finder windows open, one pointing to your Unity row's asset folder, the other to your plugin's bundle
1) Close Unity
2) Build bundle in Xcode
3) Delete old bundle from Unity's asset folder, including .meta file
4) Copy new bundle to Unity's asset folder
5) Relaunch Unity, should be good to go. One caveat: although your new bundle will run, the gui for your fx won't be updated. Seems the fx has to be removed and Unity restarted for the GUI to register params anew.

Other tip: the processing callback's outbuffer isn't cleaned between calls( which makes sense if you need a bit of history... ). memset early in the callback when testing to avoid nasty surprises...

 

Other tip:

If you don't plan on supporting PS3, the EffectData struct becomes MUCH more readable - and so does all code which needs to access it:

Code (CSharp):

1. struct EffectData
2. {
3.         float p[P_NUM];
4.         void * otherData;
5. };
6.  
7. UNITY_AUDIODSP_RESULT UNITY_AUDIODSP_CALLBACK ProcessCallback(UnityAudioEffectState* state, float* inbuffer, float* outbuffer, unsigned int length, int inchannels, int outchannels)
8. {
9.       #ifdef SAFE_OUT_BUFFER
10.       memset( outbuffer, 0, length * outchannels * sizeof( float ) );
11.       #endif
12.       EffectData * effectData = state->GetEffectData<EffectData>();
13.       float param0 = effectData->p[0]; //exposed float params
14.       void * otherData = effectData->otherData;
15. }
16.  

 

Just checking in on this thread to see if anyone has been able to deploy their native audio plugins to mobile platforms (interested in iOS specifically). I'm trying to figure out the steps required to add the plugin registration to the startup code of the Xcode project as mentioned on in the documentation:

Looking in UnityAppController.mm, it seems the place to do the plugin registration is in -(void)preStartUnity, I'm assuming using UnityRegisterAudioPlugin() but it's not clear how to use this function with my plugin's static library.

Has anyone successfully deployed a native audio plugin to a mobile platform yet?

 

Just wanted to update everyone in this thread looking for instructions on how to get native audio plugins working on iOS. I've outlined some steps which work for me in this thread.

 

How to identify data is of which audio source in process callback in native audio plugin sdk. Its only float* inBuffer which contains the audio data of all audio sources. How to identify them separately. For now , i have 2 sources which attached audio clips and after process callback ends, both audio clips starts playing on both sources. How to identify them individually. Please Provide an example. i want something like audio source id attached to input buffer to identify which audio to process in my plugin (same as of unreal provides)

 

Hi Gregzo,
Iknow this is an old thread, but great stuff!!

I'm trying to implement it in 2017.3 but I get :

EntryPointNotFoundException: __CleanRegisteredEffects
NativeAudioHelper..ctor () (at Assets/NativeAudioHelper.cs:69)
NativeAudioHelper.get_SharedInstance () (at Assets/NativeAudioHelper.cs:30)
nakedeffectTest.Start () (at Assets/nakedeffectTest.cs:17)

I'm not that familiar with c++ but I was able to build a new audioplugindemo.dll but it seems I need to build a separate audiopluginnaked.dll, but I don't know how or with which files.

Do you have a demo project with this stuff working?

regards,
Marald

 

Using :

Code (CSharp):

1.  
2. extern "C" UNITY_AUDIODSP_EXPORT_API float __GetEffectFloatParamValue( const void * effectData, unsigned int paramIndex )
3.     {
4.         float * fxParams = ( float * )effectData;
5.         return fxParams[ paramIndex ];
6.     }
7.  
8.     // idem
9.  extern "C" UNITY_AUDIODSP_EXPORT_API   void __SetEffectFloatParamValue( const void * effectData, unsigned int paramIndex, float value )
10.     {
11.         float * fxParams = ( float * )effectData;
12.         fxParams[ paramIndex ] = value;
13.     }
14.  
15.     // Called by the managed NativeAudioHelper singleton
16.  extern "C" UNITY_AUDIODSP_EXPORT_API   void __RegisterManagedCallback( void ( * callback )( int fxID, void * fxData ) )
17.     {
18.         __registrationCallback = callback;
19.     }
20.  
21.     // idem, clean up when entering play mode is necessary.
22. extern "C" UNITY_AUDIODSP_EXPORT_API    void __CleanRegisteredEffects()
23.     {
24.         if( __fxInstances != NULL )
25.         {
26.             memset( __fxInstances, 0, sizeof( void * ) * MAX_FX_INSTANCES );
27.         }
28.     }
29.  

in the nakedFXmanager seems to do the trick. But although it works now, it chrashes as soon as I stop play mode.