Question Musical instrument in unity

Hi everyone,
I am trying to create a new kind of music instrument in VR. One that is easy to learn and teaches you some musical theory. But now I have a huge issue with the most basic feature. I don’t know how to code a good audio output for a musical instrument in unity.


I found some basic concepts

1. I could use a tool in unity, which uses Midi and Soundfonts to get started. I have read the reviews and it is pointed out there that it is not accurate in it’s output, because it is not framerate independent. Becaus I am decveloping for a mobile device (Oculus Quest, so on android), I am afraid, that this will be an issue.

2. I could use an audio middleware like wwise, juce or fmod. On my research I haven’t found the implementation of a synthesizer in the VST or Soundfond format using those tools either though.

Did anyone have the same problem? Do you have any recommendations which path should I try?
Any help would be greatly appreciated.




#latency #android #vst #audiomiddleware #VR

 

What kind of interactions does the user perform in VR to "play" your instrument? Are you attempting to model an existing physical instrument? Expressive virtual instrument designs are possible with middleware like Wwise, in my experience using combinations of MIDI, audio samples, and the various Wwise game syncs.

 

The Programm basically looks for collisions with gameobjects and returns a value of a note that should be played, at which volume and witht the possibility to add effects.
It is not representing a physical instrument, but I guess there is not much difference in coding it.
So basically I would need to trigger a midi event with the collision? Can I do everytihing with midi, that I described above?

 

And 2nd: Does the game sync allow me to have more controll over the instrument than MIDI does? Or in other words, what is the value of having an audio middleware in that case vs. having a script like C# Synth?

 

I'll try again since my previous post was removed for spam :/

1. The most basic thing you can do to create a synth is to implement the OnAudioFilterRead method of an attached AudioSource.
This method is executed automatically (on a different thread than the main one), and each time it runs it gives you 1024 (not sure if this value can change) samples per channel to do something with them. You have to set the value of each sample to create sound.
The AudioSource will output the audio result of your code.
Notice that if you have an Audioclip playing in that AudioSource, the samples you get reflect its content. So you can also modify them, to apply some kind of effect.
But you'll have to know how dsp works, and also creating a simple oscillator is not that easy if you have no experience with dsp.

2. Use an existing audio library which has been ported/interfaced to Unity: good examples of this are Pure Data and Csound: they both have good communities, so you will find lots of examples; they are also very mature (30+ years of active development), so they are very stable and with lots of features. They are written in C, so they are also pretty fast.
Other good examples are Faust, Supercollider and Chuck, but I don't know if there is some interface/package/library that wraps them to be easily used in Unity.

3. The Asset Store is full of packages that handle audio syntesis. One example (free I think) is Audio Helm, but you can only use the features of their synth to create sound.

No links to avoid being accused of spamming
But happy to help if you need more info!

 

Sorry for the late reply! I'm not exactly a development whiz, but I just had to chime in because your project sounds seriously awesome! While I might not be coding-savvy, I'm a total music enthusiast and I'm always up for learning new instruments. The idea of a VR instrument that's both educational and musical has me super intrigued. Speaking of instruments, I've recently taken on the challenge of learning the concertina accordion. Let me tell you, it's been quite the journey – not as easy as I thought it would be! But you know what's really special? Holding an instrument, feeling its weight, and producing live music. It's an experience that no amount of modern tech can replicate entirely.