How to make a virtual musical instrument with Unity and Leap Motion sensor?

I would like to experiment with creating virtual musical instruments, and I'd like some suggestions about where to start learning Unity. The idea I have is to use the Leap Motion sensor data to pluck "strings" that are visible on the screen. The strings would trigger the appropriate sounds depending on which string is plucked. Basically, an air harp (instead of an air guitar) that actually makes sound.

I have already looked at the micro games tutorials, but they don't seem to be helpful for what I am trying to do. I have a lot of experience with coding, but I know very little about Unity.

So far, I have been able to get the Leap Motion sensor demo working so that I can see a pair of hands on the screen when I wave my hands over the Leap Motion sensor.

Next I need to learn how to get the data from the sensors, or figure out how to create objects that behave like strings and send a message when "plucked." I am planning on using OSC to send the pluck data to another program that will create the sound (SuperCollider).


What tutorials or concepts should I take a look at to get myself started? What kind of project? What modules are needed?

 

...

To figure out how to pluck strings, you'll need to read leapmotion documentaiton, pretty much and there's a chance that it won't be possible.

Musical Synth should be doable in unity, but you might run into sound latency problem due to the way sound system works.

https://docs.unity3d.com/ScriptReference/AudioSettings-dspTime.html
https://forum.unity.com/threads/do-...e-is-can-someone-explain-how-it-works.402308/
https://designingsound.org/2016/09/14/making-a-music-system-part-3/
https://docs.unity3d.com/ScriptReference/AudioSource.PlayScheduled.html

Please note that a harp is the kind of instrument where you REALLY want to see the strings and have tacticle feedback. With invisible strings it would be easier to make a virtual theremin or something simlar.

 

Whilst I agree with you for a real harp, if, as I suspect, the inspiration is actually the laser harp a la JMJ, then no, not necessarily.

 

Laser harp has visible "strings" i.e. beams.

Spoiler

 

Yes, Im fully aware of that, but it wasnt the 'visibiliity' part of your sentence I was addressing. I did highlight 'and tactile feedback' when I replied, because that's what I was responding to. It looks like the forum eats formatting in quotes, though.

 

Yep, pretty much did this a couple of years ago to make a virtual theremin with the Leap! Just use the Leap Motion examples, attach colliders to the fingers and put your strings on screen with their own colliders.

As has been mentioned, the lack of tactile feedback makes a theremin extremely difficult to play. My Leap theremin worked just like a real one - it was really difficult to play!

I hooked it up to G-Audio at the time, and I know there are good C# synth's for Unity on Git that you could also hook up to. The latency isn't a problem for harp-type instruments with a lot of resonance. I don't think I would attempt to build something that requires perfect synchronization.

 

Maybe that wasn't the best way to phrase it, but the general idea is that you need a way to find the strings.

On real musical instruments, depending on the nature of instrument you usually use tacticle feedback and muscle memory, with ocasional visual cues. Basically, a training musician can play piano blindfolded once they found the middle C or something they can use as a reference point to align their hands to the instrument.

I have no experience with a harp, but in case of harp you can't rest hands on strings, meaning importance of visual information would be higher. In comparison, in case of instruments from zither family ( https://en.wikipedia.org/wiki/Zither )

Spoiler

it should be possible to play many of those blindfolded, as long as you can rest the hand on the body of the instrument, as it creates an "anchor point" for your muscle memory, like shape keys on the piano.

In case of harp, due to the nature of the instrument, you'll likely need both visual and tacticle cues and importance of visual cues would be higher.

In case of Laser Harp, tacticle cues are gone, and the player is relying on visual cues only, which is why there is a small laser of beams (it is also using diatonic scale with about five or seven notes total).

And in case of leap motion controllers there aren't any visual or tactcile cues, which will make the "harp" incredibly difficult to play.

Theremin should be fairly easily playable by a person with absolute pitch or very highly trained relative pitch. They'll miss the advanced instrument techniques, but should be able to play basic tunes in matter of minutes.

Basically, the idea would be to assume the same position relative to the instrument each time, then you'll be able to benefit from muscle memory to the degree. Overall, it shoudl be somewhat similar to playing a musical saw.
https://en.wikipedia.org/wiki/Musical_saw

Spoiler

Meaning after some practice you'll know approximate "position" for every sound, and then you correct/adjust the actual sound based on your hearing once you start playing.

Without absolute pitch or highly training musical training, this one would be very difficult to play.

 

@neginfinity Have you actually played a Theremin or talked to anyone who has about the process?

 

Says who? The OP said up front there will be a visual representation onscreen. You're basically saying that this visual representation wouldnt be sufficient, right? Despite not knowing what that might be.

There are no tactile cues as to what a mouse is actually causing onscreen, but people manage all the same. Why? Because they can associate their unseen hand movements, onto the displaced and abstracted visual representation of its effect. Same with a graphics tablet.

Are you saying that in the specific case of musical instruments, we lose this ability to compensate for the visual separation between a controller versus its onscreen representation? Or that we cant ever present the onscreen representation of a musical instrument? Or what?

BTW, a lot of 'classical' thereminists started off as string players, and that helps because they're fretless instruments. But plenty of people learn the violin etc without prior musical training. Also, perfect pitch or whatever is only part of the picture; there's no parallel to the volume control of a theremin, in any other instrument.

My own theremin is reserved for atonal 50s scifi swoops, though; Im tonedeaf in my right hand.

 

No, but I happen to be a professionally trained musician with an absolute pitch (or its functional equivalent). Likely an "ex-musician" at this point....

Like with majority of other tonal instruments, I'd be able to awkwardly play basic tunes on it in matter of minutes.

And I haven't noticed that part. With onscreen strings It would be awkward, but playable. Barely.

 

True, but this person also has to remain absolutely still, almost like a robot to play it well. The instrument works through radio interference and the water in our bodies provide the interference. The slightest movement of virtually any part of your body affects the sound. So technically, I guess a leap theremin is slightly easier to play.

 

I had to triple-check it to be sure, but it does not act on principle of radio interference, and instead the player acts as a capacitator. So it is not as bad as whole body affecting it, as the nearest portion of the body will have the most impact.

 

I am happy to see that I inspired a lively discussion! I actually was able to make a theremin-like instrument with the Leap Motion sensor before. I used Open Sound Control and a sound program called SuperCollider, with which I did not have any latency problems that I noticed. Because I was using the LM API only in Python, there was no visual feedback. Mostly I am concerned with figuring out how to model something that resembles a harp. It could be anything really - steel drums, whatever. I'd like to experiment with using hand movements to trigger and control sound. All of this was inspired by the theremin. It's the best of both worlds -- the endless possibilities of digital sound coupled with the organic movement of the human body.

I would like to use Unity because it seems that you can use Orion, which, when I tried the demo, seemed a lot smoother and reliable. Also, Unity seems to be really good for creating a 3-D world.

This laser harp you all described seems like a good start. How would I go about creating something like this? Would I poll the values coming from Leap Motion and detect when the hand goes past a certain coordinate, or is there a better way with Unity?

 

This sounds promising. Can you recommend tutorials or documentation that would help me with colliders and putting strings on the screen? How would I go about putting strings on the screen?