UnitySynth - full Xplatform midi synth

Hi,
I posted recently about this very cool C#Synth project at http://csharpsynthproject.codeplex.com/

Project Description
A synthesizer made in C# that will allow midi functionality using only the compact framework for any .net device: (windows pc, xbox, or phone).

Find attached a port to Unity + simple sample scene. Read the comments for a guide.
The C# Express 2010 .net framework 3.5 project is inside the plugins folder, so you may want to move it out before rebuilding and just point the build directory to the Unity project plugins folder. I left a number of Unity debug statements turned on so it runs slightly slower but if you check the console you can see what is going on on load/play.

Make sure the project you import the package into a full .net 2.0 under the build settings, not the subset or you will get some errors.

Edit: Q: Does this work on Unity Free? A: I don't know. It should, as it's a C# plugin not native, but you have all you need to test it for yourself. Please report your findings below for others. ty!

The porting process was quite interesting and my notes are below for those who would like to do it themselves as a useful learning exercise in creating an external C# dll. If you are interested in contributing please contact the developer or just post anything in here. Have fun!

To begin download the latest non-Unity version from http://csharpsynthproject.codeplex.com/
(the version in the attached package does NOT include his demo files or instrument builder)

1. Open the C#Synth project in C# Express 2010
2. Right click, select properties, change to Target Framework .Net Framework 3.5
   You can also set the build directory to your Unity plugins folder here.
3. Close/Reopen project - Check references for yellow flags and remove if necessary

1. Add UnityEngine as a reference. You will need to add using UnityEngine to all files you modify with Unity specific code.
2. Immediately see 2 errors on BW.Dispose in WaveFileReader/Writer - comment them out for now. BW.Close() ok.
   See http://sshnet.codeplex.com/discussions/269633 for discussion on this.

1. Tag all changes with //UnitySynth so we can do a global find at any time to track changes we made

Next step is analyse the example demo project WindowsDirectSoundTest classes Form1.cs and SynthThread.cs to see how it works and emulate the same behaviour in Unity.

1. Find all System.Diagnostics.Debug.WriteLine replace with Debug.Log
2. Basically we have two things to load. A soundbank, which can be dsp or sample based, and a midi file. So we
   need to locate the asset loading code in the project and replace it with Unity specific asset loading code, also keeping in mind the correct extensions needed for the text assets to load correctly
   http://unity3d.com/support/documentation/Components/class-TextAsset.html
3. Firstly we need to allocate a StreamSynthesizer.cs. The default here is to calculate the samplebuffer size based on
   a parameter bufferSizeInMilliseconds but lets just change this to a simple int parameter of buffersize to be consistent with the unity audio allocation. Keeps it simple.
4. Next we need to convert the sample byte[] methods to float[] to match the Unity methods
   See public void GetNext(byte[] buffer) and private void ConvertBuffer(float[,] from, byte[] to) for the float equivalents.

1. Next, lets look at the loading of the sample definition files and sample loading.
   First up is InstrumentBank.cs
   public void loadBank(string bankfile) needs to load a stream from Assets, as does public void loadBank(byte[] Programs, byte[] DrumPrograms)

1. Next is SFZInstrument.cs where we need to convert the constructor to load a UnityAsset
2. We also need to check the Sample.cs and WaveFileReader.cs loading, convert to Unity asset loading format
3. Finally MidiFile constructor needs to load the file as a UnityAsset
4. Additionally a small file reader modification to FMInstrument.cs was made if you want to test the fm instrument

The core Unity function to get the audio to play is by using the OnAudioFilterRead method. See the C#Synth Demo project SynthThread.fillerThread() method for the equivalent code. By routing the buffer into OnAudioFilterRead you can do procedural audio and synthesis on the fly. See http://unity3d.com/support/documentation/ScriptReference/MonoBehaviour.OnAudioFilterRead.html

Now go make some noisy apps!

 

I got basic Midifile reading working from the original CSharpsynth project but your port is going to be such a time-saver. Thanks so much for this. I'm downloading now and checking it out.

much appreciated!

 

No problem! I've learn't so much from the Unity community it's about time I put something back.
re: midi file reading suggest you also check out the NAudio implementation. It's very good, lots of helper classes, and easy enough to integrate into unity.

 

I might be a bit confused here. Does your sample scene support iOS? it plays fine in the editor in Mac but fails when I try to build to iPad - actually the build succeeds but when it goes to run the app crashes Xcode with an EXC_BAD_ACCESS.

I'm just running the scene as is after moving the csharpsynth folder out of the plugins as you suggested (it wont build without doing that).

What I'm trying to achieve is for me to put an mp3 (of a midi file, but very nicely produced with nice instruments) in the scene and play it back while simultaneously playing back its midi file at the same time (but silently) and animating its midi note_ons and note_offs in unity code. So in theory I'd like to eventually use an IK multimedia iRig MIDI device and turn this into a music tutor. But to do that I need to be able to play back actual audio in time with the midi - so I can sync animation to the instruments and eventually MIDI input via that device.

Does this .NET playback of audio not work on iOS? Also, is NAudio another C# library? what is the difference between this and that?

I've bought MIDI Unified from the asset store but realized it requires Unity Pro which I'm not getting for a few months...

Any ideas are appreciated

 

I have not tried it on iOS as yet, so there may be some settings to change - check the .net version (full, not sub) and things like Fast vs Slow. I have Unity Pro, so cannot test free. Afaik it should work on desktop , web, iOS and any platform it can compile to.

I can't really offer you support beyond that, it's open source and I just did this as a one off - I have to get back to my project now.
Have a play and report back. You have all the source!

I'm actually doing a music instrument tutor app at the moment which does more or less what you say. Unfortunately it's not for public release as it's in development, tho I may release it after the app is done. Will see.

re: NAudio google is your friend. It's also on codeplex, and C#Synth uses some of if I remember.

 

Thanks. I wasn't trying to get you to do my app for me or anything ;-)

I was mostly wondering about the build settings. and if .NET works differently on iOS if you build to different iOS version support? Also I have paid for copies of iOS and Android basic not the free version if that matters. Its the midiStreamSynthesizer that is failing. I'll have a play and see what parts work and what parts fail in iOS. Appreciate the help tho.

Good luck with your project - I had a look and will get ChordSkilz as a thank you - it looks interesting and was wondering if it would help teach chord progressions.

 

Hi,
is it possible to do something like a little piano in game and tied midi notes to a button press ? Like when the user presses the piano a note will play, I'm still new at coding and i'm a little lost on how this all works-regardless , thanks sonic for your contribution !!

 

I'll try and run it on iPad2 later, my 2 year old is in my lap watching youtube on my MBP at the moment so I interrupt upon pain of ear death.

I have Unity Pro and iOS basic (paid). iOS build settings can be finicky, but it seems related to the .net full/sub and fast/slow from my experience with other iOS Exec_Bad_access crashes after deployment to the iPad,

 

Sure, that's essentially what the demo keyboard is doing. The A to K keys mimic a piano octave. All you need to do is build a 3d model, fire collision events from pressing the keys (via mouse/finger), animate the key and more or less use the same code in the demo to fire the noteon.
Might want to do a sustain if you keep pressed down and fire the noteoff on release, or use a coroutine timer to fire the corresponding noteoff, but up to you how the interaction will work.

 

I got it working, You will be in the credits of my game as a special thanks Sir !

For some reason it didn't import the scene the first time, second time is the charm though !

 

OK, it looks like it's throwing a JIT wobbly on an aot method only, must be something iOS related as I got it on all the settings.
Let me see...

EDIT: Fixed tested on iPad. BPM was slow, but maybe that was all the debugs being printed to console. You might want to take them out for iOS testing.
New package with fix uploaded to the first post.

Problem was:
//UnitySynth
//public struct NoteRegistryKey : IEquatable<NoteRegistryKey>
public struct NoteRegistryKey

 

Hi psDev- I got the code to run on Android .

Here's some quick and dirty GUI code to test it with a touch screen
Put this inside of the Unity Synth Test code , inside of the on GUI function

Code (csharp):

1.  
2.  
3.  
4.         if (GUI.Button(new Rect(100, 70, 100, 300), "A Note"))
5.         {
6.            
7.             midiStreamSynthesizer.NoteOn (1, midiNote, midiNoteVolume, midiInstrument);
8.             //yield return new WaitForSeconds(1);
9.             midiStreamSynthesizer.NoteOff (1, midiNote);
10.             //aOFF == 1 ;
11.             //midiSequencer.Play ();
12.         }
13.        
14.         if (GUI.Button(new Rect(230, 70, 100, 300), "W Note"))
15.         {
16.            
17.             midiStreamSynthesizer.NoteOn (1, midiNote + 1, midiNoteVolume, midiInstrument);
18.             //yield return new WaitForSeconds(1);
19.             midiStreamSynthesizer.NoteOff (1, midiNote + 1);
20.             //aOFF == 1 ;
21.             //midiSequencer.Play ();
22.         }
23.        
24.         if (GUI.Button(new Rect(360, 30, 100, 150), "S Note"))
25.         {
26.            
27.             midiStreamSynthesizer.NoteOn (1, midiNote + 2, midiNoteVolume, midiInstrument);
28.             //yield return new WaitForSeconds(1);
29.             midiStreamSynthesizer.NoteOff (1, midiNote + 2);
30.             //aOFF == 1 ;
31.             //midiSequencer.Play ();
32.         }
33.  
34.  

Unity isn't letting me yield time inside of the GUI function, so the notes only play for half a sec, I don't have a mac to build to iphone- but it works fine, with a drop in audio quality on my andriod phone .

 

Awesome, I'll try it right away. thanks!

Btw chordskilz is telling me I don't have as perfect pitch as I thought ;-)

 

thanks keithsoulasa, I'll check it out as soon as I can I don't have my android devices with me, but thanks for letting me know.

 

That would be the inversions doing your head in;-)
I made it a little hard in that regard, I'll do an update with an non-inversion option when I get this next app out.

Thanks for trying it out!

 

Hiya,

Firstly, thanks for the excellent share! That is pretty damn generous, and has inspired me to share some of my own code too. I guess it's time to give back.

Does this library actually support midi events such as note in/out, duration and so forth so we can either send midi data or receive midi data in realtime? I don't think it does, but I thought I would ask!

 

Not really, it's more one way. You can create midi events on the fly and send them to the synth, tho the current set of events is small.
Still it's open source, start a hackin! That said, you can parse midi files and with some extra coding all sorts of things are possible.

As mentioned previously, http://naudio.codeplex.com/ has a very good midi implementation so worth looking at that imo. The audio output of NAudio is pretty windows specific.

I only did the port to Unity of C#Synth btw. The C#Synth developer is [FONT="]Alex Veltsistas , you can contact him on the website [/FONT][URL]http://csharpsynthproject.codeplex.com/[/URL] so please drop by it and say thanks if you find it useful. It's still early days and needs more development. Still, it's a good start! Happy Hacking![URL="http://csharpsynthproject.codeplex.com/"][/URL]

 

Is their anyway to record the midi notes and play them back on something like a 16 step, or a 32 step pattern .
Check my game in my sigh( forgive the placeholder art ...) to get an idea of what I mean.

I could do it via physics, like I do now , but I'd prefer to program this some how and trigger it via code .

I'm finally taking steps into C#, 99% of my game is Javascript- but this midi is worth switching to C# over

 

I'm only drawing some GUI buttons, so it should still work on Iphone
Is their anyway to yield time inside of the GUI command, i'd like to be able to adjust the time the notes play

 

Sure, though you will have to write it custom. There's free/low cost projects on the asset store I think that will give you a head start, but you will have to hack them. Best way to learn though!

 

OK- this will take some time considering most of my games in Java and ill need to re-write code, but I have alot of free time so why not .

 

Nice work on this, sonicviz - I got it working easily, and the potential is there.

 

I was able to get the sample project running in Android but the audio track is not playing correctly on my Nexus One. It's all very loud and harsh crackles and pops. When I play it on PC the audio track works perfectly.

I have very limited knowledge of Audio DSP, but my initial thought is the WAV bytes are not being read correctly? How do I check this? Is it possible that the Endian of the Android HW is causing this to happen?

 

Does it work with mp3?

 

Much thanks for this , I'm toying with the code for a mini game collection of a few music based games .
For FREE( although i'm going to go broke if i want to get it on IOS )

 

Ok, so if i was a noob and i wanted to just DL the package and run it on a new unity project... could you give a walkthrough because it's throwing up errors.

-Make new unity project without any assets like trees, materials etc.

-run your file in new project.

-press play.

This happens: (see attached debug image)

 

Check your project's .Net Api level
Class: InvalidDataException (namespace System.IO, assembly System)
Search MSDN .NET 2.0 2.0 subset Web player Micro http://docs.unity3d.com/Documentation/ScriptReference/MonoCompatibility.html

 

First off, big thanks for this! I got the unity package working with no problems.

My question is, how can I do a simple pitch bend on a per note basis? I've found the SetPitchWheelRange and setPitchBend functions, but I'm not sure how to use them.

EDIT:

I found that the SetTunePosition[] function will tune a synth globally and in semi tone steps. I'd like to find something like a portamento function that will bend notes from one to the other.

 

Hi,
You'll have to write it yourself if it doesn't exist. or ask the original developer.
I'm tied up at the moment and don't have time to look at this now. Sorry!

 

Great work SonicViz! Thank you very much for putting this out.

Has anyone figured out how to separate the sequencer from the synthesiser? I am using my own synthesiser; I'm simply using this library to load and play a MIDI file, and then I am hooking into the note-ON events.

Problem is that these two components seem to be very intimately entangled with one another. Of course I can just comment out the lines that copy audio into the audio listener's buffer, but this still means that the synthesiser is doing a huge amount of unnecessary work, calculating the entire output soundwave.

I've been looking at MIDI Toolkit -- it is superbly clean and modular code; the author is a true master of his craft. I don't think it includes a synthesiser.

Does anyone fancy having a go at porting it over?

π

 

UnityMidi (on the AssetStore) is a port to unity of Sanfords midi sequencer, but I'm not sure if source is included. Email the dev and ask!

 

There appears to be no such asset. Do you mean simpleMIDI?

That would make sense; I've just run into trouble trying to port Sanfords MIDI engine because it uses Win API calls to generate timer ticks.

 

https://www.assetstore.unity3d.com/#/content/576

 

Oh! MIDI Unified, yes, I did find that.

Are you sure that he is porting Sanford?

A few reasons that make me think he isn't: I downloaded his other MIDI asset (UnitySynth) and it is practically identical to your own port. Also I've tried the Sanford port and it's very tricky. Also, he is one of the principal contributors to some MIDI open source project -- I've lost the link but I'm pretty sure it isn't the Sanford source.

π

PS for anyone following the thread, I've successfully ported N-Audio's MIDI sequencer. which is what CSharpSynthProject leverages. It is a straightforward port.

 

The original version of Midi unified was a sanford port, not sure about now.

http://csharpsynthproject.codeplex.com/ actually has Unity support built in now actually, so you should probably download the latest source from there.
He seems to have been working on it up till august http://csharpsynthproject.codeplex.com/SourceControl/list/changesets

 

Interesting, I found the link btw http://code.google.com/p/csharpsynth/ ... Marek (the Midi Unified guy) is principal contributor to that project. Source code doesn't seem to be public on that project any more. Latest activity seems to be much 2012.

Your changelog link doesn't seem to relate to Marek / MIDI Unified... all of the recent changelogs are coming from a guy named Alex who seems to be the principal developer for that project.

I've put up the results of my research here: http://mathpad.wikidot.com/unitymidi (if anyone would like write access, just message me -- in fact it probably belongs on the Unity wiki, maybe I will move it over at some point)

BTW SonicViz, have you ever looked at getting input from an external MIDI keyboard into Unity Indie?

 

the source code is here, where it says "Source Code" ;-)
http://csharpsynthproject.codeplex.com/SourceControl/latest

I originally put Marek onto csharpsynth as it happens. He's switched Midi Unified to using BASS library though, but it has a few issues imo. Please update your docs re: ", as far as I can see all he has done is taken SonicViz's port,"
Marek independently ported csharpsynth and then started working with the developer so your text is wrong.
I don't think they are working on it anymore as he switch to BASS as mentioned above.

So current csharpsynth has Unity support due to the above interactions so I don't think you need to use either mine or mareks, just get the original developers now.

>>BTW SonicViz, have you ever looked at getting input from an external MIDI keyboard into Unity Indie?

I have pro, so have not looked at it. Your best bet would be wrap it with OSC I think.

Your cheapest fastest bet would be to buy Midi Unified and ask Marek nicely for the source of the Sanford port.
$90 is an hour of good programmer time, 2 of a medium one and still only 3 or 4 hours of a noobs so what's your time worth?

It does work as I'm using an old version (with mods I made) of it in HarpNinja for all my midi and it works on PC/OSX/iOS/Android.
That said, on occasions I've had it not start so there's some weird combination of things I haven't tracked down that screws the timers up I think.

But maybe the latest version fixes that, not sure as I have not upgraded it as yet.

 

Wait, your link is to CSharpSynthProject, and mine is to CSharpSynth
As far as I can see they are completely separate projects

There's a lot of name clashing with these midi frameworks; there is a MIDI.NET and a MidiDotNet; two completely unrelated frameworks

Thanks for reading through my wiki page and giving that correction. Updated!

I've also come to the same conclusion about OSC -- I found three projects exposing it in Unity... what I can't see is how to get a MIDI keyboard creating OSC events on the network.

The Sanford project wouldn't help me for this enquiry; it uses WinAPI to connect with MIDI hardware, as does pretty much every other solution I've looked at.

 

1. CSharpSynthProject, and mine is to CSharpSynth

They are the same, but the latest one from the original developer would be the links I gave. Midi Unified comes with it's own forked source version of it.

2. The Sanford project was modified by Marek to be platform independent, at least for midi playback, which is all I'm using it for atm

To do a hardware only connection you will have to at some point interface with a platform dependent piece of code for that specific platform.

 

https://github.com/keijiro/unity-midi-input

 

This looks pretty cool, but the code goes way over my head and has some issues. For one, the crashing - constant, constant crashing.
For another, when it is working, compared to Windows Media Player it's WAY higher pitch and has a much higher tempo.

 

This appears to be a sampling problem. On Windows, Unity has a sample rate of 48khz, while the synthesizer is sampling with 44.1khz
But for some reason if I change the sample rate it completely breaks it (sounds like musical white noise, essentially - although it plays at the correct tempo so it's a start I suppose). It also does this in the non-Unity version, so I'm guessing for some reason the synthesizer can only handle 44.1khz? That's too bad.

EDIT: OK, I solved it with perhaps the biggest hack I've ever made.
Since the synthesizer seems to only be able to handle 44.1khz, basically what I'm doing now is changing the synthesizer's buffer size based on the difference between 44.1khz and the actual sample rate (so on windows, buffer size is 1024 * ( 44.1khz / 48kz ) ). Then in OnAudioFilterRead I basically stretch or shrink the output buffer into the output "data" array.'
I tested this on both Windows and Android and it seems to work perfectly. There's probably some quality loss, but my guess is if you're using MIDI files you don't really care about top quality music

EDIT 2: Okay, WTF? I started writing a music manager around this.... in order to get the music even playing I had to attach an audio source. Suddenly, for some reason, the sample rate problem went away (so I had to get rid of those hacks) - I guess Unity audio sources are 44.1khz by default? Is it like that for all platforms?

 

One more question. I hope this thread hasn't completely died...

What format is the bank file in? I can't seem to make much sense of it... I'd like to test some SFZ soundfonts but they don't have bank files, just a loose collection of SFZ and WAV files. Should I just copy+paste+edit the included GM bank file?

 

Dead topic?

 

You should just email the developer directly, as he now supports Unity. The link to the website source is a few posts back and you'll be able to contact him through that.

I'm not currently using this and don't have time to look into it.

 

Can I get data from specific channels? Data like frequency, Db, pitch or whatever?

 

Has anyone been able to get the newest C# Synth download to work? I emailed the developer and didn't hear back. I can't seem to get the buffer into OnAudioFilterRead.

 

Thanks for a great tool!

Works great except when I try using other midi files other than the ones supplied. It can't read them. Is there any special way they need to be formatted? I see they are xxxx.mid.txt - I have tried re-naming my files but doesnt work!

 

Hi all,

Sorry for the noob question.

I've imported CSharpSynth as a plugin into unity and can run UnitySynthTest.cs and everything.

I'm now trying to access it from a js script instead of C#, cos I can't write C#

Can someone tell me why the following is not working?

Code (JavaScript):

1. #pragma strict
2.  
3. private var midiStreamSynthesizer : CSharpSynth.Synthesis.StreamSynthesizer;
4. private var midiSequencer : CSharpSynth.Sequencer.MidiSequencer;
5.  
6.  
7. function Start () {
8. }
9.  
10. function Update () {
11.     midiStreamSynthesizer = GetComponent(CSharpSynth.Synthesis.StreamSynthesizer);
12.     midiSequencer = GetComponent(CSharpSynth.Sequencer.MidiSequencer);
13.     midiSequencer.LoadMidi("Midis/Mary.mid", false);
14.     print("test123");
15.  
16. }

Warnings returned:

Error returned:

 

appending *.txt worked for me