Maestro Midi Player Tool Kit - Good news for your rhythm game !

Hi,
I am using another plugin to setup the Bluetooth Midi connection between the Android and the Bluetooth Midi instrument device. From it, I can read the Midi events coming from the instrument.
I'm trying to get Maestro to be the audio player - to play audio from Midi file in the app, and also audio from the Midi events from the Bluetooth Midi instrument device.

Regards,
Edgar

 

No, there is any re-routing MIDI in Maestro. All the stuff is done with Unity standard API for playing sound. Rather, I suspect some conflict between Unity Audio and the MIDI plugin.
1) Could you build a simple Unity app witch continuously play a sound and check with your MIDI plugin?
2) Eventually, check that pause when focus loss is unchecked in MidiFilePlayer inspector (and MidiStream)

3) Unity Audio Setting is checked in Synth parameters


4) Check Logs is checked and verify with Android Logcat if MIDI events are logs.


Regards
Thierry

 

Hi Thierry,
Thanks for the quick response. After some more testing, I realized that it is the particular Bluetooth Midi instrument device that I was previously testing with that gave the issue. I used a different instrument device (different brand, model), and it worked well; I could get the Bluetooth Midi input events, and play them out correctly with Maestro plugin.
Seems like the previous device was "incorrectly" setup as a Bluetooth Midi output device, hence the Android mobile did not play audio.

Thanks again! Really glad to be using this Maestro plugin, it's well organized and versatile.

Regards,
Edgar

 

Hello Edgar, thank you for keeping me informed.
It would be really nice to describe what you have used successfully. Maybe I will do a direct Maestro integration later.

Regards
Thierry

 

Hi everyone, I bought the Pro version since the Free Demo has me convinced (and I was interested to connect a MIDI keyboard and use the plugin to read my keyboard's notes), I just have a little issue with Unity 2022.3.11f1.

When deleting the previous asset and reimporting the Pro version, I now have these errors:


It doesn't appear with the Free version so I'm assuming it's from the scripts of the Pro version, does someone have an idea how to fix this ?

The only other tool I'm using is DOTween, I don't know if it has something to do with it.

 

Hello,
Thank for your interest in Maestro!
Try this: delete the full folder Asset/MidiPlayer and reinport the Maestro Pro (sometimes the Unity importer has strange behaviors)
Or try with a new Unity project.

Just for information, it's possible to keep the free version and import the pro on it. The pro is just an extension of the free.

Regards
Thierry

 

Consider the following steps: Firstly, delete the entire Asset/MidiPlayer folder and re-import the Maestro Pro. Occasionally, the Unity importer may exhibit unexpected behaviors. If this doesn’t resolve the issue, you might want to try creating a new Unity project. It’s worth noting that you can maintain the free version and import the Pro version on top of it. Essentially, the Pro version is an extension of the free one.

My previous answer corrected with ChatGPT

 

I just found the problem, it was related to one of my script that has the same name as one of the scripts in the Pro version Thank you so much for your quick answer !!

 

Good! Thank to inform me.
Regards
Thierry

 

Hi Thierry,
thank you for creating this amazing toolkit. I have a question - is portamento currently functioning? Additionally, how can I modify pitch or frequency after a NoteOn? Will the pitch wheel work?

 

Hello, portamento is not yet available in MPTK. But, I'm currently working on a MPTK core synth updating which is based on fluidsynth 1.4 to fluidsynth 2.3.
And the result is great! I can now say that this FluidSynth 2.3 core, rewritten in C#, sounds like its brother in C when using the same SoundFont.
The next MPTK version (2.12) will not yet include portamento (there is a lot of stuff to do) but that will help for adding portamento in a close future.
On the other hand, pitch wheel work well with the current MPTK version.

 

Hello,

I'm building a MIDI player for music teaching. The strategy I'm using is to pre-render all notes from a selected track. They become visible as the music plays/scrolls. I'm reading the notes from a combination of MidiEvent/NoteOnEvent and MPTKEvent (for unique indexes) list of event objects. But I'm not sure which class, attribute, or method I should use to get from relative time to absolute time and at the same time accommodate tempo and signature changes.

For example, with 60 BPM a note has x ticks, 1s duration, and 1.0 scale Z; and with a 120 BPM change, a note with same x ticks would now have 0.5s duration and 0.5 scale Z. Is there a pre-calculated absolute time for note position and duration? If not, how to retrieve the corresponding ticks and current BPM to calculate that for each note? Is this the best route? Please advice.

BTW, great work done so far!

Leandro

 

Hello Leandro,

As you probably know, a MIDI event is defined in time by the number of ticks with its predecessor (for the same track). It is therefore a relative number, independent of tempo. Converting a number of ticks into time depends on the current BPM.
It's the same for duration which is expressed in ticks. The MPTK_DeltaTicksPerQuarterNote which is a fixed value for each MIDI and the current BPM are used to convert ticks to duration or time position .

Calculating the real time of an event is quite complex, but fortunately this is done by MPTK when the MIDI is loaded.

With MPTK Each MIDI event is represented by the MPTKEvent class.

• With attribute Tick : time in tick of the event since the start of playing the MIDI file. This time is independent of the tempo.
• But also RealTime : time in milliseconds of this MIDI Event from the start of the MIDI which take into account the tempo changes.
• Duration : duration of the note in millisecond related also to the tempo change.
  
• You also could be interested by Beat and Measure attributes which are related to the time signature change event.

So normally you shouldn't need to convert yourself the ticks values.

Also: midiFilePlayer.MPTK_MidiLoaded is useful to get MPTK_TempoMap and MPTK_SignMap
See MaestroOnEventStartPlayMidi() method in Assets\MidiPlayer\Demo\FreeDemos\Script\TestMidiFilePlayerScripting.cs for example.

Your project is very interesting! Please keep us informed.

Regards
Thierry

 

Hello all,
I have often this question (see above) and I decided to create a post, look here:
MIDI ticks, duration and real-time with MPTK - Maestro MPTK (paxstellar.fr)

Have you say in case of lack of clarity or error detected!

 

Hey @BachmannT
I'm just wondering if it's ever possible to convert MIDI file to MP3 in Unity Runtime.

 

Hi, not with MPTK ...
Why do you want to convert to MP3? I could add this feature ... one day!

 

I will let users to upload their midi file and convert it to mp3 file.

 

Oh really?
I would really love it. Thanks!

 

How do I get the number of notes in MidiFile from script?

 

you get this kind of information:

And the count of notes by channel:


But with no API for doing that, it's quite easy.
Look at Assets\MidiPlayer\Scripts\Editor\MidiFileSetupStat.cs

 

Code (CSharp):

1.  for (int channel = 0; channel < 16; channel++)
2.      for (int note = 0; note < 128; note++)
3.          if (stat_note[channel, note] > 0)
4.              stat_channel[channel]++;

 

Hi Thierry,

Thank you for your answer. Since the notes have a constant speed and don't react to BPM changes, I'm sticking with RealTime/Duration which takes care of that providing the final time calculation. Nice to have those!

Unrelated question, where should I start to support VST?

Also thanks for putting together a post on that! I'll let you know when we get to a stable version.

 

Hello
What do you mean by support VST ? Be able to host VST component in Unity ?
It's terribly complex, I tried last year, I was only able to load the VST in the Editor, not using it in Unity or that will take a lot of energy and time.
More information here:
https://github.com/obiwanjacobi/vst.net
https://stackoverflow.com/questions/34001818/make-a-vst-with-unity3d

And of course, the steinberg documentation:
https://steinbergmedia.github.io/vs...al+Documentation/API+Documentation/Index.html

Courage!

 

Hello, I noticed that when playing MIDI events with low volume, the volume level in Unity does not match that of external players and MIDI editors. Specifically, when the volume is set to single digits, the sound of the notes is almost inaudible in external players, but in Unity, it is quite noticeable if you listen carefully. What could be causing this discrepancy, and how should I adjust it?

 

If I want to keep the sound played in Unity as closely matched to the sound in external MIDI editors as possible, what settings should I pay attention to? Additionally, if I use the stream player to play an entire song through events from the MIDI file player, will this cause any issues? I've noticed occasional irregularities in the rhythm.

 

Since my English skills are not strong, I've used machine translation. I hope you can understand my meaning

 

Hello, I understand perfectly. But also your last deleted message, thanks to deepl ;-)

Question about volume:
The current version of MPTK is based on an older version of fluidsynth (v1.4) rewritten in C#. This makes a big difference when playing.
I'm working on an update with the latest version of fluidsynth (v2.3), which is a lot of work! But the global volume could be fixed with the volume available in the prefab (first parameter in the inspectors MidiFilePlayer and MidiStreamPlayer).

 

Second question: You're using MidiFilePlayer to read MIDI events but not play them. You send these events to MidiStreamPlayer to play them?
In this case, the tempo depends on the Unity Update cycle, which can be irregular.

 

Thank you for your response. I switched to a new set of soundfonts and found it helpful for the current issue (differences in fade-ins at low volumes when maintaining consistent overall volume). I believe I can solve this problem by adjusting the soundfonts or, if necessary, dynamically adjusting the overall volume at runtime. Making slight modifications after completing the MIDI is not a problem for me. However, I'm still not sure if the issue I encountered is indeed real. If it is, I'll look forward to future optimizations. Thanks again for your hard work!

 

Hi again @BachmannT
Can you help me on converting to mp3? I can help coding it if you’re busy. I just need to know how I can do it.
Is there a way to make it faster than realtime?
I mean, if there is a midi with 5 minutes, can it export it to mp3 faster like 1 minute or so?

 

Hello,
I haven't started thinking about it yet!
Sometimes, finding the best solution takes longer than building it.

MIDI is the best solution to played faster a music without changing the sound.
If you really need a MP3, a MIDI accelerated (see setting speed) and a converter to MP3 will doing the job.

 

We have a need for a 15 track midi song, with different configurable BPM's per track/instrument. Is it possible to have individual BPM control per track? I wasn't able to find anything in the documentation. Or perhaps play 15 midi files (each with one track) at different BPMs at the same time?

 

Hello,
Sadly no. By design with MIDI music, BPM is related to all the tracks and channels (it's not a MPTK limitation).
Yes, you could play 15 MIDI files with 15 MidiFilePlayers prefab with 15 different BPMs ... if you are not too demanding about the synchronization between each MIDI files.

 

Thanks for the information. Right now I wouldn't know where to start with that, sounds really complicated indeed. The motivation would be for higher sound quality and a more wide variety.

 

Hi BachmannT,
I gave up on converting to MP3. It's probably a lot of work and MIDI is more portable and configurable. So I choose MIDI approach.

Now, with the change of plans, I'm trying to merge AudioSource and (External)MidiPlayer logic with an interface.
For example:

Code (CSharp):

1. public class BAudioClip : IAudioSource
2. {
3.     public AudioSource audioSource;
4.     [ValueRequired] public AudioClip audioClip;
5.  
6.     public float length => audioClip.length;
7.  
8.     public float pitch {
9.         get => audioSource.pitch;
10.         set => audioSource.pitch = value;
11.     }
12.  
13.     public bool loop {
14.         get => audioSource.loop;
15.         set => audioSource.loop = value;
16.     }
17.     public float time
18.     {
19.         get => audioSource.time;
20.         set => audioSource.time = value;
21.     }
22.  
23.     public void Play()
24.     {
25.         audioSource.Play();
26.     }
27.     public void Pause()
28.     {
29.         audioSource.Pause();
30.     }
31.     public void Stop()
32.     {
33.         audioSource.Stop();
34.     }
35.     public void UnPause()
36.     {
37.         audioSource.Play();
38.     }
39. }

Code (CSharp):

1. public class BMidiSource : IAudioSource
2. {
3.     public byte[] midiSource;
4.     [ValueRequired] public MidiExternalPlayer externalPlayer;
5.  
6.     public float length => (float) externalPlayer.MPTK_Duration.TotalSeconds;
7.  
8.     public float pitch {
9.         get => externalPlayer.MPTK_Speed;
10.         set => externalPlayer.MPTK_Speed = value;
11.     }
12.  
13.     public bool loop
14.     {
15.         get => externalPlayer.MPTK_Loop;
16.         set => externalPlayer.MPTK_Loop = value;
17.     }
18.  
19.     public float time {
20.         get => (float) externalPlayer.MPTK_PlayTime.TotalSeconds;
21.         set => ??? // HOW DO I SET AN EXACT TIME???
22.     }
23.  
24.     public void Play()
25.     {
26.         externalPlayer.MPTK_Play();
27.     }
28.     public void Pause()
29.     {
30.         externalPlayer.MPTK_Pause();
31.     }
32.  
33.     public void Stop()
34.     {
35.         externalPlayer.MPTK_Stop();
36.     }
37.  
38.     public void UnPause()
39.     {
40.         externalPlayer.MPTK_UnPause();
41.     }
42. }

The problem is, I can't set an exact seconds in the MidiPlayer.

I need to go forward and backward in time while it's playing like an AudioSource (e.g. `audioSource.time = 1.2f`)

How can I do that?
Thanks!

 

Hi BachmannT, I'm currently having our app tested in multiple Android devices and in some of them the execution speeds up in some parts of the music, it seems it's trying to keep up with real time (e.g. Samsung A12 works, Samsung A22 has issues). A few questions:

1. It feels like a performance issue related to specific devices? Maybe the low level audio track or thread is being interrupted by something or not prioritized?
2. If so, is Oboe a way out? Any thoughts?
3. Could some of the calculations when synching up visual notes be causing the delay? (see below)

To sync up the notes I'm reading the following attributes from within a Unity Update() (~30 fps):

Code (CSharp):

1. public MidiFilePlayer midiFilePlayer;
2. ...
3. public TimeSpan CurrentTime
4. {
5.     get
6.     {
7.         var ticks = midiFilePlayer.midiLoaded.MPTK_TickPlayer;
8.         var ms = midiFilePlayer.midiLoaded.MPTK_CurrentTempoMap.CalculateTime(ticks);
9.  
10.         return TimeSpan.FromMilliseconds(ms);
11.     }
12. }
13.  
14. public TimeSpan TotalTime => midiFilePlayer.MPTK_Duration;

Please see the attached video for the issue. Notice whenever the audio is running behind the visual notes it seems it tries to compensate.

Thanks!

 

Hello MptkEvent.realtime

Hello
Have you tried MPTK_Position ?
Maestro - Midi Player Tool Kit for Unity: MidiPlayerTK.MidiFilePlayer (paxstellar.com)

 

Hello
1) A separate thread runs the MIDI sequencer (which reads MIDI events) and the MIDI synth. Moreover, these threads are not dependent on the Unity thread. But of course, a more privileged system thread could interrupt these threads.
2) Oboe is perfect for reducing latency. No idea about performance.
3) Hard to answer without a good understanding of your architecture! Have you tried using FixedUpdate() instead of the framerate dependent Update() method?

 

Thanks, I modified the code a bit and it worked.

 

@BachmannT I found a bug in `FindSegment` function and fixed it.

Posting the updated code for you:

Code (CSharp):

1.         public static int FindSegment(List<MPTKTempo> tempoMap, float timeSearch, int fromIndex = 0)
2.         {
3.             if (tempoMap == null || tempoMap.Count == 0)
4.                 return 0;
5.  
6.             int indexTempo = fromIndex;
7.  
8.             while (indexTempo < tempoMap.Count)
9.             {
10.                 if (timeSearch < tempoMap[indexTempo].FromTime +
11.                     (tempoMap[indexTempo].ToTick - tempoMap[indexTempo].FromTick) * tempoMap[indexTempo].Pulse)
12.                 {
13.                     break;
14.                 }
15.                 indexTempo++;
16.             }
17.  
18.             if (indexTempo >= tempoMap.Count)
19.                 indexTempo = tempoMap.Count - 1;
20.  
21.             return indexTempo;
22.         }

 

Hello, thanks a lot. I will look and check this update.
Regards
Thierry

 

Hello, validated! Thanks a lot, that will be available for a very soon release.
For those interested, below the scripts for testing:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using MidiPlayerTK;
5.  
6. public class TestTempoMap : MonoBehaviour
7. {
8.     public MPTKWriter mfw;
9.  
10.     // Start is called before the first frame update
11.     void Start()
12.     {
13.  
14.     }
15.  
16.     // Update is called once per frame
17.     void Update()
18.     {
19.  
20.     }
21.     public void Check()
22.     {
23.         mfw = new MPTKWriter();
24.  
25.         int ticksPerQuarterNote = mfw.DeltaTicksPerQuarterNote;
26.         int tick = 0, note = 0, beatsPerMinute = 30;
27.  
28.         // Add 6 measures with each one tempo change and four quarter
29.  
30.         AddMeasure(tick, note, beatsPerMinute);
31.         tick += 4 * ticksPerQuarterNote; note += 4;
32.  
33.         AddMeasure(tick, note, beatsPerMinute * 2);
34.         tick += 4 * ticksPerQuarterNote; note += 4;
35.  
36.         AddMeasure(tick, note, beatsPerMinute * 3);
37.         tick += 4 * ticksPerQuarterNote; note += 4;
38.  
39.         AddMeasure(tick, note, beatsPerMinute * 4);
40.         tick += 4 * ticksPerQuarterNote; note += 4;
41.  
42.         AddMeasure(tick, note, beatsPerMinute * 5);
43.         tick += 4 * ticksPerQuarterNote; note += 4;
44.  
45.         AddMeasure(tick, note, beatsPerMinute * 6);
46.         tick += 4 * ticksPerQuarterNote; note += 4;
47.  
48.         // Calculate time, measure and beat for each events
49.         mfw.CalculateTiming(logDebug: true, logPerf: true);
50.  
51.         // Show MIDI events created
52.         mfw.MPTK_MidiEvents.ForEach(t => Debug.Log(t));
53.  
54.         // Show tempo map
55.         mfw.MPTK_TempoMap.ForEach(t => Debug.Log(t));
56.  
57.         // Check index tempo map found for some time (milliseconds)
58.         FindIndexTempoMapFromTime(0f,0);
59.         FindIndexTempoMapFromTime(2500f,0);
60.         FindIndexTempoMapFromTime(4000f, 0);
61.         FindIndexTempoMapFromTime(4500f, 0);
62.         FindIndexTempoMapFromTime(10000f, 1);
63.         FindIndexTempoMapFromTime(12000f, 1);
64.         FindIndexTempoMapFromTime(12001f, 2);
65.         FindIndexTempoMapFromTime(16666f, 3);
66.         FindIndexTempoMapFromTime(16667f, 4);
67.         FindIndexTempoMapFromTime(20000f, 5);
68.  
69.     }
70.  
71.     // Add a tempo change and 4 quarters, so one measure with signature = 4/4
72.     private void AddMeasure(int tick, int note, int beatsPerMinute)
73.     {
74.         int ticksPerQuarterNote = mfw.DeltaTicksPerQuarterNote;
75.  
76.         mfw.AddTempoChange(track: 1, tick: tick, microsecondsPerQuarterNote: MPTKEvent.BeatPerMinute2QuarterPerMicroSecond(beatsPerMinute));
77.  
78.         mfw.AddNote(track: 1, tick: tick, channel: 0, note: note, velocity: 50, length: ticksPerQuarterNote);
79.         tick += ticksPerQuarterNote; note++;
80.         mfw.AddNote(track: 1, tick: tick, channel: 0, note: note, velocity: 50, length: ticksPerQuarterNote);
81.         tick += ticksPerQuarterNote; note++;
82.         mfw.AddNote(track: 1, tick: tick, channel: 0, note: note, velocity: 50, length: ticksPerQuarterNote);
83.         tick += ticksPerQuarterNote; note++;
84.         mfw.AddNote(track: 1, tick: tick, channel: 0, note: note, velocity: 50, length: ticksPerQuarterNote);
85.         tick += ticksPerQuarterNote; note++;
86.     }
87.  
88.     // Find index and check
89.     private void FindIndexTempoMapFromTime(float time, int expected)
90.     {
91.         int indexTempo = MPTKTempo.FindSegment(mfw.MPTK_TempoMap, time, fromIndex: 0);
92.         string result = indexTempo != expected ? "<color=red>ERROR</color>" : "";
93.         Debug.Log($"MPTKTempo.FindSegment find at time {time,10:F2} ms {indexTempo} expexted:{expected} {result}");
94.     }
95. }
96.  

 

I tested using the same MIDI in Samsung A55, A22, and A12, and am getting divergent outcomes.
Please see attached the 3 recordings and the original MIDI (the melody track was purposefully muted when recording).

It feels to me there's some lagging sometimes or some notes are being dropped depending on the device (conf. video from my previous post). Not really sure Oboe would address this somehow. Willing to give it a try.

On a side note, I didn't get the chance to look too deep into Maestro's code, but was wondering if that could be related to the DSPs while writing to buffers during execution (e.g. effects, linear interpolation, etc.) That being the case, maybe performance could be further improved by running analysis on the song, first thing at play start/init, to gather only the note samples used by it, preprocess all of them and store them in memory to be sent to final buffer afterwards while playing. So that could potentially eliminate some of the real time processing?

Any thoughts or any other route I should explore? Please advise.

 

Hello
Hard to answer.

I'm not convinced by an issue on the DSP side. The DSP write the buffers at each OnAudioFilterRead call. An overrun on this call create distorted sound, creaking and weird noises. Not missing notes. Try playing with the demo 3 and 6 to explore DSP overrun.


So, it's perhaps more on the MIDI reader.

Really, I need more information on your solution. In private if you prefer.

How the MIDI is played ?
Directly from the MidiFilePlayer ?
or do you process the MIDI notes in the Update() Unity method before playing them ?

In the meantime, I'm exploring some solutions.

 

Thanks for your prompt reply! That makes sense, maybe it could be something in the surroundings of when it triggers the OnEventNotesMidi event?

Regarding my integration w/ Maestro, I'm using the MidiFilePlayer (currently w/ preloaded MIDIs) and reading the MIDI data only for rendering and synching the notes. For playing I basically:

1. Set MPTK_MidiIndex
2. Call MPTK_Load()
3. Call MPTK_Play()

I'm sending you in private my main class, which is pretty much a façade for Maestro's MIDI data, actions, and events; and it's consumed by both UI controls and internals (e.g. note synchronizer, game business logic, shortcuts, etc.)