Using audio with FTM's Gesture Follower - Semesteroppgave v?r 2009

The Process

    I spent hours and hours trying to get a hold on how the patch works, using my knowledge of Max/MSP and Score Following, but nothing came out of it. It's really complex.
The only thing I got from this was that the zl-object made a list out of the positions of the mouse inside the lcd. I had to make a list from analyzer~, so i patched it into zl.
Analyzer~ is an object which sends pitch out the first outlet and you need some sort of audio in.
This didn't work so i spent some time trying to find out what was wrong. Once again searching around the patch for clues.
After a while, I went online to search for solutions. To my astonishment, there's, by the looks of it, only one examples of using audio in the Gesture Follower. This was from a FTM workshop at Bergen Center for Electronic Art (Bergen senter for elektronisk kunst – BEK), where they used gbr.yin instead of analyzer~. I downloaded their example, but there was something very wrong with their whole patch. A lot of objects were renamed, so that it wasn't objects anymore, and when i renamed them back, some connections were lost. I tried to but just the analyzer part inside the original patch, but it didn't work anymore than analyzer~ did.
One of the reasons that nothing worked, was that I had deleted many objects I thought belonged to the lcd-object only. When I tried to use the whole original patch with analyzer~, the system responded, but different errors started appear in the message window. There was something about matrixes that didn't get the right input or something. Once again I went through the patch trying to find the objects and message-boxes that had the problems, and then I followed the patch-cables trying to find where the reasons for the problem were. Nothing came out of this, except that I knew the path where to search. Since I knew there was some sort of matrix-problem, I started looking for possible matrixes in the path. There were some message-boxes with the words «rows» and «cols» inside mnm.winfilter, inside the patcher «preprocess», and since the only number to tweak was the 4 in the object name, I tried to use 1, 2, 5, 10 and 20 to see what happened.
Nothing happened and I thought maybe the problem was the output from analyzer~. I went into analyzer~.help only to see that the first outlet put out both MIDI and frequency pitch and that it is unpacked to display both. I copied this part and used it in my project.
This didn't solve it, but then it came to my mind that the lcd outputs two numbers, corresponding to the two dimensions. Analyzer~ had to also put out two numbers so that the list zl made was more like the list from the lcd. Using a scale-object the pitches was scaled to a number between 0 and 250, the same values as from the lcd. Then the number goes into a multislider with two sliders that both use the same input. Then I patched the multislider to zl.
From this point the patch was supposed to be working. It responded to my input, but even when I was matching the same input on both the recorded and the played gesture, the likelihood was very unstable and didn't come up with the same answer every time. I spent some time tweaking the number on the nmn.filter and noticed the change. The problem was that the change too was very unstable. Sometimes better, sometimes worse and it was impossible to make any sense of it.
Up until now I have used single notes played by a guitar as the input. I spent some time on the lcd again. Trying to find out how unstable the likelihood was this time. I saw what happened and found that the positions and the direction of the movement had less to say than the distance between the blue dots. No wonder that my single notes had been hard to analyze.
So I recorded new examples, this time using melodic sequences. This was better, but still the likelihood was unstable and the result wasn't the same every time. The analyzer with gbr.yin was a little bit better, so I decided to use that one for the remaining programming.
The tones played by the guitar range from about 146Hz to about 330Hz, so I tried to scale the pitches directly inside the yin-patcher. From 100 and 400 to 0 and 250. When I did this, the result from the matching was for the most part reversed. I decided to use the complicated scaling I was using from before, as it worked just fine. This was originally designed before I tried to look at the yin-patcher.
Then I cleaned up the patch, deleting everything I didn't need. I removed them one by one and checked that it still worked.
To make the patch actually do something i added a player that plays different audio-clips triggered by the matching of the input. Because of the problems with the monitor the two players were turning themselves on and off all the time, so I had to make something to stop the number from «likeliest» inside mnm.follower.monitor.Kand554 affecting the players all the time.

How To Use The Patch

First you need to download and unzip Audio-follower.zip and place the folder “ftm-and-co” into the search-patch of Max/MSP. This folder contains both the Gesture Follower and the Gabor-package.

1.Turn the audio on.
2.Choose between the examples or incomming audio, and choose the example you want to teach the program with (if you're using the examples).
3.Turn learn on.
4.Push Play to record the first sequence in one chosen phrase.
5.Choose another phrase and another example and record the second sequence.

To match audio..

6.Turn learn off.
7.Choose example or change to incoming audio.
8.Push play.

When using incoming audio, the play button will push itself when you play.

You can see the matching on the monitor, under «likelihood».
If you turn up the gain, the triggered audio-clips should work.

The examples 1 and 1_2 are different recordings of the same sequence. So is 2 and 2_2. 3 is corresponding to 1 and 1_2, and 4 to 2 and 2_2.

References

The original Gesture Follower is made by Frederic Bevilacqua and collaborators. It's a part of FTM & Co, which is developed in the Real-Time Musical Interactions team at IRCAM. It can be downloaded from: http://ftm.ircam.fr/index.php/Gesture_Follower
This package also includes the Gabor-object, gbr.yin, used inside the sub patch “yin”.
One of the patches used inside the Gesture Follower, mnm.follower.monitor.maxpat, is modified by me, for this patch. I renamed it mnm.follower.monitor.Kand554.maxpat.

The sub patch “yin” is taken from the patch mnm.follower-example-sound.help used at the FTM Workshop at BEK: http://groupware.bek.no/groups/ftm/wiki/57c80/