Public defence: ?arab and AI: Maq¨¡m Music Generation through Machine Translation and Motion Capture

This multidisciplinary dissertation explores computer music generation and motion capture-informed personal adaptation in maq¨¡m improvisation. Maq¨¡m music, prevalent in the Middle East and North Africa, is characterized by audience interactivity, fostering intense emotional and meaningful physical engagement and enchantment (?arab). The dissertation explores how artificial intelligence and modern sensing technologies can enhance the experience of ?arab.

The first part of the dissertation is based on machine translation (MT) to generate real-time instrumental improvisatory phrases for the non-metric maww¨¡l and taq¨¡s¨©m traditions, viewing these phrases as ¡°sentences¡± through the lens of natural language processing (NLP). I developed tools for curating textual datasets from vocal and MIDI inputs, and then created a dataset with local musicians in Amman. This dataset consists of 6,991 parallel maww¨¡l sentences for machine translation (from vocal calls to instrumental responses), with maq¨¡m improvisation being treated as an under-resourced language. Using scale degrees and quantized durations, I reduced the vocabulary to 15 distinct elements. This approach yielded successful results comparable to typical NLP applications, using a small, diverse dataset (8 maq¨¡m¨¡t). MT results, both neural (NMT) and statistical (SMT), showed that merging different maq¨¡m¨¡t sub-datasets doesn¡¯t necessarily improve outcomes; it can favor shorter, less sophisticated sentences in both approaches, especially in NMT. My discussion highlights that user musical background, performance decisions, as well as the training dataset size and performance quality are crucial factors in system design. Additionally, I proposed a method to transform frequent patterns¡ªextracted from a taq¨¡s¨©m dataset I commissioned for this project¡ªinto elaborate sentences using iterative translation, which enables real-time adaptation.

The second part of the dissertation examines perceivers¡¯ subtle music related body motion to gauge their engagement with human-performed taq¨¡s¨©m, considering their cultural background.  I conducted a cross-cultural perception test with 60 participants in Amman and Oslo. Using accelerometer-based motion capture (mocap) to measure the quantity of motion (QoM), I decoded significant engagement patterns. Combining QoM signals into trend curves, I was able to illustrate key gradual and abrupt engagements with both structural and non-structural aspects of the taq¨¡s¨©m. Conventional classification and clustering models made informed guesses about participants¡¯ cultural backgrounds and emphasized the importance of music training. This demonstrates the potential of accelerometry mocap to enhance real-time music generation and perception, to achieve ?arab. The overall results in generation and perception could significantly impact the future of interactive instrumental artworks. I conclude by proposing a preliminary algorithm for mocap-adapted taq¨¡s¨©m generation, paving the way for further research.

Trial lecture

Designated topic: Decolonizing Music Technology: Crafting Culturally-Sensitive AI for Interactive Music Generation and Perception

Time and place: Thursday 25 September, 14.10 PM

The trial lecture will be streamed. 

Evaluation committee

• Associate professor, Andr¨¦ Holzapfel, KTH Royal Institute of Technology (first opponent)
• Professor Farid Meziane, University of Derby, UK (second opponent)
• Professor Patrice Bellot, Aix-Marseille University, France
• Director of research Fran?ois Charpillet, INRIA, France
• Associate Professor Stefano Fasciani, University of Oslo (committee administrator)

Chair of the defence

• Anja Nylund Hagen, Head of Department, Department of Musicology

Supervisors

• Professor Alexander Refsum Jensenius, University of Oslo

• Professor Kamel Smaili, University of Lorraine