Annual Report 2022

This was a year of transition. Many doctoral and postdoctoral fellows have moved on, and another generation of RITMOanians has entered the centre.

This was yet another year that began with a corona-related lock-down. Fortunately, restrictions were lifted during the spring, and everyone could finally return to the offices and labs. Still, most activities have been run in hybrid mode, and we have tried to bring with us the best experiences from the corona period.

In October, RITMO celebrated its fifth birthday. That means we move from phase one to phase two and have a significant turnover of people. As the first group of RITMOanians are about to leave, we have had many PhD celebrations and goodbye sessions. It is sad to see people go but exciting to see the activities of our growing group of alumni. We are also excited to welcome a new generation of doctoral and postdoctoral fellows; their research will shape RITMO in the future.

Here, we present some highlights from 2022; in rhythm, time, and motion.

Anne Danielsen and Alexander Refsum Jensenius

Table of Contents

RITMO at a Glance

Rhythm is everywhere, from walking, talking, dancing and playing to telling stories about the past and predicting the future.

Our heartbeat, brain waves, and other bodily cycles all work through rhythm. Rhythm is a crucial aspect of human action and perception. Our human rhythm is also in complex interaction with the world's various cultural, biological, and mechanical rhythms.

At RITMO, we research rhythmic phenomena and the complex relationships between the human body's rhythms and the brain. The central idea is to establish a link between features of rhythmic phenomena in the world and within the (embodied) mind. The aim is to understand our ability to perceive rhythm and how this affects our actions and experiences.

RITMO is a highly interdisciplinary centre, combining perspectives and methods from music and media studies, philosophy and aesthetics, cognitive neuroscience, and informatics. Our research employs several state-of-the-art technologies, including motion capture, neuroimaging, pupillometry and robotics.

Structure & Aesthetics Cluster

We investigate the structure and aesthetics of musical rhythm and time. We are interested in rhythmical experiences and practices and how these can provide insights into perceptual processes, meaning-making, and sociocultural contexts. We examine the role of rhythm, temporal structure, and timing in the experience of music, both in terms of its aesthetic dimension and the cognitive processes underpinning the experience.

Highlights

Ragnhild Br?vig and colleagues have interviewed Norwegian EDM producers and analysed their music to find out how they shape the microlevel aspects of their sound and groove. In an article entitled A Grid in Flux (Music Theory Spectrum), they demonstrate how minute timing adjustments and the choice and processing of rhythmic sounds contribute microrhythmic nuance to grid-based EDM grooves.
Using advanced clustering techniques, Gui Schmidt C?mara and George Sioros map how drummers express different timing styles (laid-back, on-beat, pushed) via stroke onset timing and intensity features. The method captures the microtiming of the kick, snare, and hi-hat drum onsets among one another, identifying each drummer��s "microrhythmic fingerprint."
RITMO research into hip-hop is growing in number and scope. Postdoc Kjell Andreas Oddekalv has co-edited a collection on Norwegian rap flows entitled Flytsoner [Flow zones]. In another contribution, he delves into the interaction of musical and speech rhythms in rap. Anne Danielsen and Thea Paulsrud analyse a track by Azealia Banks in an anthology dedicated to exploring how recording has shaped music over the last hundred years. In the same collection, Ragnhild Br?vig analyses the aestheticisation of musical fragmentation in a track by Kid Koala.
RITMO researcher Rainer Polak received funding from the Research Council of Norway��s Groundbreaking Research programme for a four-year interdisciplinary project called DjembeDance. The project group will study multimodal rhythms in djembe music and dance from Mali.

A Comprehensive AI-based System for Advanced Music Analysis

The project MIRAGE - A Comprehensive AI-Based System for Advanced Music Analysis aims to improve computers' capability to listen to and understand music. A goal is to develop technologies that facilitate understanding and appreciation of music by generating rich and detailed descriptions. These encompass various dimensions, such as rhythm, tonality, formal structure, melodic shape, sound colour, and emotions. Effort is dedicated to designing applications of value for musicology, music cognition, and the general public. One main focus is the study of Norwegian Hardanger fiddle music.

Musical Fragmentation

The project Musical Fragmentation investigates the aesthetics and pleasures of musical rhythms and grooves that are experienced as fragmented or related to machines. The focus is sample-based music, cut-and-paste and glitch aesthetics, and other experimental music expressions. The aim is to explore how listeners make meaning out of musical fragmentation and machine rhythms and how these experiences are related to cognitive processes.

Musical Microrhythm

The project TIME: Timing and Sound in Musical Microrhythm investigates interactions between temporal and sound-related features at the micro-level of rhythm in musical genres where rhythm is a core dimension. The aim is to gain new insights into the micro-level of auditory perception and the role of cultural background and/or training in music in this regard. This is based on qualitative and quantitative methodologies from musicology, ethnomusicology, music psychology, and motion research.

Musical Time and Form

The project Musical Time and Form investigates salient differences in musical interpretation between recorded performances of orchestral compositions in Western Art Music from the last two centuries. Compositions are performed differently depending on the conductor, orchestra, historical and cultural tradition, and other factors such as the time and place of the performance. There is also a focus on the listening experiences of those performances, intending to improve the precision with which we discuss musical performance and the listening experience.

Structure & Cognition Cluster

The goal is to expand our understanding of how rhythm and sense of time are constructed in the human mind. We study the influence of rhythm and time on perception, attention and effort, imagery and illusions, prediction and cognitive control processes across sensory modalities.

Highlights

In a study published in Scientific Reports, doctoral research fellow Connor Spiech and colleagues used pupillometry to show that moderately complicated drum rhythms hold attention longer than both less and more complex rhythms. They also found that a note on an unstressed beat followed by a stressed note increases attention but not groove ratings, whereas an unstressed note followed by silence (i.e., a syncopation) yields the opposite pattern. These findings indicate these two musical features' complementary roles in predictive processes.
Postdoctoral researcher Maja Dyhre Foldal recorded participants�� brain activity (with EEG) while listening to rhythmic sound streams in different tempi presented to separate ears. The results, published in Cerebral Cortex, revealed that brain waves at the beta frequency matched the pace of the attended sound stream more strongly than the pace of the unattended stream. It was concluded that this reflects how the brain also utilizes the timing of sounds to direct attention.
In Frontiers in Neuroinformatics, researcher Alejandro Blenkmann and colleagues present a new platform for modelling intracranially implanted electrodes in epilepsy patients undergoing assessment for surgical treatment. The method can be used clinically to increase the precision in localising intracranial electrodes. The platform is an essential research tool for RITMO��s cognitive intracranial EEG program and improves conclusions about the anatomical foundations of brain functions. The methods and a database with ~50.000 simulations are freely available via open repositories.

Mental Effort

The Mental Effort project investigates the engagement of attentional resources while carrying out a cognitive task. Mental effort can be measured behaviorally, subjectively and physiologically. Our approach focuses on the psychophysiological index provided by pupillometry and brain activity with fMRI. We are particularly interested in the inherent rhythmicity of attention, the effort related to various forms of temporal violation of regularity, and the effort used when processing information in different sensory modalities.

The Neural Basis of Temporal Prediction

A primary goal of The neural basis of temporal prediction project is to understand the neurocognitive basis of predictive processing in the human brain with particular attention to auditory stimuli and rhythm. We aim to decipher the neural mechanisms involved by using scalp and intracranial EEG, eye tracking and pupillometry concomitantly with cognitive tasks performed by healthy participants and individuals with brain disorders.

Neurophysiological Mechanisms of Human Auditory Predictions

The ability to predict upcoming events is a core feature of human cognition. In a broad sense, predictive processes incorporate knowledge from the past to predict future states of the body and the environment, shaping how we perceive the world. The project AudioPred - Neurophysiological Mechanisms of Human Auditory Predictions: From population- to single neuron recordings utilizes the unprecedented timing and anatomical detail offered by intracranial EEG recordings to uncover how auditory predictive processes are computed in the human brain.

Rhythm as an Individual Ability

The goal of the project Rhythm as an individual ability is to understand how rhythmic capabilities vary between people. To accomplish this, we investigate the basic cognitive, environmental, and genetic factors that shape rhythmicity. We are developing a test battery to cover a broad area of rhythm topics, such as meter, beat, and microtiming.

Interaction & Pleasure Cluster

We seek to understand why rhythms make us move and how rhythm facilitates entrainment and interaction. We also explore the affective and social outcomes of musical and rhythmic processes. These phenomena are studied through the experience of absorption in musical performance and perception, music-related effort and skilled action, and the dynamics of music-evoked pleasure and social bonding.

Highlights

Using pupillometry to measure attentional processes, doctoral research fellow Merve Ak?a and colleagues investigated the temporal limits of selective attention, comparing the sounds of cello tones, dogs, and human voices. The latter were more likely to escape the "attentional blink" phenomenon, where detecting a particular second sound is impaired by correctly detecting a first sound in a rapid serial presentation of stimuli. The results are published in Brain and Cognition and suggest humans are susceptible to human voices.
In a study published in PLoS One, Associate professor Jonna Vuoskoski and colleagues asked participants to continuously rate how moved they felt while listening to a series of musical excerpts. Both sad and joyful music were associated with experiencing a sense of social connection and beauty. The participants�� level of the personality trait ��empathy�� predicted how strongly they were moved. The findings support the central role of empathic processes in music listening.
In the Oxford Handbook of Music Performance, RITMO researcher Laura Bishop and professor Peter Keller from the Music in the Brain centre at Aarhus university discuss the psychological mechanisms supporting instrumental ensemble performance, a creative process involving real-time interpersonal coordination of sounds, gestures, and musical ideas by two or more musicians. They outline implications for music education, networked ensemble playing, and technologies for musical interaction.

Effort and Attention in Musical Experience

Musical experiences often provide sensations of affect and pleasure, as well as feelings of sound-producing and sound-accompanying body motion. The project Effort and Attention in Musical Performance Experience aims to enhance our understanding of the attention and effort, both physical and mental, involved in producing and perceiving music-related sound and associated multimodal phenomena.

Engagement and Absorption

Do time and timing matter for musical engagement and absorption? Music is well known for its ability to engage, yet the nature of this engagement remains elusive. The project Engagement and Absorption explores rhythm, temporality, and meaning in musical engagement, such as listening and performing. The project encompasses various research angles, including musical sense-making and ethics, absorption in musical performance, spatiality and immersion, and social interaction.

Entrainment, Social Bonding and Pleasure

The project Entrainment, Social Bonding and Pleasure aims to advance our understanding of how entrainment, synchronised behaviour and other related processes lead to prosocial outcomes such as social bonding. The project seeks to explore the role of empathy in interpersonal synchronisation and uncover the factors contributing to synchronous movement and social bonding in concerts. It also explores whether entrained listening can evoke affiliation towards fellow listeners, musicians, or social groups. The sub-project Quarantine Concerts project aims to explore the social and emotional effects of virtual concerts.

Bodily Entrainment to Audiovisual Rhythms

Much focus has been devoted to understanding the "foreground" of human activities, such as the things we say, our actions, and the sounds we hear. The AMBIENT project will study the sonic and visual "background" of indoor environments, such as the sound of a ventilation system in an office, the footsteps of people in a corridor, or people fidgeting in a classroom. The project focuses on how different auditory and visual stimuli combine to create rhythms in various environments and how such rhythms influence people's bodily behaviours and feelings.

Musical Chills

The Musical Chills project investigates the brain processes and musical triggers underlying the intense, pleasurable 'chills' responses that music evokes. It explores the role of other neuromodulators (particularly the endogenous opioid system), via pharmacological agents, in the pleasurable response to music. Moreover, the project investigates what specific musical features can trigger chills.

MusicLab: Researching real-life concerts

MusicLab is an innovation project by RITMO and the University Library. The aim is to explore new methods for conducting research, research communication and education. Each MusicLab event is organised around a concert in a public venue, which is also the study object. The events also contain an edutainment element through panel discussions with world-leading researchers and artists and "data jockeying" in the form of live data analysis of recorded data.

Interaction & Robotics Cluster

We investigate aspects of rhythm and motion through robotics and technology. This includes creating interactive music systems focusing on multi-dimensional mapping from sensors to musical parameters. Moreover, we design and prototype adaptive robot systems and explore human-robot and robot-robot interactions in collective settings.

Highlights

In his new book Sound Actions: Conceptualizing Musical Instruments (MIT Press), Alexander Refsum Jensenius writes about relationships between actions and sound and how new technologies are shaping the future of musicking.
The Embodied Perspectives on Musical AI (EmAI) workshop drew around 100 participants in person and several hundreds of online participants for lectures, performances, and thematic sessions on musical artificial intelligence's past, present, and future through the lenses of embodied cognition.
During the NIME conference, research fellow ?a?r? Erdem performed with the artificial musical agent CAVI and RITMO��s self-playing guitars. The performance was the culmination of several years of developing AI-driven systems for musical interaction.

Modelling and Robots

The Modelling and Robots project's core activity is the investigation of rhythm and motion through the design and construction of models and robots. This has applications to understanding natural processes and the creation of technological systems.

Musical Human-Computer Interaction

The core activity of the project Musical Human-Computer Interaction is investigating aspects of rhythm and motion through designing and constructing new interfaces for musical human-computer interaction. This includes studying and designing both acoustic instruments and completely digital systems. We are particularly interested in various types of electroacoustic devices in which we explore the complexity of human motion in musical experience and practice.

Music for stem cells: Aiming to cure diabetes with sound waves

The convergence environment ABINO - Artificial Biomimetic systems �C the Niche of Islet Organoids wants to develop future models for diabetes research by using new strategies for stem cell differentiation. RITMO contributes with a sub-project on how audible sound impacts cell cultures grown in the laboratory.

Predictive and Intuitive Robot Companion

The project Predictive and Intuitive Robot Companion (PIRC) combines sensing across multiple modalities with learned knowledge to predict outcomes and choose the best actions in robotic systems. The project applies machine learning and robotics expertise and collaborates with researchers in cognitive psychology to use recent human prediction models and action decision-making to perception-action loops of intelligent robot assistants.

Collective Robotics

Humans and many other vertebrates contain oscillatory neural networks known as central pattern generators that generate movement patterns. These networks can entrain to rhythms and adapt to social information. However, we still do not understand the underlying mechanisms very well. The project COllective ROBOtics through Real-time Entrainment of Evolved Dynamical Systems (COROBOREES) uses evolutionary methods to create complex dynamical systems to be used as central pattern generators with tunable social responsiveness in legged robots.

Platforms

Dr. Squiggles is an interactive musical robot that plays rhythms by tapping. The RITMO-designed robot listens for tapping produced by humans or other musical robots and attempts to play along and improvise its rhythms based on what it hears.

How does a musical robot��s motion affect how its human musical partners interact with it? Robots, including those that play music, are stereotypically stiff in their movements. Human musicians, by contrast, use a wide variety of expressive actions, gestures, and signals when playing together. To explore this, we built a prototype guitar-plucking robot called Professor Plucky, designed to have visually appealing movement, even at the expense of mechanics.

The Mini Soundbox is a mini acoustic chamber for small-scale tests and experiments.

The self-playing guitars are mechatronic/robotic devices that produce sounds utilizing a classic guitar body's acoustics. Each guitar can sense audio and proximity activity and has a battery-powered onboard processing platform. The devices can thus be deployed in a decentralized, multi-guitar setup.

The ZRob is a robotic platform for exploring AI-based control of drum playing while exploiting the natural dynamics of the drum membrane and the drumstick.

RITMO in Numbers

1 book, 25 journal articles, 25 book chapters, 7 artistic outputs, 2 innovations, 27 media appearances, 109 conference presentations

Reported Research Results 2022

RITMO's researchers have published many scientific articles and book chapters in 2022, as well as one monograph. They have also given talks and presentations at several conferences. The Centre's research results also include several artistic outputs, media contributions and innovations. You can see all of RITMO's results reported in CRIStin.

Icons showing 23 women, 1 non-binary and 33 men

People

A total of 57 people were employed at RITMO in 2022, 23 women, 1 non-binary and 33 men. We work to achieve gender balance in our recruitments.

World map showing the countries RITMO's staff and guests come from.

The World at RITMO

RITMO is genuinely an international working environment. RITMO scholars come from all over the world: Argentina, Australia, Austria, Bangladesh, Brazil, Canada, China, Denmark, Ecuador, Finland, France, Germany, Greece, Iceland, India, Iran, Italy, Japan, Jordan, Mexico, Montenegro, the Netherlands, Norway, Pakistan, Portugal, Romania, South Korea, Spain, Sweden, Turkey, United Kingdom and the United States.

World map showing the countries where RITMO researchers have partners and collaborations.

RITMO in the World

RITMO has research cooperation with partners from across the world: Argentina, Australia, Austria, Belgium, Brazil, Canada, Chile, Colombia, Denmark, Finland, France, Germany, Greece, Iceland, India, Israel, Italy, Japan, Jordan, Lithuania, Mali, Mexico, Netherlands, Poland, Portugal, Romania, Singapore, Spain, Sweden, Switzerland, United Kingdom, United States and Uruguay.

RITMO Highlights

At a research-intensive centre of RITMO's size, numerous things are always happening. In 2022 we were particularly proud of three things.

RITPART International Rhythm Rising Workshop

Running an international partnership project during Corona was challenging. We had to cancel two planned RITPART workshops in Montreal and Toyoashi but fortunately managed to host one final workshop in Oslo. After a successful Berkeley workshop in 2019, we were finally able to invite a group of researchers affiliated with UC Berkeley's Embodied Design Research Laboratory to Oslo.

RITPART

TIME - Timing and Sound in Musical Microrhythm

The project Timing and Sound in Musical Microrhythm (TIME) conducted comparative investigations of four different rhythmic genres �C jazz, electronic dance music, R&B/hip-hop, samba and Scandinavian fiddle music �C in order to gain new insights into the relationship between temporal and sound-related aspects of musical perception and performance. The project ran from 2017 through 2022 and was funded by the Research Council of Norway and the University of Oslo.

TIME

Pupillometry Online Course

RITMO continues developing educational material on research methods. This year, we teamed up with researchers from the Center for Multilingualism in Society across the Lifespan, another Centre of Excellence at UiO. Together, we have developed the course Pupillometry: The Eye as a Window Into the Mind, which will run on the FutureLearn platform with our other courses. Pupillometry is a reasonably new research method with wide-ranging applications within psychology, neuroscience, and beyond. This six-week course teaches pupillometry and eye-tracking techniques before investigating some exciting and unusual applications of this far-reaching research method.

Pupillometry

RITMO Stories

A project is always more than what you can read from a project page. Here, some of RITMO's researchers present their own research.

Remy Martin is a postdoctoral researcher at RITMO studying musical engagement and the ethics of musical experience.

Marieke van Otterdijk is a doctoral fellow at RITMO studying human-robot interactions and social robotics.

Sabine Leske is a postdoctoral researcher at RITMO studying the cognitive mechanisms underlying the ability to predict events.

Henrik Herrebr?den is a doctoral fellow at RITMO studying cognitive mechanisms underlying rhythmical movements and good performance.

Connor Spiech is a doctoral fellow at RITMO studying the neurophysiological mechanisms of groove.

RITMO Seminar Series

The RITMO Seminar Series is a venue for presenting world-leading scholars worldwide. We usually invite our lecturers to stay for a few days to make the most of their visit to RITMO. In 2022, we were still affected by the pandemic, and some of the seminar series were held virtually. These events are open to the public and usually live-streamed, and have attracted participants from many parts of the world.

"Aesthetics of Machine Learning Art and Music in Embodied Agents" Sofian Audry, University of Quebec in Montreal

"Interdisciplinary Research into the Effects of Music: Diversity and Specificity", Katie Overy, University of Edinburgh

"Interpersonal Synchrony in Sound, Brain, and Body" Caroline Palmer, McGill University

"Music in Augmented Reality" Nicola Dibben, University of Sheffield

"Rhythm in contexts", Birgitte Stougaard Pedersen, Aarhus University

Disputations

More and more doctoral fellows from the first part of RITMO's life now begin to complete their dissertations. We were happy to host five successful disputations in 2022:

?a?r? Erdem: Controlling or Being Controlled? Exploring Embodiment, Agency and Artificial Intelligence in Interactive Music Performance
Kjell Andreas Oddekalv: What Makes the Shit Dope? The Techniques and Analysis of Rap Flows
Merve Ak?a: Attending to Sounds In the Blink of An Eye
Qichao Lan: Exploring Collaboration in Computer Music Systems for Live Coding
Ulf Holbrook: Objects and Structures: Aesthetical inquiry and artistic experimentation into the relationships between sound objects and spatial audio

RITMO Behind the Scenes

Research is essential to RITMO, but there is so much more. Here we give a few glimpses into everyday life and internal events of 2022.

Behind-the-scenes

RITMO 2.0

RITMO turned five years in 2022 and we are now entering the second phase.

visualization of the thematic areas of phase two

Throughout 2022, we evaluated the first five years and made plans for the next five. The whole centre was involved in the process, from small cluster discussions to plenary sessions during our annual retreats, Tour de RITMO. We also consulted with our Scientific Advisory Board and had lively discussions in RITMO's Board.

The evaluation resulted in a "sharpening" of our priorities, with a focus on five thematic areas:

Starting on the small scale, in the thematic area of Microrhythm, we are interested in beat perception and groove-related rhythmic processes at the millisecond level of music.
A second theme is the role of rhythm in the processes of Prediction and Attention. Here we focus on more complex rhythmic patterns found in more naturalistic (musical) stimuli.
In the thematic area Time and Pleasure, we address how fragmented rhythmic textures and manipulation of time structures in audiovisual modes of expression (remixes, film) are experienced.
The area Interaction and synchronization is focused on the multimodal interplay between performers, dancers and audiences in various contexts, from the djembe tradition of Mali to the synchronization of heartbeats between musicians in a classical orchestra, to social bonding between people enjoying the same music.
The last thematic area, Entrainment, addresses synchronization processes in biological systems and human-machine and human-environment systems.

Recognition

Doctoral research fellow Dongho Kwak's article Music for Cells? A Systematic Review of Studies Investigating the Effects of Audible Sound Played Through Speaker-Based Systems on Cell Cultures was selected for the anniversary edition of Music & Science by The Society for Education, Music and Psychology Research (SEMPRE).

Researcher Ulysse C?t��-Allard received the 1st Prize in the 2022 IEEE Engineering in Medicine and Biology Prize Paper Award for the article Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning.

RITMO People

Scientific Advisory Board

The members of RITMO's Scientific Advisory Board are highly acclaimed researchers within their respective fields. The Board is vital to providing feedback on on-going and future research, and its members have been carefully chosen to support the different parts of RITMO.