Annual Report 2023

Structure & Aesthetics Cluster

We investigate the structure and aesthetics of musical rhythm and time. We are interested in rhythmical experiences and practices and how these can provide insights into perceptual processes, meaning-making, and sociocultural contexts. We examine the role of rhythm, temporal structure, and timing in the experience of music, both in terms of its aesthetic dimension and the cognitive processes underpinning the experience.

Highlights

• In Parody in the Age of Remix: Mashup Creativity vs. the Takedown (MIT Press), Ragnhild Br?vig delves into the practice of making music by juxtaposing fragments from different times and spaces into a coherent piece. The book discusses the producers¡¯ motivations and the listening effects of experiencing a piece as both new and old, coherent and fragmented. It also addresses current institutional assumptions about creativity, showing an evident bias towards music consisting of a collage of prior music. This monograph is one of the key contributions of the Musical Fragmentation project.

• Through music analysis and interviews with musicians and producers, Anne Danielsen and colleagues explored how sounds are shaped at the micro level of rhythm in five groove-based musical genres. The study documents the multiparameter nature of musical performance and experience, highlighting genre-typical temporal and sonic features configurations. The article summarises the qualitative research in the TIME project, adding a focus on sound to the often time-oriented scholarly discourse on groove.

• In a motion capture study by Mari Romarheim Haugen and colleagues, skilled guitarists and bassists were instructed to perform under three different timing-style conditions. The results show that performers generally leaned forward when instructed to play slightly ahead of the pulse. These findings support the view that musicians¡¯ non-sound-producing body movements are closely related to the music¡¯s microrhythmic feel. 

• In an article on expressive microtiming in music, Danielsen, Johansson, and Stover critique traditional music theory's conceptualisation of beats as points in time. They present three alternative theories that were designed to engage a particular musical ¨C beat bins for Neo-Soul, hip-hop, and other contemporary Black musical expressions, beat span for ¡°timeline musics¡± from Africa and the African diaspora, and rhythmic tolerance for Scandinavian fiddle music ¨C and consider how the three theories can productively coexist in a shared music-analytic space.

A Comprehensive AI-based System for Advanced Music Analysis

The project MIRAGE - A Comprehensive AI-Based System for Advanced Music Analysis aims to improve computers' capability to listen to and understand music. A goal is to develop technologies that facilitate understanding and appreciation of music by generating rich and detailed descriptions. These encompass various dimensions, such as rhythm, tonality, formal structure, melodic shape, sound colour, and emotions. An effort is dedicated to designing applications for musicology, music cognition, and the general public. One main focus is the study of Norwegian Hardanger fiddle music. 

Musical Fragmentation

The project Musical Fragmentation investigates the aesthetics and pleasures of musical rhythms and grooves experienced as fragmented or related to machines. The focus is sample-based music, cut-and-paste and glitch aesthetics, and other experimental music expressions. The aim is to explore how listeners make meaning out of musical fragmentation and machine rhythms and how these experiences are related to cognitive processes.

Musical Microrhythm

The project TIME: Timing and Sound in Musical Microrhythm investigates interactions between temporal and sound-related features at the micro-level of rhythm in musical genres where rhythm is a core dimension. The aim is to gain new insights into the micro-level of auditory perception and the role of cultural background and/or training in music in this regard. This is based on qualitative and quantitative methodologies from musicology, ethnomusicology, music psychology, and motion research.

Musical Time and Form

The project Musical Time and Form investigates rhythm, meter, and expressivity in classical-romantic Western Art Music, focusing in particular on the music of Franz Schubert. The aim is to better understand the under-investigated rhythmic qualities of this repertoire, where the analytical focus has mainly been harmony and form.

Multimodal rhythm in music and dance from West Africa

The project DjembeDance ¨C Multimodal rhythm in music and dance from West Africa studies rhythm in djembe drumming and dance from Mali. The aim is to understand more about rhythm as a multimodal phenomenon. The collaborative project builds on a large dataset of live music and dance performance multimedia recordings, combining multitrack audio, multi-camera video, and 3D motion tracking data.

Structure & Cognition Cluster

The goal is to expand our understanding of how rhythm and sense of time are constructed in the human mind. We study the influence of rhythm and time on perception, attention and effort, imagery and illusions, prediction and cognitive control processes across sensory modalities.

Highlights

• Information theory is a robust framework for studying brain processes. However, its use is uncommon and comparisons with more classical approaches are rare. By combining intracranial EEG recordings from humans and marmoset monkeys, Fuhrer et al. compared the applicability of the "Encoded Information" metric with more well-established alternatives. Encoded Information emerged as a novel and powerful procedure to assess the similarity of brain responses across experimental task conditions.

• Traditionally, attention has been conceptualized as a constant spotlight that amplifies behaviorally-relevant information. A new study challenges this view. Examining attention on a fine-grained temporal scale with EEG revealed that fluctuations in task performance closely aligned with rhythmic brain oscillations. This indicates that attention is a discrete process that alternates between periods of enhanced or reduced attention. Importantly, patients with focal lesions to the frontal-parietal brain network had time-specific behavioural deficits tied to their brain rhythms. The findings provide causal evidence that the functional architecture of attention is inherently rhythmic.

• The FlashLag illusion is the inability of the brain to perceive simultaneity between a predictable object motion (along a path and at constant velocity) and another visual event that is instead unpredictable (e.g., appearing suddenly). Using lateralized stimulus presentation and pupillometry, a study revealed the hemispheric specializations behind the perception of illusory motion and judgments of timing and synchrony in the visual modality.

• Little is known about the mechanisms by which the brain simultaneously encodes task-relevant variables while minimizing interference from task-irrelevant features. In a study using intracranial EEG recordings, we demonstrate that competition between coexisting representations of past and present task features incurs a "switch cost" observed in behaviour. Our results shed light on the mechanisms by which such behavioural switch costs are diminished. These findings uncover a fundamental coding mechanism underlying flexible cognitive control.

• A comprehensive methodological review introduces and discusses novel, advanced techniques of time series analyses in psychophysiological measurements. Particular attention is given to eye-pupil measurements while processing extended stimuli that continuously require adjustments in attentive resources, for example when listening to an engaging piece of music. Pupillometry is a primary method used by several RITMO researchers in various studies and applications.

Mental Effort

The Mental Effort project investigates the engagement of attentional resources while carrying out a cognitive task. Mental effort can be measured behaviorally, subjectively and physiologically. Our approach focuses on the psychophysiological index provided by pupillometry and brain activity with fMRI. We are particularly interested in the inherent rhythmicity of attention, the effort related to various forms of temporal violation of regularity, and the effort used when processing information in different sensory modalities.

The Neural Basis of Temporal Prediction

A primary goal of The neural basis of temporal prediction project is to understand the neurocognitive basis of predictive processing in the human brain with particular attention to auditory stimuli and rhythm. We aim to decipher the neural mechanisms involved by using scalp and intracranial EEG, eye tracking and pupillometry concomitantly with cognitive tasks performed by healthy participants and individuals with brain disorders.

Neurophysiological Mechanisms of Human Auditory Predictions

The ability to predict upcoming events is a core feature of human cognition. In a broad sense, predictive processes incorporate knowledge from the past to predict future states of the body and the environment, shaping how we perceive the world. The project AudioPred - Neurophysiological Mechanisms of Human Auditory Predictions: From population- to single neuron recordings utilizes the unprecedented timing and anatomical detail offered by intracranial EEG recordings to uncover how auditory predictive processes are computed in the human brain.

Rhythm as an Individual Ability

Humans have an exceptional ability to perceive and produce rhythm, but rhythmic skills can differ substantially between people. The goal of the project Rhythm as an individual ability (INDRA) is to understand how and why rhythmic capabilities vary. To accomplish this, we investigate the basic cognitive, environmental, and genetic factors that shape rhythmicity. We are developing a test battery to cover various rhythm features, such as meter, beat, and microtiming.

Interaction & Pleasure Cluster

We seek to understand why rhythms make us move and how rhythm facilitates entrainment and interaction. We also explore the affective and social outcomes of musical and rhythmic processes. These phenomena are studied through the experience of absorption in musical performance and perception, music-related effort and skilled action, and the dynamics of music-evoked pleasure and social bonding.

Highlights

• In October 2021, a large team of RITMO researchers went to Copenhagen to run MusicLab Copenhagen, a unique concert experiment featuring the Danish String Quartet. The experiment only took a day, but the data has been under active research ever since. On the day two years after the event, we launched a Special Collection in Music & Science, with ten articles shedding light on different facets of the concert.

• Live concerts normally involve gathering at the same time and place. Participants may gather in time but not in space in live-streamed concerts, providing a natural comparison for studying live concert experiences. Swarbrick and Vuoskoski studied both live and live-streaming audiences¡¯ emotions during a concert with the Danish String Quartet. The data show that the classical concert audience actively engages with the music and the associated socioemotional experience based on genre-specific norms and expectations.

• Resilience to changes in performance environments is a hallmark of expertise in music performance. Bishop and colleagues have conducted an experiment with The Danish String Quartet (DSQ), playing with and without visual contact. The results show that the DSQ maintained the quality of their musical sound and interactive body motion across disruptive and non-disruptive conditions. However, mental effort (indexed by pupil size) was greater in non-disruptive conditions. In comparison, a student quartet moved less overall than the DSQ, moved less when they could not see each other and did not show differences in pupil size across conditions.

• There has been a belief that musical absorption includes processes of both focused attention and mind wandering. H?ffding, Nielsen, and Laeng examine this paradox, integrating concepts from cognitive psychology with phenomenology. They propose the concept of ¡°mind surfing¡± to explain how one can be both focused and freely ¡°surfing¡± on a ¡°musical wave¡± at the same time. The discussion includes consideration of qualitative and phenomenological work conducted with the Danish String Quartet.'

Bodies in Concert

Concerts bring audiences and musicians together in time, in place, and in music. Bodies in Concert is a project exploring the movement and physiology of concert participants through various technologies adapted to the study of the musical experience.

Engagement and Absorption

Do time and timing matter for musical engagement and absorption? Music is well known for its ability to engage, yet the nature of this engagement remains elusive. The project Engagement and Absorption explores rhythm, temporality, and meaning in musical engagement, such as listening and performing. The project encompasses various research angles, including musical sense-making and ethics, absorption in musical performance, spatiality and immersion, and social interaction.

Entrainment, Social Bonding and Pleasure

The project Entrainment, Social Bonding and Pleasure aims to advance our understanding of how entrainment, synchronised behaviour and other related processes lead to prosocial outcomes such as social bonding. The project seeks to explore the role of empathy in interpersonal synchronisation and uncover the factors contributing to synchronous movement and social bonding in concerts. It also explores whether entrained listening can evoke affiliation towards fellow listeners, musicians, or social groups. The sub-project Quarantine Concerts project aims to explore virtual concerts' social and emotional effects.

Bodily Entrainment to Audiovisual Rhythms

Much focus has been devoted to understanding the "foreground" of human activities, such as the things we say, our actions, and the sounds we hear. The AMBIENT project will study the sonic and visual "background" of indoor environments, such as the sound of a ventilation system in an office, the footsteps of people in a corridor, or people fidgeting in a classroom. The project focuses on how different auditory and visual stimuli combine to create rhythms in various environments and how such rhythms influence people's bodily behaviours and feelings.

Musical Chills

The Musical Chills project investigates the brain processes and musical triggers underlying the intense, pleasurable 'chills' responses that music evokes. It explores the role of other neuromodulators (particularly the endogenous opioid system), via pharmacological agents, in the pleasurable response to music. Moreover, the project investigates what specific musical features can trigger chills.

MusicLab: Researching real-life concerts

MusicLab is an innovation project by RITMO and the University Library. The aim is to explore new methods for conducting research, research communication and education. Each MusicLab event is organised around a concert in a public venue, which is also the study object. The events also contain an edutainment element through panel discussions with world-leading researchers and artists and "data jockeying" in the form of live data analysis of recorded data.

Interaction & Robotics Cluster

We investigate aspects of rhythm and motion through robotics and technology. This includes creating interactive music systems focusing on multi-dimensional mapping from sensors to musical parameters. Moreover, we design and prototype adaptive robot systems and explore human-robot and robot-robot interactions in collective settings.

Highlights

• For a robot to be both autonomous and collaborative, it requires the ability to adapt its movement to various external stimuli. Typically, legged robots have oscillation periods explicitly defined as a control parameter, limiting the adaptability of walking gaits. Alexander Szorkovszky and colleagues demonstrate a virtual quadruped robot employing a bio-inspired central pattern generator that can spontaneously synchronize its movement to a range of rhythmic stimuli. This can be used to facilitate coordinated movement despite differences in morphology and to learn new movement patterns.
• Dongho Kwak and colleagues present an innovative biological experimental setup. The cells were stimulated with four types of vibration, with and without a regular pulse. Accelerometers and fluorescence microscopy were used to analyse the cellular motion. The findings indicate a consistent trend across all vibrated cell cultures, revealing the filaments' size reduction and altered orientation. This study paves the way for future investigations and provides a compelling framework for comprehending the intricate cellular responses to rhythmic mechanical stimulation.
• The co-adaptive audiovisual instrument CAVI uses deep learning to generate control signals for sound-processing modules based on muscle and motion data of a performer's actions. ?a?r? Erdem and colleagues report on an evaluation of CAVI after it had been used in two public performances. The findings suggest that the perceived agency changes throughout a performance. Vital was the balance between surprise and familiarity.
• The development of compact and energy-efficient wearable sensors has increased the availability of biosignals. Knut Strommen and colleagues propose a novel unsupervised segmentation algorithm. The approach can accurately segment time series data in real-time, making it well-suited for applications where timely change detection is critical. It performs equally or better than other change-point detection algorithms in offline and real-time settings.

Modelling and Robots

The Modelling and Robots project's core activity is the investigation of rhythm and motion through the design and construction of models and robots. This has applications to understanding natural processes and the creation of technological systems.

Musical Human-Computer Interaction

The core activity of the project Musical Human-Computer Interaction is investigating aspects of rhythm and motion through designing and constructing new interfaces for musical human-computer interaction. This includes studying and designing both acoustic instruments and completely digital systems. We are particularly interested in various types of electroacoustic devices in which we explore the complexity of human motion in musical experience and practice.

Music for stem cells: Aiming to cure diabetes with sound waves

The convergence environment ABINO - Artificial Biomimetic systems ¨C the Niche of Islet Organoids wants to develop future models for diabetes research by using new strategies for stem cell differentiation. RITMO contributes with a sub-project on how audible sound impacts cell cultures grown in the laboratory.

Predictive and Intuitive Robot Companion

The project Predictive and Intuitive Robot Companion (PIRC) combines sensing across multiple modalities with learned knowledge to predict outcomes and choose the best actions in robotic systems. The project applies machine learning and robotics expertise and collaborates with researchers in cognitive psychology to use recent human prediction models and action decision-making to perception-action loops of intelligent robot assistants.

Collective Robotics

Humans and many other vertebrates contain oscillatory neural networks known as central pattern generators that generate movement patterns. These networks can entrain to rhythms and adapt to social information. However, we still do not understand the underlying mechanisms very well. The project COllective ROBOtics through Real-time Entrainment of Evolved Dynamical Systems (COROBOREES) uses evolutionary methods to create complex dynamical systems to be used as central pattern generators with tunable social responsiveness in legged robots.

Integrated technologies for tracking organoid morphogenesis

There is a significant need for developing reliable human organ representations (termed organoids) for drug development, personalized drug testing, and in the longer run organ transplantations. Integrated technologies for tracking organoid morphogenesis (ITOM) is a UiO:Life Science convergence environment project focused on advancing imaging technology and data analysis to improve stem cell-derived organ representations.

AUTORHYTHM ¨C the role of autophagy in healthy aging

Autophagy is the natural degradation of a cell that removes unnecessary or dysfunctional components through a regulated mechanism. AUTORHYTHM ¨C the role of autophagy in healthy aging is a UiO:Life Science convergence environment project studying the recycling system of our cells in time and space to improve people's health and life span.

Platforms

The self-playing guitars are mechatronic/robotic devices that produce sounds utilizing the acoustics of a classic guitar body. Each guitar can sense audio and proximity activity and has a battery-powered onboard processing platform. Thus, the devices can be deployed in a decentralized, multi-guitar setup.

The ZRob is a robotic platform for exploring AI-based control of drum playing while exploiting the natural dynamics of the drum membrane and the drumstick.

Reported Research Results 2023

RITMO's researchers have published many scientific articles and book chapters in 2023, as well as one monograph. They have also given talks and presentations at several conferences. The Centre's research results also include several artistic outputs, media contributions and innovations. You can see all of RITMO's results reported in CRIStin.

People

A total of 60 people were employed at RITMO in 2023, 28 women, 1 non-binary and 31 men. We work to achieve gender balance in our recruitments.

The World at RITMO

RITMO is genuinely an international working environment. RITMO scholars come from all over the world: Argentina, Australia, Austria, Brazil, Canada, China, Denmark, Ecuador, Finland, France, Germany, Greece, Hungary, Iceland, India, Iran, Italy, Jordan, Mexico, the Netherlands, Norway, Pakistan, South Korea, Spain, Sweden, Turkey, United Kingdom and the United States.

RITMO in the World

RITMO has research cooperation with partners from across the world: Argentina, Australia, Austria, Belgium, Brazil, Canada, China, Denmark, Estonia, Finland, France, Germany, Greece, Iceland, India, Israel, Italy, Japan, Jordan, Lithuania, Mali, Mexico, Netherlands, Poland, Portugal, Singapore, Spain, Sweden, Switzerland, United Kingdom, and United States.