After the startup phase, we had planned that 2020 should be a consolidation year at RITMO. Unfortunately, the pandemic changed all our plans.
Closed labs, cancelled experiments, postponed activities, guests unable to come to Oslo, researchers that could not travel elsewhere. The year 2020 was different for everyone, including all of us working at RITMO.
Fortunately, we had already invested in video conferencing solutions before the pandemic and had a lot of internal know-how in organizing online activities. This meant that we quickly managed to convert most of our activities to online and hybrid formats. Several of RITMO's researchers also turned around to utilise the situation's unique research potential.
Despite our relative success in keeping the centre activities going, we all long for getting back to a more normal situation. Meanwhile, here we present some highlights from 2020, in rhythm, time, and motion.
Anne Danielsen and Alexander Refsum Jensenius
RITMO at a Glance
Rhythm is everywhere, from how we walk, talk, dance and play, to how we tell stories about the past and predict the future.
Our heartbeat, brain waves, and other bodily cycles all work through rhythm. Rhythm is a crucial aspect of human action and perception. Our human rhythm is also in complex interaction with the world's various cultural, biological and mechanical rhythms.
At RITMO, we research rhythmic phenomena and the complex relationships between the human body's rhythms and the brain. The central idea is to establish a link between features of rhythmic phenomena in the world and within the (embodied) mind. The aim is to understand our ability to perceive rhythm and how this affects our actions and experiences.
RITMO is a highly interdisciplinary centre, combining perspectives and methods from music and media studies, philosophy and aesthetics, cognitive neuroscience, and informatics. We employ several state-of-the-art technologies in our research, including motion capture, neuroimaging, pupillometry and robotics.
Structure & Time Cluster
We investigate the structure and aesthetics of musical rhythm and their link to perceptual, cognitive and motor mechanisms. We are also interested in the role of rhythm, temporal structure and timing in the experience of long-lasting musical form and more fragmented, audiovisual musical formats. Overarching questions are how genre familiarity, technology and socio-cultural context interact with perceptual and cognitive mechanisms in the experience of rhythm and time.
Highlights
- Timing one’s playing to a beat is something that most musicians practice. In a series of performance experiments, PhD fellow Guilherme Schmidt C?mara and colleagues showed that musicians also use features other than pure timing. Intensity, timbre and duration are all used to communicate whether a beat is played early, behind or on-the-beat. Studies of instructed timing in guitar performance (JASA) and drumming (Music Perception) support previous work showing that perceived timing at the micro-level of rhythm is closely related to the sound's shape.
- Olivier Lartillot and collaborators have developed a new method for computational analysis of sound (JASA). Previous attempts, based on the reduction of the audio waveform into an envelope curve, decreases its temporal resolution. The proposed approach detects the attack region directly from the audio waveform and shows a significant increase in precision.
- The seminar Muddy Rhythms and Broken Beats gathered a group of scholars to focus on experimental grooves within hip-hop music, neo-soul, urban and electronic dance music. They discussed questions such as: How do producers experiment with rhythm and groove via the use of processing effects, sequencer tools and samples? What is the relationship between interface design and groove aesthetics? And how do listeners make sense of muddy rhythms and broken beats? By identifying “seasick-, fragmented-, cut-up-, staggering and irregular rhythms” as a topic and gathering pioneering scholars within this field, RITMO has laid the ground for a new subfield within rhythm studies.
Musical Microrhythm
The project TIME: Timing and Sound in Musical Microrhythm investigates interactions between temporal and sound-related features at the micro-level of rhythm in musical genres where rhythm is a core dimension. The aim is to gain new insights into the micro-level of auditory perception and the role of cultural background and/or training in music in this regard. This is based on qualitative and quantitative methodologies from musicology, ethnomusicology, music psychology and motion research.
Musical Fragmentation
This project examines the pleasure of musical fragmentation and the related play with time, focusing on sample-based and cut-and-paste music. In the MASHED project, the focus is on the underlying aesthetics and temporality of so-called Mashup music.
Musical Time and Form
The project Musical Time and Form investigates salient differences in musical interpretation between recorded performances of orchestral compositions in Western Art Music from the last two centuries. Compositions are performed differently depending on the conductor, orchestra, historical and cultural tradition, and other factors such as time and place of the performance. There is also a focus on the listening experiences of those performances, intending to improve the precision with which we discuss musical performance and the listening experience.
A Comprehensive AI-based System for Advanced Music Analysis
The aim of the project MIRAGE - A Comprehensive AI-Based System for Advanced Music Analysis is to improve the computers' capability to listen to and understand music. A goal is to develop technologies that facilitate understanding and appreciation of music through generating rich and detailed descriptions. These encompass a broad range of dimensions, such as rhythm, tonality, formal structure, melodic shape, sound colour, and emotions. Significant effort is dedicated to designing applications of value for musicology, music cognition and the general public. One main focus is the study of Norwegian Hardanger fiddle music.
Structure & Cognition Cluster
The goal is to expand our understanding of how rhythm and our sense of time are constructed in the human mind. We study the influence of rhythm and time on perception, attention and effort, imagery and illusions, prediction and cognitive control processes across sensory modalities.
Highlights
- In an article about rhythmic predictability (Scientific Reports), Maja Foldal and colleagues reported on findings from scalp-EEG recordings in healthy adult participants who listened to rhythmic sound stimuli. Participants were instructed to pay attention to either the left or the right ear and ignore the sounds in the other ear. This is the first study to show that the brain predicts the timing of sounds even when they occur in the background and attention is directed elsewhere.
- Using pupillometry and functional MR imaging, RITMO researchers investigated the mental effort of a professional pianist while playing, listening to, and imagining music (Frontiers in Neuroscience). They found that the motor activity of playing music increases mental effort, whereas levels of effort during imagery and listening are almost similar. Also, they discovered that the music's technical difficulty is reflected in activations of the brainstem’s locus coeruleus as well as in pupil diameters. This combined approach of psychophysiology and neuroimaging revealed the mental work going on during different musical activities.
- Researcher Alejandro Blenkmann was awarded funding from the Research Council of Norway to uncover the neurophysiological basis of the brain’s ability to predict upcoming auditory events. To do this, he will be using intracranial EEG and single-neuron recordings. Single-neuron recordings in humans have not previously been conducted in Scandinavia. The findings are expected to expand our understanding of how the brain predicts future events.
Mental effort and cognition
The Mental Effort and Cognition project investigates the use or consumption of attentional resources while carrying out a cognitive task. Mental effort can be measured behaviorally, subjectively and physiologically. Our approach focuses on a) the psychophysiological index provided by pupillometry, a well-established method for measuring physiological arousal, and b) brain activity with fMRI. We are, among other things, interested in the inherent rhythmicity of attention, effort related to different forms of temporal violation of regularity, and the effort used when processing information in different sensory modalities.
Multimodal perception of rhythm, time and illusory time
In the project Multimodal perception of rhythm, time and illusory time we try to understand how the brain keeps track of and makes meaning out of temporal information to produce stable percepts of the world. We use illusions (the Risset illusion and the flash-lag effect) to reveal how predictions can influence the sensory system and override the characteristics of actual physical stimuli. We also study how the brain makes sense of regular accelerations in tempo compared to irregular changes.
The neural basis of temporal prediction.
A primary goal of the project The neural basis of temoral prediction is to understand the neurocognitive basis of predictive processing in the human brain with special attention to auditory stimuli and rhythm. Using scalp and intracranial EEG, eye tracking and pupillometry, we aim to decipher the neural mechanisms involved.
Rhythm as an individual ability
The goal of the project Rhythm as an individual ability is to understand how rhythmic abilities vary between individuals. To do this we investigate the basic cognitive, environmental and genetic factors that shape rhythmicity. We are developing a test battery to cover a broad area of rhythm topics, such as, meter, beat, and microtiming.
Interaction & Pleasure Cluster
We seek to understand why rhythms make us move and how rhythm facilitates entrainment and interaction. We also explore the affective and social outcomes of such processes. These are studied through the experience of absorption in musical performance and perception, music-related effort and skilled action, and the dynamics of music-evoked pleasure.
Highlights
- PhD fellow Agata Zelechowska and colleagues published several papers on human, music-related micromotion. It was shown that people move more when listening to music on headphones than on speakers (Frontiers in Psychology). They also found that people's empathic concern can predict spontaneous movement responses to music (Music & Science).
- The Oslo Standstill Database was released, containing motion capture data of more than 600 people standing still in silence or listening to music. The database marks RITMO's investment in developing Open Science practices and is a resource for other researchers interested in human, music-related micromotion.
- When reconsidering her research activities in March 2020, PhD fellow Dana Swarbrick turned around and set up the Quarantine Concerts project in collaboration with several partners in the music industry. The aim is to explore the social and emotional effects of virtual concerts. The unique data that has been collected will help to understand more about “liveness” in online concerts and how performers can enhance engagement with online audiences.
Entrainment, Social Bonding and Pleasure
The project Entrainment, Social Bonding and Pleasure aims to advance our understanding of how entrainment, synchronised behaviour and other related processes lead to prosocial outcomes such as social bonding. Work within the project seeks to uncover the factors contributing to synchronous movement and social bonding in concerts and explores whether entrained listening can evoke affiliation towards fellow listeners, the musicians, or their social groups. The sub-project Quarantine Concerts project aims to explore the social and emotional effects of virtual concerts.
Effort and Attention in Musical Performance and Perception
Musical experiences often provide sensations of affect and pleasure and sensations of sound-producing and sound-accompanying body motion. The project Effort and Attention in Musical Performance and Perception aims to enhance our understanding of the attention and effort, both physical and mental, involved in producing and perceiving music-related sound and related multimodal phenomena.
Engagement and Absorption
Does time and timing matter for musical engagement and absorption? Music is well known for its ability to engage, yet the nature of this engagement remains elusive. The project Engagement and Absorption explores aspects of rhythm, temporality, and meaning in musical engagement, such as listening and performing. The project works from various research angles, including musical sense-making and ethics, absorption in musical performance, spatiality and immersion, and relational interaction.
Human Music-related Micromotion
The project MICRO - Human Bodily Micromotion in Music Perception and Interaction investigates how music influences micromotion, the smallest bodily motion that it is possible to produce and perceive. Data is collected using motion capture and physiological measurements. One of the annual highlights is the Norwegian Championship of Standstill, which is both important for data collection and serves as a dissemination activity.
Musical Chills
The Musical Chills project investigates the brain processes and musical triggers underlying the intense, pleasurable ‘chills’ responses sometimes evoked by music. It explores the role of other neuromodulators (particularly the endogenous opioid system), via pharmacological agents, in the pleasurable response to music. Moreover, the project seeks to investigate what specific musical features can trigger chills.
Interaction & Robotics Cluster
We investigate aspects of rhythm and motion through robotics and technology. This includes creating interactive music systems with a focus on multi-dimensional mapping from sensors to musical parameters. Moreover, we design and prototype adaptive robot systems and explore both human-robot and robot-robot interactions in collective settings.
Highlights
-
Postdoctoral fellow Charles Martin and Professor Jim T?rresen performed a data-driven analysis on more than 1600 musical touchscreen performances (Computer Music Journal). This was based on MicroJam, a social media-inspired smartphone app. Their work on AI-powered musical interaction was also reported in a Frontiers in AI article.
-
The Musical Gestures Toolbox is a software collection for creating video visualisations. It was primarily developed for music research, but is also used for sports, dance, healthcare, architecture, and interaction design. Alexander Refsum Jensenius and colleagues have worked on a Python implementation of the toolbox, which was released in 2020.
-
Professor Jim T?rresen received funding for the new project PIRC - Predictive and Intuitive Robot Companion from the Research Council of Norway. The project targets a psychology-inspired computing breakthrough by combining insight from cognitive psychology with computational intelligence to build models that forecast future events and respond dynamically.
Modelling and Robots
The Modelling and Robots project's core activity is the investigation of rhythm and motion through the design and construction of models and robots. This has applications to understanding natural processes as well as for the creation of technological systems.
Musical Human-Computer Interaction
The core activity of the project Musical Human-Computer Interaction is investigating aspects of rhythm and motion through the design and construction of interfaces for musical human-computer interaction. This includes the study and design of both acoustic instruments and completely digital systems. We are particularly interested in various types of electroacoustic instruments, in which we explore the complexity of human motion in musical experience and practice.
PIRC - Predictive and Intuitive Robot Companion
Humans anticipate future events more effectively than computers. The Predictive and Intuitive Robot Companion project combines sensing across multiple modalities with learned knowledge to predict outcomes and choose the best actions. The project will apply machine learning and robotics expertise and collaborate with researchers in cognitive psychology to apply recent human prediction models and action decision-making to perception-action loops of intelligent robot assistants. The work will allow such robots to adapt and act more seamlessly with their environment than the current technology.
Self-playing Guitars
The self-playing guitars are mechatronic/robotic devices capable of producing sounds utilising a classic guitar body's acoustics. Each guitar can sense audio and proximity activity and has a battery-powered onboard processing platform. The devices can thus be deployed in a decentralized, multi-guitar setup.
Dr. Squiggles
Dr. Squiggles is an interactive musical robot that plays rhythms by tapping. The RITMO-designed robot listens for tapping produced by humans or other musical robots and attempts to play along and improvise its rhythms based on what it hears.
RITMO in Numbers
Reported Research Results 2020
RITMO's researchers have published many scientific articles and book chapters in 2020, as well as two anthologies. They have also given talks and presentations at several conferences, although the numbers are lower due to the pandemic. The Centre's research results also include several artistic outputs and media contributions. You can see all of RITMO's results reported in CRIStin.
People
A total of 62 people were employed at RITMO in 2020, of which there were 26 women and 36 men. We work to achieve gender balance in our recruitments.
The World at RITMO
RITMO is genuinely an international working environment. RITMO scholars come from all over the world: Argentina, Australia, Canada, China, Colombia, Denmark, Finland, France, Germany, Greece, Hungary, India, Iran, Italy, Japan, Jordan, Mexico, Montenegro, Norway, Poland, Romania, Serbia, South Korea, Spain, Sweden, Turkey, United Arab Emirates, United Kingdom and the United States.
RITMO in the World
RITMO has research cooperation with partners from across the world: Argentina, Australia, Austria, Belgium, Brazil, Canada, Chile, Denmark, Finland, France, Germany, Iceland, India, Italy, Japan, Jordan, Netherlands, Poland, Sweden, Switzerland, United Kingdom and the United States.
RITMO Highlights
At a centre with around 60 employees, there are always numerous things happening. In 2020 we were particularly proud of three things.
Muddy Rhythms and Broken Beats
In February 2020, RITMO organized the seminar Muddy Rhythms and Broken Beats focusing on experimental grooves. Such experimental grooves may include seasick rhythms, fragmented or cut-up rhythms, staggering rhythms, pushed and dragged rhythms, syncopated rhythms, or various forms of rhythmic irregularities.
Dr. Squiggles
Postdoctoral researcher Michael Krzyzaniak has developed Dr. Squiggles, an interactive musical robot that listens to rhythms and plays along by tapping on the table.
Quarantine research
RITMO's researchers had to be innovative when faced with closed labs and constantly changing Corona regulations.
.jpg&display=gallery)
RITMO Stories
A project is always more than what you can read from a project page. Here some of RITMO's researchers present their own research.
Georgios Sioros is a postdoctoral researcher at RITMO studying musical rhythm.
Agata Zelechowska is a doctoral fellow at RITMO, studying human music-related micromotion.
Alejandro Blenkmann is a researcher at RITMO, studying attention and prediction.
Dana Swarbrick is a doctoral fellow at RITMO, studying entrainment and pleasure.
RITMO Seminar Series
The RITMO Seminar Series is a venue for presenting world-leading scholars from around the world. These events were also affected by the current situation, and there were fewer seminars organised this year.
Disputation
PhD fellow Agata Zelechowska successfully defended her dissertation Irresistible Movement: The Role of Musical Sound, Individual Differences and Listening Context in Movement Responses to Music in an online disputation.
RITMO Behind the Scenes
The year turned out to be very different than everyone had expected. Here are some snapshots of physical and online activities during 2020.
.jpg&display=gallery)
Data management
Data management was an important centre topic in 2020. Many of RITMO's researchers collect or generate data, and it is pivotal that these data are handled well.
RITMO's labs are run by two lab engineers, who also act as data managers. The data manager role is new at the University of Oslo, and RITMO has been involved in several university-wide initiatives to develop data handling policies and procedures. This includes:
- Developing storage solutions that are both secure and accessible. In collaboration with UiO's Center for Information Technology, we have set up a virtual desktop infrastructure (VDI) that allows all researchers to access and process data securely. This turned out to be crucial once everyone had to work from home. Even without access to labs, RITMO researchers could access their data and utilize processing resources. Since many of the researchers work with heavy processes, and many work with video files, it was beneficial to have hardware-accelerated processing (GPUs) on the virtual machines.
- Leading the cross-university Human Time Data network project, aiming at connecting research groups working on similar types of human time-series data. There are many differences and commonalities between data coming from motion capture systems, physiological sensors, eye trackers, and various types of brain imaging. The network has identified best practices and suggested future data management procedures.
- Developing a data management handbook, with guidance for researchers on the steps necessary to collect, store, and archive data. There are many policy documents and generic tools available. However, in our experience, researchers need more concrete advice targeted at the applications at hand.
- As part of the Open Science innovation project MusicLab, we have worked with the University Library on handling ethical and legal issues related to data collection in public spaces. This includes developing solutions for handling consent forms concerning privacy issues according to the GDPR. We have also started discussions with relevant actors when it comes to recording and sharing copyrighted musical material.
While we did a lot in 2020, there are still many unresolved matters. So data management will surely continue to be an important development topic in the years ahead.
Awards
Two RITMO papers received awards at international conferences:
- The article Towards Playing in the 'Air': Modeling Motion-Sound Energy Relationships in Electric Guitar Performance Using Deep Neural Networks by Cagri Erdem, Julian Fuhrer, Charles Martin, Jim T?rresen, and Alexander Refsum Jensenius received the best paper award at the 17th Sound and Music Computing Conference.
- The article “Standstill to the ‘beat’: Differences in involuntary movement responses to simple and complex rhythms” by Agata Zelechowska, Victor Gonzalez Sanchez and Alexander Refsum Jensenius received the best paper award at the Audio Mostly Conference.
RITMO People
Scientific Advisory Board
The members of RITMO's Scientific Advisory Board are highly acclaimed researchers within their respective fields. The Board is vital to providing feedback on on-going and future research, and its members have been carefully chosen to support the different parts of RITMO.
Board
Management
Postdoctoral Fellows and Researchers
PhD Fellows
Assistants
Technical/Administrative Staff
Guest researchers
- Cancino-Chacón, Carlos Eduardo (Austrian Research Institute for Artificial Intelligence, Austria)
- Milojevic, Andrija (Lut University, Finland)
- Minami, Tetsuto (Toyohashi University of Technology, Japan)
- Murayama, Rintaro (Toyohashi University of Technology, Japan)
- Upham, Finn (McGill University, Canada)
A Centre of Excellence in Norway
A collaboration between
