Another year with partial lock-down has passed. At RITMO, 2021 was a consolidation year during which we managed to get back to a more steady rhythm.
The year started with a complete lock-down. Planned experiments had to be cancelled again, meetings postponed, and seminars converted to a hybrid format. Fortunately, restrictions were lifted during the spring, making it possible to get back to more normal operations.
Research-wise, we now start to see the fruits of our efforts to bring the various researchers and disciplines at the centre together. Serendipitous encounters at the coffee machine lead to new experiments. Lively seminar discussions end up as multi-authored manuscripts. Virtual visits lead to new project ideas.
It is fascinating to see the creativity and enthusiasm among our researchers and visitors. Here we present some highlights from 2021; in rhythm, time, and motion.
Anne Danielsen and Alexander Refsum Jensenius
RITMO at a Glance
Rhythm is everywhere, from how we walk, talk, dance and play, to how we tell stories about the past and predict the future.
Our heartbeat, brain waves, and other bodily cycles all work through rhythm. Rhythm is a crucial aspect of human action and perception. Our human rhythm is also in complex interaction with the world's various cultural, biological and mechanical rhythms.
At RITMO, we research rhythmic phenomena and the complex relationships between the human body's rhythms and the brain. The central idea is to establish a link between features of rhythmic phenomena in the world and within the (embodied) mind. The aim is to understand our ability to perceive rhythm and how this affects our actions and experiences.
RITMO is a highly interdisciplinary centre, combining perspectives and methods from music and media studies, philosophy and aesthetics, cognitive neuroscience, and informatics. We employ several state-of-the-art technologies in our research, including motion capture, neuroimaging, pupillometry and robotics.
Structure & Time Cluster
We investigate the structure and aesthetics of musical rhythm and their link to perceptual, cognitive and motor mechanisms. We are also interested in the role of rhythm, temporal structure, and timing in the experience of long-lasting musical form and more fragmented, audiovisual musical formats. Overarching questions are how genre familiarity, technology and socio-cultural context interact with perceptual and cognitive mechanisms in the experience of rhythm and time.
Highlights
- In the opening article of the Journal of Music Theory's special issue on music and dance, Mari Romarheim Haugen argues that dance and other genre-typical movements are integral to understanding musical rhythm. The study advocates a holistic view of music and dance, the importance of insider experience, and the role of embodied experience in guiding our understanding of music.
- New research from the TIME project, published in Attention, Perception and Psychophysics, shows that jazz musicians, Scandinavian folk fiddlers and hip-hop/EDM producers hear the timing of sounds differently. This indicates that expertise in a specific musical genre affects our basic perceptions of sounds. The study provides new insights into the effects of active long-term musical enculturation on timing perception and synchronization. The results shed light on the question of how nature and nurture intersect in the development of our perceptual systems.
- The MIRAGE project has released a Hardanger fiddle dataset with recordings of polyphonic performances spanning five different emotional expressions. The recordings have been annotated by the performers and these scores have been used to develop software to make it easier to analyze, understand, and appreciate the music. The project also organized the first MIRAGE symposium in June 2021, with participants from all over the world. The symposium focused particularly on strengthening the dialogue between computer science and musicology.
- The MASHED project has investigated music that juxtaposes musical excerpts into a new coherent whole (so-called "mashups"). The main focus has been to understand more about the "dual plane experience" of such mashup music; it is experienced as something old and new at the same time. The project also investigated mashups' legal ramifications and how content moderation affects producers' creative decisions and distribution strategies (published in New Media and Society).
A Comprehensive AI-based System for Advanced Music Analysis
The aim of the project MIRAGE - A Comprehensive AI-Based System for Advanced Music Analysis is to improve the computers' capability to listen to and understand music. A goal is to develop technologies that facilitate understanding and appreciation of music through generating rich and detailed descriptions. These encompass a broad range of dimensions, such as rhythm, tonality, formal structure, melodic shape, sound colour, and emotions. Significant effort is dedicated to designing applications of value for musicology, music cognition and the general public. One main focus is the study of Norwegian Hardanger fiddle music.
Musical Fragmentation
This project examines the pleasure of musical fragmentation and the related play with time, focusing on sample-based and cut-and-paste music. In the MASHED project, the focus is on the underlying aesthetics and temporality of so-called Mashup music.
Musical Microrhythm
The project TIME: Timing and Sound in Musical Microrhythm investigates interactions between temporal and sound-related features at the micro-level of rhythm in musical genres where rhythm is a core dimension. The aim is to gain new insights into the micro-level of auditory perception and the role of cultural background and/or training in music in this regard. This is based on qualitative and quantitative methodologies from musicology, ethnomusicology, music psychology and motion research.
Musical Time and Form
The project Musical Time and Form investigates salient differences in musical interpretation between recorded performances of orchestral compositions in Western Art Music from the last two centuries. Compositions are performed differently depending on the conductor, orchestra, historical and cultural tradition, and other factors such as time and place of the performance. There is also a focus on the listening experiences of those performances, intending to improve the precision with which we discuss musical performance and the listening experience.
Structure & Cognition Cluster
The goal is to expand our understanding of how rhythm and our sense of time are constructed in the human mind. We study the influence of rhythm and time on perception, attention and effort, imagery and illusions, prediction and cognitive control processes across sensory modalities.
Highlights
- Anne-Kristin Solbakk was the guest editor of the special issue Prefrontal cortex and executive functions in the Journal of Cognitive Neuroscience, published in honour of Donald T. Stuss. The special issue focused on how anticipation, monitoring, and evaluation of the outcome of one's actions are at the core of proactive control. Individuals with lesions to the orbitofrontal cortex often demonstrate behaviours that indicate a lack of recognition or concern for the negative effects of their actions. Altered action timing has also been reported in these patients.
- In one of the articles in the special issue, entitled "Monitoring of self-paced action timing and sensory outcomes after lesions to the orbitofrontal cortex" Solbakk and colleagues reported EEG and behavioural data that show how damage to a part of the frontal lobes (orbitofrontal cortex) interferes with the ability to control the timing of actions and monitor whether the outcome corresponds to the expected.
- Former RITMO postdoctoral research fellow Anais Llorens is the first author of a Neuron article that addresses the ways in which gender bias negatively impacts careers, work-life balance, and mental health of underrepresented groups in neuroscience. The author list includes 36 researchers from leading research institutions within the field, among them Anne-Kristin Solbakk and RITMO Scientific Advisory Board member and collaborator Robert Knight (UC Berkeley). In the article, they disentangle the complexity of gender bias and propose concrete solutions that can be adopted by individuals, academic institutions, and society.
Mental Effort
The Mental Effort project investigates the use or consumption of attentional resources while carrying out a cognitive task. Mental effort can be measured behaviorally, subjectively and physiologically. Our approach focuses on the psychophysiological index provided by pupillometry and brain activity with fMRI. We are, among other things, interested in the inherent rhythmicity of attention, effort related to different forms of temporal violation of regularity, and the effort used when processing information in different sensory modalities.
The Neural Basis of Temporal Prediction.
A primary goal of the project The neural basis of temporal prediction is to understand the neurocognitive basis of predictive processing in the human brain with special attention to auditory stimuli and rhythm. Using scalp and intracranial EEG, eye tracking and pupillometry, we aim to decipher the neural mechanisms involved.
Neurophysiological Mechanisms of Human Auditory Predictions
The ability to predict upcoming events is a core feature of human cognition. The project AudioPred - Neurophysiological Mechanisms of Human Auditory Predictions: From population- to single neuron recordings contributes to understanding how auditory predictive processes are computed in the human brain. In a broad sense, predictive processes incorporate knowledge from the past to predict future states of the body and the environment, shaping how we perceive the world.
Rhythm as an Individual Ability
The goal of the project Rhythm as an individual ability is to understand how rhythmic abilities vary between individuals. To do this we investigate the basic cognitive, environmental and genetic factors that shape rhythmicity. We are developing a test battery to cover a broad area of rhythm topics, such as meter, beat, and microtiming.
Interaction & Pleasure Cluster
We seek to understand why rhythms make us move and how rhythm facilitates entrainment and interaction. We also explore the affective and social outcomes of such processes. These are studied through the experience of absorption in musical performance and perception, music-related effort and skilled action, and the dynamics of music-evoked pleasure.
Highlights
- Can sight change what we hear? The McGurk effect is a classic example of cross-modal integration in speech. Bruno Laeng and colleagues investigated whether such a cross-modal integration of sight and sound also applies to vocal articulations by singers. They found that switching videos of sung musical intervals systematically changes the estimated distance between two notes of a musical interval. Pairing the video of a smaller sung interval to a relatively larger auditory led participants to rate the interval as smaller than it actually was, and vice versa. The results are published in Scientific Report and indicate that strong integration of sound and sight can occur beyond the articulations of natural speech.
- In the article "Musical and Bodily Predictors of Mental Effort in String Quartet Music", Laura Bishop, Alexander Refsum Jensenius and Bruno Laeng report on a study combining eye-tracking and motion capture technologies in a real-world concert setting. The results show that bodily effort and expressive difficulty influence the performers' pupil diameters.
- The social distancing measures associated with the COVID-19 pandemic led artists worldwide to perform their music online in live-streamed concerts. Dana Swarbrick and colleagues took advantage of the extraordinary circumstances by investigating how virtual concerts may contribute to listeners' experiences of social connection. The results revealed that the live-streamed concerts produced more social connections than pre-recorded concerts.
- The project AMBIENT - Bodily Entrainment to Audiovisual Rhythms was funded by the Research Council of Norway. The project will investigate how rhythms in everyday environments influence people's bodily behaviour and will be led by Alexander Refsum Jensenius.
Effort and Attention in Musical Experience
Musical experiences often provide sensations of affect and pleasure and feelings of sound-producing and sound-accompanying body motion. The project Effort and Attention in Musical Performance Experience aims to enhance our understanding of the attention and effort, both physical and mental, involved in producing and perceiving music-related sound and related multimodal phenomena.
Engagement and Absorption
Does time and timing matter for musical engagement and absorption? Music is well known for its ability to engage, yet the nature of this engagement remains elusive. The project Engagement and Absorption explores aspects of rhythm, temporality, and meaning in musical engagement, such as listening and performing. The project works from various research angles, including musical sense-making and ethics, absorption in musical performance, spatiality and immersion, and relational interaction.
Entrainment, Social Bonding and Pleasure
The project Entrainment, Social Bonding and Pleasure aims to advance our understanding of how entrainment, synchronised behaviour and other related processes lead to prosocial outcomes such as social bonding. The project seeks to uncover the factors contributing to synchronous movement and social bonding in concerts. It explores whether entrained listening can evoke affiliation towards fellow listeners, the musicians, or social groups. The sub-project Quarantine Concerts project aims to explore the social and emotional effects of virtual concerts.
Human Music-related Micromotion
The project MICRO - Human Bodily Micromotion in Music Perception and Interaction investigates how music influences micromotion, the smallest bodily motion possible to produce and perceive. Data is collected using motion capture and physiological measurements. One of the annual highlights is the Championship of Standstill, which is important for both data collection and dissemination.
Musical Chills
The Musical Chills project investigates the brain processes and musical triggers underlying the intense, pleasurable 'chills' responses that music evokes. It explores the role of other neuromodulators (particularly the endogenous opioid system), via pharmacological agents, in the pleasurable response to music. Moreover, the project investigates what specific musical features can trigger chills.
MusicLab: Researching real-life concerts
MusicLab is an innovation project by RITMO and the University Library. The aim is to explore new methods for conducting research, research communication and education. Each MusicLab event is organised around a concert in a public venue, which is also the study object. The events also contain an edutainment element through panel discussions with world-leading researchers and artists and "data jockeying" in the form of live data analysis of recorded data.
Interaction & Robotics Cluster
We investigate aspects of rhythm and motion through robotics and technology. This includes creating interactive music systems focusing on multi-dimensional mapping from sensors to musical parameters. Moreover, we design and prototype adaptive robot systems and explore human-robot and robot-robot interactions in collective settings.
Highlights
-
The paper "Real-world embodied AI through a morphologically adaptive quadruped robot" was published in Nature machine intelligence. The study shows the feasibility and benefits of adaptation to new environments through changes in a robot's body.
-
Michael Krzyzaniak has published an article in Make magazine with complete instructions on building the music robot Dr. Squiggles. He has also written about how these robots interact in an article in the Journal of New Music Research.
-
Qichao Lan has developed the programming language and platform GLICOL for real-time programming of music. The web portal allows users to collaborate on coding, and it also has an interactive training section.
-
The project COROBOREES - Collective Robotics through Real-time Entrainment of Evolved dynamical Systems, funded by the European Union's Marie Sklodowska-Curie Individual Fellowships programme, started in September. Alexander Szorkovszky studies the synchronization of complex movement patterns by implementing social feedback into robot control systems.
Modelling and Robots
The Modelling and Robots project's core activity is the investigation of rhythm and motion through the design and construction of models and robots. This has applications to understanding natural processes and the creation of technological systems.
Music for stem cells: Aiming to cure diabetes with sound waves
The convergence environment ABINO - Artificial Biomimetic systems – the Niche of Islet Organoids wants to develop future models for diabetes research by using new strategies for stem cell differentiation.
Musical Human-Computer Interaction
The core activity of the project Musical Human-Computer Interaction is investigating aspects of rhythm and motion through the design and construction of interfaces for musical human-computer interaction. This includes studying and designing both acoustic instruments and completely digital systems. We are particularly interested in various types of electroacoustic instruments, in which we explore the complexity of human motion in musical experience and practice.
PIRC - Predictive and Intuitive Robot Companion
Humans anticipate future events more effectively than computers. The Predictive and Intuitive Robot Companion project combines sensing across multiple modalities with learned knowledge to predict outcomes and choose the best actions. The project will apply machine learning and robotics expertise and collaborate with researchers in cognitive psychology to apply recent human prediction models and action decision-making to perception-action loops of intelligent robot assistants. The work will allow such robots to adapt and act more seamlessly with their environment than the current technology.
Platforms
Dr. Squiggles is an interactive musical robot that plays rhythms by tapping. The RITMO-designed robot listens for tapping produced by humans or other musical robots and attempts to play along and improvise its rhythms based on what it hears.
How does the way a musical robot moves affect how its human musical partners interact with it? Robots, including those that play music, are stereotypically stiff in their movement. Human musicians, by contrast, use a wide variety of expressive actions, gestures, and signals when playing together. To explore this, we built a prototype guitar plucking robot called Professor Plucky, designed to have visually appealing movement, even at the expense of mechanics.
The Mini Sound box is a mini acoustic chamber for small-scale testings and experiments.
The self-playing guitars are mechatronic/robotic devices capable of producing sounds utilizing a classic guitar body's acoustics. Each guitar can sense audio and proximity activity and has a battery-powered onboard processing platform. The devices can thus be deployed in a decentralized, multi-guitar setup.
RITMO in Numbers
Reported Research Results 2021
RITMO's researchers have published many scientific articles and book chapters in 2021, as well as one monograph and one anthology. They have also given talks and presentations at several conferences, although the numbers are lower due to the pandemic. The Centre's research results also include several artistic outputs and media contributions. You can see all of RITMO's results reported in CRIStin.
People
A total of 55 people were employed at RITMO in 2021, of which there were 24 women, 1 non-binary and 31 men. We work to achieve gender balance in our recruitments.
The World at RITMO
RITMO is genuinely an international working environment. RITMO scholars come from all over the world: Argentina, Australia, Austria, Bangladesh, Brazil, Canada, China, Denmark, Ecuador, Finland, France, Germany, Greece, Hungary, India, Iran, Italy, Japan, Jordan, Mexico, Montenegro, Norway, Pakistan, Poland, Romania, South Korea, Spain, Sweden, Turkey, United Kingdom and the United States.
RITMO in the World
RITMO has research cooperation with partners from across the world: Argentina, Australia, Austria, Belgium, Brazil, Canada, Chile, China, Denmark, Finland, France, Germany, India, Israel, Italy, Japan, Jordan, Netherlands, Poland, Singapore, Spain, Sweden, Switzerland, United Kingdom and the United States.
RITMO Highlights
At a research-intensive centre of RITMO's size, there are always numerous things happening. In 2021 we were particularly proud of three things.
MusicLab Copenhagen
After two years in the making, MusicLab Copenhagen finally happened in October 2021. Working with The Danish String Quartet, one of the world's finest ensembles, a team of researchers set out to understand how the mind and body are engaged in musicians and audience members in intense states of shared musical absorption. The event combined music, science, and science communication in a novel format. The audience (local and online) enjoyed the concert while also being part of an experiment aimed at understanding more about the minds and bodies of expert musicians.
Rhythm Production and Perception Workshop
The 18th Rhythm Production and Perception Workshop brought together 200 researchers for an intensive hybrid conference experience in June 2021. Researchers from a wide range of disciplines engaged in discussions about the scientific study of rhythm. The program featured talks, poster sessions, keynotes, music performances, and installations.
Motion Capture Online Course
After a long production phase, RITMO launched the online course Motion Capture: The art of studying complex human activity on the FutureLearn platform in December 2021. The course is for everyone interested in human motion capture. It starts with the basics of human anatomy and biomechanics. Then it moves on to setting up, calibrating, and recording with an infrared optical motion capture system. The last week covers the ethical and legal challenges of working with human motion capture.
RITMO Stories
A project is always more than what you can read from a project page. Here some of RITMO's researchers present their own research.
Benedikte Wallace is a doctoral researcher at RITMO studying music, dance, and artificial intelligence.
Guilherme Schmidt C?mara is a postdoctoral fellow at RITMO studying musical groove.
Laura Bishop is a postdoctoral researcher at RITMO studying music performance.
Martin Plei? is a doctoral fellow at RITMO, studying music and virtual reality.
RITMO Seminar Series
The RITMO Seminar Series is a venue for presenting world-leading scholars worldwide. We usually invite our lecturers to stay for a few days to make the most of their visit to RITMO. Sadly, this was difficult in 2021 due to the pandemic. We therefore decided to postpone most of this year's seminars. Still, we were lucky to have one online session in the autumn.
Danica Kragic, School of Computer Science and Communication at the Royal Institute of Technology, KTH
Disputations
Doctoral fellows from the first part of RITMO's life now begin to complete their dissertations. We were happy to host two successful disputations in 2021:
RITMO Behind the Scenes
Through the year, the corona measures came and went and then came back again. Luckily we managed to keep the good RITMO spirit alive, both online, in hybrid format and in real life.
Diversity
Diversity was an important centre topic in 2021, and this topic was discussed at both the spring and autumn retreats.
One important contribution was the publication of the paper Gender bias in academia: A lifetime problem that needs solutions, which was fronted by former RITMO postdoctoral fellow Ana?s Llorens.
The diversity workshops resulted in a revised code of conduct and a diversity action plan that will guide our continued operation. We have already implemented several measures to increase the diversity among RITMO staff and guests.
However, diversity measures such as working towards gender balance and bridging the gap between the global north and south is not only about recruitment. Equally important is the choice of research topics, experimental stimuli, and participant recruitment. Raising awareness about the importance of cultivating diversity at all levels will continue within RITMO in the years to come.
RITMO’s Career Development Programme
Systematic and comprehensive career support for early career researchers is increasingly important, especially at Centres of Excellence such as RITMO, because of the high percentage of young researchers at these centres.
RITMO’s Career Development Programme aims to help our doctoral and postdoctoral research fellows build skills and experience that are relevant and valuable both within and outside academia and enable them to document these skills systematically. Each recruit has a personal career mentor who helps them map their career goals and make a personalised career development plan. In addition, RITMO organises biannual workshops focused on academic and generic skills.
In 2021 we published a report on the programme's background, aims, and structure. We hope that this portfolio will serve as an inspiration and possibly a guide for other centres, university departments, and entities seeking to improve their career development support for early career researchers.
Awards
RITMO people received many distinctions during the year:
- Simon H?ffding, RITMO, and the Danish String Quartet received the Danish Broadcast Corporation's P2 prize for "Happening of the year" for MusicLab Copenhagen.
- Chris Stover received the 2021 Steve Larson Award for Jazz Scholarship for his article "The Queer Rhythm of Cecil Taylor's 'Enter Evening'" in Deleuze and Guattari Studies 15 (3).
- Tor Endestad, Ingrid Funderud, Anne-Kristin Solbakk, Venke A. Grane, Andras Puszta, and Rene Huster received The Helgeland Hospital Trust research award for their internationally recognized studies on the neurophysiological basis of attention and cognitive control in adult ADHD, and for developing an interdisciplinary team which includes national and international collaborators.
- Bálint Laczko and Alexander Refsum Jensenius received the best presentation award during the Nordic Sound and Music Computing Conference.
- The work ‘Aporia’ that Martin Pleiss created for/with artist Patricia Detmering won the 2nd place in the first German VR art prize.
- Benedikte Wallace was invited to hold a TEDx talk titled What Happens When Art Meets AI?
RITMO People
Scientific Advisory Board
The members of RITMO's Scientific Advisory Board are highly acclaimed researchers within their respective fields. The Board is vital to providing feedback on on-going and future research, and its members have been carefully chosen to support the different parts of RITMO.
Board
Management
Postdoctoral Fellows and Researchers
PhD Fellows
.jpg)
Technical/Administrative Staff
Assistants
|
|
Guest researchers
- Bucio, Diego Antonio Marin (Choreomundus Programme (International Masters in Dance Knowledge, Practice and Heritage) NTNU)
- Cancino-Chacón, Carlos Eduardo (Austrian Research Institute for Artificial Intelligence)
- Clarke, Eric (University of Oxford)
- Martinez, David Ricardo Quiroga (Center for Music in the Brain, Aarhus University)
- Memi?, Emin (Ankara University)
- Milojevic, Andrija (Lut University, Finland)
- Potel, Pierre (?cole Nationale Supérieure de Techniques Avancées (ENSTA Paris))
-
Ratzinger, Carolin (Universit?t für Musik und darstellende Kunst Wien)
-
Remache, Paul (Universidad de Málaga)
A Centre of Excellence in Norway
A collaboration between
