Augmented Reality Apps

Alexander Scriabin Sonata No.5, RIAM Dublin, April 2019

Dr. Svetlana Rudenko, Technological University Dublin post-doctoral researcher / Prof. Maura McDonnell (www.mauramcdonnell.com) MMT Trinity College Dublin, (published in “Music & Synaesthesia”, University of Vienna, IASAS, 2020)

Harmonic music analysis is an important field in musicology and a lot of academic research has been conducted in this area.  However, there is little discussion and analysis on the sensory aspects of the music composition content and the musical narrative nor on the cross-modal associations that can arise as a result of the organization of musical texture in music composition. Cognitive Musicology is a relatively new discipline, that uses computational tools to reflect on musical structures. This paper investigates a methodology of creating visuals for classical music, in the genre of a music concert that will consist of presenting a music performance and a simultaneous visual artwork, what one of the contributor’s names as a visual music art.  In this project, the musical narrative explores an alternative form of music analysis that is based on archetypes of musical texture, cross-modal associations, synaesthetic art, and appropriate symbolistic artworks from the time of the music composer.

 There are two components to the new type of music analysis undertaken in this project and these are: 1.) music performance analysis and 2.) a visual artwork informed by the music performance analysis.

1)   Music performance analysis: The enhancement of “musical events” through the instrumentalist’s performance will be examined and becomes part of the performance as the imaginative visualization of musical narrative benefits the performance of the music by instrumentalist as a result of the variety of gestures, sounds and physical actions that the performance can contribute. This is also used by the animator to create an animated visualization of similar gestures and physical actions.  The music and visual art together so closely construed enhances the imagination of the listener.  The music analysis for an art visualization of Scriabin’s Sonata N5 is based on visually and audibly recognizable archetypes of musical texture and consists also of including in that analysis the complex philosophical beliefs of a symbolist composer.  The source images for the animation come from various sources, some images are historical (for example the image by the artist Jean Delville for the Prometheus score and Queen Swan by symbolist artist Mikhail Vrubel) and some images are by artists synaesthetes Timothy Layden theshapeofsounds.com and Svetlana Rudenko.  Other images are created by visual music artist Maura McDonnell in the process of animating the source images and constructing the final visual animation.

2) Visual artwork informed by music analysis: A visualization of the music analysis through the creation and inclusion of several visual artist artworks is included in the project.  The identification and examination of the synaesthetic and multisensory associations that arise in the visual modality as a result of the music and its performance will be examined and used as the basis for creating visualizations. Bulat Galeyev called cross-modal associations an “essential component of musical thinking”, and experienced consciously or stored as a cross-modal secondary response at the level of the subconscious sensory system. Synaesthetes are people who are more aware of their sensory pairings.  For this project, the research field of synaesthesia multisensory perception is taken as the model from which to assess music analysis and musical narrative in the composition. A related field to that of a synaesthete approach to visualizing music is that of visual music where an artist composes visuals according to a musical expression, a form of multi-sensory design.  The animator brought a visual music approach to the animation in this project.

The project objective is to demonstrate through the performance with visualizations the new music analysis approach and an expected outcome of this demonstration is that music audiences and scholars could benefit from having an artistic visualization of musical narrative and it could provide additional information that is easy to understand and grasp to the content of the composition.

European Research Night, Trinity College Dublin, 28th September 2018 - PROBES

Tetractys is a multi-disciplinary research project that brings together science, music, and visual art. The research is grounded on synaesthesia – a benign condition that is characterized by cross-modal sensory pairings – in addition to perceiving one stimulus, people with synaesthesia may have other sensory effects to the experience – a color, a smell, or feeling. The color patterns and shapes of the artworks by Sofia Areal are musically interpreted by the pianist Dr. Svetlana Rudenko and digitally animated by Dr. João Cabral, into a synchronized cross-modal perception experience.

Here is a collection of pieces for animated Art in Music:

• The Journey Blue, Archetypes of Musical textures

• The Green Rose - The feeling of Texture

• Hearts - What music cannot explain without words