Emotional Response was conceived of as a piece of interactive generative music, where the music responds in real time to a changing input. The audience, attending in-person at the interactive installation, were imagined to provide the input in the form of their facial expression, interacting playfully with a webcam, causing the nature of the music being produced to match the emotions that they are expressing. This would make the audience active participants in the creation of the music, widen participation in an activity often reserved for the elite, which is one of the project’s design pillars.
Between submitting this idea and starting to work on it, the UK entered into lockdown. The intention is still to tour Emotional Response as an interactive installation, however, it did not make sense to focus on the facial recognition and in-person interaction aspects of the project while in-person meetings were not possible, and time-frames for the easing of lockdown were impossible to predict. I decided to focus my efforts into the design and development of the generative music system and how it would respond to changing inputs.
Generative music is any music where compositional choices or actions are taken out of the hands of the composer, and are given to some other agent. This agent could be a computer, a human improvisor, or the rules of a game. In the case of Emotional Response, some delegated compositional actions are performed by the computer according to a program, and others are made at the time of performance by the members of the audience interacting with the camera. This delegation of ‘composerly’ actions leads to their automation.
Emotional Response is designed to be an open-ended composition; producing sound and music indefinitely, and changing in character along with changes of emotion as picked up through facial expression recognition. I planned to create ‘sections’ that correspond to seven basic emotions and facial expressions: happiness, sadness, anger, fear, surprise, disgust and neutral. The overall structure of the piece, with transitions between structures, was programmed using FMOD (audio middleware often used for computer games) and existing music from my catalogue that matched each emotion was used as placeholder.
It would be possible for the music to continue playing for a long time within the same section, and so generative processes would be utilised within each section to keep things unpredictable and varied. I decided to engage deeply with the design of the generative music for one of these ‘sections’: Disgust. I wanted all possible characteristics of the music and the sounds that would make it up to be disgusting. To ensure this, I created sample-based instruments out of sounds that are deemed to be disgusting, and used them to create unpleasant dissonant harmonies. I avoided melody or anything that would become familiar or pleasant over time. I used very high frequency, harsh noises and low drones.
In order to help choose my disgusting sounds, I invited another generative process into my practice: I asked my social media followers to suggest sounds that they found disgusting, making those decisions for me. A selection of these sounds were then recorded, and made into sample-based virtual instruments.
The next stage was to create musical gestures from these disgusting instruments. In order to maximise their disgusting qualities, the gestures avoided any pleasant intervals and were largely non-melodic. Many gestures made use of unpleasant textures and very high frequencies to strengthen the impact of the unpleasant source recordings.
The final stage for composing the Disgust section of Emotional Response was to build a system in FMOD out of the prepared gestures. During this process, it became clear that even the most dissonant chord progressions I had written would grow familiar over time, when looped, and the same with the melodies. The more textural gestures were therefore favoured. It is hard to be disgusted by something familiar, and so unpredictability became an important tool. Using probabilistic triggering of sounds, and randomising which sample out of a group would be triggered at specific points, made the piece much less predictable. This also made it easier for the Disgust section to be open-ended (which is very important for this project), as it is unlikely to become repetitive or familiar, especially in the short to medium term.
Beyond the scope of the R&D period, Emotional Response remains in development. The next steps for the project include constructing the generative music systems for the remaining six emotions, to replace the placeholder, fixed-media loops already in place. I plan to tackle anger first, as a way to process my own experiences, thoughts and feelings during these difficult times. After this, happiness will be a pleasant change from exploring negative emotions. My hope is still that Emotional Response will be toured in physical spaces with an in-person audience able to interact with the finished installation. In the mean-time, the process of working on this project has inspired and informed some other project ideas that better suit a more socially distanced, often isolated lifestyle like we are all experiencing right now.
Bedroom Producer is another project based around generative music, where the compositional process is distilled down to a series of creative decisions. If you can make a choice, you can make music.
Switching Off is a narrative-focused game where a conversation serves as the primary interaction. A character is trying to convince the protagonist, controlled by the player, to delete all of their social media, citing reasons such as the behaviour-modification model of data monetisation alongside many others. The dialogue choices made by the player will influence the emotional state of both characters, and also of the adaptive music score.