Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers

14 Sep 2021
Tim Murray-Browne and Panagiotis Tigas
Academic journal article

This paper gives the philosophical details behind Sonified Body. I explain what I mean by supplanting abstract manipulation with embodied resonance. Published in the peer reviewed journal Applied Sciences.

Human-computer interaction is currently dominated by a paradigm where abstract representations are manipulated. We propose an approach paradigm of emergence and resonance, which can be realised through unsupervised machine learning. This shift requires us to release the requirement that interactions with computers be precise and explainable, and instead embrace vagueness, ambiguity and embodied cognition. This makes little sense in traditional computing tasks originating in the office, but it offers new opportunities for embodied activities like creative expression, dance and social connection.

In Sonified Body, we explore this approach by using a Variational Autoencoder to devise an interface based on an individual's existing vocabulary of movement, and test out its artistic potential with some dancers in the studio.

There are broader political implications to this approach. It may help to reduce the potential of interfaces to uniformly impose the mindset of its designer onto its users.

abstract

Abstract

Most human–computer interfaces are built on the paradigm of manipulating abstract representations. This can be limiting when computers are used in artistic performance or as mediators of social connection, where we rely on qualities of embodied thinking: intuition, context, resonance, ambiguity and fluidity. We explore an alternative approach to designing interaction that we call the emergent interface: interaction leveraging unsupervised machine learning to replace designed abstractions with contextually derived emergent representations. The approach offers opportunities to create interfaces bespoke to a single individual, to continually evolve and adapt the interface in line with that individual’s needs and affordances, and to bridge more deeply with the complex and imprecise interaction that defines much of our non-digital communication. We explore this approach through artistic research rooted in music, dance and AI with the partially emergent system Sonified Body. The system maps the moving body into sound using an emergent representation of the body derived from a corpus of improvised movement from the first author. We explore this system in a residency with three dancers. We reflect on the broader implications and challenges of this alternative way of thinking about interaction, and how far it may help users avoid being limited by the assumptions of a system’s designer.

bibtex

@article{murraybrowne2021emergent-interfaces,
    author = {Murray-Browne, Tim and Tigas, Panagiotis},
    journal = {Applied Sciences},
    number = {8531},
    title = {Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers},
    volume = {11},
    year = {2021}
}

reference

T. Murray-Browne and P. Tigas, “Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers,” Applied Sciences, 11(18): 8531, 2021.

Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders

14 Jun 2021
Tim Murray-Browne and Panagiotis Tigas
Academic conference paper

A short technical paper for the NIME conference that describes the approach Panagiotis and I developed in Sonified Body to interpret human movement. We use an AI model trained entirely on an individual's movement to form a compressed representation of how their body moves. We use this to generate a primary mapping for transforming movement into sound that is based entirely on their existing vocabulary of movement.

In the paper we argue why we feel this approach is effective in open ended creative work, such as the real-time transformation of a dancer's movement into sound, as we're doing.

full text

abstract

Abstract

In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator [1] and MIMIC [2] allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.

bibtex

@inproceedings{murray-browne2021latent-mappings,
    author = {Murray-Browne, Tim and Tigas, Panagiotis},
    booktitle = {International Conference on New Interfaces for Musical Expression},
    day = {29},
    doi = {10.21428/92fbeb44.9d4bcd4b},
    month = {4},
    note = {\url{https://doi.org/10.21428/92fbeb44.9d4bcd4b}},
    title = {Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders},
    url = {https://doi.org/10.21428/92fbeb44.9d4bcd4b},
    year = {2021},
}

reference

T. Murray-Browne and P. Tigas, “Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders,” in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME), online, 2021.

Who is that voice in your head speaking to?

22 Jan 2019
Tim Murray-Browne
Essay

There seems at first something a bit pathetic about spending your time rehearsing conversations you might have with people. And yet, I recently observed that this seems to be how my thinking process operates. My internal monologue, the inner voice of the mind, is near exclusively in the form of an imaginary conversation with somebody. I have a hunch that this might be quite common. After all, it took me a while to admit…

The Cave of Sounds: An Interactive Installation Exploring How We Create Music Together

30 Jun 2014
Tim Murray-Browne, Dom Aversano, Susanna Garcia, Wallace Hobbes, Daniel Lopez, Tadeo Sendon, Panagiotis Tigas, Kacper Ziemianin and Duncan Chapman
Academic conference paper

A paper at the NIME conference describing the experimental creative process of non-hierarchical organic creation that led to Cave of Sounds.

abstract

Abstract

The Cave of Sounds is an interactive sound installation formed of eight new musical instruments exploring what it means to create instruments together. Each instrument was created by an individual but with the aim of forming a part of this new ensemble, with the final installation debuting at the Barbican in London in August 2013. In this paper, we describe how ideas of prehistoric collective music making inspired and guided this participatory musical work, both in creation process and in the audience experience of musical collaboration. Following a detailed description of the installation itself, we reflect on the successes, lessons and future challenges of encouraging creative musical collaboration among members of an audience.

bibtex

@inproceedings{murray-browne2014cave-of-sounds,
    address = {London, UK},
    author = {Murray-Browne, Tim and Aversano, Dom and Garcia, Susanna and Hobbes, Wallace and Lopez, Daniel and Sendon, Tadeo and Tigas, Panagiotis and Ziemianin, Kacper and Chapman, Duncan},
    booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
    pages = {307-310},
    title = {The {C}ave of {S}ounds: An Interactive Installation Exploring How We Create Music Together},
    year = {2014},
}

reference

T. Murray-Browne, Dom Aversano, S. Garcia, W. Hobbes, D. Lopez, P. Tigas, T. Sendon, K. Ziemianin, D. Chapman, “The Cave of Sounds: An Interactive Installation Exploring How We Create Music Together,” in Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 307-310, London, UK, 2014.

Interactive Music: Balancing Creative Freedom with Musical Development

1 Oct 2012
Tim Murray-Browne
PhD thesis

I completed my PhD in 2012 at the Centre for Digital Music, Queen Mary University of London, under the invaluable supervision of Mark D. Plumbley and Nick Bryan-Kinns. It was examined by Atau Tanaka and Francois Pachet.

My thesis explores how we can create interactive music experiences that let you be creatively involved in what you hear, but also draw you in to the composer’s musical world, maintaining the hypnotic connection we are familiar with from linear music. For me, interactive music experiences are at their best when they are a shared creation between composer and audience where music is something that happens with you rather than to you. In this way, composing interactive music is as much about musical actions as it is about sound. It requires you to move and to consider how your behaviour affects the environment around you.

Creating a captivating interactive music experience is challenging. How can we create a musical narrative and shape our audience’s experience without reducing their sense of creative freedom? Addressing this question has led me to examine musical structure and the perception of skill through perspectives rooted in information theory, social psychology and human-computer interaction. My thesis draws upon a number of fields and methodologies and considers composed instruments, interactive music systems, narrative structures within interactive art, the perception of agency within music and a brief analysis of conversational interaction.

I created a number of artworks as part of the PhD: the Serendiptichord, the Manhattan Rhythm Machine and finally IMPOSSIBLE ALONE, which encapsulated many of the ideas on narrative and agency that I had developed.

Transcripts of interviews referred to within the thesis may be found here.

abstract

Abstract

This thesis is about interactive music – a musical experience that involves participation from the listener but is itself a composed piece of music – and the Interactive Music Systems (IMSs) that create these experiences, such as a sound installation that responds to the movements of its audience. Some IMSs are brief marvels commanding only a few seconds of attention. Others engage those who participate for considerably longer. Our goal here is to understand why this difference arises and how we may then apply this understanding to create better interactive music experiences.

I present a refined perspective of interactive music as an exploration into the relationship between action and sound. Reasoning about IMSs in terms of how they are subjectively perceived by a participant, I argue that fundamental to creating a captivating interactive music is the evolving cognitive process of making sense of a system through interaction.

I present two new theoretical tools that provide complementary contributions to our understanding of this process. The first, the Emerging Structures model, analyses how a participant's evolving understanding of a system's behaviour engages and motivates continued involvement. The second, a framework of Perceived Agency, refines the notion of ‘creative control’ to provide a better understanding of how the norms of music establish expectations of how skill will be demonstrated.

I develop and test these tools through three practical projects: a wearable musical instrument for dancers created in collaboration with an artist, a controlled user study investigating the effects of constraining the functionality of a screen-based IMS, and an interactive sound installation that may only be explored through coordinated movement with another participant. This final work is evaluated formally through discourse analysis.

Finally, I show how these tools may inform our understanding of an oft-cited goal within the field: conversational interaction with an interactive music system.

bibtex

@phdthesis{murray-browne2012phd,
    Author = {T. Murray-Browne},
    School = {Queen Mary University of London},
    Title = {Interactive music: Balancing creative freedom with musical development},
    Year = {2012}
}

reference

T. Murray-Browne. Interactive Music: Balancing Creative Freedom with Musical Development. PhD thesis, Queen Mary University of London, 2012.