Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers

Academic journal article
Catriona Robertson dancing with the Sonified Body system by Tim Murray-Browne and Panagiotis Tigas during a residency at the Centre for Contemporary Arts, Glasgow.

This paper gives the philosophical details behind Sonified Body. I explain what I mean by supplanting abstract manipulation with embodied resonance. Published in the peer reviewed journal Applied Sciences.

Human-computer interaction is currently dominated by a paradigm where abstract representations are manipulated. We propose an alternative paradigm of emergence and resonance, which can be realised through unsupervised machine learning. This shift requires us to release the requirement that interactions with computers be precise and explainable, and instead embrace vagueness, ambiguity and embodied cognition. This makes little sense in traditional computing tasks originating in the office, but it offers new opportunities for embodied activities like creative expression, dance and social connection.

In Sonified Body, we explore this approach by using a Variational Autoencoder to devise an interface based on an individual's existing vocabulary of movement, and test out its artistic potential with some dancers in the studio.

There are broader political implications to this approach. It may help to reduce the potential of interfaces to uniformly impose the mindset of its designer onto its users.

Read the full paper (pdf).


T. Murray-Browne and P. Tigas, “Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers,” Applied Sciences, 11(18): 8531, 2021.



Most human–computer interfaces are built on the paradigm of manipulating abstract representations. This can be limiting when computers are used in artistic performance or as mediators of social connection, where we rely on qualities of embodied thinking: intuition, context, resonance, ambiguity and fluidity. We explore an alternative approach to designing interaction that we call the emergent interface: interaction leveraging unsupervised machine learning to replace designed abstractions with contextually derived emergent representations. The approach offers opportunities to create interfaces bespoke to a single individual, to continually evolve and adapt the interface in line with that individual’s needs and affordances, and to bridge more deeply with the complex and imprecise interaction that defines much of our non-digital communication. We explore this approach through artistic research rooted in music, dance and AI with the partially emergent system Sonified Body. The system maps the moving body into sound using an emergent representation of the body derived from a corpus of improvised movement from the first author. We explore this system in a residency with three dancers. We reflect on the broader implications and challenges of this alternative way of thinking about interaction, and how far it may help users avoid being limited by the assumptions of a system’s designer.

    author = {Murray-Browne, Tim and Tigas, Panagiotis},
    journal = {Applied Sciences},
    number = {8531},
    title = {Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers},
    volume = {11},
    year = {2021}