Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers

Peer-reviewed journal article
Catriona Robertson dancing with the Sonified Body system by Tim Murray-Browne and Panagiotis Tigas during a residency at the Centre for Contemporary Arts, Glasgow.

This paper gives the philosophical details behind Sonified Body. I explain what I mean by supplanting abstract manipulation with embodied resonance. Published in the peer reviewed journal Applied Sciences.

Human-computer interaction is currently dominated by a paradigm where abstract representations are manipulated. We propose an alternative paradigm of emergence and resonance, which can be realised through unsupervised machine learning. This shift requires us to release the requirement that interactions with computers be precise and explainable, and instead embrace vagueness, ambiguity and embodied cognition. This makes little sense in traditional computing tasks originating in the office, but it offers new opportunities for embodied activities like creative expression, dance and social connection.

In Sonified Body, we explore this approach by using a Variational Autoencoder to devise an interface based on an individual's existing vocabulary of movement, and test out its artistic potential with some dancers in the studio.

There are broader political implications to this approach. It may help to reduce the potential of interfaces to uniformly impose the mindset of its designer onto its users.

Read the full paper (pdf).

  • pdf
reference

T. Murray-Browne and P. Tigas, “Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers,” Applied Sciences, 11(18): 8531, 2021.

abstract

Abstract

Most human–computer interfaces are built on the paradigm of manipulating abstract representations. This can be limiting when computers are used in artistic performance or as mediators of social connection, where we rely on qualities of embodied thinking: intuition, context, resonance, ambiguity and fluidity. We explore an alternative approach to designing interaction that we call the emergent interface: interaction leveraging unsupervised machine learning to replace designed abstractions with contextually derived emergent representations. The approach offers opportunities to create interfaces bespoke to a single individual, to continually evolve and adapt the interface in line with that individual’s needs and affordances, and to bridge more deeply with the complex and imprecise interaction that defines much of our non-digital communication. We explore this approach through artistic research rooted in music, dance and AI with the partially emergent system Sonified Body. The system maps the moving body into sound using an emergent representation of the body derived from a corpus of improvised movement from the first author. We explore this system in a residency with three dancers. We reflect on the broader implications and challenges of this alternative way of thinking about interaction, and how far it may help users avoid being limited by the assumptions of a system’s designer.

bibtex
@article{murraybrowne2021emergent-interfaces,
    author = {Murray-Browne, Tim and Tigas, Panagiotis},
    journal = {Applied Sciences},
    number = {8531},
    title = {Emergent Interfaces: Vague, Complex, Bespoke and Embodied Interaction between Humans and Computers},
    volume = {11},
    year = {2021}
}
Published:
Updated:
Tim Murray-Browne
hi ‹at› timmb.com
Glasgow • London • Bucharest
© 2009-24 Tim Murray-Browne

This site sets no cookies one cookie __cf_bm, which is an essential functional cookie set by Cloudflare for 30 minutes to protect against bot traffic (more info). It‘s part of the embedded Vimeo videos. If you catch any embedded content trying to set any other cookies please let me know.

Newsletter subscriber data is controlled by Tim Murray-Browne and processed by Preverbal (my own studio) and Substack.